LeaderRebalancerTest.AddTserver: /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:435: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260809 23:22:23.478675 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.482823 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.436054  6005 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:23.486609 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.486948 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
I20260809 23:22:23.487257  5959 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260809 23:22:23.488219  5993 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260809 23:22:23.487830  5801 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.492897 24430 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:23.383841  5903 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.515775 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 274437us
I20260809 23:22:23.507028 24429 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:23.510653 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 313493us
I20260809 23:22:23.508610  5646 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:23.510320  5887 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:23.583627  5801 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:23.514801  5965 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.798s	user 0.013s	sys 0.000s
I20260809 23:22:23.571179  6000 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.572680 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.573217  6004 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:23.574075  6009 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260809 23:22:23.576088  5311 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.519025  5611 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:23.431536  5813 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.580227  5939 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.539031  5646 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:23.587134 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 75648.9us
I20260809 23:22:23.628423 24312 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:23.595412 24430 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:23.596429 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.004s	sys 0.015s
W20260809 23:22:23.629412 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.005s	sys 0.015s
W20260809 23:22:23.588063  5484 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.002s
W20260809 23:22:23.629705 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 573863us
I20260809 23:22:23.633074  5752 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.639685 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:23.645903 24431 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:23.620985  5796 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.019s
I20260809 23:22:23.602186  5887 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:23.635450  5646 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:23.614874  5643 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:23.533638  5951 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.660742  5796 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.006s	sys 0.019s
I20260809 23:22:23.686021 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.586902  5652 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.690122 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.690333 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.672762  5681 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:23.634611  5801 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260809 23:22:23.714970 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.626202  6000 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.639520  5611 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:23.719354 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 299669us
I20260809 23:22:23.603413  5965 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.681766  5280 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:23.726415 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 305890us
I20260809 23:22:23.639371  5484 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.736714 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.738590 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3992) took 1001 ms. Trace:
I20260809 23:22:23.736860 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.602639  5839 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.766969 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.716234  5889 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.603101  5194 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.801338 24428 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:23.729562  6007 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:23.745415 24535 rpcz_store.cc:276] 0809 23:22:22.737526 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.743251 (+  5725us) service_pool.cc:227] Handling call
0809 23:22:22.794523 (+ 51272us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:22.816318 (+ 21795us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:22.816377 (+    59us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:22.882668 (+ 66291us) log.cc:842] Serialized 52 byte log entry
0809 23:22:23.062226 (+179558us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:23.062266 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:23.062332 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:23.738352 (+676020us) raft_consensus.cc:1671] finished
0809 23:22:23.738376 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:23.738531 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15460,"raft.queue_time_us":9100,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11727,"thread_start_us":152000,"threads_started":1,"wal-append.queue_time_us":313274}
I20260809 23:22:23.804179 24423 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:23.792273 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55505.8us
W20260809 23:22:23.726063  5622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.017s	sys 0.006s
I20260809 23:22:23.816903 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.813858  6007 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.735868  5681 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:23.820524  6015 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260809 23:22:23.822569  6033 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
W20260809 23:22:23.824086  6023 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.164s	user 0.001s	sys 0.001s
W20260809 23:22:23.824636  6024 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260809 23:22:23.747213  5670 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.768784  5751 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.767810  6012 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:23.855382 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 338492us
I20260809 23:22:23.789916  5882 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.858345 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:23.828424  5622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.017s	sys 0.013s
I20260809 23:22:23.812127  5891 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.833596  6007 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
W20260809 23:22:23.825008  6014 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260809 23:22:23.878301 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:23.745046  5280 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.853546  5903 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.893253 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:23.894836 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:23.901643 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 231682us
I20260809 23:22:23.902303 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:23.739203  5887 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.913410 24540 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:23.875607  6033 consensus_queue.cc:1225] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 23.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260809 23:22:23.915169 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62108.1us
I20260809 23:22:23.913244 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.930428 25363 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260809 23:22:23.906440  5718 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.171s	user 0.012s	sys 0.006s
I20260809 23:22:23.891026  5813 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.934631 25371 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:23.898769  5643 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.932808 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.936451 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.939640  5652 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:23.939791  6025 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260809 23:22:23.883687  5751 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.897285  5752 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:23.946283  6034 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:22:23.946826  5965 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.947094  6044 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:23.902015  6012 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.951933 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.952694 24540 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:23.954459 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.951231 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3997) took 1141 ms. Trace:
I20260809 23:22:23.963209 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.956023 24546 rpcz_store.cc:276] 0809 23:22:22.809733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.832328 (+ 22595us) service_pool.cc:227] Handling call
0809 23:22:22.847762 (+ 15434us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:22.896787 (+ 49025us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:22.896833 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:22.915868 (+ 19035us) log.cc:842] Serialized 26 byte log entry
0809 23:22:23.261559 (+345691us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:22:23.328543 (+ 66984us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:23.328602 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:23.328694 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:23.937213 (+608519us) raft_consensus.cc:1671] finished
0809 23:22:23.937243 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:23.951144 (+ 13901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48582,"raft.queue_time_us":60618,"raft.run_cpu_time_us":4531,"raft.run_wall_time_us":351357,"spinlock_wait_cycles":18720128,"thread_start_us":306418,"threads_started":1,"wal-append.queue_time_us":595422}
I20260809 23:22:23.933082  5801 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:23.948787  5951 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.973982 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.965590  5652 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260809 23:22:23.975108 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.901669  5891 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.967406  5484 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:23.998751 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.999787 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:24.001497  6037 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260809 23:22:24.004925  5278 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.335s	user 0.004s	sys 0.008s
I20260809 23:22:24.005460  5278 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.007010  5611 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.007524  5801 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.009810  5801 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.013893  5891 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:24.014170 24580 heartbeater.cc:644] Failed to heartbeat to 127.21.255.62:37921 (6 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.255.62:37921 timed out after 15.000s (SENT)
I20260809 23:22:24.016083 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.953548  5643 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.024909 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.026441 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:24.027539 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 194051us
I20260809 23:22:24.028445 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.028579 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54716.9us
I20260809 23:22:24.029302 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:24.031395 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61115.8us
W20260809 23:22:24.036319 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80660.3us
I20260809 23:22:23.995239  6000 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.938246  5718 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.034350 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.976434  5752 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
W20260809 23:22:24.043547 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164602us
I20260809 23:22:23.964524  5681 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.038452 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.997948  5484 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.032624  6011 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.036432  5889 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:24.059067  6002 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.055159  6048 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:24.052137  5280 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:24.061259  6050 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260809 23:22:24.061748  6060 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:24.065778  5545 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.887s	user 0.009s	sys 0.001s
I20260809 23:22:24.068233 24542 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:24.070034 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.105535 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.066607  6063 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.073510  6036 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:24.074024  6011 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.074069 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.096298 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.071476  5889 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.114408 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.070658  6012 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.116148  5939 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.120935 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.123797 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.112726  6019 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:24.117069  5889 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:24.129225 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60509.5us
I20260809 23:22:24.113128  5280 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.116531  6063 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.084865  5646 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.119472  5278 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.113014  6000 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
I20260809 23:22:24.117997  6036 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.110424  5545 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.141469 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.129454  5813 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:24.128274  5891 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.130198  6019 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.160467 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.164692 25368 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:24.171254 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59932.7us
I20260809 23:22:24.173527 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.162559 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.125793  6012 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260809 23:22:24.176153 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.129801  5939 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:22:24.153739  6063 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.178469 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.173388 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 33) took 1408 ms. Trace:
I20260809 23:22:24.180207 24422 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:24.183043 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.183672 25363 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:24.183772 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.180781 24427 rpcz_store.cc:276] 0809 23:22:22.755411 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.806244 (+ 50833us) service_pool.cc:227] Handling call
0809 23:22:24.160711 (+1354467us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.163386 (+  2675us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2259555200}
I20260809 23:22:24.159117  6036 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:24.187577 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.161414  5813 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.193085 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.176807  6019 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:24.178776  6005 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.000s
I20260809 23:22:24.180047  6003 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:24.180080  5891 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.207235 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.187477  5903 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:24.196075  5813 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.218909 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.235951 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.223649 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.238463 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.244891 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.199496  6000 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.220818 25365 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:24.264786 25372 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:24.184358  6033 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260809 23:22:24.201032  5718 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.274729 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.275638 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94594.2us
W20260809 23:22:24.223644  6066 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:22:24.207785  5643 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:24.207412  6003 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:24.248034  6024 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260809 23:22:24.281635 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.220800  5903 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.233178  5646 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:24.286664 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99134us
I20260809 23:22:24.297780 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.298596 24425 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.229090  5545 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.291611 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.229233  6073 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.272275  5891 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:24.308753 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 191236us
I20260809 23:22:24.275291  6012 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.314291 25371 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: [23.18-25.19]
I20260809 23:22:24.315164 24427 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:24.323482 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.207731  6005 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.325003 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:24.287467  5571 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.872s	user 0.006s	sys 0.006s
I20260809 23:22:24.287508  6003 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:24.328286 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.331285 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.331328 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:24.302078  5903 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.336786 25365 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:24.224881  6069 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.170s
W20260809 23:22:24.335988 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50922.7us
W20260809 23:22:24.344091 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.344867 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.335704 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.348418 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.334024 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.285066  5961 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260809 23:22:24.351318 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.351480 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62321.3us
I20260809 23:22:24.352635 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.359344 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.326843  6073 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.372815 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.346814  5545 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.373667 25372 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:24.375422 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:24.375882 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90983.7us
W20260809 23:22:24.378032 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83450.8us
I20260809 23:22:24.381245 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.348673  5959 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:24.396077 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.341962  6068 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.399139 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.339313  5571 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.402076 25365 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:24.374202  6073 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:24.404973 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64233us
I20260809 23:22:24.349380  6012 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.410277 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.412885  6011 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:24.413573 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.422056 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.399029  5545 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.416059 25372 raft_consensus.cc:2362] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:24.422247 25363 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
W20260809 23:22:24.432132 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88092us
I20260809 23:22:24.420392  5885 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.424186 25370 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.430168 25365 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 30: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:24.403280  5959 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.401039  5194 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:24.433197 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.434235 25371 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.401600  6068 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:24.406957  6060 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
W20260809 23:22:24.447511 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.434736 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3594) took 1128 ms. Trace:
I20260809 23:22:24.399767  6033 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260809 23:22:24.459204  6033 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:24.459812  6033 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.460181  6033 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:24.461426  6012 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.464389 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.461336  6033 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.466575 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.451073  5194 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.467191 25375 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.467331  5194 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:24.471220  6057 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.471758  6057 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:24.474834  6007 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.565s	user 0.007s	sys 0.005s
I20260809 23:22:24.475457 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.475304  6007 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.476125 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:24.477576  6007 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.478567 24544 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.481992  5278 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.482375  5278 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.483158  6060 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.483410 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.485285 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:24.453421 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:24.454388  6088 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260809 23:22:24.491714 25366 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.488180 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.489041 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.434875  6086 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.490540 24427 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:24.453476 24429 rpcz_store.cc:276] 0809 23:22:23.306409 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.530895 (+224486us) service_pool.cc:227] Handling call
0809 23:22:23.588501 (+ 57606us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:23.588736 (+   235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:23.588763 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:23.589247 (+   484us) log.cc:842] Serialized 26 byte log entry
0809 23:22:24.020204 (+430957us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:24.045691 (+ 25487us) log.cc:842] Serialized 14 byte log entry
0809 23:22:24.045889 (+   198us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.045960 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:24.127817 (+ 81857us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18001c7e60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:24.423320 (+295503us) raft_consensus.cc:1671] finished
0809 23:22:24.434478 (+ 11158us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.434657 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2478,"prepare.queue_time_us":24535,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":1187,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6620,"spinlock_wait_cycles":15863552,"thread_start_us":430534,"threads_started":1,"wal-append.queue_time_us":505169}
I20260809 23:22:24.490691 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:24.499112 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.450642  6071 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:24.466418  6033 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.511029 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.446645  5885 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.515740  5885 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.516824  5885 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:24.519387 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71889.9us
I20260809 23:22:24.520542 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1857) took 1773 ms. Trace:
I20260809 23:22:24.521098 24431 rpcz_store.cc:276] 0809 23:22:22.747057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.760569 (+ 13512us) service_pool.cc:227] Handling call
0809 23:22:22.761074 (+   505us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.646435 (+885361us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.520184 (+873749us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.520470 (+   286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2034286464}
W20260809 23:22:24.526400 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 4054) took 1520 ms (client timeout 1895 ms). Trace:
I20260809 23:22:24.526117  5889 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.527076  5898 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:24.528982  6019 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.531033 24424 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.531312  6019 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.450523  6067 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.244s
I20260809 23:22:24.531973  6019 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:24.535341  5622 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:24.535792  5622 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.536296  5545 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.536801  5622 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.538257  5622 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:24.538746  6019 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:24.539222  6019 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.522619 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.547537 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:24.526760 24422 rpcz_store.cc:269] 0809 23:22:23.006257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.007283 (+  1026us) service_pool.cc:227] Handling call
0809 23:22:24.526117 (+1518834us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.526255 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2640194432}
I20260809 23:22:24.541407 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.450253  6011 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.548919 24425 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:24.561161 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.507297 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.557014 25370 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:24.490407  6035 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.464761  5571 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.558504 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:24.576428 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 246984us
I20260809 23:22:24.562338 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.579252 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.579633  6069 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:24.568590 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.544659 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50054.4us
I20260809 23:22:24.580689 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.580886 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.453344  6068 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.581672 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.581801  5796 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.582545 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.582749  5796 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.583535  5796 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:24.583724 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.584069  5796 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:24.585412 24269 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.866s (SENT)
I20260809 23:22:24.586200  5801 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.587157  5801 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:24.588303  6000 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.585316  5796 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.589514  6069 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:24.581866 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.590583 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.594862  6000 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.540233  6086 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.598492 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.599670 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.600720 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.434362  6092 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.193s
I20260809 23:22:24.489707  6036 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.548513  5666 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:24.606287 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.617745 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.629662 24425 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260809 23:22:24.596563  5946 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.003s	sys 0.005s
I20260809 23:22:24.612138 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1863) took 1011 ms. Trace:
I20260809 23:22:24.631639 24425 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260809 23:22:24.565232  6005 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.483182  5278 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.564893  5545 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:24.646212 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.646669 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.647580  6078 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260809 23:22:24.647675  6031 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:24.648434  6031 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.648882  5311 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.649163  6102 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.633524  5571 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:24.631088 24430 rpcz_store.cc:276] 0809 23:22:23.600406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.729071 (+128665us) service_pool.cc:227] Handling call
0809 23:22:23.729403 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.826359 (+ 96956us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:23.854763 (+ 28404us) log.cc:842] Serialized 14 byte log entry
0809 23:22:24.519672 (+664909us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b180051b860. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:24.611693 (+ 92021us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:24.611762 (+    69us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:24.611823 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.611920 (+    97us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.612067 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2381,"prepare.queue_time_us":28198,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":15352,"spinlock_wait_cycles":20539264,"thread_start_us":81837,"threads_started":1}
I20260809 23:22:24.649402  6031 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.639109  6094 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.649910  6102 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.651597  6094 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.652551  6102 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.653491 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.653669 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.679009 25372 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:24.654186  6057 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.655104 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3591) took 1349 ms. Trace:
I20260809 23:22:24.655522  5751 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:24.656423 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2046) took 1578 ms. Trace:
I20260809 23:22:24.627854  5666 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.685667 24542 rpcz_store.cc:276] 0809 23:22:23.078501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.078697 (+   196us) service_pool.cc:227] Handling call
0809 23:22:23.079088 (+   391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.108747 (+1029659us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.656169 (+547422us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.656357 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2274281728}
I20260809 23:22:24.660604 24430 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260809 23:22:24.663439  6074 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.504s	user 0.003s	sys 0.000s
I20260809 23:22:24.663749  5889 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.602046  5891 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260809 23:22:24.672951  6104 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.609860  6004 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.676756 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.678217 24425 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:24.636421  5946 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.003s	sys 0.005s
I20260809 23:22:24.680455 24535 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:24.680866 25370 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.678428 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.681777 24424 rpcz_store.cc:276] 0809 23:22:23.305549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.493517 (+187968us) service_pool.cc:227] Handling call
0809 23:22:24.530820 (+1037303us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.531494 (+   674us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.654923 (+123429us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400062050. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.655054 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2384870784}
I20260809 23:22:24.684207 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.657505 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3301) took 1283 ms. Trace:
I20260809 23:22:24.608296  6091 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.563616  6011 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.606331  5670 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.068s
I20260809 23:22:24.641163  5652 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.701462 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.702544  6051 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:24.714478 24312 rpcz_store.cc:276] 0809 23:22:23.374241 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.374398 (+   157us) service_pool.cc:227] Handling call
0809 23:22:23.374752 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.628948 (+254196us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.657254 (+1028306us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.657446 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582845696}
I20260809 23:22:24.603063  6035 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.696179  5666 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.797135 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.637979  6060 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.798583 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.799556 25369 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:24.690661  6044 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.110s
I20260809 23:22:24.801474  6044 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.801836  6044 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:24.802314 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1861) took 2053 ms (client timeout 1803 ms). Trace:
I20260809 23:22:24.637053  5813 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:24.802871 24428 rpcz_store.cc:269] 0809 23:22:22.748562 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.749693 (+  1131us) service_pool.cc:227] Handling call
0809 23:22:24.801773 (+2052080us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.802081 (+   308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2266494208}
I20260809 23:22:24.761210  5194 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.803439  6044 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.804483 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.805279  5194 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.804930  6086 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.806581  6086 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.808360  5666 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.809569 24427 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260809 23:22:24.810460  5666 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.643039  6019 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.813730  6126 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:24.724498  6104 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.820418  6104 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.776161  6048 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.822036  6048 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.773519  6100 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.779129  5278 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.823284 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.822997  6048 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.824097  6048 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.684979  5545 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.827471  5545 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.781059 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.828992 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.829478 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61609.3us
I20260809 23:22:24.830300 24425 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.773012 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.795728  5670 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:24.796233  6011 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.804594  5813 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.772446  6020 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.805459 24428 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:24.763118  6102 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.833637 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.834523 24425 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:24.834456 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.781322  5959 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.775863  6073 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.810382  5903 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.840183 25371 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260809 23:22:24.797089  6036 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.854068 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.838538 25372 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:24.764936  5652 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.881768 24544 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 30. Current term is 31. Ops: []
I20260809 23:22:24.864243 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.870409 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.848927  6020 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:24.860397 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.766876  6004 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:24.769944  6051 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.885380  5903 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:24.886008  6003 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260809 23:22:24.885620  6051 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:24.760805  6002 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.652s	user 0.009s	sys 0.003s
I20260809 23:22:24.886462  6003 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.887171  6002 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.888298 24428 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:24.888828  6003 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.862164  6102 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.889827 24534 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.838380  6035 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.846416  5946 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.046s
I20260809 23:22:24.893050  5946 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.892660  6020 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.769564  5718 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.110s
I20260809 23:22:24.894027  6044 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.895197  5652 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.896175  6073 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.896821 24429 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.898459  5959 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:24.814445  6126 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.900617 24425 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260809 23:22:24.901547  6126 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.901989 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.904217 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.886411  6053 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260809 23:22:24.909714 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 298225us
I20260809 23:22:24.910425 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.891335  6102 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.893409  5946 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.904541 24424 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
W20260809 23:22:24.859398  6063 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
I20260809 23:22:24.884119  6004 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.843010  5670 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.875473  5571 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:24.915602 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51470us
I20260809 23:22:24.918031 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.920305 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.922505 24535 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260809 23:22:24.922895 24535 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
W20260809 23:22:24.920501 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 333796us
I20260809 23:22:24.847079  6005 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260809 23:22:24.894687  6020 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.923146 24535 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260809 23:22:24.967839 24535 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:24.919656  6102 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:22:24.966336  6073 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.006s
I20260809 23:22:24.971999 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.972472 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.025622 25366 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.972959 25375 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260809 23:22:24.973595  6019 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:24.974004  6039 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260809 23:22:25.032121 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50333.6us
I20260809 23:22:24.975167 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.977278 24430 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.978358 24428 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.979458  5946 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:24.983553  6117 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:22:25.036968 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.823009  6100 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.988076  6000 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:24.992053 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.997370 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1871) took 1012 ms. Trace:
I20260809 23:22:24.998284 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1867) took 1340 ms. Trace:
W20260809 23:22:25.044682 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52921.3us
I20260809 23:22:25.044867 24429 rpcz_store.cc:276] 0809 23:22:23.658350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.556210 (+897860us) service_pool.cc:227] Handling call
0809 23:22:24.556616 (+   406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.897206 (+340590us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.998062 (+100856us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.998229 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781969152}
I20260809 23:22:25.009339  6068 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.010062 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.012630  6007 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:25.030558 25373 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.031991 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.032882 24424 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.974535 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:25.034984 24378 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.043174 24428 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:25.030351  5959 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.061503 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.044153 24422 rpcz_store.cc:276] 0809 23:22:23.984935 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.602390 (+617455us) service_pool.cc:227] Handling call
0809 23:22:24.832021 (+229631us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.832185 (+   164us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:24.832209 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:24.832248 (+    39us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:24.832276 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.832329 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.997187 (+164858us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540012fcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.997302 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472936704}
I20260809 23:22:25.048017 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.053081 24429 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.045827 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.064771 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:25.066802  5646 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:25.067062  5166 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:25.054059 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.075251 24378 leader_election.cc:334] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.075938 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.077519 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.079284 24378 leader_election.cc:334] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.079946 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.081878 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.961104  5643 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260809 23:22:25.084265 24423 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.963065  5571 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.086545 24378 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.918551  6063 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.965216  5545 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.087s
I20260809 23:22:25.043846  6000 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.091547 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.100204 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.052446  6011 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:25.042716  6065 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.057s
W20260809 23:22:25.127907 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.128396 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.129372 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.129693 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.097558  6095 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.050453  6067 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260809 23:22:25.083834  6073 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.139840 24542 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260809 23:22:25.084527  5946 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.096907  5965 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:25.138813 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47349.7us
I20260809 23:22:25.206032 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.058591  6012 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.079s
I20260809 23:22:25.208948 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:25.098562  6039 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:25.098784  5752 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:25.103968  6036 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20260809 23:22:25.135289  5813 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.135111  6091 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:25.121917  5646 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.091s
W20260809 23:22:25.156288  5622 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.002s
I20260809 23:22:25.136912  6126 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:25.234280  6124 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260809 23:22:25.154836  6104 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:22:25.154620  6127 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.120817  6031 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.095s
I20260809 23:22:25.256791  6012 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 LEADER]: Stepping down as leader of term 30
W20260809 23:22:25.256959 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46363.6us
W20260809 23:22:25.263273  6033 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.002s
I20260809 23:22:25.257316  6129 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:22:25.258009  5752 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.259572 25375 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260809 23:22:25.104329  5959 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.263723 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57777.8us
I20260809 23:22:25.299638 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.216070  6011 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.283572 25366 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:25.228576  6039 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:25.291905  6003 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.284951  6004 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.284948  6118 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:25.289320  6012 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.334530 25375 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.348094 24422 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:25.354825 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:25.356563  6136 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260809 23:22:25.356961  6135 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260809 23:22:25.357543  5670 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.331214  6048 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.003s
I20260809 23:22:25.359442 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1864) took 1756 ms. Trace:
W20260809 23:22:25.360515 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.361819  5278 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.000s
I20260809 23:22:25.245121  6000 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.362632  6000 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:25.350342 24422 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:25.369912 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 239343us
I20260809 23:22:25.359791 24435 rpcz_store.cc:276] 0809 23:22:23.603035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.958118 (+355083us) service_pool.cc:227] Handling call
0809 23:22:23.970437 (+ 12319us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:23.980087 (+  9650us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:23.980125 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:24.665873 (+685748us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:24.693250 (+ 27377us) log.cc:842] Serialized 26 byte log entry
0809 23:22:25.091428 (+398178us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:25.091459 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:25.091584 (+   125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:25.359123 (+267539us) raft_consensus.cc:1671] finished
0809 23:22:25.359145 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:25.359378 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6616960,"thread_start_us":386941,"threads_started":1,"wal-append.queue_time_us":626002}
W20260809 23:22:25.370560 24378 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.370971 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.361529  6011 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.374856 24424 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
I20260809 23:22:25.381510 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:25.378832  5752 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.384654  6100 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.385445 25363 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:25.381884 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:25.386461 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.387138 25370 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:25.391916 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.379817 24422 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.396400 24427 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:25.395792 24424 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:25.397684 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:25.400019  5571 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.402482 24427 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
W20260809 23:22:25.372021 24378 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.393601 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.408923 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.409377 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.410915 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.402827 24427 pending_rounds.cc:76] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:22:25.379899  5796 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:25.387310  5813 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:25.418428 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46076.3us
I20260809 23:22:25.415444 24427 pending_rounds.cc:97] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260809 23:22:25.433226 24427 raft_consensus.cc:2866] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:25.424156 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.420279 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.396965  6068 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:25.439682 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.418378  6065 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:25.449761 24435 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.444034 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.430648  6091 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.448688 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.437501  5666 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:25.431370  6063 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:25.441654 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122370us
I20260809 23:22:25.402983  6095 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:25.412147  5718 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260809 23:22:25.478276 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.421095  5939 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260809 23:22:25.420961  6036 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
W20260809 23:22:25.493458 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44846.7us
I20260809 23:22:25.494973 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:25.329501  6053 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.148s
W20260809 23:22:25.510407 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.511464 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.514034 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.515403 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.468344  5887 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.542894 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.545671 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.546463 25375 raft_consensus.cc:2354] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.441998  6069 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.050s
W20260809 23:22:25.547241 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.453421  5796 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.559722 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.573454 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:25.563376 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.451900  6068 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:25.473613  5666 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:25.574831 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.579310 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.581141 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.501663  6063 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.577734 24378 leader_election.cc:398] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.588132 24378 leader_election.cc:401] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:25.477273  6065 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:25.383620  6039 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.561865  6005 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:25.594308 25370 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: [28.20-28.20]
W20260809 23:22:25.574609  5652 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.002s
I20260809 23:22:25.578950  5611 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260809 23:22:25.576583  5813 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260809 23:22:25.598305  5751 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:25.576306  5545 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:25.590324 24435 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.440932  6012 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.604389 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 182502us
I20260809 23:22:25.560305  6057 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.559089  6099 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.559214  5959 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.576268  6033 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.581372  5666 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:25.631911 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3610) took 1314 ms. Trace:
I20260809 23:22:25.658746 24423 rpcz_store.cc:276] 0809 23:22:24.317903 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.377891 (+ 59988us) service_pool.cc:227] Handling call
0809 23:22:25.631655 (+1253764us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540017f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:25.631826 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1577490816}
I20260809 23:22:25.585919  5887 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.636934 24539 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.608466  6068 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.623915  5571 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.505908  6095 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.635612  6140 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
W20260809 23:22:25.656083 24498 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.632625  6004 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:25.635303  6146 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260809 23:22:25.655233  6039 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:25.633795  5652 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.644378  5813 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:25.661828  6012 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:25.680563 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.674243  5571 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.688935  6095 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.689390  6004 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.677111  6065 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.700031 24498 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.705001 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.705965 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.706292 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.677666  6036 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.708307 25372 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:25.698474  5813 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:25.709190 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:25.721097 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.721557 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.659808  5801 consensus_queue.cc:1159] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 30 status { last_received { term: 28 index: 16 } last_committed_idx: 15 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:25.723484  5813 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.723959 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.724153 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.726207 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.726948 25363 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.726909 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.730947  6057 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.731465  6057 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.733554  6057 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:25.735230 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2124) took 1863 ms (client timeout 1866 ms). Trace:
W20260809 23:22:25.735761 25373 rpcz_store.cc:269] 0809 23:22:23.872257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.872367 (+   110us) service_pool.cc:227] Handling call
0809 23:22:24.796983 (+924616us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:22:25.734989 (+938006us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:25.735163 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514528256}
I20260809 23:22:25.738735  5882 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:25.738917  6005 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:25.716236  6036 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:25.743655 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1865) took 2131 ms. Trace:
I20260809 23:22:25.744369  6036 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.744660  6002 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:25.745103  6002 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:25.746409  6019 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.708s	user 0.009s	sys 0.002s
I20260809 23:22:25.747277  5571 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.748808  5882 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:25.672811  5796 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.745986  6002 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.772034 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 208707us
W20260809 23:22:25.772938 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.773447 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.744043 24431 rpcz_store.cc:276] 0809 23:22:23.611995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.557912 (+945917us) service_pool.cc:227] Handling call
0809 23:22:24.602914 (+ 45002us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.800625 (+197711us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:24.801014 (+   389us) log.cc:842] Serialized 14 byte log entry
0809 23:22:25.097871 (+296857us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:25.097927 (+    56us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:25.097970 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:25.743400 (+645430us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:25.743448 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:25.743597 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2246,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":5690,"spinlock_wait_cycles":464166016,"thread_start_us":296364,"threads_started":1,"wal-append.queue_time_us":445493}
W20260809 23:22:25.766877 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43531.4us
W20260809 23:22:25.767239 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46171.5us
I20260809 23:22:25.774411 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:25.775363 24498 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.674820  5887 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:25.775776 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.787992 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.789054 24545 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260809 23:22:25.787472 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.803912 24543 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260809 23:22:25.806147 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:25.810793 25323 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.811277 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.771091  5946 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.812918 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.814211 24427 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:25.814141 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.744715  5652 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:25.818702 24498 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.815840 24425 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260809 23:22:25.816681 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.802049 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.821014 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.773228  6069 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.822127 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.825371 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.834110 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.826870 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:25.835824 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.839972 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.836838 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.840606 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.824226  5751 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.818212  6002 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:25.823788  6086 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.979s	user 0.011s	sys 0.001s
W20260809 23:22:25.866729 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.824532  6005 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.836458  6165 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.854727  6167 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:25.857342  6057 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.905781 24543 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:25.859431  6032 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.914952 24543 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:25.876135  5882 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:25.927227 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110564us
I20260809 23:22:25.875154  5611 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:25.933710 25373 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260809 23:22:25.919694 24543 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.936810 24543 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.944334 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.947237 24535 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 30. Current term is 31. Ops: []
I20260809 23:22:25.945221 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
W20260809 23:22:25.952688 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 117222us
I20260809 23:22:25.953917 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.954715 24431 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260809 23:22:25.941565  6032 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.957662 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.950461  6000 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.587s	user 0.008s	sys 0.016s
W20260809 23:22:25.960721 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.968174  5666 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.968855  6162 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260809 23:22:25.971304 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.971388  5666 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.972572 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.974032 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.974404 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46449.6us
W20260809 23:22:25.974987 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.975911 25323 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.933373  5611 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.976776 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.977921  5611 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.977187 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.947606  6004 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:25.971869 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 371367us
I20260809 23:22:25.981712 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.982350 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.958565  6020 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.943s	user 0.012s	sys 0.009s
I20260809 23:22:25.990536 24430 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:25.995108 24431 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:25.982134 24378 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.996799 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.003098  5571 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.003682  5571 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.006022  5571 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.006891 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.007503 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.007745 24430 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 29: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:26.008196 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.008692  5751 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260809 23:22:26.011672 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.012327  5882 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.014616  5751 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.013887 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.021502 25363 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:26.021930 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.021380 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.022647  5751 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260809 23:22:26.023746 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.024199 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.025031  5611 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:26.024175 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.026394  5882 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.995631  5194 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.029892 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.031486 24541 pending_rounds.cc:76] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 16
I20260809 23:22:26.031414 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.031889 24541 pending_rounds.cc:97] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260809 23:22:26.020927  6167 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:26.032218 24541 raft_consensus.cc:2866] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:25.964375 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:26.033072 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68766.5us
I20260809 23:22:26.035717 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.036384 24534 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:26.036464 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54370.3us
W20260809 23:22:26.037889 24378 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.038266 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.039176 24545 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
W20260809 23:22:26.041149 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.041112 25375 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260809 23:22:26.042145 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.042701 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.048487 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.034328  6181 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260809 23:22:26.034183  6189 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260809 23:22:26.033412  6163 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260809 23:22:26.063854 24378 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.033560  6178 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:22:26.059394 24431 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:26.054836 25366 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.064378 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.068352  6190 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:26.066203  6068 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.005s
I20260809 23:22:26.071545  6068 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:26.071887  6068 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:26.072718  6068 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.073946  6068 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.047317  5891 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.075624  6011 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:26.076290  5891 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.078121 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.078480  6188 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.079116  6126 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.079655 24378 leader_election.cc:334] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.035277  6031 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.020s
I20260809 23:22:26.080155 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.083861 24543 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260809 23:22:26.084272 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:26.085116 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.085729 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.087157 24378 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.090606 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.091176 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.091245 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.093677 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.094276 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.048874  6167 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:26.047526  6020 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.097829 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.047233  6035 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.040251  5887 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.122036 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.123507 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:26.124625 24378 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.124743 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.125056 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.126148 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.126784 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.137975 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.079195  6036 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.079681  5611 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.066623  5751 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:25.966459  6032 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.098544  6126 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.125891  6129 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.121737  6185 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.123306  6073 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.158638  6190 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.146191  5887 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.150565  6060 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.209049 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 122412us
I20260809 23:22:26.123939  6033 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.207374 25369 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.145541  6020 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.145308  6091 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.153077  5965 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.217408 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.204887  5194 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.204459  5611 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.212718  6129 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.229576  5194 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:26.232215  6195 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260809 23:22:26.235095  6158 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.600s	user 0.003s	sys 0.000s
I20260809 23:22:26.212862  6203 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.249240  6203 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.249584  6203 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.206068  5751 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.079191  6188 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.227540  6005 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.243362  6129 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.237946  6060 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.245522  5891 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.224191  6020 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.225440  5887 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.237694  6033 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260809 23:22:26.264904  6203 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.348747 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:26.343448  6012 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.638s	user 0.010s	sys 0.002s
I20260809 23:22:26.359288  6188 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.360332  6069 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.000s
I20260809 23:22:26.360778  6069 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.363766  6196 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260809 23:22:26.357985  5484 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:22:26.365393  5484 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:26.366955  5611 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.271785  6005 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.369042 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.369843  6005 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.372135 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.363864  6032 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.292749  6060 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.375646  6036 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.369817 24542 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:26.377254  6060 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.378592 24498 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.366619  5484 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.380916 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:26.382505 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 338733us
W20260809 23:22:26.403331 24498 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.411551 25369 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:26.414179 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 187124us
W20260809 23:22:26.382771  6197 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260809 23:22:26.419355 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.424345 24542 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
W20260809 23:22:26.427316 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.427938 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.431555 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.434180 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.435240 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.437953 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.418227  5965 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.439479  5965 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.441702  5965 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.442812 24429 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:22:26.447845 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 223930us
I20260809 23:22:26.438544 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:26.415200  6005 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:26.417445  6151 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.464941 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.450420 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.436025 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.466404 24431 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
W20260809 23:22:26.469517 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.465623 24427 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.469986 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.440004  5484 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:26.473564 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.429646  5959 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.447515  6091 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.495752 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260809 23:22:26.491595 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.463532  5611 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.465848  6005 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:26.493487 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.465458  6032 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.491151 25368 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260809 23:22:26.501549 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.492373 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.506448 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.507469 24541 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 28 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:26.509991  6031 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:26.501588  5194 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.497586  5959 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.493399  6202 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260809 23:22:26.511520  6032 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.492391  5311 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.513028  5311 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.513342  5959 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.499650  6209 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.513358  5311 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.474328  6151 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.517042  6151 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.518594  5751 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.519024  6020 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.523034 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.500973  5611 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.528131 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:26.529229 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.529618 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.530751 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60291.8us
I20260809 23:22:26.530773 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.532444 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:26.479496  6210 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.537077 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.501111  6005 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.538024 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.538287 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.491801  5278 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.513470  5887 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.513622  6032 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.545688 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.546079 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.547521 24541 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 30: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:26.534652  5952 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:26.551785 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.527762  6020 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.539732  6206 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.555538 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83914.5us
W20260809 23:22:26.559520 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.554104  6210 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.565083 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.552289  6036 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:26.550737  6151 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.569526 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.500171  6091 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.570195 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.554751  5959 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.571403 25368 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:26.572880 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.574838 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.554473  5952 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:26.559180  6020 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.549576  5278 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.574949  6151 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.566129  6210 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.587141 24425 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:26.576674  6216 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260809 23:22:26.587808 25373 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260809 23:22:26.576767 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.588740 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.589398 25368 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:26.589951  6033 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.578998  5611 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.590435  6033 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.583799  6091 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.585341  6210 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.591809  6005 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.592830  5952 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.594091  5952 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.594605  6005 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.594651  6213 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.595508  6213 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.597482  6213 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.586556  6151 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.601900 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2146) took 1671 ms. Trace:
I20260809 23:22:26.604274 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.605733 25369 raft_consensus.cc:2354] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:26.586452  6068 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.002s
I20260809 23:22:26.564189  6202 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:26.602555 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.607388 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:26.608225  5885 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260809 23:22:26.620834  6209 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:26.602269 25371 rpcz_store.cc:276] 0809 23:22:24.927050 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.927198 (+   148us) service_pool.cc:227] Handling call
0809 23:22:25.753084 (+825886us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:25.753413 (+   329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:25.753441 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:25.779580 (+ 26139us) log.cc:842] Serialized 26 byte log entry
0809 23:22:25.962029 (+182449us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:25.987765 (+ 25736us) log.cc:842] Serialized 14 byte log entry
0809 23:22:26.034644 (+ 46879us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.034714 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:26.216045 (+181331us) raft_consensus.cc:1671] finished
0809 23:22:26.216065 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:26.598722 (+382657us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b54007f03d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:26.598865 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53945,"prepare.queue_time_us":66880,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":47100,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":18627,"spinlock_wait_cycles":1898590464,"thread_start_us":108937,"threads_started":1,"wal-append.queue_time_us":161683}
I20260809 23:22:26.581390  6020 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.624416 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.584048  5278 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.626397 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.614560 24427 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260809 23:22:26.627336 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.627558 24427 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 19
I20260809 23:22:26.627943 24427 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260809 23:22:26.628274 24427 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:26.616664 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.629377 24541 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.617475  6129 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.590847  6033 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.622475  5670 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.537410  6154 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260809 23:22:26.641607 25363 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
W20260809 23:22:26.642973 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.643723 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.644613 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.645392 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
W20260809 23:22:26.651425 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.651738 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.646178 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.627246  6202 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.640343  5801 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.012s
W20260809 23:22:26.653309 24378 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.653785 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.655834 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.656186 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.627602  6005 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.640277  6216 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:26.640614  5611 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:26.660507 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.586913  5311 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.660574 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.652470  5494 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.662223  6216 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.662375  6005 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.662803  6068 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:26.671036  6166 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.683800  6129 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.655205 24428 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:26.655217  6202 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.622777  6210 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.661800  6057 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
W20260809 23:22:26.692162 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.664822  5611 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.687343  6095 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.689357  6216 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:26.690657  6213 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.693928  6220 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.705070 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.698024  5611 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.708938 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.709800 24435 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:26.698000  6226 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.700547  5889 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.690060  5494 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.716972  6005 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.719712 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:26.704643  6213 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.739274 24422 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:26.739400  6213 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.726904 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.740262 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.741729 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.721287  5278 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.746728 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.747902 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.749128 24544 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:26.707650  6220 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.758706 24542 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.758563 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:26.764753 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.765894  5311 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.768112 25371 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260809 23:22:26.772297 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:26.752285  6011 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.772588 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.775683 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.776158 24431 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.772303 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.774428 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.752126  6005 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:26.777446 24430 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.783133  6068 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.773053  5887 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.783635  6068 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.784322  5887 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.786634  5887 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.788292  6118 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:26.793917 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.794692 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.796201 25371 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:26.797124 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.797300 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.791004  6068 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.799963  5494 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.804500  6033 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.777311  6011 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.805727  6011 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.806196  5718 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:26.806746  5718 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.809455  5718 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.777841  6020 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260809 23:22:26.810292  5946 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.810907  5646 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:26.736714  5889 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.806649  6033 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.881119 24428 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.813210  5611 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.885417 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.886252 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.816830 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.888052 24544 raft_consensus.cc:2354] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:26.887697 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111954us
I20260809 23:22:26.817994 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.836622 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 4044) took 1424 ms. Trace:
I20260809 23:22:26.885460  6051 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.866421 24422 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
W20260809 23:22:26.839192 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 180132us
I20260809 23:22:26.848384 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:26.863140 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.879956 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.816438 24541 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.818773 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.819235 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.909690 24542 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:26.909976 24534 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:26.820115 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.827320 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.832934 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:26.890655 24535 rpcz_store.cc:276] 0809 23:22:25.412545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.413031 (+   486us) service_pool.cc:227] Handling call
0809 23:22:25.413327 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.955135 (+541808us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.836393 (+881258us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:26.836565 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1227300224}
I20260809 23:22:26.838364 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.915912 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:26.850448 24423 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:22:26.863950 24378 leader_election.cc:398] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:26.840535 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1984) took 1913 ms. Trace:
I20260809 23:22:26.831207 24430 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:26.878072 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.863387 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:26.919735 25373 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.917752 24378 leader_election.cc:401] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:26.906495 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.904896 24422 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 17
I20260809 23:22:26.925443 24422 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260809 23:22:26.925796 24422 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:26.925828 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.926525 24544 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.926602 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4080) took 1598 ms. Trace:
I20260809 23:22:26.853412 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.891238 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:26.929965 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66239.8us
I20260809 23:22:26.918067 25372 rpcz_store.cc:276] 0809 23:22:24.926865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.927005 (+   140us) service_pool.cc:227] Handling call
0809 23:22:24.927293 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.716586 (+789293us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.840204 (+1123618us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:26.840460 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1795622144}
I20260809 23:22:26.934592 24380 leader_election.cc:398] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:26.939769  6030 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.927052 24422 rpcz_store.cc:276] 0809 23:22:25.328264 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.328751 (+   487us) service_pool.cc:227] Handling call
0809 23:22:25.347960 (+ 19209us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:26.738747 (+1390787us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b5800a4fcb0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:22:26.926354 (+187607us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.926543 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":982050560}
I20260809 23:22:26.940598  6030 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.941264  6207 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260809 23:22:26.843704  6133 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.598s	user 0.005s	sys 0.005s
W20260809 23:22:26.873941  6199 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.520s	user 0.000s	sys 0.001s
I20260809 23:22:26.898430  5280 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
W20260809 23:22:26.906514 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58219.3us
I20260809 23:22:26.950079  6002 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.955979 25371 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:26.955713  6002 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.957839  6002 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.855152  5643 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.945559 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.866106  5278 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.823637  5646 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:26.834095  6205 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.484s	user 0.001s	sys 0.000s
I20260809 23:22:26.862720  6011 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.976646  6011 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.977094  6030 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.947094 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.824687  6220 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.951713 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2155) took 1228 ms. Trace:
I20260809 23:22:26.827070  6211 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:26.978644  6211 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.979077  6211 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.917166  6007 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.949906  6044 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.982157  6044 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.984234  6044 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.846019  5813 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.985824 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.986711 24427 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:26.987393 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:26.870998  6067 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:26.828019  5889 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.820073  6118 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:26.989114  6067 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:26.990123  6067 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.991850  6067 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.982175  6007 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.989084 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.995577 24427 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:26.995824  6007 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.996069  5959 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.996616  5959 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.990185  6118 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.997978  6118 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:26.999269  5611 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.943064  5965 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:22:27.000490 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.002169  6051 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:27.002573  6051 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.002883  6051 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.841444  6216 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.005187  6216 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:27.005836  6234 consensus_queue.cc:1159] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 30 status { last_received { term: 29 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:26.867940  6057 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:27.007248  5646 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.007642 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60678.3us
I20260809 23:22:27.009591  6216 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.010317  6216 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.010782  6216 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.011597  5278 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.929586  6003 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:26.945122 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53964.5us
I20260809 23:22:26.843816  6086 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.872633  5891 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:26.810890  5946 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.890125  5752 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.019585 25373 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:26.814989  6210 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.967182 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.971279  5643 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.977932 25366 rpcz_store.cc:276] 0809 23:22:25.723671 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.723833 (+   162us) service_pool.cc:227] Handling call
0809 23:22:26.951498 (+1227665us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:26.951638 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584899456}
I20260809 23:22:27.033203 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.034148 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.034893 25366 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.001904  5951 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260809 23:22:26.930554  6071 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:26.963357 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.970549 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.040293 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.015923 25372 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:27.017014 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.042613 24430 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.017325 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.044839 24423 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.965966 25363 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.020576 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.021381 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.023164 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.023854 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.021457  5752 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.019428  5891 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.024721 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.966075  5882 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.961885  5796 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.038223 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:27.057703 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94407.9us
I20260809 23:22:26.868017  6202 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.050609 25373 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.059329 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.050647 24535 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.052446 24541 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.054482 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.050871 24427 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.060693 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:27.067288 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.065541 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2156) took 1340 ms. Trace:
I20260809 23:22:27.024809  5311 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260809 23:22:27.054328  5752 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.079857 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.097288 24534 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.082206 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.086100 24544 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:27.070319 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.089243 24430 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:27.099242 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61151.9us
I20260809 23:22:27.051499  6071 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:27.051735  6127 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.063407  5882 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.106474  5882 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.098404 25369 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.022073  6086 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.118083 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:27.122269 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52063.9us
I20260809 23:22:27.123615 24427 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.125603 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.112478 24535 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:27.102563 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.166396  5494 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.120400  5278 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.168994  5494 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.170302 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59396.1us
I20260809 23:22:27.114059 25366 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
I20260809 23:22:27.019294  6003 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.170964 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.133379  6007 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.115351 24534 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
I20260809 23:22:27.111383  6071 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:27.171582  6007 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.172013  6007 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.172845  6207 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.173344  6207 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.100365  6053 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.174186  6207 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.122045 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2162) took 1093 ms. Trace:
I20260809 23:22:27.141630  6086 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.072731  6033 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.118687  5946 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.151638 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.151829  6051 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.158299  5965 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:27.158833  5891 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.071256  6020 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:27.163501 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.227296 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63949.4us
I20260809 23:22:27.164183  5889 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:22:27.166429 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50688.9us
I20260809 23:22:27.171048  5494 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.174773  6053 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.118202  5796 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.167315  5813 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.177572 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.167320 24424 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260809 23:22:27.230034 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.231971 24424 pending_rounds.cc:76] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 16
I20260809 23:22:27.232226 24424 pending_rounds.cc:97] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260809 23:22:27.232499 24424 raft_consensus.cc:2866] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:27.132525  6044 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:27.233110  6202 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.233678  6044 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:27.234115  6044 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:27.127919  6127 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.235066  6044 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.129130 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.179088 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.235913  5813 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.236737  6127 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.236923  6220 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:27.237507  6220 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.237921  5813 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.238469  6051 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.223166  6210 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.239445  6210 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.240561 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.241190 24543 raft_consensus.cc:2362] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27. Re-sending same reply.
I20260809 23:22:27.241475  6210 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.246064 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.246464  5946 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:22:27.246969 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.247027  6053 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.246721 24543 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:27.248375  5889 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.248593 24435 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
I20260809 23:22:27.249336  5889 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.126552 24423 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.252599  6216 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.253674 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.252727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.254979 24431 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.134171  6211 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.257393  6211 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.257907  6007 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.183169 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.189651  6068 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.214500  5903 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.264369 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:27.239887  6142 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.611s	user 0.002s	sys 0.000s
I20260809 23:22:27.238402  6220 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.249719 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.236541 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57536.5us
I20260809 23:22:27.262282 24427 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:27.171741  6102 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 30 is higher.
I20260809 23:22:27.226991  6020 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.180217 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.269451 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.285128 24423 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:27.274199 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:27.276258 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46313.8us
I20260809 23:22:27.278584 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.192844 25365 rpcz_store.cc:276] 0809 23:22:26.028618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:26.029004 (+   386us) service_pool.cc:227] Handling call
0809 23:22:26.029344 (+   340us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:26.037943 (+  8599us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:26.037973 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:26.038349 (+   376us) log.cc:842] Serialized 26 byte log entry
0809 23:22:26.157990 (+119641us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:26.158050 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.158144 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:26.873380 (+715236us) spinlock_profiling.cc:243] Waited 9.75 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:27.121741 (+248361us) raft_consensus.cc:1671] finished
0809 23:22:27.121768 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:27.121982 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15570,"raft.queue_time_us":10568,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8819,"spinlock_wait_cycles":22414592,"thread_start_us":74090,"threads_started":1,"wal-append.queue_time_us":211034}
I20260809 23:22:27.281342 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.281400 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.278302  5903 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.246965  5946 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.305294 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.306406 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.294250 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.309007 24430 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.316486 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.317317 24534 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.318501 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.318760 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.280498  6210 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.281008  6151 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:27.289043  6020 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:27.277910  6202 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:27.278028  6068 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.312688  6099 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.329631 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.315685 24431 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.323463 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.336962 24544 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.311504  5946 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.338290 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.300395  6003 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.311714  6216 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.342658 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.319005  5278 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.324234  6086 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.331465  5494 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:27.320770  6057 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.327353  6210 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.331699  6151 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:27.353165 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.300701  5752 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.360955 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.376447 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.377373 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.352046  6067 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.364024 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.366636  6068 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.360477  6210 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.379889 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.380774 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.383531 25375 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.377976 25373 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.389361 25369 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.372256  6151 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.392168 25368 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:27.363150 24544 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.378517  6048 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.402838  6211 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:27.403672  6211 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.405264  6011 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.405738  6011 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.408226  6000 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:27.383122  6002 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.412292  6002 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:27.412923 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.413319  6002 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.442863 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63074.3us
I20260809 23:22:27.388803 24430 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:27.378666  5494 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.354790  6086 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.457765 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.469508 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.360221  6057 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.473090 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:27.352156  5796 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.469079  6011 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.482170 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.449719  5611 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.503568 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.476760  6000 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:27.508001 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.472666  5622 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.480939  6019 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:22:27.513095 24378 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.510946 24430 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:27.483435  5903 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.500226  5280 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:27.515218 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.513923  6211 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.517098  5622 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.524398  5622 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.529596  6204 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:27.530122  6204 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.530519  6204 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.519495  6000 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.531466  6000 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.532459  6053 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.533833  6000 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:27.550915 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 173391us
I20260809 23:22:27.554879 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.466634 25373 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:27.565858 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.518289  6033 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.507256  5611 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.549796  6053 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.548815  6019 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.577442 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.579700 25373 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.583222  6033 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.509938  5494 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.521446  6071 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.554674  5280 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.587288  5752 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.617576  5280 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.619117 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 729928us
I20260809 23:22:27.622367 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.624305  6086 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.630005 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:27.631238  5484 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:27.632601  5484 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.633056  5484 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:27.633908  5484 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.633073 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.635515 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3638) took 1512 ms. Trace:
I20260809 23:22:27.635841 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:27.635847 24424 rpcz_store.cc:276] 0809 23:22:26.123406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:26.123953 (+   547us) service_pool.cc:227] Handling call
0809 23:22:26.124250 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:27.232942 (+1108692us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:27.635278 (+402336us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:27.635458 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2398001920}
I20260809 23:22:27.641500 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.646972 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.622778 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
W20260809 23:22:27.655084 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 211159us
I20260809 23:22:27.656610 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.657253 24424 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.661680 24543 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.661798 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.663581  6067 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:27.664507  6067 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.665344  5752 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.666137  5752 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.675750 24424 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:27.684237 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.685186 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.685497  5611 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.686213 24423 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:27.688990 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.689857  6245 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.697700 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.659304  6068 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.590893  6019 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.595700  6053 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.087s
I20260809 23:22:27.661393  6086 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.058s
I20260809 23:22:27.707428  6071 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.703325 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:27.744434 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.745371 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.757515 24544 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.706055  6057 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.679390  6020 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.760937  6020 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.708703  5946 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.762334 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.734298 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.767076 25369 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:27.747802 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.766299 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.785727 24422 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:27.787467  5889 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:27.754796 25373 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:27.777843 25369 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
W20260809 23:22:27.790026 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 280123us
I20260809 23:22:27.790442 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.768181 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64964.3us
I20260809 23:22:27.767400 24424 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.769131 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.738644  5646 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.733144  6234 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.706964  5752 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.772693 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.795215 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.732599  6068 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.799039  6256 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:27.801898 24535 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.801898 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.795068  6020 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.809689 24424 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:27.796196  6003 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:27.808352 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:27.815506  6245 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:27.816020  6245 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.816403  6245 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.816519 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.816784 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.817485 25369 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:27.822417 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.806437  6102 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.811311  6020 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.835125 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.825721  6003 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.802973  5646 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.829972  6263 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260809 23:22:27.874572  5946 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.875301 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.810689  6256 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.831686  6020 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:27.870985  5278 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.007s
I20260809 23:22:27.888659 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.804791  6234 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.892879 24429 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:27.893532 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58507.1us
W20260809 23:22:27.910342 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:27.911124 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:27.911913 24498 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.912154 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.877977  6102 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:27.906177 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.913100  6207 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:27.913704 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.914768  6266 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.838754  6210 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.914077 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.928373 25369 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:27.903038  6251 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:27.915700  5278 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.920601  5646 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.920939  6068 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.929528  6210 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.924149 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.895421  5751 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.931094  5751 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.931629  6003 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.933334  5751 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.935691 25375 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:27.936501 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.937093 24546 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.943481 25369 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:27.944422 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 200971us
I20260809 23:22:27.944839 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.945474  5278 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.945830 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:27.946153  5278 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.948239 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.948284  6204 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.948995  5278 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.956748  6204 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.956985  6256 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.903223  6057 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.957975  6057 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.960879  6057 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.071131 25370 rpcz_store.cc:276] 0809 23:22:25.725435 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.754635 (+ 29200us) service_pool.cc:227] Handling call
0809 23:22:25.805205 (+ 50570us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.805480 (+   275us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:26.867056 (+1061576us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b180031bb80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:22:26.867216 (+   160us) log.cc:842] Serialized 14 byte log entry
0809 23:22:27.065069 (+197853us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:27.065154 (+    85us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:27.065222 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:27.065310 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:27.065469 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25045,"prepare.queue_time_us":140245,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":1767,"raft.queue_time_us":175,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":15044,"spinlock_wait_cycles":11444480,"thread_start_us":309411,"threads_started":2}
I20260809 23:22:27.918118  6019 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:27.991389 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.915413  6207 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:28.001758 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.005379 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.939570  5889 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.007782 24546 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:28.014519 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.962026  6251 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:28.022482 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.996630  6207 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.996193  6167 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.040274 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.041168 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.056913 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:28.045334 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 222990us
I20260809 23:22:28.053983  5643 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.058179 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:28.066692 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.013628  6019 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:28.040169  6224 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.125578  6224 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.024307  5622 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:28.127275  5622 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:28.128255  5622 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.130090  6007 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.131565 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.076460 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:28.133145 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66511.9us
I20260809 23:22:28.134315  6091 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:28.077306 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:28.136106 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58907.6us
I20260809 23:22:28.064821 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.059749  6129 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:28.084108 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60030.5us
I20260809 23:22:28.089208  5952 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
W20260809 23:22:28.093127  6151 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.695s	user 0.015s	sys 0.007s
I20260809 23:22:28.071702 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.022047  5889 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.039961  6207 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.123662  6245 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.120983  6251 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:28.139529 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74822.4us
I20260809 23:22:28.078157  5494 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:28.144147 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:28.146564 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 234638us
I20260809 23:22:28.150657 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.068101  5484 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:28.120431  6234 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.173913 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:28.176755 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.177309 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.177553 24545 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:22:28.171448 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.180310 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.183236 24541 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260809 23:22:28.187727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.145339  5952 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:28.188736  6234 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:28.184090 25370 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:28.143932  6033 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.155323  5959 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.152172  5280 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.152750  6151 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.197589 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.220537 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.203352 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.204216  6220 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:28.206159 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.194268  6086 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.210630 24431 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:28.153321  6267 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.217854 24540 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:28.183027  6204 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:28.257398  5952 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.201455  5643 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260809 23:22:28.267256 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.270588 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.219424  6102 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.274564 24542 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 31.
I20260809 23:22:28.274212 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.251780  5751 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.252935  6168 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:28.295914 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 115642us
I20260809 23:22:28.285137 25368 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260809 23:22:28.350835 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.352185 24431 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:28.266214  6086 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.274660  6224 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.344508  5952 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:28.370949 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195842us
I20260809 23:22:28.335400  6003 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.350783  6030 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:28.382095  6295 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260809 23:22:28.382433  6030 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.384661  6030 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.382126  6033 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.004s
I20260809 23:22:28.387401  6033 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.387856  6033 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.389786  6033 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:28.392584 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:28.395186 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.269726  6267 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.421291 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.434533 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.435500 24544 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:28.439162 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:28.441965 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.442613 24544 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:28.339320  6168 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.452057 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 31.
I20260809 23:22:28.453454 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:28.258697  6019 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.465204  6204 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.472409 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.473652  5952 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:28.474100  5611 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:28.474303  6293 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260809 23:22:28.474651  6012 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:28.476486  6102 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:28.477119  6102 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:28.481667  5611 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.482735  5891 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:28.490298  6086 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.491328  5903 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:28.491976  5903 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:28.492918  5903 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.500684 24378 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 146 similar messages]
I20260809 23:22:28.498771 24545 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:28.440974  6245 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.497515 25368 pending_rounds.cc:76] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 17
I20260809 23:22:28.512714 25368 pending_rounds.cc:97] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260809 23:22:28.513028 25368 raft_consensus.cc:2866] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:28.452557  5889 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.416946  6003 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.496945  6102 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:28.518695 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.519364  6316 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.512013  6057 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:28.520706  6267 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:28.541411 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:28.557477 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 136338us
I20260809 23:22:28.564745 24544 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:28.557772 24425 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
W20260809 23:22:28.561934 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4046) took 2846 ms (client timeout 1852 ms). Trace:
W20260809 23:22:28.516104  6295 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260809 23:22:28.566542 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:28.566210 24539 rpcz_store.cc:269] 0809 23:22:25.696806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.696957 (+   151us) service_pool.cc:227] Handling call
0809 23:22:28.542673 (+2845716us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.542833 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2919915392}
I20260809 23:22:28.535822  6102 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.562881  6151 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.574476  6245 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.576292  6102 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:28.579164  5796 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
W20260809 23:22:28.589347 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 136015us
I20260809 23:22:28.595448 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:22:28.598832 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.590024 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.596902  6020 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.000s
I20260809 23:22:28.593117 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.571201  6316 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.589416  5751 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.604923  5891 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260809 23:22:28.616772 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.588001  6319 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.619144 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.626758 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:28.627672  5891 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.612176  6020 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.634263  6207 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.643045  6319 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.599432  6245 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.621812  6320 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.611755  6168 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.673609 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.680601 24546 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:28.669952 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.686165 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.691102 24542 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:28.666915  5891 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:28.694362 24422 raft_consensus.cc:2354] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
W20260809 23:22:28.688843 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69877.4us
I20260809 23:22:28.706975  5889 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.724108 24380 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:28.724448 24380 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:28.710436 24542 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:28.692081  6322 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.668921  6316 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.669024  6207 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.720750  6020 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.726416 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:28.727671 24378 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.734675 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.735795  6003 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.735803  5751 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.736475  5751 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.736724 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.738806  5751 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.745124  6303 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.018s
W20260809 23:22:28.755206 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.741375  6305 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.747293 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.766431 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
W20260809 23:22:28.767177 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.767719 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:28.770752 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3668) took 1502 ms. Trace:
I20260809 23:22:28.771174  6303 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:28.771142 24435 rpcz_store.cc:276] 0809 23:22:27.268658 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.277224 (+  8566us) service_pool.cc:227] Handling call
0809 23:22:27.306226 (+ 29002us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:27.350391 (+ 44165us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:22:27.364557 (+ 14166us) log.cc:842] Serialized 14 byte log entry
0809 23:22:28.048153 (+683596us) spinlock_profiling.cc:243] Waited 7.47 ms on lock 0x7b180032f380. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:28.215585 (+167432us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:28.217210 (+  1625us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:28.217280 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:28.770535 (+553255us) spinlock_profiling.cc:243] Waited 43.9 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:28.770583 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:28.770697 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13879,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3121,"raft.queue_time_us":2296,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":31924,"spinlock_wait_cycles":118229632,"thread_start_us":167133,"threads_started":1,"wal-append.queue_time_us":1010082}
W20260809 23:22:28.772575 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 597288us
W20260809 23:22:28.773224 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.773834 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4091) took 1445 ms. Trace:
W20260809 23:22:28.774111 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.774224 24535 rpcz_store.cc:276] 0809 23:22:27.328767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.329084 (+   317us) service_pool.cc:227] Handling call
0809 23:22:28.773641 (+1444557us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.773772 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069730688}
I20260809 23:22:28.775460 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4082) took 1459 ms. Trace:
I20260809 23:22:28.732752  6306 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.775403  5891 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.776026 24543 rpcz_store.cc:276] 0809 23:22:27.315685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.318240 (+  2555us) service_pool.cc:227] Handling call
0809 23:22:28.775286 (+1457046us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400270bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.775408 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784302848}
I20260809 23:22:28.776407  6306 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.776881 24425 raft_consensus.cc:2354] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
I20260809 23:22:28.777864  5891 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.778458  6306 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.778786 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.683671  6325 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:28.780524 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.782727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.786743 24498 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:28.787911 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.766124  6305 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:28.755048 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.730187  6323 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.747643  6044 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:28.799026 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.810324 24435 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:28.745177  5646 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.816355 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.793968  6099 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.819128 25373 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
I20260809 23:22:28.771682  6303 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.820333 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.730816  5889 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.778744  6102 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.829128 22524 thread.cc:536] Waited for 1000ms trying to join with heartbeat (tid 24356)
W20260809 23:22:28.843526 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.828908  6303 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:28.848399 24424 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:28.795316  6325 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.856812  6102 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.858306 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1920) took 1170 ms. Trace:
I20260809 23:22:28.864574  6325 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.863843 24423 rpcz_store.cc:276] 0809 23:22:27.687496 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.687630 (+   134us) service_pool.cc:227] Handling call
0809 23:22:27.687905 (+   275us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:27.688192 (+   287us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:27.688214 (+    22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:28.648197 (+959983us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:28.648740 (+   543us) log.cc:842] Serialized 26 byte log entry
0809 23:22:28.781300 (+132560us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:28.781360 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:28.781452 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:28.857926 (+ 76474us) raft_consensus.cc:1671] finished
0809 23:22:28.857951 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:28.858228 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":324,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":11399424,"thread_start_us":132114,"threads_started":1,"wal-append.queue_time_us":172244}
I20260809 23:22:28.868345 25365 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:28.869563  5889 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.871151 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.871950 24541 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 31: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:28.875771  6319 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:28.885749  6285 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.879s	user 0.000s	sys 0.006s
I20260809 23:22:28.896502  6305 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:28.899175 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110382us
W20260809 23:22:28.900269 24498 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.826129  5646 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.904138 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.908155 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.908881 25369 raft_consensus.cc:2354] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:28.911520 25369 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:28.912200 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.912403 25370 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:28.919412 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.921950  5646 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.924355 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.925354 25371 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.927680 25366 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:28.929885 25366 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:28.931274 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.923039  6303 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.935571 25370 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:28.937703 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.936764 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.943602 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.945454 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.945955 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.956172 25363 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:28.960384 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:28.971796 24541 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260809 23:22:28.972307 24541 pending_rounds.cc:76] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 15
I20260809 23:22:28.972579 24541 pending_rounds.cc:97] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260809 23:22:28.972855 24541 raft_consensus.cc:2866] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:28.978327 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.979553  6224 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:28.979924  6224 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.980247  6224 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:28.981228 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.983448  5611 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:28.984575  5278 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:28.985450  6303 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.987682  6207 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:22:28.990108 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44291.5us
I20260809 23:22:28.991106  6213 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.996532 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 4118) took 1125 ms. Trace:
I20260809 23:22:28.997298 24422 rpcz_store.cc:276] 0809 23:22:27.871335 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.871563 (+   228us) service_pool.cc:227] Handling call
0809 23:22:28.996163 (+1124600us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.996309 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1833029376}
I20260809 23:22:28.999296 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.999914 24542 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:28.945173  6000 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260809 23:22:28.978539  6263 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:29.003440  6263 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.003844  6263 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:28.956609  6091 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260809 23:22:28.961369  5643 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.005681  5889 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:29.006372 24540 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:29.008004  5280 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:28.947809  6323 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
W20260809 23:22:29.008734 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 162716us
I20260809 23:22:28.978370 24534 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 27 index: 24. (index mismatch)
I20260809 23:22:29.025513 24431 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260809 23:22:29.028427 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.035451 24542 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:29.037029 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:29.036374 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.043553 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:29.053077 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.055277 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.010164  6277 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.070760  6266 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:29.075687  6086 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:29.076207  6086 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:29.076346 24496 leader_election.cc:398] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
I20260809 23:22:29.076686 24496 leader_election.cc:401] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.076563  6277 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.077132  6086 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.078773  6167 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:29.080726 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.089993 24498 leader_election.cc:398] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:29.090588 24498 leader_election.cc:401] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.068316  6166 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:29.094683 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.040155  6000 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
W20260809 23:22:29.115561 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 347778us
W20260809 23:22:29.117462 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:29.123816 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.132845  5484 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.117250  5311 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:29.133409  5484 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.136276  5484 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.139683 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.149374 24496 leader_election.cc:398] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
I20260809 23:22:29.149745 24496 leader_election.cc:401] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.090816  5278 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260809 23:22:29.165365  6245 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.168174  6033 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.171653 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:29.171939 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49081.7us
W20260809 23:22:29.174366 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48376.2us
I20260809 23:22:29.180693 25371 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:29.117650  6166 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:29.183781 25368 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:29.124742  6211 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:29.190130 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:29.194779 24424 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:29.174623  6325 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.174319  5752 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260809 23:22:29.206270  6274 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:29.214988  6003 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:29.186625  6033 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.215294  6245 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:29.232446 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50374.5us
I20260809 23:22:29.198280  6048 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.029s
I20260809 23:22:29.217775  6102 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.219681  5891 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:29.235083 24378 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.230804  6003 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:29.228585  6207 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.230248  6216 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.230229  5951 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260809 23:22:29.221923  6325 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.244133  6044 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.028s
I20260809 23:22:29.245190  6245 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.245442  6102 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.251034  5891 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.279120 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.291311 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:29.301491 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 532874us
W20260809 23:22:29.301928 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.262517  6003 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:29.304397 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.305032 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.302237 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:29.310909 24498 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.270689  6263 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.320116 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.313673  6262 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:29.282055  6207 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:29.328054 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.319904  6234 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.332024  6234 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.333470  5891 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.334303  6234 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:29.334534  6358 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260809 23:22:29.339645  6102 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.341349  6033 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.289639  6216 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.344710  6216 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.316192  6249 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.349792 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.351223  6211 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:29.352075  6325 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.352182 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.355916  6263 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.242332  6166 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.329753 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:29.366266 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.366753 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.325383  6364 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.377243  6224 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.381821  6167 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
W20260809 23:22:29.385751 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:29.387233 25323 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.387871 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.392994 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.369594  6086 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.404321 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.415148 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:29.329864  6289 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:29.442260 24431 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:29.382521  6277 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.422472 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.455144 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.456943 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.478885 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:29.483618 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 176004us
W20260809 23:22:29.490628 24498 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.491226 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.490655 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.492779 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.398702  6305 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:29.492833 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:29.416991  6303 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.496940 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:29.504588  6033 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.508306 24428 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 27 index: 24. (index mismatch)
I20260809 23:22:29.418002  6319 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:29.518443 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.519110 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:29.519516 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.519936 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.520310 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:29.521128 24542 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:29.523476 24428 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:29.523377 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.527539 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.517169 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.535363 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.537468 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.470103  6364 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.533131 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.620851 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.397507  6086 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.565984 24545 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260809 23:22:29.453361  6314 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:29.633586 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94783us
I20260809 23:22:29.508273  6216 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:29.509713  6366 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.619578  6224 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:29.680168 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 147129us
I20260809 23:22:29.675570 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:29.666838 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 159556us
I20260809 23:22:29.668076 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2119) took 1294 ms. Trace:
I20260809 23:22:29.683622 24534 rpcz_store.cc:276] 0809 23:22:28.373544 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:28.373727 (+   183us) service_pool.cc:227] Handling call
0809 23:22:28.381563 (+  7836us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:29.009993 (+628430us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:29.660601 (+650608us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:29.668016 (+  7415us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1371737984}
I20260809 23:22:29.669800 24535 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:29.659588 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.490082  6003 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.686534 24542 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:29.627787  6325 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.695003 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.729732 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.527725  6303 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.676772  6167 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.528160  6033 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.620339  6044 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260809 23:22:29.652731  6319 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.786288 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.787987 24431 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 31: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:29.792759 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 273240us
I20260809 23:22:29.758550  6102 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:29.800539 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.676522  6314 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.732182  6325 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:29.841614 22524 thread.cc:536] Waited for 2000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:29.688874  6366 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.677030  6364 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.792037  6319 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.948511 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 264971us
W20260809 23:22:29.943259 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 207260us
I20260809 23:22:29.767163  6167 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.958920 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 2018) took 1040 ms. Trace:
I20260809 23:22:29.959335 25370 rpcz_store.cc:276] 0809 23:22:28.917999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:28.931019 (+ 13020us) service_pool.cc:227] Handling call
0809 23:22:29.958709 (+1027690us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:29.958845 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8089856}
I20260809 23:22:29.960501 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.963378  6086 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.963943  6086 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.964573  6086 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.966509  6086 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.963992  6224 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:29.972597 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.975616 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.959460  5718 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:29.988380  5718 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:29.988814  5718 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:29.992756  6224 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.997845 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:29.939625  6207 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.003338  6365 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:30.003993  6376 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.214s
W20260809 23:22:30.004859  6379 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
W20260809 23:22:30.014521  6370 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260809 23:22:30.028695 25371 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260809 23:22:30.025709 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.029778 25370 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:29.951395  6057 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.063s
I20260809 23:22:30.044296  6366 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.044764  6057 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.045436  6057 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.046584  6319 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.047154  6319 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.047361  6003 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.047659  6319 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.049620  6319 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.999951  6091 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:30.035944  6314 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.014873  6377 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.037338  6167 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.037880  6033 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.038094  6303 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.003548  6289 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260809 23:22:30.040325  6364 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.041396  6325 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.042060  6393 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.019601  6365 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:30.019438  6376 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:30.013947  6298 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:30.074755 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.049896  6274 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.817s	user 0.008s	sys 0.004s
I20260809 23:22:30.096638 25370 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
W20260809 23:22:30.112140 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.114289 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:30.103006  6224 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.110958  6393 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.125324  6393 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.114042  6365 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:30.127018  6365 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.120285  6274 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
W20260809 23:22:30.133723  6378 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.331s	user 0.001s	sys 0.001s
I20260809 23:22:30.135874  5718 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.142364 24544 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 32.
W20260809 23:22:30.146517 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.148890 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 176328us
I20260809 23:22:30.150846 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.145202  6057 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:30.153788 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 333202us
I20260809 23:22:30.153131 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.155292 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 335888us
W20260809 23:22:30.167338 24378 leader_election.cc:334] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.167799 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.146167  6305 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.021s	sys 0.005s
I20260809 23:22:30.174012 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.171269 24378 leader_election.cc:334] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.172328 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:30.171696 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.177520 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.177795 24428 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:30.178638 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.179766  6391 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.180486  6391 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.173967 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.181959 24422 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260809 23:22:30.183424 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.184262 25375 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:30.164703  6129 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:30.191965  6376 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:30.192364  6373 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.173s	user 0.002s	sys 0.000s
I20260809 23:22:30.155808  6310 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.196872  5718 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.197512  6289 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.207968 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.208840 25366 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
W20260809 23:22:30.222159 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.223004 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.225497 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.227519 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.231652 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.234162  6204 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.238703 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.240710 24535 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 18. (term mismatch)
I20260809 23:22:30.241209 24535 pending_rounds.cc:76] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 17
I20260809 23:22:30.220878  6151 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.241562 24535 pending_rounds.cc:97] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260809 23:22:30.241876 24535 raft_consensus.cc:2866] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:30.242440 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.234813  6204 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.219775  6289 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:30.251346  6369 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.609s	user 0.000s	sys 0.002s
I20260809 23:22:30.252264 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.253424 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.253569 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.256408 24539 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:30.236482  6393 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.257007  6393 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.260061  6393 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.260644  6289 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.262656  6289 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.262951 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:30.182780  6391 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.268234 24422 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.264564  6204 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.265668 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.273166 24428 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:30.275051 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.277838 24422 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:30.283571 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.284569 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:30.287079 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.287204 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.288888 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.290755 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.291570 24545 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:30.293288 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.294019 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.294126 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.294797 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.313452 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.335041 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.291353  6204 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:30.341583 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.296168  6289 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.342124  6408 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.295223  5646 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:30.390167 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51379.8us
I20260809 23:22:30.385376  6224 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:30.399238  6387 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260809 23:22:30.314400  6413 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:30.343627  6415 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:30.379622 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.380391  6033 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.380162  5622 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:30.405004 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 375566us
I20260809 23:22:30.420284 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.461480 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:30.432926 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 222190us
I20260809 23:22:30.416371  6289 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.450454  6245 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.464824  6249 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:30.467716 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.468819  6408 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:30.470077  6274 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
W20260809 23:22:30.471472 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91925.4us
I20260809 23:22:30.394135  6401 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.113s
I20260809 23:22:30.420491  6204 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:30.337833  6364 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.454116  6415 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:30.462309  5796 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.493911 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.477044  6249 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:30.477108  6309 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:30.489100  6274 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:30.486721  6320 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.475520  5622 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.473891  6033 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.522477 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:30.484715  6204 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.494741  6263 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.573974  6310 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.575886  6400 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260809 23:22:30.517730  6057 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.059s
I20260809 23:22:30.576323  6400 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.576893  6400 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:30.579653  5622 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.495834  6364 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.587651  6364 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.484040  6413 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.491158  6415 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:30.591080  6413 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.515374  6394 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.495252  5796 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.593947  5796 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.515373  6249 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.520745  5887 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.599334  5887 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.573349  6263 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.601696  5887 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.601752  6263 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.574505  6310 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.576264  6401 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:30.602900  6430 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:22:30.603266  6401 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.604964 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.605810  6401 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.606057  6410 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:30.606554  6410 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:30.608321  6310 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:30.610900 24167 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 244) took 12342 ms (client timeout 15000 ms). Trace:
I20260809 23:22:30.612020  6289 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.616720  6033 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:30.617892  6427 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:22:30.619565  6204 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.620743 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 4050) took 1102 ms. Trace:
I20260809 23:22:30.517678  6391 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.621106 24544 rpcz_store.cc:276] 0809 23:22:29.518580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.518836 (+   256us) service_pool.cc:227] Handling call
0809 23:22:30.620591 (+1101755us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:30.620689 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409682176}
I20260809 23:22:30.621526  6391 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.623610  6391 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.623692 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.624522 24539 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 32: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:30.626145 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:30.630682 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:30.572386 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50034.5us
I20260809 23:22:30.633162 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.626751 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:30.612106 24167 rpcz_store.cc:269] 0809 23:22:18.268421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.274317 (+  5896us) service_pool.cc:227] Handling call
0809 23:22:30.610793 (+12336476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:23.822940 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:23.823572 (+   632us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:23.823625 (+    53us) write_op.cc:435] Acquired schema lock
0809 23:22:23.823639 (+    14us) tablet.cc:660] Decoding operations
0809 23:22:23.966958 (+143319us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:23.986759 (+ 19801us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:23.998275 (+ 11516us) tablet.cc:683] Acquiring locks for 21 operations
0809 23:22:24.226514 (+228239us) tablet.cc:699] Row locks acquired
0809 23:22:24.226565 (+    51us) write_op.cc:260] PREPARE: finished
0809 23:22:24.226746 (+   181us) write_op.cc:270] Start()
0809 23:22:24.226923 (+   177us) write_op.cc:276] Timestamp: P: 1786317744226673 usec, L: 0
0809 23:22:24.226987 (+    64us) op_driver.cc:331] REPLICATION: starting
0809 23:22:24.227756 (+   769us) log.cc:842] Serialized 8188 byte log entry
0809 23:22:25.211765 (+984009us) op_driver.cc:444] REPLICATION: finished
0809 23:22:25.666216 (+454451us) write_op.cc:301] APPLY: starting
0809 23:22:25.677042 (+ 10826us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:26.372663 (+695621us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:26.372780 (+   117us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:26.716034 (+343254us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:26.716479 (+   445us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=0,mrs_lookups=0
0809 23:22:26.716542 (+    63us) write_op.cc:312] APPLY: finished
0809 23:22:26.737139 (+ 20597us) log.cc:842] Serialized 188 byte log entry
0809 23:22:26.954289 (+217150us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:26.954502 (+   213us) write_op.cc:454] Released schema lock
0809 23:22:26.954577 (+    75us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:26.954958 (+   381us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:26.954986 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18263,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":399735,"apply.run_cpu_time_us":78820,"apply.run_wall_time_us":1310652,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91823,"num_ops":21,"prepare.queue_time_us":590547,"prepare.run_cpu_time_us":22360,"prepare.run_wall_time_us":483228,"raft.queue_time_us":584768,"raft.run_cpu_time_us":18002,"raft.run_wall_time_us":190163,"replication_time_us":930849,"thread_start_us":1070166,"threads_started":5,"wal-append.queue_time_us":537830,"wal-append.run_cpu_time_us":39929,"wal-append.run_wall_time_us":2446634}]]}
I20260809 23:22:30.638525 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2213) took 1564 ms. Trace:
I20260809 23:22:30.650219 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.651175 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:30.652544 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.640066 25372 rpcz_store.cc:276] 0809 23:22:29.074691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.074878 (+   187us) service_pool.cc:227] Handling call
0809 23:22:29.081180 (+  6302us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:29.082511 (+  1331us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:29.082578 (+    67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:29.083113 (+   535us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.250211 (+1167098us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b18001a49a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:30.409689 (+159478us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:30.417767 (+  8078us) log.cc:842] Serialized 14 byte log entry
0809 23:22:30.422013 (+  4246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:30.422240 (+   227us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:30.627203 (+204963us) raft_consensus.cc:1671] finished
0809 23:22:30.638320 (+ 11117us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:30.638460 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3918,"prepare.queue_time_us":6109,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":4859,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":10005,"spinlock_wait_cycles":13354624,"thread_start_us":159276,"threads_started":1,"wal-append.queue_time_us":1390902}
I20260809 23:22:30.655925 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.680495 24539 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:30.639293  5891 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election won for term 32
I20260809 23:22:30.681834 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.678589  6033 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260809 23:22:30.685979 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.699777 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.702754  6319 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.712111  6319 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.675662  6262 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.675225  5952 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.698736  6033 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.703305  6401 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:30.672895  6415 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.708197  5891 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:30.726025  5887 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:30.748376 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 229125us
I20260809 23:22:30.751593 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.738646  6033 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:30.754842 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.766450 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.767236 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.714816  6319 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:30.754976  6211 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.040s	sys 0.073s
I20260809 23:22:30.702594  6274 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.774724 24431 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:30.776288 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.730525  6262 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.729816  5952 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:30.774638  6211 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.040s	sys 0.073s
I20260809 23:22:30.785993  6400 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.811599 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.783614 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.724414  6410 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.24, Last appended by leader: 24, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.785236 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.810597 25366 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260809 23:22:30.744638  6401 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:30.816665 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66854.1us
I20260809 23:22:30.820720 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:30.825067 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:30.828670  6438 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260809 23:22:30.829162 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.829546  5952 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.753249  5887 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.819751  6204 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.792711  6210 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:30.676205  6249 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.835923 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:30.842847 22524 thread.cc:536] Waited for 3000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:30.844666 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:30.845010  6435 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260809 23:22:30.835348  5891 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.848147 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.850003 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.850306  6019 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:30.850890  6019 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:30.855798  6415 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.857884  6401 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.859423 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:30.860459  6262 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.778189  6437 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.083s
I20260809 23:22:30.862895 24544 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:30.863456  6249 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.864506  6410 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.866971  6400 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.815371  6339 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.867606  6339 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.850421  5887 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.869706  6339 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:30.873853 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.835850  6266 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.882328 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.887457 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.888442 25373 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.888592 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.884721 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.893685 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.893760 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:30.899603 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.902673 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.904194 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.905962 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.835290  6129 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
W20260809 23:22:30.945564 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50707.4us
I20260809 23:22:30.960678 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
W20260809 23:22:30.968525 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 206221us
W20260809 23:22:30.973867 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67996.5us
I20260809 23:22:30.939066  5643 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.991780 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.991827 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.993304 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:30.944350  6339 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:30.944554  6437 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:30.997969 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.999934 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:30.851998  6019 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.008605 24431 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.980954  6445 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:31.013556 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.906332  6266 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.041426  6266 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.016420 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.024523  6298 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.025206 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.010126  5643 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:31.034796  6216 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.079082 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.035948  6415 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.045794 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.005223  6166 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:31.081295 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64378.4us
I20260809 23:22:31.013229  6448 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:31.107822 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:31.074162  6298 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:31.144768 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65784.7us
I20260809 23:22:31.147352 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.149869 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4143) took 1725 ms. Trace:
I20260809 23:22:31.150333 24435 rpcz_store.cc:276] 0809 23:22:29.424542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.424720 (+   178us) service_pool.cc:227] Handling call
0809 23:22:29.496986 (+ 72266us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.157410 (+660424us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:30.157454 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.220193 (+ 62739us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.426115 (+205922us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:30.426171 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.084322 (+658151us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:31.084374 (+    52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.084419 (+    45us) raft_consensus.cc:1671] finished
0809 23:22:31.084431 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:31.149786 (+ 65355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4005,"raft.queue_time_us":43000,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":1146937088,"thread_start_us":308665,"threads_started":2,"wal-append.queue_time_us":284961}
I20260809 23:22:31.082870  5494 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:31.143747  6032 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.155614  6412 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:31.186321 24540 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:31.048557  6204 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.119s
I20260809 23:22:31.157087  6118 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:31.192795 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.159673  6057 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:31.195014 24431 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:31.169183  6263 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.091652  6216 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.212596  6216 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.193813  6032 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.204213  6412 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:31.215999  5280 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.215821  6412 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.216393  6032 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.216689  5280 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.218523  6032 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.220600 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.208884  6298 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:31.144115  6166 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.222126 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.223088  6166 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.223362  6129 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.224741  6449 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.224956 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:31.209930  6452 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:22:31.188084  6448 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.228896  6032 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.230139  6033 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.232718 24422 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:31.145380  6415 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.236526  6415 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.220829  6204 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Advancing to term 32
W20260809 23:22:31.251200 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96415.1us
W20260809 23:22:31.254530 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.255070 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:31.258095 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.263586 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.263748  6166 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:31.268221  6033 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.259423  6032 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.266788 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.242056  6057 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:31.274833  6436 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.500s	user 0.000s	sys 0.005s
I20260809 23:22:31.256757  5280 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.250442  6263 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.285980 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.293576 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.312700 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.313758 25369 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.318058 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:31.322849  6263 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.324079 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.324316 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:31.325148 25372 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.325532 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.325953 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.326481 24378 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.288092  5494 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.328280 24378 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.328783 25369 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 31: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:31.331079 24378 leader_election.cc:401] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:31.332660 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.335151 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.338094 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.342041 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.347718 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.330001  5891 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.329509  6415 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:31.402470  6371 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:31.335440  6325 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.336084  6316 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:31.372613  6057 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.407027 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 61) took 1649 ms. Trace:
I20260809 23:22:31.434265  6204 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:31.435807  6213 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.412611 24429 rpcz_store.cc:276] 0809 23:22:29.757518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.757657 (+   139us) service_pool.cc:227] Handling call
0809 23:22:31.378805 (+1621148us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:31.406912 (+ 28107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1460627712}
I20260809 23:22:31.441056 25370 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:31.448740  6086 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:31.453197  6213 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:31.457629  6428 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.789s	user 0.003s	sys 0.015s
I20260809 23:22:31.462404 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 4138) took 1236 ms. Trace:
I20260809 23:22:31.462796 24543 rpcz_store.cc:276] 0809 23:22:30.226245 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.226393 (+   148us) service_pool.cc:227] Handling call
0809 23:22:30.226762 (+   369us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.232694 (+  5932us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:30.232722 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.233290 (+   568us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.471548 (+238258us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:30.471590 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:30.471658 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.456967 (+985309us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:31.462085 (+  5118us) raft_consensus.cc:1671] finished
0809 23:22:31.462113 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:31.462343 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5518,"raft.queue_time_us":6771,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":30719232,"thread_start_us":237843,"threads_started":1,"wal-append.queue_time_us":434614}
W20260809 23:22:31.468988  6464 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260809 23:22:31.466743  6213 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:31.483464 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 225809us
W20260809 23:22:31.584039 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 420901us
I20260809 23:22:31.588675 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.589560 24422 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:31.634631 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:31.639411 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 373868us
I20260809 23:22:31.630987  6012 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.645416  6249 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:31.661893 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:31.662592 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 312337us
I20260809 23:22:31.673983  6249 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:31.672053  6086 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:31.706084 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.659147  6012 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.732679  6012 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.709218  6249 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:31.716955 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.680155  5752 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260809 23:22:31.743691 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:31.748842 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.749204 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:31.751905 24498 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.654309 24162 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:31.752383 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.716180  6289 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:31.755102 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:31.759837  6274 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:31.735648  6057 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:31.746189  6303 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:31.752846  5311 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.753252  6262 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.787674 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:31.810770 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50503us
W20260809 23:22:31.807423  6465 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260809 23:22:31.816216 24498 leader_election.cc:334] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.816649 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.776801  6289 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.823452 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.823899 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.824349 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.891635 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.825399 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.892419 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:31.830298 24430 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
W20260809 23:22:31.843751 22524 thread.cc:536] Waited for 4000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:31.893509 24430 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 19
I20260809 23:22:31.894009 24430 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260809 23:22:31.894277 24430 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:31.783696  6057 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.898687 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:31.898898 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82703us
W20260809 23:22:31.903329 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.903848 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.904580 24546 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260809 23:22:31.906795 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.785027  6303 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.911687 24380 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
W20260809 23:22:31.916541 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:31.895766  6488 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260809 23:22:31.887404  6377 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:31.924420 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.928493 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.928053 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.895505  5643 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.958916 24540 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260809 23:22:31.927839 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.952557  6412 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260809 23:22:31.977125  6303 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:31.979403  6057 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.980566 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.981057  5643 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.955096  6491 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260809 23:22:31.927345  5718 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.915144  6371 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:31.945238  6480 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260809 23:22:31.999755  5643 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.001638 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2157) took 1119 ms. Trace:
I20260809 23:22:31.949745  6413 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:32.012591 24541 rpcz_store.cc:276] 0809 23:22:30.882489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.883063 (+   574us) service_pool.cc:227] Handling call
0809 23:22:30.883502 (+   439us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.883932 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:30.883970 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.886585 (+  2615us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.413402 (+526817us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b18001c6720. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:31.686182 (+272780us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:31.686226 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.686291 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.986904 (+300613us) raft_consensus.cc:1671] finished
0809 23:22:31.986949 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.001552 (+ 14603us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8581,"raft.queue_time_us":1562,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":9488,"spinlock_wait_cycles":47628160,"thread_start_us":272595,"threads_started":1,"wal-append.queue_time_us":1057083}
I20260809 23:22:31.949085  6277 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:32.019075 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.029834 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.959936  6377 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.040191 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.039973 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.055299 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.058215 24546 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 31. Current term is 32. Ops: []
I20260809 23:22:32.058847 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.060729 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.992762  6415 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260809 23:22:32.075011 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.085650 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.098255 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.034492  6413 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.015528  6371 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.067047  6404 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.052610  5752 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260809 23:22:32.037601  6277 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.123524  6210 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:32.130316 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.122807 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.100154  6220 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:32.131712 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 222109us
W20260809 23:22:32.134404 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 223031us
I20260809 23:22:32.132354  6266 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:32.144351 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.140659  6210 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:32.149814 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:32.152009  6494 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260809 23:22:32.151782  6413 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.150329  6289 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.152680  6277 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.163442  6371 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.215435 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.217549 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.239564 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:32.223052 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.162986  6377 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.236110 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.147791  6249 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.244287 24539 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:32.251070 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.272132 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122216us
I20260809 23:22:32.274757  6388 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.277601  6388 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.289733  6204 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:32.268785  6478 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:32.288909 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 115264us
I20260809 23:22:32.269348  5494 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:32.300254  5494 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:32.308593  5494 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.311282  5494 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:32.320770 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112838us
I20260809 23:22:32.323115 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.325167 24429 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:32.328109  6129 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:32.332417  6388 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.332623 24428 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 28 index: 24. (index mismatch)
W20260809 23:22:32.335942  6487 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.254s	user 0.004s	sys 0.000s
I20260809 23:22:32.339568 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 2033) took 1637 ms. Trace:
I20260809 23:22:32.340005 25371 rpcz_store.cc:276] 0809 23:22:30.702318 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.725754 (+ 23436us) service_pool.cc:227] Handling call
0809 23:22:30.726176 (+   422us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.734256 (+  8080us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:30.734302 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.860474 (+126172us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b1800544c00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:22:31.240378 (+379904us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:31.265576 (+ 25198us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.676823 (+411247us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:31.676890 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.677050 (+   160us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:32.334687 (+657637us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:32.339071 (+  4384us) raft_consensus.cc:1671] finished
0809 23:22:32.339098 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.339343 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8876,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":5523,"spinlock_wait_cycles":14096000,"thread_start_us":358560,"threads_started":1,"wal-append.queue_time_us":692731}
I20260809 23:22:32.346273  6249 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:32.361657 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.367313 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.391145 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.393060 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.394433 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.395081 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 89541.9us
W20260809 23:22:32.396430 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.401791 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.404045 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.393208  6466 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.707s	user 0.008s	sys 0.016s
I20260809 23:22:32.368861  5611 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:32.436852 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2242) took 1265 ms. Trace:
W20260809 23:22:32.439419 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67953.1us
I20260809 23:22:32.439785 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.441371  6513 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260809 23:22:32.444420 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:32.445909  6305 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:32.446527  6305 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:32.447000  6305 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:32.453802  5571 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:32.454244  6216 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:32.467288  6224 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Stepping down as leader of term 31
I20260809 23:22:32.467716  6224 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:32.467952  6404 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:32.470098 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 330510us
I20260809 23:22:32.469012  6224 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.463320 25372 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:32.464749 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.473834 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84088.3us
I20260809 23:22:32.476781  6388 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:32.480340  6249 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.481158  6249 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.492116 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.492645  6249 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:32.437309 25368 rpcz_store.cc:276] 0809 23:22:31.171602 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:31.171770 (+   168us) service_pool.cc:227] Handling call
0809 23:22:31.172126 (+   356us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:31.504328 (+332202us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:31.504572 (+   244us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:31.506763 (+  2191us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.683203 (+176440us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:31.772246 (+ 89043us) log.cc:842] Serialized 14 byte log entry
0809 23:22:31.783565 (+ 11319us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.783654 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:32.280752 (+497098us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b180039c880. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:32.413932 (+133180us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b180039c880. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:32.426653 (+ 12721us) raft_consensus.cc:1671] finished
0809 23:22:32.426674 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.436638 (+  9964us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26683,"prepare.queue_time_us":105757,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":3735,"raft.queue_time_us":48866,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":6065,"spinlock_wait_cycles":12565632,"thread_start_us":544814,"threads_started":3,"wal-append.queue_time_us":262547}
I20260809 23:22:32.434645  6509 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.500777 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.510437 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.514490 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.529086 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:32.530094 24429 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:32.530720 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54753.1us
W20260809 23:22:32.532071 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:32.547485  6069 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:32.550037  6224 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:32.567332 24428 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:32.596573  5796 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:32.610335 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109681us
I20260809 23:22:32.612303 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.612890 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.619158  6151 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.628940 24429 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:32.623279 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.636260  5796 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.662389  5796 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.652503  6249 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.022s
I20260809 23:22:32.683290  6069 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:32.683676 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74386.5us
I20260809 23:22:32.687310 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.693315 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.698143 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105408us
I20260809 23:22:32.710012  6151 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:32.710150 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:32.711360  6069 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:32.711932  6069 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:32.696000  6249 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.713099  6069 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.716137  6305 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:32.716701 24378 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.717201 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.717623 25368 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:32.719986  6289 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260809 23:22:32.743723 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.747092 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55874.9us
I20260809 23:22:32.771229 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:32.777947 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:32.754482 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.756775  6305 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.791687 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.787509 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.788736 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:32.799002 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.827929  6404 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:32.828626  6404 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.831250  5494 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:32.832180  6404 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.831923  5494 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.807327  6305 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:32.842675  6210 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.843317  6210 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.845826  6210 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.852869 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.869455 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.875296 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.877349  6535 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260809 23:22:32.853696  5494 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.889990  5311 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.040s
W20260809 23:22:32.891348 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80519.8us
I20260809 23:22:32.873498 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.903223 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:32.908583 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44954.5us
W20260809 23:22:32.912554 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.915452  6266 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:32.907150 22524 thread.cc:536] Waited for 5000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:32.918547 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.921921 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.922899  6537 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260809 23:22:32.927076 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.891348  5311 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 27.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260809 23:22:32.936424 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.953984  6069 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Stepping down as leader of term 28
W20260809 23:22:32.954291 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:32.959802  6540 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.004s
I20260809 23:22:32.911401  6151 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:32.986300 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 269661us
W20260809 23:22:32.960593  6536 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.146s	user 0.000s	sys 0.005s
I20260809 23:22:32.979526 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.983884  6069 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:33.007499  6069 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 28.24, Last appended by leader: 24, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.010692  6069 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:33.010910  5494 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.008481 24378 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.026888 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.028059  5611 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.028407 25368 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: [28.24-28.24]
W20260809 23:22:33.029014 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43777.3us
I20260809 23:22:33.030432  5494 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.031064  5494 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.033392  5494 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.035909 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.036337 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.039675  6210 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.041451 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.042121 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.052834 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:33.060859 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73370.6us
I20260809 23:22:33.062774  6547 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.063490 24378 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.067407 24428 raft_consensus.cc:2354] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.069443 24266 leader_election.cc:398] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.069774 24266 leader_election.cc:401] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:33.071059  6389 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 27
W20260809 23:22:33.077822 24378 leader_election.cc:334] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.078313 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.081027  6389 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.103340  6547 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.104939 24378 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.105733 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.107532  6547 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:33.108847 24378 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.109485 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.110246  6547 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.112910 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:33.116578 24378 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:33.117723 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.119830 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:33.126448 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.122603  6548 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.139048 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.154325 24378 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.156031 24425 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:33.157546  6388 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:33.163096 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.163568  6389 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:33.165787 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.165974  6389 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.169703  6389 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.171317 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.176307 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:33.182981 24162 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 344) took 84335 ms (client timeout 15000 ms). Trace:
W20260809 23:22:33.197124 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.166697  6548 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:33.191025 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.166587 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.201431  6547 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.203711  5611 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:33.206966 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.207442 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:33.197330 24162 rpcz_store.cc:269] 0809 23:21:08.847371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.847795 (+   424us) service_pool.cc:227] Handling call
0809 23:22:33.182719 (+84334924us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:32.266446 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:32.295419 (+ 28973us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:32.295494 (+    75us) write_op.cc:435] Acquired schema lock
0809 23:22:32.295543 (+    49us) tablet.cc:660] Decoding operations
0809 23:22:32.314798 (+ 19255us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:32.369300 (+ 54502us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:32.371144 (+  1844us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:32.379373 (+  8229us) tablet.cc:699] Row locks acquired
0809 23:22:32.379408 (+    35us) write_op.cc:260] PREPARE: finished
0809 23:22:32.379596 (+   188us) write_op.cc:270] Start()
0809 23:22:32.379797 (+   201us) write_op.cc:276] Timestamp: P: 1786317752379513 usec, L: 0
0809 23:22:32.379858 (+    61us) op_driver.cc:331] REPLICATION: starting
0809 23:22:32.380657 (+   799us) log.cc:842] Serialized 538 byte log entry
0809 23:22:32.750848 (+370191us) op_driver.cc:444] REPLICATION: finished
0809 23:22:32.876396 (+125548us) write_op.cc:301] APPLY: starting
0809 23:22:32.876587 (+   191us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:32.878646 (+  2059us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:32.878792 (+   146us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:32.956589 (+ 77797us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:32.956936 (+   347us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:32.957008 (+    72us) write_op.cc:312] APPLY: finished
0809 23:22:32.957420 (+   412us) log.cc:842] Serialized 25 byte log entry
0809 23:22:32.957646 (+   226us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:32.957757 (+   111us) write_op.cc:454] Released schema lock
0809 23:22:32.957827 (+    70us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:32.958272 (+   445us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:32.958306 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":121662,"apply.run_cpu_time_us":5694,"apply.run_wall_time_us":85704,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"num_ops":1,"prepare.queue_time_us":49935,"prepare.run_cpu_time_us":14780,"prepare.run_wall_time_us":261550,"raft.queue_time_us":120455,"raft.run_cpu_time_us":28477,"raft.run_wall_time_us":69222,"replication_time_us":370866,"spinlock_wait_cycles":69504,"thread_start_us":271858,"threads_started":4,"wal-append.queue_time_us":216334}]]}
I20260809 23:22:33.210044  6547 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.213048 24160 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.211228 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.230824 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.233274 24378 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.233705 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.234567  6547 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:33.234823 24160 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.246454 24160 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 21, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:33.248780 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.249289 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.252386  6548 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.248389 24160 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 29 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.264187 24160 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 23 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.266642 24160 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 26. New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.283761 24160 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.296262 24160 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 22 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 25 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.304912 24160 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.311363 24160 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.316315 24160 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 24, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.325826 24160 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.331486 24160 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 22, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 22 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.339792 24160 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.351215 24160 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.354609 24160 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.358318 24160 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.369275 24160 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.371737 24160 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.421535  5611 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:33.426075  5611 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.428381  5611 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.432139 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.433792 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.438099 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.441735  6310 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.464329 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:33.511065 24498 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 181 similar messages]
W20260809 23:22:33.551164 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.563117  6476 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:33.563803  6476 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:33.570225 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.565973  6476 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:33.571396 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.572208 24428 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:33.573170 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.574523  6476 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:33.575297  6151 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.575901  6151 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.578122  6151 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.580253 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:33.581096 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.584632 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.586999  6511 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260809 23:22:33.592298  6019 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.592808  6019 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.595399 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.596411 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:33.597617 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:33.594871  6019 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.599439 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.602469  5494 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:33.606878  5494 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:33.607272  5494 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.623620  6476 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.631971 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:33.636865 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:33.645004  5494 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.647739  5611 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:33.648365  5611 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.650461  5611 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.650523 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:33.651225 24422 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.652364 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.660213 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.660899  6310 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.662225  6310 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.664343  6310 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.665740 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.666440 24540 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 32.
I20260809 23:22:33.667258 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.667735 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:33.669189  5611 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260809 23:22:33.669301 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:33.669688  5611 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:33.670110 24540 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.670166  5611 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Advancing to term 33
I20260809 23:22:33.671049  5494 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.676338 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.685043  6019 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:33.715680 24422 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
W20260809 23:22:33.729779 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68102.2us
I20260809 23:22:33.731758  5611 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.737345 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:33.738037 24543 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Advancing to term 33
I20260809 23:22:33.738463  6389 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.739032  6389 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.742165 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.743080  6389 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.746140  5611 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.718369 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:33.747252 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:33.756639 24543 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 33.
I20260809 23:22:33.735044 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:33.760687 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.761198  6310 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:33.761765  6310 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:33.762367  6019 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Leader election won for term 33
I20260809 23:22:33.762845  6310 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.762849  6019 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:33.765111  6019 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.773491 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.781836 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.785145  5611 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.871856  6310 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.872498  6310 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.876086 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.878782 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.880358 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.881260  5611 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:33.881780  6310 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.884275 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:33.886662 24160 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 346) took 22446 ms (client timeout 15000 ms). Trace:
I20260809 23:22:33.901090 24168 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 21 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.903200  6168 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:33.903781  6168 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.904572 24168 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 27, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.906558 24168 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.908046 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.911813 24539 raft_consensus.cc:2354] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.913343 25325 leader_election.cc:398] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.913657 25325 leader_election.cc:401] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:33.914244  6210 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Advancing to term 33
W20260809 23:22:33.915429 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:33.916522 22524 thread.cc:536] Waited for 6000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:33.916232 24168 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.899339 24160 rpcz_store.cc:269] 0809 23:22:11.440358 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.443092 (+  2734us) service_pool.cc:227] Handling call
0809 23:22:33.886557 (+22443465us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:33.432698 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:33.433198 (+   500us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:33.433237 (+    39us) write_op.cc:435] Acquired schema lock
0809 23:22:33.433252 (+    15us) tablet.cc:660] Decoding operations
0809 23:22:33.454477 (+ 21225us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:33.454558 (+    81us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:33.454586 (+    28us) tablet.cc:683] Acquiring locks for 19 operations
0809 23:22:33.456049 (+  1463us) tablet.cc:699] Row locks acquired
0809 23:22:33.456068 (+    19us) write_op.cc:260] PREPARE: finished
0809 23:22:33.456206 (+   138us) write_op.cc:270] Start()
0809 23:22:33.456332 (+   126us) write_op.cc:276] Timestamp: P: 1786317753456162 usec, L: 0
0809 23:22:33.456357 (+    25us) op_driver.cc:331] REPLICATION: starting
0809 23:22:33.468360 (+ 12003us) log.cc:842] Serialized 7439 byte log entry
0809 23:22:33.497156 (+ 28796us) op_driver.cc:444] REPLICATION: finished
0809 23:22:33.528120 (+ 30964us) write_op.cc:301] APPLY: starting
0809 23:22:33.528214 (+    94us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:33.554960 (+ 26746us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:33.555080 (+   120us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:33.609570 (+ 54490us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:33.609904 (+   334us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0809 23:22:33.609955 (+    51us) write_op.cc:312] APPLY: finished
0809 23:22:33.618885 (+  8930us) log.cc:842] Serialized 172 byte log entry
0809 23:22:33.619089 (+   204us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:33.619292 (+   203us) write_op.cc:454] Released schema lock
0809 23:22:33.619358 (+    66us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:33.619829 (+   471us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:33.619866 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":208200,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":30613,"apply.run_cpu_time_us":25397,"apply.run_wall_time_us":92310,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83471,"num_ops":19,"prepare.queue_time_us":37878,"prepare.run_cpu_time_us":5972,"prepare.run_wall_time_us":35958,"replication_time_us":40691,"thread_start_us":54300,"threads_started":2}]]}
I20260809 23:22:33.920423  6310 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.924724  6310 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:33.918517 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.927464  6310 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.928100 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.928798 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:33.932441 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:33.906723  6168 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.954596 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 356) took 49932 ms (client timeout 15000 ms). Trace:
W20260809 23:22:33.957064 24164 rpcz_store.cc:269] 0809 23:21:43.994543 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.995054 (+   511us) service_pool.cc:227] Handling call
0809 23:22:33.926771 (+49931717us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":390267,"reported_tablets":47}
I20260809 23:22:33.970374 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.926033 24168 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 25 to 29, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.984217 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.946923  6210 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.988791 24168 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 20 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.988209  5611 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.999294 24168 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 24 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.007360 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:34.018430 24168 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 22 to 27, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.017037 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81797.1us
I20260809 23:22:34.020311 24168 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 17 to 24, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:34.024153 24168 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 23 to 29, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:34.029932 24168 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 19, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:34.038329 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.054908 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:22:34.059844 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.070209 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.078979 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.097782 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.103645 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.105207 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.113830 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.114900 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.120951 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.135015 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.139583 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.141441 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.157366 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.160562  6476 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.169438  6476 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.171576  6476 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.172479 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.173185 25363 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:34.174351 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:34.176972  6388 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:34.177361  6388 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:34.177685  6388 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.178741 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.192281 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.194655  5611 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.199609  5611 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:34.200837 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.201893  5611 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:34.200029  6388 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.204206 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.204676 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.207928 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:34.211799 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.219064  5611 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:34.219703  6388 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.222671 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.224121 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.224833 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.226666 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:34.227291 25363 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.250648 25365 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260809 23:22:34.264129  6310 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260809 23:22:34.271066 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.281069 25363 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
I20260809 23:22:34.282426 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.272230 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.283747  6388 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:34.284674  6388 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:34.285945  6388 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 28.24, Last appended by leader: 24, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.290477 24539 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260809 23:22:34.292757  6310 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:34.298154 24160 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.300511 24160 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.310020 24160 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.318240 24422 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260809 23:22:34.319933  5611 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:22:34.325454 24160 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.329660 24160 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.323369 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.336717 24160 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.407714 24160 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 28, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 28 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.424365 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:34.449347 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:34.456379 24160 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.463016 24160 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.339670 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.465458 24539 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260809 23:22:34.473237 24160 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:34.348452 24168 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 375) took 15313 ms (client timeout 15000 ms). Trace:
I20260809 23:22:34.475011 24160 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 30, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 30 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.482393 24160 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.489338 24160 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 17 to 21, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 21 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.498561 24160 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.501935 24160 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.496490 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:34.480362 24168 rpcz_store.cc:269] 0809 23:22:19.035244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.424771 (+3389527us) service_pool.cc:227] Handling call
0809 23:22:34.348356 (+11923585us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:34.087512 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:34.088040 (+   528us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:34.088079 (+    39us) write_op.cc:435] Acquired schema lock
0809 23:22:34.088099 (+    20us) tablet.cc:660] Decoding operations
0809 23:22:34.102776 (+ 14677us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:34.102859 (+    83us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:34.105608 (+  2749us) tablet.cc:683] Acquiring locks for 11 operations
0809 23:22:34.106378 (+   770us) tablet.cc:699] Row locks acquired
0809 23:22:34.106391 (+    13us) write_op.cc:260] PREPARE: finished
0809 23:22:34.106584 (+   193us) write_op.cc:270] Start()
0809 23:22:34.106690 (+   106us) write_op.cc:276] Timestamp: P: 1786317754106539 usec, L: 0
0809 23:22:34.106707 (+    17us) op_driver.cc:331] REPLICATION: starting
0809 23:22:34.107405 (+   698us) log.cc:842] Serialized 4359 byte log entry
0809 23:22:34.143278 (+ 35873us) op_driver.cc:444] REPLICATION: finished
0809 23:22:34.188441 (+ 45163us) write_op.cc:301] APPLY: starting
0809 23:22:34.188532 (+    91us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:34.216895 (+ 28363us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:34.216988 (+    93us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:34.228954 (+ 11966us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:34.229199 (+   245us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0809 23:22:34.229241 (+    42us) write_op.cc:312] APPLY: finished
0809 23:22:34.229528 (+   287us) log.cc:842] Serialized 105 byte log entry
0809 23:22:34.229693 (+   165us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:34.229830 (+   137us) write_op.cc:454] Released schema lock
0809 23:22:34.229876 (+    46us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:34.273203 (+ 43327us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:34.273233 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1558,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":44845,"apply.run_cpu_time_us":13745,"apply.run_wall_time_us":85206,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50063,"num_ops":11,"prepare.queue_time_us":33103,"prepare.run_cpu_time_us":4017,"prepare.run_wall_time_us":20280,"replication_time_us":36463,"thread_start_us":43993,"threads_started":2}]]}
I20260809 23:22:34.470132 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.516921 22524 pending_rounds.cc:62] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:34.517303 22524 pending_rounds.cc:68] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 24 index: 16 } timestamp: 7316757417841188864 op_type: NO_OP noop_request { }
I20260809 23:22:34.507297  6548 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:34.518114  6548 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.503242  6310 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.520364  6310 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.522692 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.523665 25363 raft_consensus.cc:2354] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
I20260809 23:22:34.524444 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.521692  6548 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.506659  6575 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:34.525832 24378 leader_election.cc:398] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
I20260809 23:22:34.526635 24378 leader_election.cc:401] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:34.527243  6289 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.529284 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.529731 22524 raft_consensus.cc:2866] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:34.530160 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.535301  6289 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
W20260809 23:22:34.543311 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:34.527781 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.545163 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:34.546092  6576 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:34.547153  6310 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.550236 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.551420 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.552910 24160 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.556409 24160 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 20, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 20 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.558145 24160 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.564730 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.559458 24160 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 28 to 31, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 31 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.628209 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.628741 24160 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 26, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.632680 24160 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 27, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.630689 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.671950 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.670305 24160 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.677029 24160 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 29, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 29 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.682758 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:34.691299 25368 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 24. Preceding OpId from leader: term: 30 index: 25. (index mismatch)
I20260809 23:22:34.693650  6511 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:34.695259  6511 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 27.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260809 23:22:34.698892  5611 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.699859  5611 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.702895  5611 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.704401 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.678807 24160 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.709007 25363 pending_rounds.cc:76] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 23
I20260809 23:22:34.709429 25363 pending_rounds.cc:97] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 27.24
I20260809 23:22:34.709699 25363 raft_consensus.cc:2866] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:34.709100 24160 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.719734 24160 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.723531 24160 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.720412 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.706739 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.713778 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.766287 24160 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 28, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.778240 24160 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 22 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.782852 24160 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.785995 24160 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.786702 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:34.787859 24160 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 26, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.789600 24160 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 28. New cstate: current_term: 28 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.799962  6575 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.800702  6575 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:34.802883  6575 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:34.804674 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78685.7us
I20260809 23:22:34.810873  6576 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:34.812700 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.813710 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.818921 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.820966 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.821338 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.822506  5611 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:34.822569 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.825990 24422 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 24. Preceding OpId from leader: term: 30 index: 25. (index mismatch)
I20260809 23:22:34.827529 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.830721 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.830960  6476 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260809 23:22:34.831799 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.832520 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.843828 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.846410 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.848670 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.858829 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.859973 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.861119 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.864145 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
W20260809 23:22:34.864353 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.865272 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.865927 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.869275 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.870375 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.871033 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.874086 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.875133 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.876456 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.879393 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.880374 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.881433 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.896770 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.897845 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.898507 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:34.943428 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.943956 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.945009 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.947778 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.950793 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.951663 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.952718 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:34.968154 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.975467 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.976557 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.977228 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.980154 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.980978 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.981473 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.004189 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.005425 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.009296 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.014169 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.014959 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.016088 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.035631 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.036562 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.037309 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:35.043433 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.068485 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.070729 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.071517 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.084659 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.085600 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.086125 22524 pending_rounds.cc:62] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.086355 22524 pending_rounds.cc:68] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7316757421635989504 op_type: NO_OP noop_request { }
I20260809 23:22:35.086725 22524 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.087080 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.089757 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.090556 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Raft consensus shutting down.
W20260809 23:22:35.107147 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:35.111717 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.137553 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.138500 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.139192 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.142078 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.142840 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.160346 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.163899 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.164777 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.165933 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.173728 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.174503 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.175022 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.177639 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.178387 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.178917 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.187598  6546 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.189232  6546 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.191287  6546 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.195863 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:35.202307 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.202744 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.208686  6583 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:35.209959 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.212376 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.217483 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.225839 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.226766 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.234946  6578 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.235450 22524 pending_rounds.cc:62] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.235755 22524 pending_rounds.cc:68] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 21 index: 16 } timestamp: 7316757418382639104 op_type: NO_OP noop_request { }
I20260809 23:22:35.235602  6578 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.236261 22524 raft_consensus.cc:2866] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.236730 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.240646 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.241293 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.241568 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.242435 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:35.252081 24378 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.252592 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:35.253300  6289 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:35.238222  6578 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.259779 24162 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.261426 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
W20260809 23:22:35.254814 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48934.8us
I20260809 23:22:35.262295 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.262534 24162 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 32, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 32 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:35.263720 22524 pending_rounds.cc:62] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
W20260809 23:22:35.264333 24244 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.268397 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.266391 24160 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 245) took 2168 ms. Trace:
I20260809 23:22:35.270129 22524 pending_rounds.cc:68] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 27 index: 16 } timestamp: 7316757451460698112 op_type: NO_OP noop_request { }
I20260809 23:22:35.277489 22524 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.277961 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.270037 24162 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.281641 24162 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 21, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.283018 24160 rpcz_store.cc:276] 0809 23:22:33.098300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:33.929738 (+831438us) service_pool.cc:227] Handling call
0809 23:22:35.266286 (+1336548us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:34.918474 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:34.918991 (+   517us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:34.919020 (+    29us) write_op.cc:435] Acquired schema lock
0809 23:22:34.919036 (+    16us) tablet.cc:660] Decoding operations
0809 23:22:34.921413 (+  2377us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:34.921497 (+    84us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:34.921526 (+    29us) tablet.cc:683] Acquiring locks for 33 operations
0809 23:22:34.923901 (+  2375us) tablet.cc:699] Row locks acquired
0809 23:22:34.923917 (+    16us) write_op.cc:260] PREPARE: finished
0809 23:22:34.924158 (+   241us) write_op.cc:270] Start()
0809 23:22:34.924274 (+   116us) write_op.cc:276] Timestamp: P: 1786317754924105 usec, L: 0
0809 23:22:34.924298 (+    24us) op_driver.cc:331] REPLICATION: starting
0809 23:22:34.925018 (+   720us) log.cc:842] Serialized 12814 byte log entry
0809 23:22:34.996697 (+ 71679us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.007085 (+ 10388us) write_op.cc:301] APPLY: starting
0809 23:22:35.007179 (+    94us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.124837 (+117658us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.124961 (+   124us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.196594 (+ 71633us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.197049 (+   455us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=66,key_file_lookups=66,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.197099 (+    50us) write_op.cc:312] APPLY: finished
0809 23:22:35.197602 (+   503us) log.cc:842] Serialized 284 byte log entry
0809 23:22:35.197791 (+   189us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.198078 (+   287us) write_op.cc:454] Released schema lock
0809 23:22:35.198142 (+    64us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.198489 (+   347us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.198516 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25119,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":10080,"apply.run_cpu_time_us":42262,"apply.run_wall_time_us":216551,"cfile_cache_hit":134,"cfile_cache_hit_bytes":141935,"num_ops":33,"prepare.queue_time_us":19039,"prepare.run_cpu_time_us":6912,"prepare.run_wall_time_us":27450,"replication_time_us":72298,"thread_start_us":25991,"threads_started":2}]]}
I20260809 23:22:35.286283 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.293496 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.294718 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.299090 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.300700 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.301681 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.304896 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.291471 24162 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.305661 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.306314 22524 pending_rounds.cc:62] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.306541 22524 pending_rounds.cc:68] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 21 index: 16 } timestamp: 7316757424355483648 op_type: NO_OP noop_request { }
I20260809 23:22:35.306913 22524 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.309151 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.311884 24162 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.318416 24162 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.320855 24162 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 30, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 30 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:35.336479 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.340116 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.340792 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.342522 24162 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 31, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 31 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:35.346459 24162 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.355672 24162 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 28 to 31, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 31 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.356930  6590 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.357523  6590 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.358466 24162 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.359537  6590 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.360682 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.360231 24162 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 31, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 31 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.361375 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:35.362690 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:35.363768  6511 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:35.364192  6511 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.364583  6511 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:35.366276  6575 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:35.366884  6575 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.369247 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.371471 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.372548 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.373456 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.373839  6575 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.374027 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.375664 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.378789 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.381240  5494 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:35.383103  6590 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.383703  6590 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.378711 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.385547 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:35.385967 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.386277 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.389950 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.390201 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.391129 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.391690 22524 pending_rounds.cc:62] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.391916 22524 pending_rounds.cc:68] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7316757361437347840 op_type: NO_OP noop_request { }
I20260809 23:22:35.392222 22524 raft_consensus.cc:2866] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.392534 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.395521 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.396215 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.397284 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.385716  6590 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.414729  6511 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.450383  6578 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.451052  6578 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.453132  6578 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.455536 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.457063 24160 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:35.459204 24378 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.458737 24160 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 30, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 30 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:35.459692 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.460813  6578 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:35.467430 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:35.472098 24498 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.487828 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.487991 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:35.485352  6511 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.492914 24428 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:35.502970  6511 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:35.507578 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:35.508711 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:35.509583  6511 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:35.512233  6511 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:35.513407  6578 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.513958  6578 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.513226  6511 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.516213  6578 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.518570 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.519200 25363 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:35.520252 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.521492  6578 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:35.521880  6578 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.522257  6578 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.534355 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.536732  6578 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.542387  6546 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.543262  6583 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.543774  6583 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.545656 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.546646 25363 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:35.547940  6546 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.552997 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.553603 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:35.545899  6583 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.556864 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.554803  6546 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.561569 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.560022 25363 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:35.538774  6578 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.565009 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.566811  6583 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:35.567649  6583 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.567152  6289 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:35.568080  6583 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.568519 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.576258  6583 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.584391 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.616110 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:35.616767 24428 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.625474 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.626462 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.611150  6590 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.624655 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
W20260809 23:22:35.628047 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.597301  6583 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.634922 24167 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 17 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.627224  6590 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.638888  6590 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.639832 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.648012 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.648675 25363 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:35.649897 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:35.648633 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.650529 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:35.655076 24162 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 347) took 2868 ms. Trace:
I20260809 23:22:35.655686 24162 rpcz_store.cc:276] 0809 23:22:32.774420 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:33.233555 (+459135us) service_pool.cc:227] Handling call
0809 23:22:35.642213 (+2408658us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:35.377969 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:35.379344 (+  1375us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:35.379393 (+    49us) write_op.cc:435] Acquired schema lock
0809 23:22:35.379411 (+    18us) tablet.cc:660] Decoding operations
0809 23:22:35.381792 (+  2381us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:35.381854 (+    62us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:35.382057 (+   203us) tablet.cc:683] Acquiring locks for 13 operations
0809 23:22:35.386723 (+  4666us) tablet.cc:699] Row locks acquired
0809 23:22:35.386746 (+    23us) write_op.cc:260] PREPARE: finished
0809 23:22:35.386911 (+   165us) write_op.cc:270] Start()
0809 23:22:35.387050 (+   139us) write_op.cc:276] Timestamp: P: 1786317755386864 usec, L: 0
0809 23:22:35.387074 (+    24us) op_driver.cc:331] REPLICATION: starting
0809 23:22:35.396204 (+  9130us) log.cc:842] Serialized 5159 byte log entry
0809 23:22:35.399868 (+  3664us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.402489 (+  2621us) write_op.cc:301] APPLY: starting
0809 23:22:35.402585 (+    96us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.409404 (+  6819us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.409498 (+    94us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.434380 (+ 24882us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.434606 (+   226us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.434649 (+    43us) write_op.cc:312] APPLY: finished
0809 23:22:35.434977 (+   328us) log.cc:842] Serialized 121 byte log entry
0809 23:22:35.435159 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.435316 (+   157us) write_op.cc:454] Released schema lock
0809 23:22:35.435376 (+    60us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.441910 (+  6534us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.441951 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43714,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":2375,"apply.run_cpu_time_us":16524,"apply.run_wall_time_us":40439,"cfile_cache_hit":52,"cfile_cache_hit_bytes":54288,"num_ops":13,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":4848,"prepare.run_wall_time_us":21837,"replication_time_us":12698}]]}
W20260809 23:22:35.656059 24498 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.659755  6615 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:35.651388  6590 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:35.660712  6590 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.661090  6590 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:35.651957  6583 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:35.684553  6583 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:35.685562  6583 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.688078 24167 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 358) took 3766 ms. Trace:
I20260809 23:22:35.688515 24167 rpcz_store.cc:276] 0809 23:22:31.945153 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:31.945882 (+   729us) service_pool.cc:227] Handling call
0809 23:22:35.688013 (+3742131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:35.642644 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:35.643749 (+  1105us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:35.643786 (+    37us) write_op.cc:435] Acquired schema lock
0809 23:22:35.643801 (+    15us) tablet.cc:660] Decoding operations
0809 23:22:35.644151 (+   350us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:35.644223 (+    72us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:35.644263 (+    40us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:35.644502 (+   239us) tablet.cc:699] Row locks acquired
0809 23:22:35.644530 (+    28us) write_op.cc:260] PREPARE: finished
0809 23:22:35.644696 (+   166us) write_op.cc:270] Start()
0809 23:22:35.644834 (+   138us) write_op.cc:276] Timestamp: P: 1786317755644641 usec, L: 0
0809 23:22:35.644860 (+    26us) op_driver.cc:331] REPLICATION: starting
0809 23:22:35.645495 (+   635us) log.cc:842] Serialized 538 byte log entry
0809 23:22:35.647463 (+  1968us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.647858 (+   395us) write_op.cc:301] APPLY: starting
0809 23:22:35.647935 (+    77us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.648758 (+   823us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.648854 (+    96us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.649536 (+   682us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.649686 (+   150us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.649726 (+    40us) write_op.cc:312] APPLY: finished
0809 23:22:35.649935 (+   209us) log.cc:842] Serialized 25 byte log entry
0809 23:22:35.650117 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.650187 (+    70us) write_op.cc:454] Released schema lock
0809 23:22:35.650221 (+    34us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.650450 (+   229us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.650479 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2205,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":3006,"apply.run_wall_time_us":3003,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4176,"num_ops":1,"prepare.queue_time_us":137,"prepare.run_cpu_time_us":2576,"prepare.run_wall_time_us":4874,"replication_time_us":2520}]]}
W20260809 23:22:35.705089 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:35.652345  6289 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:35.719385  6546 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.727190  6546 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.729336  6546 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.730911 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.730777  6289 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.731971 24539 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:35.733510 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.734653  6546 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:35.735102  6546 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.735564  6546 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.751287 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102681us
W20260809 23:22:35.755052 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.751421 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85197.6us
W20260809 23:22:35.793335 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.801627  6615 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.803282  6615 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.804795  6289 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.805621  6615 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.805388  6289 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.801209 24167 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:35.810520 24378 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.812413  6546 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.813706 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.817761  6476 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.818284  6476 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.820315  6476 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.821280 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.813717  6289 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.827795 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.827749  6604 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.829000  6604 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.831158  6604 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.831667 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:35.833423 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.834594  6289 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:35.835019  6289 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:35.835335 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.835388  6289 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:35.835798 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.836977  6476 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:35.850704  6546 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.825924  6590 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.860164 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:35.852492 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:35.853210  6289 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.860805 24540 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.863574 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.867177 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.876185  6590 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.877460 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:38882
I20260809 23:22:35.880326 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.895244 25363 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.884996 24378 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.888592 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68793us
W20260809 23:22:35.889873 24498 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.900758 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.901346 25368 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:35.911319 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.912151 24422 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:35.912784 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.930006 24540 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:35.924342 25363 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:35.927707 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:35.931485 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.935614 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.934485 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.899040  6289 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.939424 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.940418  6623 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:35.943574  6615 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:35.940850  6611 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:35.945495  6611 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:35.941056  6623 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:35.946400  6611 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.946689  6623 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.971499  6604 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.972123  6604 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.974252  6604 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:35.976137 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:35.983026 25368 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:35.985600  6621 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:35.985627 24170 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 354) took 133327 ms (client timeout 15000 ms). Trace:
W20260809 23:22:35.986274 24170 rpcz_store.cc:269] 0809 23:20:22.635593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.635901 (+   308us) service_pool.cc:227] Handling call
0809 23:20:23.344092 (+708191us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400002dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac297bad6c 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec 00007fac27bc60a2 00007fac21c8a94c 00007fac21c89d86
0809 23:22:35.953004 (+132608912us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23270,"reported_tablets":47,"spinlock_wait_cycles":349312768}
I20260809 23:22:35.987324 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.988260 24422 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 31.
I20260809 23:22:35.989625 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.990280 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.990609  6623 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260809 23:22:35.991242  6578 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.992123  6605 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:35.991833  6578 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.992877  6605 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:35.993351 24422 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260809 23:22:35.994372  6578 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.994998  6476 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:35.992306  6590 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:36.001398 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.999325 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:22:36.002326 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.003082 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:36.003120  6590 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.991065  6623 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.004484  6623 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:36.005896  6590 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.008117 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.993909  6605 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.012653 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.958181 24167 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.015493 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:38882
W20260809 23:22:36.016188 24171 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 353) took 161118 ms (client timeout 15000 ms). Trace:
I20260809 23:22:36.017973 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:22:36.015727  6623 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.018563 24171 rpcz_store.cc:269] 0809 23:19:54.897182 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.897457 (+   275us) service_pool.cc:227] Handling call
0809 23:22:36.015976 (+161118519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119814,"reported_tablets":47}
I20260809 23:22:36.021250 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.014207 24167 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.027637  6578 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:36.028069 24167 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.036207 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.037565 24428 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:36.028033  6578 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.038506  6578 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:36.036712  6623 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.045794 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.051602  6604 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.051259 24171 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:36.052141  6604 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.054399  6604 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.055579 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.070263 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.070752 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.071091  6623 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.071837  6623 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.072467  6621 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:36.076023 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.082443 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.087495  6609 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.088160  6609 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.081195  6289 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.089676  6289 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.090298  6609 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.091645  6289 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.094422 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.098614 24428 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 32.
W20260809 23:22:36.098677 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:36.096995 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.101558 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.101734 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.100521 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.105122  6604 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Leader election won for term 32
I20260809 23:22:36.105702  6604 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:36.106614  6604 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.104352  6621 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.081449  6623 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.099349 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.116750 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.119626  6511 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:36.106655  6578 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.156307 24378 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.156899 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.163854  6605 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:36.165706 24378 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.168547  6578 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.152434  6590 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.169392  6590 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.170135  6511 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.170778  6511 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.172830  6511 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.175721 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.176280 24427 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:36.166184 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.177191 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.180070  6620 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:36.180481  6620 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.180820  6620 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.183884 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.184044 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.188560  6605 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.188885 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.189091  6605 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.189474 25368 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:36.197721 24428 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:36.198321 24170 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 31 to 32, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 32 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:36.200611 24498 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:36.201310 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.201746  6604 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:36.187054  6590 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.177799  6578 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:36.203331  6605 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.207998 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.208714 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:36.210381 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:36.223490 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:36.225907 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.228029 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.232362  6476 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:36.230269  6605 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:36.232918  6476 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.233100  6605 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.235208  6476 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.235462  6605 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:36.215785  6590 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.237402  6590 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.228766  6623 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.240052  6590 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.240192  6623 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.242966 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.245167 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.250202  6476 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.251993  6623 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:22:36.255337 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.255761 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.257892  6623 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.272430  6605 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.273008  6289 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.273583  6289 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.276041  6289 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.276110  6620 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.276057 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.277482 24545 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:36.281287  6620 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.283687  6476 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.286051 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.286693 24422 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.276126  6605 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.290731  6623 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.291280  6623 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.293473  6511 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.298276  6511 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.300885  6511 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.301213  6621 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.306694 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.307230 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.308606 24422 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:36.309873 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.310992  6590 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:36.311619  6590 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.312804  6590 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.307482 24427 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.307428 24545 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.308156 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:36.317111 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.304934  6621 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.318444  6605 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:36.320678  6621 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.320338 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.318996  6605 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.322899  6605 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.304996  6623 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.344298 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.345630 25368 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:36.347309 24167 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.348569 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.347831 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.351965 25363 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:36.348745 24167 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 30, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 30 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.358831 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.359948 25365 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
W20260809 23:22:36.365483 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.373005 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.362408  6609 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:36.355793  6578 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.384415  6578 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.379403  6289 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.408126  6578 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.412528 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
W20260809 23:22:36.420725 24161 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 357) took 27370 ms (client timeout 15000 ms). Trace:
W20260809 23:22:36.429005 24161 rpcz_store.cc:269] 0809 23:22:09.052426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.052627 (+   201us) service_pool.cc:227] Handling call
0809 23:22:36.420343 (+27367716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":920363,"reported_tablets":47}
W20260809 23:22:36.433378 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.436643 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.441308 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.431569  6609 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:36.446395 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.448930 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.449303 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.403915  6289 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.457369 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.447872  6604 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.462124 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.445593  6649 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.459635  6622 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.511236  6649 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.511426  6622 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.513427  6649 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.514315  6622 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.514091  6609 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.472074  6578 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:36.473799  6289 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.462003  6623 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260809 23:22:36.518330 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:36.498306 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.519246 24543 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:36.521322 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.522488  6622 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.523104  6622 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.523658  6622 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:36.525760 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.528643 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.531880 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86830.8us
I20260809 23:22:36.532441  6643 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.535041  6643 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.536703  6622 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.537602 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75555.7us
I20260809 23:22:36.539111  6511 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:36.539170  6622 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.539767  6511 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.541102  6511 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.547849 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.548503 24539 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.552343  6638 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.552882  6638 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.554833  6611 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.555316  6611 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.555959  6638 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.557065 24539 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:36.557173  6611 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:36.533838 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.557402 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.558146 24427 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:36.557977 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.558058 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.559263 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:36.560199 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:36.560945 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.561822 24498 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.548656 24543 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260809 23:22:36.537173  6643 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.564546  6611 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.565011 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.586023  6604 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:36.565111  6638 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:36.586555  6604 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:36.586848  6638 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.587329  6638 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.587697  6604 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.571209 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119753us
I20260809 23:22:36.590644 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.568430 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.593303 24543 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:22:36.593689 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.594177 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.594815 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.594604  6623 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260809 23:22:36.599346 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.600178  6638 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.603680 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.604434 24427 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.625108 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.627120  6605 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:36.627553  6605 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.627928  6605 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:36.625864 24427 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:36.632215 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.633133  6511 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:36.635079  6611 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.635603  6611 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.636973 25363 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260809 23:22:36.636245 24167 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:36.562849 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.638917 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:36.638266 24167 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:36.640466 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:36.640312 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77500us
I20260809 23:22:36.641043  6289 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.641340  6652 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.642187  6289 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.643447  6611 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.645609  6289 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.641197  6609 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.651358  6609 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.651741  6609 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.654838 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:36.655917 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.656436 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.655553 25363 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:36.657531 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.658427 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.637017  6605 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.663558  6643 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.664115  6643 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.665546  6605 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.666079  6643 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.667531 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.668042 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.668133 24540 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:36.672418 25368 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260809 23:22:36.673659 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.674311 25375 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 33.
I20260809 23:22:36.675932 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.658847  6289 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:36.656126  6511 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.681953  6511 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.684221  6546 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.688051 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.688912 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.690215 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.690810 25363 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:36.691114 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:36.692076 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.641054  6638 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.697469 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.684777  6546 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.699760  6546 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.696456  6605 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:36.700994  6605 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.701911  6605 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.710964 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.713560 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.714217 24540 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:36.715358 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.718187  6546 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:36.720659  6546 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.721179  6546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.719022  6638 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.719712  6604 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.727371  6604 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.727829  6578 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260809 23:22:36.730055  6604 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.716178  6289 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:36.718673 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.732263 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.733609 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.734402 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.732203 24425 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260809 23:22:36.735098 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.735172  6609 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.735589 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.735545  6289 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.736280 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.736076  6611 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:22:36.737833 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.736692  6638 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.732752 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.745674  6578 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:36.745275 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.746107  6578 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.746555  6578 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:36.745285 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.749089  6620 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:36.749820  6658 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.750195  6658 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.750486  6653 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260809 23:22:36.750560  6658 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.750998  6653 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.751415  6653 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Advancing to term 34
I20260809 23:22:36.756464  6578 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.767854 24378 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.768330  6289 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:36.768676 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.769649 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.769989 24425 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260809 23:22:36.757956  6658 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.776819 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.775240 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:36.777480 25371 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.780891 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.781744 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.782526 24545 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.744694  6609 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.798133 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.799088 24539 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.805835 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.822595 24540 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260809 23:22:36.824067 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.824853 25368 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:36.758693  6578 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.831981 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:36.844065 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.783200  6652 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:36.809720  6476 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.828142 24545 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:36.883495  6656 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.884140  6656 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.884593  6656 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:36.887162 24165 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 355) took 88418 ms (client timeout 15000 ms). Trace:
W20260809 23:22:36.887580 24165 rpcz_store.cc:269] 0809 23:21:08.468468 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.470072 (+  1604us) service_pool.cc:227] Handling call
0809 23:22:36.876112 (+88406040us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87933,"reported_tablets":47}
I20260809 23:22:36.873402  6476 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.888443  6476 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.856004  6609 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
W20260809 23:22:36.903390 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.903980 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.915472 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.916160 25365 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.872614  6658 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:36.900673 24498 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.894639 25368 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:22:36.900228 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.853753  6649 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260809 23:22:36.903615 24161 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:36.827741  6616 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.098s
W20260809 23:22:36.923344 24378 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.914608  6623 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:36.917814 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.929026 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120411us
I20260809 23:22:36.916891  6651 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.917742  6609 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.961910 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:36.964490 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63842.5us
W20260809 23:22:36.977522 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102696us
I20260809 23:22:36.961272  6658 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.966743  6638 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.968549  6643 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.977360  6647 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:37.029688 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:37.037101 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:37.043982 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.046363 24171 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.051702  6578 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.052278  6578 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.053314  6578 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.070477 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97094.2us
I20260809 23:22:37.043732 24165 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 22, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 22 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.966183  6651 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.075839 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.037113  6647 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.006388  6658 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.081799  6668 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
W20260809 23:22:37.087627 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.091292 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.092007 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.093029 24428 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:37.093523 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.020138  6546 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.104875 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.110121  6643 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:37.110556  6643 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:37.110960  6643 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:37.111582  6687 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:22:37.114306  6658 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:37.115340 24163 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 373) took 77836 ms (client timeout 15000 ms). Trace:
W20260809 23:22:37.115875 24163 rpcz_store.cc:269] 0809 23:21:19.284711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.304198 (+ 19487us) service_pool.cc:227] Handling call
0809 23:22:37.115099 (+77810901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48454,"reported_tablets":47}
I20260809 23:22:37.042243  6609 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.129631  6647 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.147698 24425 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
W20260809 23:22:37.130959 24149 connection.cc:441] server connection from 127.0.0.1:35202 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 373) could send its response
I20260809 23:22:37.155722  6620 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260809 23:22:37.158643  6546 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:37.196631 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.197602 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.198143  6476 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.202077  6653 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.205868 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.206501 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.203009  6476 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.242575 24428 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:37.243073  6651 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:37.244982 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:37.248607 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.249529  6653 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.251345  6653 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:37.263649  6643 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:37.264909 24543 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:37.265594 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.266211 25368 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:37.265735  6643 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:37.283957 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.284893 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:37.290516 24423 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:37.291605 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.292212 25363 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Advancing to term 34
I20260809 23:22:37.302677 25368 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:37.319564 22524 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 24352)
I20260809 23:22:37.355049  6656 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.357618 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 255775us
I20260809 23:22:37.359997 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:37.360692 25372 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:37.363230 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:37.363651 24539 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260809 23:22:37.363801 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.364410 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.371512 25372 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:37.375608 24543 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:22:37.379331  6649 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:37.363858 24546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:37.393527 24423 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:37.386433 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:37.399029 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84979.1us
I20260809 23:22:37.401827 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.403983 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:37.404633 24543 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:37.407214 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.414678  6706 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:37.401058  6656 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:37.423245 24378 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.423441 25363 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 34.
I20260809 23:22:37.401861  6605 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:37.466519 24543 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:37.467694 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.469203  6710 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.469887  6710 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.470853  6710 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.471802 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64669.2us
I20260809 23:22:37.472134  6620 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:37.472723  6620 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.473672  6620 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.478853 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.479689 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.435071  6643 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.465390  6605 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:37.494369 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:37.477784  6697 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260809 23:22:37.496851  6605 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:37.518592  6638 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Leader election won for term 34
I20260809 23:22:37.519358  6638 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.492197  6643 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.520453  6643 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.520485  6638 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.590126 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62443.5us
W20260809 23:22:37.590215 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54528.7us
W20260809 23:22:37.617717 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83704.4us
W20260809 23:22:37.621906 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.636514 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.637068 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.638075 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.638489 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.643994 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.645466 24378 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.645942 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:37.620541 24546 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 29.
I20260809 23:22:37.672513  6691 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:37.672003 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:37.683614 24546 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260809 23:22:37.685977 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.688076  6712 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260809 23:22:37.696260 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:37.699887 25363 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:37.680020  6676 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:37.707556  6676 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:37.708454 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.708544  6676 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.701891 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80103.7us
I20260809 23:22:37.751214  6691 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:37.732060  6605 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.766626 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.767501 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.769934 25363 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:37.796846  6605 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.777441  6656 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260809 23:22:37.814085 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.814733 25363 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:37.817849 25372 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260809 23:22:37.829679 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:37.831710 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.832746 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:37.845273 25363 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:37.849568 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.850126 24171 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 359) took 1684 ms. Trace:
I20260809 23:22:37.859333  6656 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260809 23:22:37.859534  6651 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:22:37.882771 24378 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.850490 24171 rpcz_store.cc:276] 0809 23:22:36.166218 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:36.166372 (+   154us) service_pool.cc:227] Handling call
0809 23:22:37.850073 (+1683701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:37.203714 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:37.204162 (+   448us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:37.204190 (+    28us) write_op.cc:435] Acquired schema lock
0809 23:22:37.204201 (+    11us) tablet.cc:660] Decoding operations
0809 23:22:37.229239 (+ 25038us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:37.234030 (+  4791us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:37.239407 (+  5377us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:37.271128 (+ 31721us) tablet.cc:699] Row locks acquired
0809 23:22:37.274608 (+  3480us) write_op.cc:260] PREPARE: finished
0809 23:22:37.291973 (+ 17365us) write_op.cc:270] Start()
0809 23:22:37.297712 (+  5739us) write_op.cc:276] Timestamp: P: 1786317757291903 usec, L: 0
0809 23:22:37.305003 (+  7291us) op_driver.cc:331] REPLICATION: starting
0809 23:22:37.351626 (+ 46623us) log.cc:842] Serialized 533 byte log entry
0809 23:22:37.695950 (+344324us) op_driver.cc:444] REPLICATION: finished
0809 23:22:37.756071 (+ 60121us) write_op.cc:301] APPLY: starting
0809 23:22:37.757452 (+  1381us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:37.795413 (+ 37961us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:37.795559 (+   146us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:37.796363 (+   804us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:37.796570 (+   207us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:37.796615 (+    45us) write_op.cc:312] APPLY: finished
0809 23:22:37.796811 (+   196us) log.cc:842] Serialized 25 byte log entry
0809 23:22:37.796980 (+   169us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:37.797068 (+    88us) write_op.cc:454] Released schema lock
0809 23:22:37.797104 (+    36us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:37.797424 (+   320us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:37.797450 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":78877,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":5071,"apply.run_cpu_time_us":3943,"apply.run_wall_time_us":67148,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"mutex_wait_us":23908,"num_ops":1,"prepare.queue_time_us":127342,"prepare.run_cpu_time_us":5442,"prepare.run_wall_time_us":149267,"replication_time_us":376419}]]}
I20260809 23:22:37.896363 24161 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.916560 24161 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 28 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 29 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.892040  6651 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.918140 24161 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.935763  6651 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.929203 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.940235 24161 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.946978 24161 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 24, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.952870 24161 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.954839 24161 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:37.981855 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:37.992295 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.995965 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.080034 24165 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 28 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 29 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:38.081002 25363 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260809 23:22:38.087561  6655 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:38.095188 24543 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:38.096500  6691 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:38.102751 24171 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:38.104940 24171 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 33 to 34, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 34 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:38.117668 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.136118 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.141448 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.106550 24171 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:38.169816 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.173801 24427 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:38.174258 25363 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260809 23:22:38.176508  6620 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:38.177851  6620 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260809 23:22:38.194178 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:38.197789 24165 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:38.238234 24546 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260809 23:22:38.241605  6675 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260809 23:22:38.320088 22524 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:38.397346 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:38.410494 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:38.436255 25375 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:38.437789  6691 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260809 23:22:38.489917 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.540879 25323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 369 similar messages]
W20260809 23:22:38.632645 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:38.666527 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:38.723593 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:38.828394 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.844589 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.907812 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:38.929491 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.935875 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:39.030452 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.069355 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.121943 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.163829 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.215929 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.246104 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.320679 22524 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:39.325034 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:39.332139 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.334118 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.340541 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.374466 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.486325 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.497105 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.541884 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45124.4us
W20260809 23:22:39.631208 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.647257 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.691574 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.762539 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.779362 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.840782 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.903658 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.018432 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.019141 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.293251 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.305536 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.321506 22524 thread.cc:536] Waited for 4000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:40.331434 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.342473 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.365185 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.382272 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:40.414088 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.504138 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.522156 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.533262 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.542294 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.550172 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.653672 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.654194 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.679430 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.771974 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.983645 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.048519 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.067188 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:41.204190 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.257122 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.265594 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.322227 22524 thread.cc:536] Waited for 5000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:41.378995 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.379693 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.523564 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76082.5us
W20260809 23:22:41.537828 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.545823 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.644548 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.656816 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.699756 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.738806 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.774240 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.858539 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.872458 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:41.906867 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.915017 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.916898 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.929023 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.932312 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.984566 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.026434 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.050009 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.264060 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.270288 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.358246 22524 thread.cc:536] Waited for 6000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:42.400383 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.424820 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:42.444995 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.507004 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.513021 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.642519 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.653707 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.776948 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.785567 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.787222 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:42.796777 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.995898 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:43.043123 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:43.067287 24427 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:43.111157 25363 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260809 23:22:43.116122  6763 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260809 23:22:43.122372 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.142885 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:43.152338 24544 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260809 23:22:43.163671 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 23:22:43.169204  6763 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:43.173656  6786 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:43.175432 25375 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:43.194276 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 23:22:43.214780 24171 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:43.224139 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 15)
I20260809 23:22:43.234522 24544 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:43.246618 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.277037 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:43.348836 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.360848 22524 thread.cc:536] Waited for 7000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:43.465983 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.481552 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:43.506009 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:43.546617 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 475 similar messages]
W20260809 23:22:43.613065 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.695175 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.734637 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:43.762704 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.949922 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.971514 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.981755 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.071445 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.185603 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.204026 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.205547 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.214458 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.251688 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.288883 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.295830 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.323282 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.361466 22524 thread.cc:536] Waited for 8000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:44.422550 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.429253 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.524637 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.534207 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.590778 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.639392 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.650266 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.656983 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:44.703169 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.913066 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.916213 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.975437 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.976068 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.031840 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.065470 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:45.102443 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.223168 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.272696 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.274536 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.362315 22524 thread.cc:536] Waited for 9000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:45.403436 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.410341 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.547221 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.603015 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.645450 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.650213 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.694408 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.772217 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.799011 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.801237 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.802277 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:45.921953 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48313us
W20260809 23:22:45.922760 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.945344 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.011148 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.203291 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.220029 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.322443 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.345176 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.363065 22524 thread.cc:536] Waited for 10000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:46.373409 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.377466 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.398969 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.649892 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.679776 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.717887 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.719462 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.880457 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.908936 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.939949 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.954097 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.019589 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.028900 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.029526 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.032310 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.047176 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.222010 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.262892 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.318364 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.363655 22524 thread.cc:536] Waited for 11000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:47.383950 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52034.5us
W20260809 23:22:47.448067 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.507507 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.536587 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.540005 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.652731 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.685117 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.687077 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.777523 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.791504 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.847868 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.972888 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.981439 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.025175 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.032135 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:48.211882 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66920.4us
W20260809 23:22:48.223740 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.248937 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.260950 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08004028c0 after lost signal to thread 22536
W20260809 23:22:48.262603 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc3a0 after lost signal to thread 24220
W20260809 23:22:48.267364 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.267450 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800439ca0 after lost signal to thread 24469
W20260809 23:22:48.269533 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08003723a0 after lost signal to thread 24579
W20260809 23:22:48.270995 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb860 after lost signal to thread 25410
W20260809 23:22:48.307157 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.334534 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.347944 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:48.357765 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:48.364295 22524 thread.cc:536] Waited for 12000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:48.394030 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.549158 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 463 similar messages]
W20260809 23:22:48.551823 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.585500 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.750501 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.753178 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:48.806869 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.823016 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:48.893935 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.953387 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.100961 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.112106 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.123922 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.308080 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.364880 22524 thread.cc:536] Waited for 13000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:49.367897 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.389302 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:49.446261 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:49.471419 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.543403 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.528348 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.554752 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.604717 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.631367 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.694960 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:49.775035 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.853561 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.866576 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:50.043990 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:50.077615 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.091914 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.166442 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.247498 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59173.2us
W20260809 23:22:50.293711 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95726.8us
W20260809 23:22:50.297850 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.330900 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.333119 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.365419 22524 thread.cc:536] Waited for 14000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:50.381793 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.382426 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.447394 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.511991 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.607074 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.635382 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.643216 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.775123 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.787914 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.838515 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.972340 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:51.011555 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46063.7us
W20260809 23:22:51.066491 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:51.133636 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.220157 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.229753 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:51.287300 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46827.5us
W20260809 23:22:51.290987 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.365943 22524 thread.cc:536] Waited for 15000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:51.389916 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08003383e0 after lost signal to thread 22536
W20260809 23:22:51.391295 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08004045c0 after lost signal to thread 24220
W20260809 23:22:51.392832 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080024d400 after lost signal to thread 24469
W20260809 23:22:51.394222 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08002669e0 after lost signal to thread 24579
W20260809 23:22:51.395615 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad4c0 after lost signal to thread 25410
W20260809 23:22:51.517055 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67637.4us
W20260809 23:22:51.542796 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.595595 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.613942 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.665479 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.673713 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.675976 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.676581 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.674353 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.701221 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.792834 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.812414 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.958516 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.966392 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.003068 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.003926 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.118799 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.142284 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.155673 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.260723 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.287057 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.366530 22524 thread.cc:536] Waited for 16000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:52.376441 24151 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 179514us
W20260809 23:22:52.424111 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.434791 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:52.478682 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.497992 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.501741 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.699298 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.760114 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.775743 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.848161 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.870779 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.878947 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.897091 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.917240 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.925702 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.050067 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.068535 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.261554 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.278112 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.315799 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.367496 22524 thread.cc:536] Waited for 17000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:53.418872 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.544785 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.545728 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.550076 25323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 447 similar messages]
W20260809 23:22:53.552659 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:53.581354 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.649751 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.651990 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.670410 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.689635 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.884833 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.925707 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.974997 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:54.009560 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.044349 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.119601 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.138975 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.216213 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.235695 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.368498 22524 thread.cc:536] Waited for 18000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:54.386703 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.443899 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.451130 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.507561 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:54.567802 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.652784 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.664502 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:54.686590 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.690105 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.718998 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53248.7us
W20260809 23:22:54.808887 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.814916 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.961126 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.998842 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.060998 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.136780 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.145308 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.200248 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.210507 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.243661 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.369002 22524 thread.cc:536] Waited for 19000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:55.393031 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.420848 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.439136 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.491663 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.496644 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.506549 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.587136 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.664705 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.887292 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.972971 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.975260 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108193us
W20260809 23:22:55.999130 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:56.003700 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:56.012493 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:56.089093 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56306.2us
W20260809 23:22:56.099349 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.109369 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.203042 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71090.3us
W20260809 23:22:56.203872 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.295034 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 157058us
W20260809 23:22:56.369570 22524 thread.cc:536] Waited for 20000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:56.367870 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.422451 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.423388 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53443.7us
W20260809 23:22:56.443636 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.482096 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.583734 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:56.621840 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.649557 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.681615 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.823266 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.825407 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.976714 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.988842 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:57.019461 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52922.8us
W20260809 23:22:57.035686 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.036978 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.081753 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44833.6us
W20260809 23:22:57.077755 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.087638 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 127227us
W20260809 23:22:57.086755 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51815.1us
W20260809 23:22:57.100490 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.158269 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.205722 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.207216 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.207732 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.313390 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.370642 22524 thread.cc:536] Waited for 21000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:57.464855 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.553798 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.638461 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.651748 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.720824 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.843542 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.843678 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08002011a0 after lost signal to thread 22536
W20260809 23:22:57.845160 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080005db60 after lost signal to thread 24220
W20260809 23:22:57.846634 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800346d60 after lost signal to thread 24469
W20260809 23:22:57.848953 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800318160 after lost signal to thread 24579
W20260809 23:22:57.850350 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08001b97a0 after lost signal to thread 25410
W20260809 23:22:57.919503 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.940189 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:57.967974 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.235687 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.282364 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.306668 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.332921 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.365753 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.375756 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.421674 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.604270 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 114436us
W20260809 23:22:58.606513 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.607425 24378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 432 similar messages]
W20260809 23:22:58.741597 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:58.748865 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 319200us
W20260809 23:22:58.776238 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:58.799105 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.845772 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.896803 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.921192 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:58.975860 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.979249 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.982692 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:59.103904 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.114744 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260809 23:22:59.115378 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
W20260809 23:22:59.131803 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.160400 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.239082 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:59.260784 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.309881 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.388428 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:22:59.407191 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
W20260809 23:22:59.420207 24496 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.464097 24496 consensus_peers.cc:487] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.470050 25321 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.495548 24496 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.496939 24496 consensus_peers.cc:487] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.499890 25321 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.514068 25321 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.516753 24496 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.522607 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:22:59.525205 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.526414 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.529441 24496 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.530012 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.533202 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.537729 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.540827 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08001718e0 after lost signal to thread 22536
I20260809 23:22:59.543989 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.544320 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08002524e0 after lost signal to thread 24220
I20260809 23:22:59.544680 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.545922 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08003b5360 after lost signal to thread 24469
I20260809 23:22:59.547734 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.548183 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08003ce780 after lost signal to thread 24579
I20260809 23:22:59.548617 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.550341 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800365ac0 after lost signal to thread 25410
I20260809 23:22:59.550669 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.554395 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.559439 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.561132 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.570411 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.571326 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.572031 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.579250 24496 consensus_peers.cc:487] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.581585 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.582652 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.583367 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.589406 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.591629 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.597581 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.597692 25321 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.602469 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.605757 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.606658 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.607934 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.610740 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.615568 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.616289 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.618481 25321 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.623008 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.625131 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.625599 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:22:59.626675 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.631801 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.636312 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.640152 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.643150 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.643992 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.644650 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.647476 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.648921 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.650209 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.652860 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.653716 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.654804 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.655813 24496 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.657420 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.658298 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.659260 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.662096 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.662906 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.664022 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.664988 24496 consensus_peers.cc:487] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.666893 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.667900 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.669112 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.671836 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.672650 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.673326 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.676057 24496 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.683148 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.694794 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.696729 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:22:59.698987 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.713644 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.714725 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 LEADER]: Raft consensus shutting down.
W20260809 23:22:59.716053 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:22:59.716061 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.719242 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.721617 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.722434 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.725143 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.728137 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.729169 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.731905 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.732774 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.733330 25321 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.734033 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.736668 24496 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.736820 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.737768 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.738510 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.741452 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.742200 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.743096 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.746194 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.747056 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.751068 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.753513 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.754313 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.755576 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.758246 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.760141 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.761615 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.768427 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.770994 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.772274 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.772671 25321 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.777433 24496 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.777819 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.778599 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.783110 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.788234 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.789211 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.789986 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.795145 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.800843 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 23:22:59.802321 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.803095 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.807219 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.808132 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.813192 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.817631 24496 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.815850 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.821504 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.822872 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.826555 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:22:59.827100 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.828007 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.829569 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.834688 24496 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.844429 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.845830 24496 consensus_peers.cc:487] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.846338 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.847942 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.852559 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.853350 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.854038 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.858114 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.858958 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.860250 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.861634 24496 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.866024 25321 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.869925 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.871073 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Raft consensus shutting down.
W20260809 23:22:59.879765 25321 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.872406 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.885178 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.887179 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.887974 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.890738 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.891458 24496 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.891503 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.896991 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.899775 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.900633 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.901857 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.905283 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.906450 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.907262 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.913030 25321 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.914628 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.915482 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.916203 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.918972 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.919704 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.920341 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.925096 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.925803 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.926590 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.929589 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.930240 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.930915 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.936053 24496 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.936856 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:59.945518 25321 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.979723 24496 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:00.151129 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:00.166225 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.263809 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.300781 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:00.404243 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:00.428694 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:23:00.429443 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:00.430051 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.471663 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 23:23:00.541488  6882 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.542562  6882 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.553126  6882 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.555037 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:00.555732 25321 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.556270 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.557637  6882 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:00.635797  6902 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.636442  6902 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:00.638468  6902 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.649678 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:00.652194 24496 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.652643 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.654052  6887 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260809 23:23:00.732260 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.739192  6902 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.739708  6902 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.743423  6902 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:00.745087 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:23:00.751788 24496 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.754261 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:00.755484  6887 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:00.771255  6898 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.772006  6898 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.785213  6887 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.774710  6898 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:00.785531 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:00.789062 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.789583 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.791497  6898 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:00.795977  6887 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:00.797997  6887 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.888159  6903 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.888962  6903 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.891700  6903 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.892477  6903 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.893114  6903 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.895524  6903 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.891587  6898 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.896302  6898 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.898367  6898 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.903151 25321 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.903923 25321 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.904454 25321 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.907500 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.907630  6898 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.908051 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.908196  6898 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.908888  6903 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:23:00.909850 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.911001 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.910209  6898 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.911926  6898 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260809 23:23:00.912752 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.913380 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:00.915791 25321 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.920619 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.922425  6898 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:00.922449 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.924201  6898 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:01.233134  6921 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.245956  6903 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.253419  6907 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.254019  6907 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.256237  6907 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.247227  6903 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.259202  6903 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:01.260947 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:23:01.266809  6882 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.267458  6882 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.268177  6920 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.268901  6920 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.272619  6920 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.279363  6887 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.283774  6887 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.286469  6887 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:23:01.282877 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:23:01.269464  6882 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:01.289903 25321 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.296285 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.297075 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.297610 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.298184 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.293606  6903 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.299203  6903 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.301291  6903 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:01.305308 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.233789  6921 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.316179  6921 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.298640 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.317857  6903 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:23:01.322595  6933 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.323462  6933 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.330801  6887 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:23:01.331334 24498 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.331377  6887 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.333429  6887 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.341257  6921 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.341847  6921 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.329253  6933 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.327505  6938 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:23:01.347081 24496 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.348506 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.357219 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.358171 25372 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:23:01.347823  6938 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:23:01.362879 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:23:01.363848  6938 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.364384  6938 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.365386 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:23:01.365628 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.366092 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:23:01.366197 25372 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
W20260809 23:23:01.367014 24496 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.367293 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:01.367820 24496 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.368237 24543 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:23:01.368726 24496 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.369769 24496 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.371579 24496 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.363823 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:23:01.347234  6921 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.372778 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:01.376107 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.376518 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.376653 24540 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:23:01.377327  6931 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:23:01.382838 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.383948  6920 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.384330  6920 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.384675  6920 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:23:01.385229 24496 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.385910 24496 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.388934 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.389747  6931 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.390152  6931 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.390553  6931 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.400089 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.400449 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.400974  6903 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:23:01.401242  6921 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:23:01.401329  6903 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.401736  6921 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.401799  6903 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:23:01.407898  6921 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:23:01.397464  6920 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.421435  6921 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.416770 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.416721  6903 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.427883 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.430756  6924 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.431238  6924 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.431596  6924 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.433043  6903 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.398324  6931 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.438524 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.440888  6938 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.441431  6938 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.446748 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.447562 24543 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:23:01.451465  6938 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.454375  6924 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.453603 24496 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.453441 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.459767  6903 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:23:01.462616 24496 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.469880 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.470679 24539 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:23:01.463440 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.472978 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:23:01.476796 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.477190 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.477878  6938 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:23:01.478281 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.478703 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.479323  6938 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:23:01.480362 24498 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.480741 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.481357  6938 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:01.457006  6921 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.474148 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:01.485071 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.475162 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.485726 24543 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.485816 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.486472 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:23:01.489825 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.490489 25372 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:23:01.477517 24496 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.474565  6924 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.485209  6903 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.467020  6931 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.492944 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.493623 25372 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 28: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:23:01.495401 24543 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:23:01.467523  6920 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.503159 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:01.503731 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.499194 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.504487 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.505231  6931 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:01.506047  6920 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:23:01.506793  6920 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:23:01.507714  6920 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.509847 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.510622  6931 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:23:01.510989  6931 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.511337  6931 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:23:01.531504 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:01.532779 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:01.535176  6931 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.536233 24163 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 27 to 28, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 28 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:23:01.537259  6931 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.537858  6938 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:23:01.538487  6938 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.540844 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.541396 25372 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:23:01.546856 24496 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.552910 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:23:01.554466 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.558862  6921 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:23:01.559374  6921 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:23:01.560575  6921 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.610781 24165 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 22 to 23, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 23 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:01.785490 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.818131 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.880503 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.898439 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.949636 24496 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:01.950184 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080023c280 after lost signal to thread 22536
W20260809 23:23:01.953676 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080023c600 after lost signal to thread 24220
W20260809 23:23:01.955268 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080031acc0 after lost signal to thread 24579
W20260809 23:23:01.960033 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800415660 after lost signal to thread 25410
W20260809 23:23:01.963610 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.964488 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.952307 25321 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:01.997169 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.005687 24496 consensus_peers.cc:487] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.012259 25321 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.017490 24543 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260809 23:23:02.020702  6903 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260809 23:23:02.041153 24496 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.077796 25321 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.118309 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.140836 25321 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.158324 25372 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:23:02.160393  6954 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:23:02.183859 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.217566 25321 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.232977 25321 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.233074 24496 consensus_peers.cc:487] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.245766 24496 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.248039 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.249629 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:02.258755 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
W20260809 23:23:02.282390 24496 consensus_peers.cc:487] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.354059 24496 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.359382  6931 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.359952  6931 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:02.362468  6931 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:02.368746 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:02.379698 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.381786 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.383431 24496 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.383806 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.384569  6931 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260809 23:23:02.395857 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
W20260809 23:23:02.404567 24244 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:23:02.404946 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:23:02.415123 24496 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.417028 25325 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.417162 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:02.421355 25325 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.432889 25321 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.461719 25321 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.464066 24496 consensus_peers.cc:487] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.467670 24496 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.478345 24496 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.482869 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:23:02.486171 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.487167 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.489130 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.492143 24496 consensus_peers.cc:487] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.493817 25321 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.495223 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.496100 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.497228 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.498071  6903 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.498652  6903 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.499749 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.500732 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.500854  6903 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.501562 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.505592 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.506458 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.506920 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.507616  6924 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.508133  6924 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:02.508841 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:02.510320  6924 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.513873 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.514623 25321 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.515173 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:02.515508 24496 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.516013  6924 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:02.516911 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.516980 25321 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.517609 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.518627 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.520867 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.521485 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.522327 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.513881 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:02.522807 25321 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.525238 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.525821 25325 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.525730 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.526111 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.526906  6924 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:02.527261 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.528868  6924 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.529390  6924 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.531260  6924 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.529377 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.532474 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.533516 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.535673 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.536342 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.537185 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.537748 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.539853 25321 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.540105 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.540343 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.540879 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.541404  6903 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:02.541581 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.544180 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.545547 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.546146 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.548327 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.548904 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.549449 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.551319 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.552006 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.552765 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.554922 25325 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.555051 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.555778 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.556672 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.558598 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.559332 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.559949 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.562451 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.562410 25321 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.563325 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.563829 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.566565 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.567341 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.568039 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.570415 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.571271 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.572266 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.573915 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.574421 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.575316 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.577057 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.577598 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.578392 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.580183 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.580725 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.581781 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.585114 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.586175 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.587050 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.587308 25325 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.587371  6924 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.588732  6924 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.590152 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.591303 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.592159 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.592218  6924 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.595479 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.596372 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.597744 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.597846 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.598364 25321 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.598829 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.599622  6924 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:02.643556 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.652478  6903 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.653954  6903 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.658028  6903 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.659961 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:02.661852 25325 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.663939 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.671710 25321 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.672771 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.674644 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.675028 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.675698 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.676906  6903 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:02.678380 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.683221 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.684638 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.688923 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.702219 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.704466 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.707968 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.713065 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.714759 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.717519 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.721788 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.722970 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.727010 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.731318 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.732726 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.734470 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.738582 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.739535 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.743685 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.746297 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260809 23:23:02.747494 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.748739 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.751199 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.754685 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.755591 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.758775 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.762682 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.763784 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.766157 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.766827 24496 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.769929 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.770967 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:02.771200 24496 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.772348 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.775756 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.776618 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.778648 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.783241 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.784272 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.785516 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.787022 25325 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.789552 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.790602 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.792835 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.796162 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.797034 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.799695 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.803699 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.804515 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.805473 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.806208 25321 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.810025 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.811159 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.814358 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.817891 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.818888 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.820811 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.824559 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.825394 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.826297 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.830163 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.830960 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.831774 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.835052 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.835568 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.836292 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.854787 25325 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.857161 25325 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.864897 25325 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.874090 25325 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.886818 25325 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.912813 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:02.938782 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.994361 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:03.132210 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260809 23:23:03.386475  6973 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:03.387143  6973 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.389140  6973 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.392467 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.392908 25325 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.393323 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.394191  6973 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:23:03.491757 24244 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.498018 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.498311 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.503813 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.504073 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.507915 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.508167 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.512689 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.512943 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.522011 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:03.555186  6973 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:03.555928  6973 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.557705  6973 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.558435  6973 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:03.559001  6973 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:03.561254 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.561653  6973 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:03.561803 25325 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.562348 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.565169  6973 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260809 23:23:03.565759 25321 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.567560 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.567965 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.568823  6973 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:23:03.614835 25321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111) [suppressed 612 similar messages]
I20260809 23:23:03.655536 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:23:03.701689  6973 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.702499  6973 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.705071  6973 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.709823 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.710400 25325 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.711035 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.712066  6972 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:23:03.714859  6972 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.715619  6972 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.718274  6972 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.722141 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:03.725029 25325 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.726444 25321 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.726981 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.727857  6972 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:03.728405  6973 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.729110  6973 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.731532  6973 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.735973 25321 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.736557 25325 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.736943 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.737687  6972 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:03.746222  6972 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.747133  6972 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.749418  6972 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.752835 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:03.753433 25325 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.753911 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.754755  6973 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:03.767558 22524 tablet_server.cc:178] TabletServer@127.21.255.4:0 shutting down...
I20260809 23:23:03.812103  6972 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.813555  6972 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.816099  6972 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:23:03.821002 25325 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.821513 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.821942 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.822822  6972 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:03.840520 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:23:03.841497 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.842864 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.844300 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.846683  6972 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.847556 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.847313  6972 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.848791 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.849376  6972 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.850423 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:03.853408 25321 leader_election.cc:334] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.854099 25325 leader_election.cc:334] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.854467 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.854560 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.855373  6972 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.855865 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.855957  6972 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.855834  6924 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:23:03.857800 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.858179  6972 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.861308 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
W20260809 23:23:03.861378 25325 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.862289 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.862330 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.862794 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.863499  6972 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260809 23:23:03.863798 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.866340 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.866984 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.867901 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.870309 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.871071 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.871683 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.874653 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.874336  6972 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.875242  6972 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.875691 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.876426 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.877514  6972 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.879837 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.880636 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:03.880837 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:03.881567 25325 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.881814 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.882278 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.883231  6972 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:23:03.884093  6972 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.885387 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.884683  6972 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.886879 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.888149  6972 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.888338 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.891499 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
W20260809 23:23:03.891474 25321 leader_election.cc:334] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.892321 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.893085 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:03.893620 25325 leader_election.cc:334] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.893991 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.894609  6972 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:03.896085 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.896886 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.898033 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.900614 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.901175 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.902297 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.904667 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.905371 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.906469 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.908741 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.909269 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.909816 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.911885 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.912432 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.913350 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.915619 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.916184 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.917100 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.919489 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.920024 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.921113 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.923267 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.923748 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.924211 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.925987 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.926509 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.927449 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.929385 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.929978 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.930434 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.932400 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.932934 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.933436 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.935463 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.936008 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.936525 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.938468 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.939014 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.939683 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.941700 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.942199 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.942730 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.945461 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.946228 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.947672 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.950340 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.951021 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.951655 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.953715 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.954239 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.955259 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.957158 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.957700 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.958184 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.960191 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.960877 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.962131 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.964967 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.965566 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.966723 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.968778 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.969347 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.969908 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.971910 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.972414 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.973270 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.975214 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.975742 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.976370 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.978348 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.978826 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.979804 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.982896 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.983464 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.984067 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.986003 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.986510 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.987120 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.989279 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.989765 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.990675 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.993103 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.993916 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.995169 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.997691 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.998189 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.999676 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:04.401579 22524 tablet_server.cc:194] TabletServer@127.21.255.4:0 shutdown complete.
I20260809 23:23:04.473778 22524 master.cc:537] Master@127.21.255.62:37921 shutting down...
W20260809 23:23:17.123988 24151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260809 23:23:17.127803 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:33.968999 24152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:33.972368 24152 catalog_manager.cc:4584] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): ChangeConfig:REMOVE_PEER RPC failed for tablet cd2b577e2c3e4021bc8ebdfa3462437d: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:37.689265 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:37.692821 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:37.693310 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: stopping tablet replica
I20260809 23:23:37.924127 22524 master.cc:552] Master@127.21.255.62:37921 shutdown complete.
I20260809 23:23:37.988503 22524 test_util.cc:175] -----------------------------------------------
I20260809 23:23:37.988709 22524 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0

[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260809 23:16:24.948139 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:41299
I20260809 23:16:24.950466 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 23:16:24.952250 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:24.986785 22544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:24.986898 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:24.987247 22541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:24.988384 22524 server_base.cc:1034] running on GCE node
I20260809 23:16:26.329490 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:26.330080 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:26.330348 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317386330298 us; error 0 us; skew 500 ppm
I20260809 23:16:26.331928 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:26.341130 22524 webserver.cc:460] Webserver started at http://127.21.255.62:35535/ using document root <none> and password file <none>
I20260809 23:16:26.342581 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:26.342840 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:26.343549 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:26.352695 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "d639e2db02b14b668b28ae8a4cfe5f8f"
format_stamp: "Formatted at 2026-08-09 23:16:26 on dist-test-slave-drl0"
I20260809 23:16:26.363902 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.010s	user 0.010s	sys 0.001s
I20260809 23:16:26.371775 22558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:26.373413 22524 fs_manager.cc:725] Time spent opening block manager: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:16:26.373955 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "d639e2db02b14b668b28ae8a4cfe5f8f"
format_stamp: "Formatted at 2026-08-09 23:16:26 on dist-test-slave-drl0"
I20260809 23:16:26.374327 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:26.421191 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:26.423322 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 23:16:26.423913 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:26.515033 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:41299
I20260809 23:16:26.515167 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:41299 every 8 connection(s)
I20260809 23:16:26.527225 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:26.555554 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: Bootstrap starting.
I20260809 23:16:26.563970 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:26.566610 22637 log.cc:824] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: Log is configured to *not* fsync() on all Append() calls
I20260809 23:16:26.572811 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: No bootstrap required, opened a new log
I20260809 23:16:26.599774 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER }
I20260809 23:16:26.600515 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:26.600783 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d639e2db02b14b668b28ae8a4cfe5f8f, State: Initialized, Role: FOLLOWER
I20260809 23:16:26.602026 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER }
I20260809 23:16:26.602604 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:16:26.602883 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:16:26.603235 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:26.607678 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER }
I20260809 23:16:26.608717 22637 leader_election.cc:302] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d639e2db02b14b668b28ae8a4cfe5f8f; no voters: 
I20260809 23:16:26.610620 22637 leader_election.cc:288] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:16:26.611075 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:26.614317 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 1 LEADER]: Becoming Leader. State: Replica: d639e2db02b14b668b28ae8a4cfe5f8f, State: Running, Role: LEADER
I20260809 23:16:26.615373 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER }
I20260809 23:16:26.616568 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:16:26.629403 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d639e2db02b14b668b28ae8a4cfe5f8f. Latest consensus state: current_term: 1 leader_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER } }
I20260809 23:16:26.630883 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d639e2db02b14b668b28ae8a4cfe5f8f" member_type: VOTER } }
I20260809 23:16:26.631889 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [sys.catalog]: This master's current role is: LEADER
I20260809 23:16:26.631148 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [sys.catalog]: This master's current role is: LEADER
I20260809 23:16:26.635993 22657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:16:26.652550 22657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:16:26.654878 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:16:26.672775 22657 catalog_manager.cc:1347] Generated new cluster ID: 079efc92673041e8b6fe4634df7fe739
I20260809 23:16:26.673115 22657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:16:26.705711 22657 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:16:26.708055 22657 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:16:26.728106 22657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: Generated new TSK 0
I20260809 23:16:26.729458 22657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:16:26.787096 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:26.798349 22668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:26.799563 22669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:26.804663 22524 server_base.cc:1034] running on GCE node
W20260809 23:16:26.804780 22673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:26.805822 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:26.806020 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:26.806334 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317386806208 us; error 0 us; skew 500 ppm
I20260809 23:16:26.807417 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:26.812201 22524 webserver.cc:460] Webserver started at http://127.21.255.1:36615/ using document root <none> and password file <none>
I20260809 23:16:26.812758 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:26.812971 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:26.813246 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:26.814597 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "e609d1a6e91147bb978f92fcd441ef58"
format_stamp: "Formatted at 2026-08-09 23:16:26 on dist-test-slave-drl0"
I20260809 23:16:26.820441 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20260809 23:16:26.825232 22680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:26.826162 22524 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260809 23:16:26.826524 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "e609d1a6e91147bb978f92fcd441ef58"
format_stamp: "Formatted at 2026-08-09 23:16:26 on dist-test-slave-drl0"
I20260809 23:16:26.827034 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:26.858104 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:26.859967 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:26.863824 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:26.870265 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:26.870632 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:26.870995 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:26.871217 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:26.984023 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:43667
I20260809 23:16:26.984266 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:43667 every 8 connection(s)
I20260809 23:16:26.992919 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:27.018138 22787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:27.020964 22788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:27.030869 22524 server_base.cc:1034] running on GCE node
W20260809 23:16:27.032608 22790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:27.033654 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:27.033897 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:27.034093 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317387034081 us; error 0 us; skew 500 ppm
I20260809 23:16:27.034827 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:27.039321 22524 webserver.cc:460] Webserver started at http://127.21.255.2:34055/ using document root <none> and password file <none>
I20260809 23:16:27.040059 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:27.040339 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:27.040695 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:27.042044 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
format_stamp: "Formatted at 2026-08-09 23:16:27 on dist-test-slave-drl0"
I20260809 23:16:27.044328 22780 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41299
I20260809 23:16:27.044814 22780 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:27.046152 22780 heartbeater.cc:511] Master 127.21.255.62:41299 requested a full tablet report, sending...
I20260809 23:16:27.051645 22584 ts_manager.cc:194] Registered new tserver with Master: e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667)
I20260809 23:16:27.052539 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.010s	user 0.007s	sys 0.004s
I20260809 23:16:27.055594 22584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50316
I20260809 23:16:27.058522 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:27.059561 22524 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.001s
I20260809 23:16:27.059882 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
format_stamp: "Formatted at 2026-08-09 23:16:27 on dist-test-slave-drl0"
I20260809 23:16:27.060259 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:27.073698 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:27.074920 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:27.076357 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:27.079511 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:27.079721 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:27.079975 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:27.080148 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:27.182443 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:45785
I20260809 23:16:27.182582 22888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:45785 every 8 connection(s)
I20260809 23:16:27.189042 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:27.200163 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:27.203006 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:27.208730 22524 server_base.cc:1034] running on GCE node
I20260809 23:16:27.209436 22898 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41299
I20260809 23:16:27.209841 22898 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:27.210763 22898 heartbeater.cc:511] Master 127.21.255.62:41299 requested a full tablet report, sending...
W20260809 23:16:27.211462 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:27.213033 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:27.213312 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:27.213507 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317387213494 us; error 0 us; skew 500 ppm
I20260809 23:16:27.213603 22584 ts_manager.cc:194] Registered new tserver with Master: f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:27.214233 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:27.215612 22584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50328
I20260809 23:16:27.217478 22524 webserver.cc:460] Webserver started at http://127.21.255.3:43867/ using document root <none> and password file <none>
I20260809 23:16:27.218201 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:27.218452 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:27.218762 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:27.220192 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "0a36468db59a4d328c8c7a060916e4bd"
format_stamp: "Formatted at 2026-08-09 23:16:27 on dist-test-slave-drl0"
I20260809 23:16:27.226603 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20260809 23:16:27.231297 22911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:27.232316 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260809 23:16:27.232766 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "0a36468db59a4d328c8c7a060916e4bd"
format_stamp: "Formatted at 2026-08-09 23:16:27 on dist-test-slave-drl0"
I20260809 23:16:27.233151 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:27.255555 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:27.257207 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:27.259227 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:27.262876 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:27.263170 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260809 23:16:27.263468 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:27.263675 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:27.368803 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:35565
I20260809 23:16:27.369431 22997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:35565 every 8 connection(s)
I20260809 23:16:27.389345 22999 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41299
I20260809 23:16:27.389816 22999 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:27.390537 22999 heartbeater.cc:511] Master 127.21.255.62:41299 requested a full tablet report, sending...
I20260809 23:16:27.392530 22584 ts_manager.cc:194] Registered new tserver with Master: 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:27.393118 22524 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018702902s
I20260809 23:16:27.394551 22584 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50332
I20260809 23:16:27.398658 22524 test_util.cc:268] Using random seed: 1834286570
I20260809 23:16:27.469544 22584 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50334:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:16:27.476267 22584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:16:27.604243 22724 tablet_service.cc:1467] Processing CreateTablet for tablet 33d3436d82c047b1b742bbb14c7cf58e (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:16:27.608976 22718 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6164b6930d4ad08cc26b898fe3837b (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:16:27.609083 22717 tablet_service.cc:1467] Processing CreateTablet for tablet d0bfd567ce714705b5383659e80041ee (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:16:27.605738 22723 tablet_service.cc:1467] Processing CreateTablet for tablet 03850407c41840899265114271b654b5 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:16:27.606169 22725 tablet_service.cc:1467] Processing CreateTablet for tablet 981102a03e6d4b0192c3bf28d0631104 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:16:27.619695 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d3436d82c047b1b742bbb14c7cf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.615440 22715 tablet_service.cc:1467] Processing CreateTablet for tablet f2831a340c2641648f0d13fc637782d8 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:16:27.615432 22721 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1aad2b54974018a9e3a15521b84c2c (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:16:27.622797 22718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6164b6930d4ad08cc26b898fe3837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.628012 22714 tablet_service.cc:1467] Processing CreateTablet for tablet 416c561d8e8e4f109692dd3f0063319d (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:16:27.628252 22838 tablet_service.cc:1467] Processing CreateTablet for tablet 981102a03e6d4b0192c3bf28d0631104 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:16:27.618228 22712 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4debc6d6aa47d181f4f8d655f5aced (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:16:27.633486 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981102a03e6d4b0192c3bf28d0631104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.636301 22835 tablet_service.cc:1467] Processing CreateTablet for tablet 03850407c41840899265114271b654b5 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:16:27.640686 22834 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1aad2b54974018a9e3a15521b84c2c (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:16:27.638734 22837 tablet_service.cc:1467] Processing CreateTablet for tablet 33d3436d82c047b1b742bbb14c7cf58e (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:16:27.643180 22835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03850407c41840899265114271b654b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:16:27.647503 22655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:16:27.644917 22833 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6164b6930d4ad08cc26b898fe3837b (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:16:27.645610 22832 tablet_service.cc:1467] Processing CreateTablet for tablet d0bfd567ce714705b5383659e80041ee (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:16:27.655452 22834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1aad2b54974018a9e3a15521b84c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.657580 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bfd567ce714705b5383659e80041ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.653303 22829 tablet_service.cc:1467] Processing CreateTablet for tablet f2831a340c2641648f0d13fc637782d8 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:16:27.660040 22828 tablet_service.cc:1467] Processing CreateTablet for tablet 416c561d8e8e4f109692dd3f0063319d (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:16:27.660583 22837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d3436d82c047b1b742bbb14c7cf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.660298 22712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4debc6d6aa47d181f4f8d655f5aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.662906 22833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6164b6930d4ad08cc26b898fe3837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.665375 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981102a03e6d4b0192c3bf28d0631104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.664762 22827 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4debc6d6aa47d181f4f8d655f5aced (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:16:27.667196 22832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bfd567ce714705b5383659e80041ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.667820 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c561d8e8e4f109692dd3f0063319d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.667853 22827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4debc6d6aa47d181f4f8d655f5aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.670156 22721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1aad2b54974018a9e3a15521b84c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.677588 22723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03850407c41840899265114271b654b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.681120 22829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2831a340c2641648f0d13fc637782d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.684206 22828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c561d8e8e4f109692dd3f0063319d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.686666 22715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2831a340c2641648f0d13fc637782d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.775221 22944 tablet_service.cc:1467] Processing CreateTablet for tablet 3a6164b6930d4ad08cc26b898fe3837b (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:16:27.785761 22945 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1aad2b54974018a9e3a15521b84c2c (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:16:27.788775 22942 tablet_service.cc:1467] Processing CreateTablet for tablet f2831a340c2641648f0d13fc637782d8 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:16:27.792255 22948 tablet_service.cc:1467] Processing CreateTablet for tablet 981102a03e6d4b0192c3bf28d0631104 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:16:27.792441 22947 tablet_service.cc:1467] Processing CreateTablet for tablet 33d3436d82c047b1b742bbb14c7cf58e (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:16:27.795046 22946 tablet_service.cc:1467] Processing CreateTablet for tablet 03850407c41840899265114271b654b5 (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:16:27.782564 22943 tablet_service.cc:1467] Processing CreateTablet for tablet d0bfd567ce714705b5383659e80041ee (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:16:27.805120 22941 tablet_service.cc:1467] Processing CreateTablet for tablet 416c561d8e8e4f109692dd3f0063319d (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:16:27.808781 22944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a6164b6930d4ad08cc26b898fe3837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.815294 22945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1aad2b54974018a9e3a15521b84c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.815459 22939 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4debc6d6aa47d181f4f8d655f5aced (DEFAULT_TABLE table=test-workload [id=9f57c28afed541339d3a20e1951a3f21]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:16:27.821563 22947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33d3436d82c047b1b742bbb14c7cf58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.824959 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981102a03e6d4b0192c3bf28d0631104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.830006 22943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0bfd567ce714705b5383659e80041ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.833707 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03850407c41840899265114271b654b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.836410 22941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 416c561d8e8e4f109692dd3f0063319d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.838310 22942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2831a340c2641648f0d13fc637782d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.846174 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4debc6d6aa47d181f4f8d655f5aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:27.946796 23031 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:27.954869 23032 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.010020 23031 tablet_bootstrap.cc:654] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.015707 23033 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.031134 23032 tablet_bootstrap.cc:654] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.032789 23033 tablet_bootstrap.cc:654] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.034907 23031 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.035521 23031 ts_tablet_manager.cc:1397] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.101s	user 0.016s	sys 0.008s
I20260809 23:16:28.039510 23031 raft_consensus.cc:348] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.040566 23031 raft_consensus.cc:374] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.040992 23031 raft_consensus.cc:729] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.041944 23031 consensus_queue.cc:260] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.044672 23032 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.045135 23032 ts_tablet_manager.cc:1397] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.091s	user 0.008s	sys 0.013s
I20260809 23:16:28.045727 23033 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.046725 23033 ts_tablet_manager.cc:1397] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.032s	user 0.008s	sys 0.010s
I20260809 23:16:28.048709 23032 raft_consensus.cc:348] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.049435 23032 raft_consensus.cc:374] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.049721 23032 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.050427 23032 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.050899 22780 heartbeater.cc:503] Master 127.21.255.62:41299 was elected leader, sending a full tablet report...
I20260809 23:16:28.052680 23031 ts_tablet_manager.cc:1428] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.005s
I20260809 23:16:28.050437 23033 raft_consensus.cc:348] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.053121 23033 raft_consensus.cc:374] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.053402 23033 raft_consensus.cc:729] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.053784 23031 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.054124 23033 consensus_queue.cc:260] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.060359 23031 tablet_bootstrap.cc:654] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.064253 22999 heartbeater.cc:503] Master 127.21.255.62:41299 was elected leader, sending a full tablet report...
I20260809 23:16:28.066090 23033 ts_tablet_manager.cc:1428] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.001s
I20260809 23:16:28.068367 23033 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.075793 23033 tablet_bootstrap.cc:654] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.097194 22898 heartbeater.cc:503] Master 127.21.255.62:41299 was elected leader, sending a full tablet report...
I20260809 23:16:28.099689 23031 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.100786 23031 ts_tablet_manager.cc:1397] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.047s	user 0.026s	sys 0.006s
I20260809 23:16:28.103998 23031 raft_consensus.cc:348] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.104807 23031 raft_consensus.cc:374] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.105129 23031 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.106142 23043 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.106146 23031 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.106714 23043 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.107599 23042 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.108018 23042 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.115973 23042 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.122217 23032 ts_tablet_manager.cc:1428] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.077s	user 0.013s	sys 0.029s
I20260809 23:16:28.123411 23032 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.124663 23031 ts_tablet_manager.cc:1428] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.006s
I20260809 23:16:28.129724 23043 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
W20260809 23:16:28.130349 23000 tablet.cc:2369] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 23:16:28.133666 23031 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.137193 23033 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.137729 23033 ts_tablet_manager.cc:1397] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.070s	user 0.006s	sys 0.006s
I20260809 23:16:28.140133 23032 tablet_bootstrap.cc:654] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.141141 23033 raft_consensus.cc:348] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.142077 23033 raft_consensus.cc:374] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.142426 23033 raft_consensus.cc:729] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.155869 23033 consensus_queue.cc:260] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.160012 23033 ts_tablet_manager.cc:1428] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.005s
I20260809 23:16:28.161193 23033 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.162343 23032 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.162846 23032 ts_tablet_manager.cc:1397] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.010s
I20260809 23:16:28.154877 23031 tablet_bootstrap.cc:654] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.165915 23032 raft_consensus.cc:348] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.175294 23032 raft_consensus.cc:374] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.175637 23032 raft_consensus.cc:729] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.176406 23032 consensus_queue.cc:260] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.186540 23032 ts_tablet_manager.cc:1428] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.023s	user 0.003s	sys 0.003s
I20260809 23:16:28.186757 23031 tablet_bootstrap.cc:492] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.187343 23031 ts_tablet_manager.cc:1397] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.002s
I20260809 23:16:28.192546 23032 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.193720 23033 tablet_bootstrap.cc:654] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.199533 23032 tablet_bootstrap.cc:654] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.200433 23031 raft_consensus.cc:348] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.201217 23031 raft_consensus.cc:374] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.201515 23031 raft_consensus.cc:729] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.202183 23031 consensus_queue.cc:260] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.204887 23043 raft_consensus.cc:482] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.205781 23043 raft_consensus.cc:504] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.206619 23031 ts_tablet_manager.cc:1428] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260809 23:16:28.207741 23031 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.210109 23043 leader_election.cc:288] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.216874 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
W20260809 23:16:28.220577 22688 leader_election.cc:341] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.221115 23033 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.221575 23033 ts_tablet_manager.cc:1397] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.061s	user 0.012s	sys 0.003s
I20260809 23:16:28.224535 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.224562 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.225595 23033 raft_consensus.cc:348] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.226307 22964 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 0.
I20260809 23:16:28.226446 23033 raft_consensus.cc:374] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.226827 23033 raft_consensus.cc:729] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
W20260809 23:16:28.228027 22802 leader_election.cc:341] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.229036 22802 leader_election.cc:302] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.228363 23033 consensus_queue.cc:260] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.233172 23043 raft_consensus.cc:2781] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.233723 23032 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.234477 23032 ts_tablet_manager.cc:1397] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.042s	user 0.012s	sys 0.004s
I20260809 23:16:28.234516 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.235348 22736 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 0.
I20260809 23:16:28.233925 23043 raft_consensus.cc:482] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.236707 23043 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.238778 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.240682 23032 raft_consensus.cc:348] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.241464 23032 raft_consensus.cc:374] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.241878 23032 raft_consensus.cc:729] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
W20260809 23:16:28.241446 22802 leader_election.cc:341] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.246088 23044 raft_consensus.cc:482] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.246565 23044 raft_consensus.cc:504] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.243333 22802 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:28.242542 23032 consensus_queue.cc:260] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.252094 23033 ts_tablet_manager.cc:1428] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.001s
I20260809 23:16:28.253072 23033 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.253592 23043 raft_consensus.cc:504] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.257967 23044 leader_election.cc:288] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.261945 23043 leader_election.cc:288] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.263525 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.264710 23044 raft_consensus.cc:482] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.269757 23043 raft_consensus.cc:2726] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:28.270131 23044 raft_consensus.cc:504] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.279034 23031 tablet_bootstrap.cc:654] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.280243 23044 leader_election.cc:288] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
W20260809 23:16:28.287221 22686 leader_election.cc:341] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.291576 22686 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58; no voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf
W20260809 23:16:28.296432 22802 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:43667, user_credentials={real_user=slave}} blocked reactor thread for 55005.6us
I20260809 23:16:28.296921 23049 raft_consensus.cc:2726] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:28.304072 23033 tablet_bootstrap.cc:654] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.304118 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.304800 22737 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.312119 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:28.312712 22964 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.319705 23032 ts_tablet_manager.cc:1428] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.085s	user 0.011s	sys 0.006s
I20260809 23:16:28.321820 22737 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 1.
I20260809 23:16:28.326179 22802 leader_election.cc:302] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.327497 23073 raft_consensus.cc:2781] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.328464 23073 raft_consensus.cc:686] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:28.331080 23032 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.334724 23073 consensus_queue.cc:237] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.337168 22964 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 1.
I20260809 23:16:28.343863 23031 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.343889 23032 tablet_bootstrap.cc:654] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.344556 23031 ts_tablet_manager.cc:1397] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.137s	user 0.020s	sys 0.013s
I20260809 23:16:28.351213 23031 raft_consensus.cc:348] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.351964 23031 raft_consensus.cc:374] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.352232 23031 raft_consensus.cc:729] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.353150 23033 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.352892 23031 consensus_queue.cc:260] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.353611 23033 ts_tablet_manager.cc:1397] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.004s
I20260809 23:16:28.357182 23033 raft_consensus.cc:348] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.358014 23033 raft_consensus.cc:374] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.358299 23033 raft_consensus.cc:729] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.359134 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.359843 22736 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.361215 22914 leader_election.cc:302] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:28.362075 23044 raft_consensus.cc:2781] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.362010 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.362532 23044 raft_consensus.cc:482] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.362887 23044 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:16:28.364244 22913 leader_election.cc:341] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.364859 23031 ts_tablet_manager.cc:1428] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260809 23:16:28.365088 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.365813 23031 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
W20260809 23:16:28.366349 22913 leader_election.cc:341] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.358990 23033 consensus_queue.cc:260] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.359236 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.368287 22737 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.369340 22914 leader_election.cc:302] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:28.372800 23044 raft_consensus.cc:504] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.375253 23031 tablet_bootstrap.cc:654] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.376200 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
W20260809 23:16:28.377914 22913 leader_election.cc:341] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.383514 23033 ts_tablet_manager.cc:1428] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.030s	user 0.011s	sys 0.000s
I20260809 23:16:28.374755 23044 leader_election.cc:288] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.384614 23044 raft_consensus.cc:2781] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.385020 23044 raft_consensus.cc:482] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.385386 23044 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.389813 22577 catalog_manager.cc:5495] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 1 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:28.396344 23031 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.396894 23031 ts_tablet_manager.cc:1397] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.009s
I20260809 23:16:28.397611 23033 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.397799 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.399360 22737 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.399361 23032 tablet_bootstrap.cc:492] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.399564 23031 raft_consensus.cc:348] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.400154 23032 ts_tablet_manager.cc:1397] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.072s	user 0.010s	sys 0.006s
I20260809 23:16:28.400372 23031 raft_consensus.cc:374] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.400815 23031 raft_consensus.cc:729] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.401505 23031 consensus_queue.cc:260] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.403362 23032 raft_consensus.cc:348] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.404037 23031 ts_tablet_manager.cc:1428] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260809 23:16:28.404224 23032 raft_consensus.cc:374] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.404150 23044 raft_consensus.cc:504] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.404702 23032 raft_consensus.cc:729] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.405014 23031 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.405655 23032 consensus_queue.cc:260] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.408161 23032 ts_tablet_manager.cc:1428] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:16:28.408015 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.409139 22737 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.409391 22736 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.410112 23032 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.414199 23031 tablet_bootstrap.cc:654] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.415797 22914 leader_election.cc:302] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:28.417480 23032 tablet_bootstrap.cc:654] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.420257 23088 raft_consensus.cc:2781] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.420975 23088 raft_consensus.cc:686] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:28.422063 23088 consensus_queue.cc:237] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.429317 23031 tablet_bootstrap.cc:492] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.429956 23031 ts_tablet_manager.cc:1397] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.012s
I20260809 23:16:28.433015 23033 tablet_bootstrap.cc:654] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.432958 23031 raft_consensus.cc:348] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.433883 23031 raft_consensus.cc:374] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.434186 23031 raft_consensus.cc:729] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.435148 23032 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.436563 23044 leader_election.cc:288] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.434917 22736 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.437707 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:28.434895 23031 consensus_queue.cc:260] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.438709 22914 leader_election.cc:302] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:28.435684 23032 ts_tablet_manager.cc:1397] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.002s
I20260809 23:16:28.441337 23031 ts_tablet_manager.cc:1428] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.003s
I20260809 23:16:28.438719 22854 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.442322 23031 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.442752 23044 raft_consensus.cc:2781] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.447712 23044 raft_consensus.cc:686] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:28.448792 23044 consensus_queue.cc:237] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.456343 22854 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.471596 23032 raft_consensus.cc:348] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.472807 23032 raft_consensus.cc:374] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.473203 23032 raft_consensus.cc:729] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.479267 23031 tablet_bootstrap.cc:654] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.474115 23032 consensus_queue.cc:260] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.499114 23033 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.499616 23033 ts_tablet_manager.cc:1397] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.102s	user 0.011s	sys 0.004s
I20260809 23:16:28.500615 23032 ts_tablet_manager.cc:1428] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.058s	user 0.003s	sys 0.006s
I20260809 23:16:28.500073 22583 catalog_manager.cc:5495] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 1 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:28.501482 23032 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.502626 23033 raft_consensus.cc:348] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.503582 23033 raft_consensus.cc:374] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.503921 23033 raft_consensus.cc:729] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.504587 23033 consensus_queue.cc:260] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.507203 23033 ts_tablet_manager.cc:1428] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260809 23:16:28.508111 23033 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.510035 23032 tablet_bootstrap.cc:654] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.517871 23031 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.518121 23033 tablet_bootstrap.cc:654] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.518317 23031 ts_tablet_manager.cc:1397] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.076s	user 0.009s	sys 0.005s
I20260809 23:16:28.521023 23031 raft_consensus.cc:348] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.521716 23031 raft_consensus.cc:374] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.522053 23031 raft_consensus.cc:729] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.522689 23031 consensus_queue.cc:260] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.525935 23031 ts_tablet_manager.cc:1428] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260809 23:16:28.526882 23031 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
I20260809 23:16:28.531613 23033 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.532301 23033 ts_tablet_manager.cc:1397] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.003s
I20260809 23:16:28.536903 23073 raft_consensus.cc:482] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.537350 23073 raft_consensus.cc:504] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.539322 23073 leader_election.cc:288] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.540946 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.541613 23033 raft_consensus.cc:348] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
W20260809 23:16:28.542259 22802 leader_election.cc:341] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.543004 23033 raft_consensus.cc:374] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.543802 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.544090 23033 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.524089 22577 catalog_manager.cc:5495] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 1 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:28.545390 23033 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
W20260809 23:16:28.546285 22802 leader_election.cc:341] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.546732 22802 leader_election.cc:302] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:28.548098 23033 ts_tablet_manager.cc:1428] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260809 23:16:28.548388 23073 raft_consensus.cc:2726] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:28.552686 23031 tablet_bootstrap.cc:654] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.553071 23032 tablet_bootstrap.cc:492] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.553645 23032 ts_tablet_manager.cc:1397] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.052s	user 0.014s	sys 0.009s
I20260809 23:16:28.556322 23032 raft_consensus.cc:348] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.557005 23032 raft_consensus.cc:374] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.557273 23032 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.557863 23032 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.560078 23032 ts_tablet_manager.cc:1428] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:16:28.561406 23033 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.567767 23033 tablet_bootstrap.cc:654] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.571017 23032 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.575986 23031 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.576535 23031 ts_tablet_manager.cc:1397] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.050s	user 0.013s	sys 0.011s
I20260809 23:16:28.577595 23044 raft_consensus.cc:482] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.578161 23044 raft_consensus.cc:504] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.580476 23032 tablet_bootstrap.cc:654] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.580542 23044 leader_election.cc:288] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.581461 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.582261 22736 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.584448 22914 leader_election.cc:302] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:28.584615 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.585244 23044 raft_consensus.cc:2781] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.585582 23033 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.585244 22854 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.585917 23044 raft_consensus.cc:482] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.586408 23044 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.586081 23033 ts_tablet_manager.cc:1397] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.000s
I20260809 23:16:28.588352 23031 raft_consensus.cc:348] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.589123 23031 raft_consensus.cc:374] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.589413 23031 raft_consensus.cc:729] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.590129 23031 consensus_queue.cc:260] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.593061 23031 ts_tablet_manager.cc:1428] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260809 23:16:28.597791 23033 raft_consensus.cc:348] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.598603 23033 raft_consensus.cc:374] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.598922 23033 raft_consensus.cc:729] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.599733 23033 consensus_queue.cc:260] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.600696 23032 tablet_bootstrap.cc:492] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.601255 23032 ts_tablet_manager.cc:1397] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.040s	user 0.014s	sys 0.000s
I20260809 23:16:28.601959 23033 ts_tablet_manager.cc:1428] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260809 23:16:28.602919 23033 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.604089 23032 raft_consensus.cc:348] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.604847 23032 raft_consensus.cc:374] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.605149 23032 raft_consensus.cc:729] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.605805 23032 consensus_queue.cc:260] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.607391 23044 raft_consensus.cc:504] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.610023 23033 tablet_bootstrap.cc:654] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.610059 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.610693 22736 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.617484 22736 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.618680 22914 leader_election.cc:302] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:28.619504 23088 raft_consensus.cc:2781] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.620028 23044 leader_election.cc:288] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.620194 23088 raft_consensus.cc:686] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:28.621400 23088 consensus_queue.cc:237] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.622596 23032 ts_tablet_manager.cc:1428] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.005s
I20260809 23:16:28.623713 23032 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.625397 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:28.626168 22854 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.631824 23032 tablet_bootstrap.cc:654] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.644521 23032 tablet_bootstrap.cc:492] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.645074 23032 ts_tablet_manager.cc:1397] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.004s
I20260809 23:16:28.647791 23032 raft_consensus.cc:348] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.648646 23032 raft_consensus.cc:374] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.648973 23032 raft_consensus.cc:729] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.649709 23032 consensus_queue.cc:260] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.650306 23031 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: Bootstrap starting.
W20260809 23:16:28.651266 22655 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:16:28.652341 23032 ts_tablet_manager.cc:1428] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:16:28.653144 23032 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: Bootstrap starting.
I20260809 23:16:28.656498 22854 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.658411 23032 tablet_bootstrap.cc:654] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.662976 23049 raft_consensus.cc:482] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.663558 23049 raft_consensus.cc:504] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.666306 23049 leader_election.cc:288] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.668174 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.669463 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.670152 22853 raft_consensus.cc:2454] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 0.
W20260809 23:16:28.671435 22688 leader_election.cc:341] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.672508 22686 leader_election.cc:302] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:28.673839 23049 raft_consensus.cc:2781] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.674222 23049 raft_consensus.cc:482] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.674615 23049 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.678778 22582 catalog_manager.cc:5495] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 1 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:28.680994 23049 raft_consensus.cc:504] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.682307 23073 raft_consensus.cc:482] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.687389 23073 raft_consensus.cc:504] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.689213 23073 leader_election.cc:288] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.690698 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.691380 22736 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 0.
I20260809 23:16:28.691697 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:28.692274 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.692890 22964 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 0.
W20260809 23:16:28.693013 22688 leader_election.cc:341] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:28.686342 23032 tablet_bootstrap.cc:492] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: No bootstrap required, opened a new log
I20260809 23:16:28.693758 23032 ts_tablet_manager.cc:1397] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent bootstrapping tablet: real 0.041s	user 0.015s	sys 0.001s
I20260809 23:16:28.695086 22802 leader_election.cc:302] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.696346 23073 raft_consensus.cc:2781] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.696235 23032 raft_consensus.cc:348] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.696744 23073 raft_consensus.cc:482] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.696977 23032 raft_consensus.cc:374] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.697212 23073 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.697304 23032 raft_consensus.cc:729] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.698035 23032 consensus_queue.cc:260] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.698313 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:28.698951 23049 leader_election.cc:288] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.699047 22854 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.703409 23088 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.703864 23088 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.704312 23073 raft_consensus.cc:504] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.707830 23088 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.709554 23032 ts_tablet_manager.cc:1428] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260809 23:16:28.709748 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.710896 22853 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.711786 23031 tablet_bootstrap.cc:654] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.713268 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.713243 22913 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.715689 22736 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:28.718221 23088 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.718665 23088 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.718556 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.719209 22737 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.719069 23088 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.724095 22854 raft_consensus.cc:2454] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 1.
I20260809 23:16:28.726066 22686 leader_election.cc:302] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:28.727185 23049 raft_consensus.cc:2781] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.733559 23033 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.734149 23033 ts_tablet_manager.cc:1397] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.131s	user 0.039s	sys 0.048s
I20260809 23:16:28.736671 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:28.737278 22964 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.738010 23073 leader_election.cc:288] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.737919 23033 raft_consensus.cc:348] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.739409 23088 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.740751 22737 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 1.
I20260809 23:16:28.741874 23088 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.742026 22802 leader_election.cc:302] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.742965 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:28.743556 22854 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.746491 22964 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 1.
I20260809 23:16:28.748100 23073 raft_consensus.cc:2781] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.750186 22854 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.750294 23073 raft_consensus.cc:686] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:28.753722 22913 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.754262 23073 consensus_queue.cc:237] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.754921 23031 tablet_bootstrap.cc:492] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: No bootstrap required, opened a new log
I20260809 23:16:28.755244 23088 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.755591 23031 ts_tablet_manager.cc:1397] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: Time spent bootstrapping tablet: real 0.105s	user 0.028s	sys 0.019s
I20260809 23:16:28.755980 23088 raft_consensus.cc:686] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:28.756572 23049 raft_consensus.cc:686] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:28.756824 23088 consensus_queue.cc:237] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.757580 23049 consensus_queue.cc:237] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.751423 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:28.738915 23033 raft_consensus.cc:374] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.758611 23031 raft_consensus.cc:348] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.759153 23033 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.759181 22737 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.759948 23033 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.762266 23033 ts_tablet_manager.cc:1428] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260809 23:16:28.763152 23033 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: Bootstrap starting.
I20260809 23:16:28.769153 23033 tablet_bootstrap.cc:654] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:28.769454 23031 raft_consensus.cc:374] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.772145 23031 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.772876 23031 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.775131 23031 ts_tablet_manager.cc:1428] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260809 23:16:28.776405 22584 catalog_manager.cc:5495] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 1 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:28.780284 22737 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:28.784322 22582 catalog_manager.cc:5495] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 0 to 1, leader changed from <none> to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:28.787079 22577 catalog_manager.cc:5495] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 1 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:28.788709 23049 raft_consensus.cc:482] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.789346 23049 raft_consensus.cc:504] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.791288 23049 leader_election.cc:288] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.792402 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.793016 22854 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 0.
I20260809 23:16:28.793910 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.794137 22686 leader_election.cc:302] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.794518 22964 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 0.
I20260809 23:16:28.796192 23033 tablet_bootstrap.cc:492] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: No bootstrap required, opened a new log
I20260809 23:16:28.796707 23033 ts_tablet_manager.cc:1397] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: Time spent bootstrapping tablet: real 0.034s	user 0.018s	sys 0.004s
I20260809 23:16:28.799088 23033 raft_consensus.cc:348] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.799712 23033 raft_consensus.cc:374] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:28.799952 23033 raft_consensus.cc:729] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Initialized, Role: FOLLOWER
I20260809 23:16:28.800625 23033 consensus_queue.cc:260] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.801139 23049 raft_consensus.cc:2781] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:28.802034 23049 raft_consensus.cc:482] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:28.802412 23049 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.802798 23033 ts_tablet_manager.cc:1428] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:16:28.807530 23073 raft_consensus.cc:482] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.808060 23073 raft_consensus.cc:504] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.810611 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.811510 22737 raft_consensus.cc:2379] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6d3e3e4c8424c91adee4f5c26e134bf in current term 1: Already voted for candidate 0a36468db59a4d328c8c7a060916e4bd in this term.
I20260809 23:16:28.812506 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd" is_pre_election: true
I20260809 23:16:28.813678 23049 raft_consensus.cc:504] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.814339 22802 leader_election.cc:302] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:28.815167 23069 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.815508 23073 leader_election.cc:288] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.816913 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:28.817508 22854 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.818459 23049 leader_election.cc:288] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:28.820039 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:28.820592 22964 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.823647 22854 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 1.
I20260809 23:16:28.824800 22686 leader_election.cc:302] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:28.825736 23049 raft_consensus.cc:2781] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:28.825860 23069 raft_consensus.cc:2726] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:28.827358 23049 raft_consensus.cc:686] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:28.827850 22964 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 1.
I20260809 23:16:28.828208 23049 consensus_queue.cc:237] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.832125 23069 consensus_queue.cc:1035] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:16:28.838832 22577 catalog_manager.cc:5495] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 0 to 1, leader changed from <none> to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:28.859064 23128 consensus_queue.cc:1035] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:16:28.911039 23088 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:28.922052 23044 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:28.955457 23044 raft_consensus.cc:482] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:28.955927 23044 raft_consensus.cc:504] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:28.957841 23044 leader_election.cc:288] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:28.959026 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:28.959147 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:28.959846 22854 raft_consensus.cc:2379] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0a36468db59a4d328c8c7a060916e4bd in current term 1: Already voted for candidate e609d1a6e91147bb978f92fcd441ef58 in this term.
I20260809 23:16:28.960860 22913 leader_election.cc:302] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd; no voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:28.961540 23044 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:28.968487 23044 raft_consensus.cc:2726] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:29.001261 23044 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.013537 23088 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.034500 23088 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.046564 23044 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:16:29.065056 23044 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:29.065482 23044 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.067188 23044 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:29.068049 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58" is_pre_election: true
I20260809 23:16:29.068223 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" is_pre_election: true
I20260809 23:16:29.068579 22737 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:29.068895 22854 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 0.
I20260809 23:16:29.069540 22914 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:29.070353 23044 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:29.070719 23044 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:29.071017 23044 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:29.075724 23044 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.077553 23044 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:29.077961 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:29.078158 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:29.078544 22737 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:29.078673 22854 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:29.082722 22854 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:29.083685 22913 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.084915 23044 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:29.085546 22737 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 1.
I20260809 23:16:29.085747 23044 raft_consensus.cc:686] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.087440 23044 consensus_queue.cc:237] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.096824 22577 catalog_manager.cc:5495] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 1 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:29.139339 23073 consensus_queue.cc:1035] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.152022 23073 consensus_queue.cc:1035] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.202183 23044 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.212798 23086 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.244858 23049 consensus_queue.cc:1035] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.258615 23049 consensus_queue.cc:1035] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.307467 23154 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.319734 23049 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.360005 22964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:29.360803 22964 raft_consensus.cc:595] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:29.366443 22964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:29.367048 22964 raft_consensus.cc:595] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:29.372421 22964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03850407c41840899265114271b654b5"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:29.372970 22964 raft_consensus.cc:595] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:29.395767 22737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:48114
I20260809 23:16:29.396399 22737 raft_consensus.cc:595] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:29.401702 23086 raft_consensus.cc:979] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.402274 23044 raft_consensus.cc:1067] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.403679 22964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:29.404068 22737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48110
I20260809 23:16:29.404304 22964 raft_consensus.cc:595] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:29.404697 22737 raft_consensus.cc:482] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.405136 22737 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.409677 22964 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:29.410262 22964 raft_consensus.cc:595] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:29.410769 22737 raft_consensus.cc:504] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.412678 22737 leader_election.cc:288] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.413945 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:29.414372 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.414919 22964 raft_consensus.cc:3032] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.414505 22854 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.415268 22964 raft_consensus.cc:729] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.416069 22964 consensus_queue.cc:260] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.417564 22964 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.421746 22854 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.423372 22686 leader_election.cc:302] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.424250 23049 raft_consensus.cc:2781] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.424830 23049 raft_consensus.cc:686] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:29.425809 23049 consensus_queue.cc:237] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.431282 22964 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.434652 22582 catalog_manager.cc:5495] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 1 to 2, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:29.440918 22854 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:43300
I20260809 23:16:29.441435 22854 raft_consensus.cc:595] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Received request to transfer leadership to TS 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:29.495752 23044 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.497974 23086 raft_consensus.cc:979] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.498505 23086 raft_consensus.cc:1067] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.500109 22854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43290
I20260809 23:16:29.500795 22854 raft_consensus.cc:482] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.501183 22854 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.511888 22854 raft_consensus.cc:504] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.513474 23044 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:29.515831 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:29.516407 22736 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.530956 22736 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.532975 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.533614 22964 raft_consensus.cc:3032] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.533113 22854 leader_election.cc:288] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.533934 22964 raft_consensus.cc:729] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.534698 22964 consensus_queue.cc:260] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.535765 22964 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.532968 22802 leader_election.cc:302] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.539891 23130 raft_consensus.cc:2781] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.540302 23130 raft_consensus.cc:686] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:29.541292 23130 consensus_queue.cc:237] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.548395 22964 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.549458 23086 raft_consensus.cc:979] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.550045 23169 raft_consensus.cc:1067] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.551673 22853 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43290
I20260809 23:16:29.551606 22577 catalog_manager.cc:5495] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 1 to 2, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 2 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:29.552238 22853 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.552620 22853 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.559010 22853 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.560848 22853 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.562484 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:29.562480 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.563103 22736 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.563191 22964 raft_consensus.cc:3032] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.563558 22964 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.564352 22964 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.565526 22964 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.567826 22736 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.568763 22802 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.569427 23130 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.569832 23130 raft_consensus.cc:686] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:29.570624 23130 consensus_queue.cc:237] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.571362 22964 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.576730 22577 catalog_manager.cc:5495] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 1 to 2, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 2 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:29.660135 23169 raft_consensus.cc:979] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.660837 23086 raft_consensus.cc:1067] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.663128 22736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03850407c41840899265114271b654b5"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48110
I20260809 23:16:29.663854 22736 raft_consensus.cc:482] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.664254 22736 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.671324 22736 raft_consensus.cc:504] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.673965 22736 leader_election.cc:288] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.674983 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:29.675712 22853 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.675549 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.676232 22964 raft_consensus.cc:3032] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.676577 22964 raft_consensus.cc:729] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.677367 22964 consensus_queue.cc:260] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.678772 22964 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.680543 22853 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.681481 22686 leader_election.cc:302] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.682147 23049 raft_consensus.cc:2781] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.682559 23049 raft_consensus.cc:686] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:29.683379 23049 consensus_queue.cc:237] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.684852 22964 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.691686 22577 catalog_manager.cc:5495] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 1 to 2, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:29.772727 23073 raft_consensus.cc:979] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:29.773241 23130 raft_consensus.cc:1067] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Signalling peer 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:29.774683 22964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:53770
I20260809 23:16:29.775308 22964 raft_consensus.cc:482] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.775665 22964 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.782312 22964 raft_consensus.cc:504] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.784365 22964 leader_election.cc:288] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 2 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:29.786212 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:29.787205 22737 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.787252 22853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4debc6d6aa47d181f4f8d655f5aced" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:29.788561 23044 raft_consensus.cc:979] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.789297 23044 raft_consensus.cc:1067] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:29.789235 22853 raft_consensus.cc:3032] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.790079 22853 raft_consensus.cc:729] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:29.791742 22736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48110
I20260809 23:16:29.792709 22736 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.793346 22736 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.792469 22853 consensus_queue.cc:260] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.794564 22853 raft_consensus.cc:3037] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.798635 22737 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 2.
I20260809 23:16:29.800288 22914 leader_election.cc:302] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:29.801153 23044 raft_consensus.cc:2781] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.801633 23044 raft_consensus.cc:686] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 2 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.802304 22736 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.802500 23044 consensus_queue.cc:237] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.804730 22853 raft_consensus.cc:2454] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 2.
I20260809 23:16:29.805351 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:29.805904 22736 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.806147 22854 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.808943 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.809520 22964 raft_consensus.cc:3032] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.809811 22964 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:29.810503 22964 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.811704 22964 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.812531 22854 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.813723 22686 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.814635 23049 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.815096 23049 raft_consensus.cc:686] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:29.815886 23049 consensus_queue.cc:237] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.817770 22964 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:29.817364 22584 catalog_manager.cc:5495] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 1 to 2, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 2 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:29.825600 22577 catalog_manager.cc:5495] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 1 to 2, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:29.847141 22964 raft_consensus.cc:1261] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:29.848840 23049 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:29.856544 22854 raft_consensus.cc:1261] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:29.862044 23049 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:29.944031 22736 raft_consensus.cc:1261] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:29.945264 23130 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:16:29.951150 23049 raft_consensus.cc:979] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.951589 23049 raft_consensus.cc:1067] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:29.954897 22853 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43282
I20260809 23:16:29.955458 22853 raft_consensus.cc:482] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:29.955366 22964 raft_consensus.cc:1261] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:29.955914 22853 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.957515 23128 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:29.961700 22853 raft_consensus.cc:504] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.963382 22853 leader_election.cc:288] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:29.964761 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:29.965670 22965 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.971967 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:29.972502 22736 raft_consensus.cc:3032] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:29.972759 22736 raft_consensus.cc:729] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:29.975339 22736 consensus_queue.cc:260] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.976271 22736 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:29.976298 22965 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.977355 22802 leader_election.cc:302] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:29.978005 23073 raft_consensus.cc:2781] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:29.978430 23073 raft_consensus.cc:686] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:29.979239 23073 consensus_queue.cc:237] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:29.981160 22736 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:29.986833 22577 catalog_manager.cc:5495] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 1 to 2, leader changed from e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 2 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:30.046592 22965 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.047853 23073 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.055207 22736 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.056664 23172 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.198467 22736 raft_consensus.cc:1261] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.199338 22965 raft_consensus.cc:1261] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.199919 23044 consensus_queue.cc:1035] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.200698 23049 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.208457 22853 raft_consensus.cc:1261] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.212651 23158 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.218205 22853 raft_consensus.cc:1261] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.219722 23044 consensus_queue.cc:1035] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.305919 22853 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.307717 23171 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.316607 22965 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.319113 23171 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.408854 22736 raft_consensus.cc:1261] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.410486 23128 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:30.418887 22965 raft_consensus.cc:1261] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:30.420564 23128 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:32.475592 22524 auto_leader_rebalancer-test.cc:269] The leader distribution is 
I20260809 23:16:32.476574 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:32.504405 22736 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
mode: GRACEFUL
new_leader_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:48126
I20260809 23:16:32.505151 22736 raft_consensus.cc:595] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Received request to transfer leadership to TS 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:32.519309 22854 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:32.519966 22854 raft_consensus.cc:595] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Received request to transfer leadership to TS 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:32.521322 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:16:32.521914 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:32.533528 23190 raft_consensus.cc:979] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:32.533980 23128 raft_consensus.cc:1067] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Signalling peer 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:32.535660 22965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:53770
I20260809 23:16:32.536083 22965 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:32.536345 22965 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.542920 22965 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.544911 22965 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:32.545674 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:32.545933 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:32.546438 22737 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.546471 22854 raft_consensus.cc:3032] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:32.546916 22854 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:32.547577 22854 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.548462 22854 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.551313 22737 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:32.552332 22914 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:32.553705 22854 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:32.553905 23044 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:32.555281 23044 raft_consensus.cc:686] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:32.556149 23044 consensus_queue.cc:237] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.564873 22584 catalog_manager.cc:5495] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 2 to 3, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 3 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:32.688141 23171 raft_consensus.cc:979] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: : Instructing follower 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:32.688585 23207 raft_consensus.cc:1067] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Signalling peer 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:32.690070 22965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:53760
I20260809 23:16:32.690647 22965 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:32.690955 22965 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.696142 22965 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.698130 22965 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:32.698614 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:32.699285 22737 raft_consensus.cc:3032] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:32.699358 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:32.699667 22737 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:32.700153 22854 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.700573 22737 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.702102 22737 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:32.705515 22854 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:32.706717 22913 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:32.707562 23044 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:32.708036 23044 raft_consensus.cc:686] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:32.708832 23044 consensus_queue.cc:237] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:32.709853 22737 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:32.715803 22584 catalog_manager.cc:5495] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 2 to 3, leader changed from e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1) to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 3 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:32.987864 22737 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:32.989675 23044 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:33.007341 22854 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:33.008675 23044 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:33.109937 22737 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:33.111460 23044 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:33.121170 22854 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:33.122611 23213 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:33.522903 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:33.529378 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:33.529767 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:34.530592 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:34.536851 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:34.537384 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:35.538616 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:35.545279 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:35.545720 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:36.546530 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:36.551200 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:36.551620 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:37.552631 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:37.559808 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:37.560212 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:38.561585 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:38.566391 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:38.566783 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:39.567638 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:39.572292 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:39.572664 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:40.573405 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:40.581214 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:40.581635 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:41.582350 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:41.586831 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:41.587226 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:43.502403 22524 auto_leader_rebalancer-test.cc:288] The leader distribution is 
I20260809 23:16:43.507153 22735 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:48114
I20260809 23:16:43.507856 22735 raft_consensus.cc:595] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:43.512871 22963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:43.513368 22963 raft_consensus.cc:595] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:43.517995 22735 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03850407c41840899265114271b654b5"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:48114
I20260809 23:16:43.518482 22735 raft_consensus.cc:595] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:43.523718 22735 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:48114
I20260809 23:16:43.524214 22735 raft_consensus.cc:595] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
I20260809 23:16:43.535459 22963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
mode: GRACEFUL
new_leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:53786
I20260809 23:16:43.535970 22963 raft_consensus.cc:595] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Received request to transfer leadership to TS f6d3e3e4c8424c91adee4f5c26e134bf
W20260809 23:16:43.757097 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080012f0a0 after lost signal to thread 22536
W20260809 23:16:43.758209 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080012f780 after lost signal to thread 22636
W20260809 23:16:43.759296 22885 debug-util.cc:398] Leaking SignalData structure 0x7b08001bc380 after lost signal to thread 22779
W20260809 23:16:43.760174 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080003e340 after lost signal to thread 22888
W20260809 23:16:43.761021 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080019f680 after lost signal to thread 22997
I20260809 23:16:43.844897 23253 raft_consensus.cc:979] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.845712 23236 raft_consensus.cc:1067] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.847455 22853 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43282
I20260809 23:16:43.848064 22853 raft_consensus.cc:482] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:43.848479 22853 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.854694 22853 raft_consensus.cc:504] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.861151 22853 leader_election.cc:288] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:43.861968 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:43.862629 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981102a03e6d4b0192c3bf28d0631104" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:43.862777 22735 raft_consensus.cc:3032] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:43.863263 22963 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.863297 22735 raft_consensus.cc:729] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:43.864303 22735 consensus_queue.cc:260] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.865528 22735 raft_consensus.cc:3037] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.870536 22735 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 3.
I20260809 23:16:43.870720 22963 raft_consensus.cc:2454] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 3.
I20260809 23:16:43.871822 22802 leader_election.cc:302] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:43.872725 23238 raft_consensus.cc:2781] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:43.873203 23238 raft_consensus.cc:686] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:43.874287 23238 consensus_queue.cc:237] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.889212 22584 catalog_manager.cc:5495] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 2 to 3, leader changed from e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 3 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:43.906920 23264 raft_consensus.cc:979] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.907608 23243 raft_consensus.cc:1067] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.909204 22853 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43290
I20260809 23:16:43.909776 22853 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:43.910130 22853 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.914469 22853 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.915998 22853 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 4 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:43.916785 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:43.917038 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:43.917433 22735 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.917611 22963 raft_consensus.cc:3032] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:16:43.917990 22963 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:43.918740 22963 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.919917 22963 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.926576 22963 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 4.
I20260809 23:16:43.926573 22735 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 4.
I20260809 23:16:43.928476 22802 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:43.929323 23238 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:16:43.929858 23238 raft_consensus.cc:686] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:43.930862 23238 consensus_queue.cc:237] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.936937 23243 raft_consensus.cc:979] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.937417 23243 raft_consensus.cc:1067] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.941051 22854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43290
I20260809 23:16:43.941850 22854 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:43.942229 22854 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.942610 23236 raft_consensus.cc:979] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.943101 23236 raft_consensus.cc:1067] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:43.945122 22850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03850407c41840899265114271b654b5"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43282
I20260809 23:16:43.945712 22850 raft_consensus.cc:482] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:43.946081 22850 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.945119 22584 catalog_manager.cc:5495] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 3 to 4, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 4 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:43.951009 22854 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.952939 22854 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 4 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:43.954310 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:43.954902 22963 raft_consensus.cc:3032] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:16:43.955250 22963 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:43.956074 22963 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.956745 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:43.957262 22963 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.957652 22735 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:16:43.962402 22850 raft_consensus.cc:504] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.964843 22850 leader_election.cc:288] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:43.965389 22735 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 4.
I20260809 23:16:43.965982 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:43.966567 22737 raft_consensus.cc:3032] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:43.966821 22737 raft_consensus.cc:729] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:43.967724 22802 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:43.967577 22737 consensus_queue.cc:260] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.968775 22737 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.969048 22963 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 4.
I20260809 23:16:43.969542 23254 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:16:43.966648 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03850407c41840899265114271b654b5" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:43.970002 23254 raft_consensus.cc:686] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:43.970117 22965 raft_consensus.cc:3037] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:43.970912 23254 consensus_queue.cc:237] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.975700 22737 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 3.
I20260809 23:16:43.976573 22802 leader_election.cc:302] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:43.977571 23238 raft_consensus.cc:2781] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:43.977818 22965 raft_consensus.cc:2454] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 3.
I20260809 23:16:43.981904 23238 raft_consensus.cc:686] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:43.983386 23238 consensus_queue.cc:237] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:43.983639 22584 catalog_manager.cc:5495] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 3 to 4, leader changed from 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 4 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:43.996862 22584 catalog_manager.cc:5495] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 2 to 3, leader changed from e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 3 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:44.065841 23236 raft_consensus.cc:979] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: : Instructing follower f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:44.066322 23253 raft_consensus.cc:1067] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Signalling peer f6d3e3e4c8424c91adee4f5c26e134bf to start an election
I20260809 23:16:44.067831 22850 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
 from {username='slave'} at 127.0.0.1:43282
I20260809 23:16:44.068301 22850 raft_consensus.cc:482] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:44.068600 22850 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:44.073201 22850 raft_consensus.cc:504] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:44.075351 22850 leader_election.cc:288] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:44.076116 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:44.076655 22737 raft_consensus.cc:3032] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:44.076946 22737 raft_consensus.cc:729] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:44.076664 22963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1aad2b54974018a9e3a15521b84c2c" candidate_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:44.077720 22963 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:44.077842 22737 consensus_queue.cc:260] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:44.079207 22737 raft_consensus.cc:3037] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:44.084386 22737 raft_consensus.cc:2454] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:44.085412 22802 leader_election.cc:302] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:44.086203 23238 raft_consensus.cc:2781] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:44.086648 23238 raft_consensus.cc:686] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Leader. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:44.086668 22963 raft_consensus.cc:2454] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3e3e4c8424c91adee4f5c26e134bf in term 2.
I20260809 23:16:44.087618 23238 consensus_queue.cc:237] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:44.096727 22584 catalog_manager.cc:5495] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf reported cstate change: term changed from 1 to 2, leader changed from e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1) to f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2). New cstate: current_term: 2 leader_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:44.322785 22963 raft_consensus.cc:1261] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:44.324061 23238 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:44.335469 22737 raft_consensus.cc:1261] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:44.339002 23267 consensus_queue.cc:1035] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:16:44.386035 22963 raft_consensus.cc:1261] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:44.387591 23267 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:44.398396 22737 raft_consensus.cc:1261] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:44.399922 23267 consensus_queue.cc:1035] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:44.433887 22963 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:16:44.435755 23267 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:16:44.450708 22737 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:16:44.452513 23267 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:16:44.536849 22963 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:16:44.538367 23267 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:16:44.551913 22737 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:16:44.553079 23238 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:16:44.563149 22735 raft_consensus.cc:1261] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:44.564399 23238 consensus_queue.cc:1035] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:44.573603 22963 raft_consensus.cc:1261] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer f6d3e3e4c8424c91adee4f5c26e134bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:44.579200 23238 consensus_queue.cc:1035] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:16:46.755254 22524 auto_leader_rebalancer-test.cc:303] The leader distribution is 
I20260809 23:16:46.756040 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:46.762795 22850 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:46.763854 22850 raft_consensus.cc:595] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:46.767313 22850 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:46.767949 22850 raft_consensus.cc:595] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Received request to transfer leadership to TS 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:46.770808 22850 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:46.771427 22850 raft_consensus.cc:595] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:46.773543 22850 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:46.774041 22850 raft_consensus.cc:595] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Received request to transfer leadership to TS 0a36468db59a4d328c8c7a060916e4bd
I20260809 23:16:46.776413 22850 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
mode: GRACEFUL
new_leader_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:43312
I20260809 23:16:46.776908 22850 raft_consensus.cc:595] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Received request to transfer leadership to TS e609d1a6e91147bb978f92fcd441ef58
I20260809 23:16:46.778551 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:16:46.779177 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:47.012925 23292 raft_consensus.cc:979] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:47.013590 23295 raft_consensus.cc:1067] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Signalling peer 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:47.015256 22964 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:53770
I20260809 23:16:47.015743 22964 raft_consensus.cc:482] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:47.016058 22964 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.019464 23292 raft_consensus.cc:979] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.019958 23292 raft_consensus.cc:1067] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.021883 22737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48102
I20260809 23:16:47.022370 22737 raft_consensus.cc:482] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:47.022727 22737 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.022734 22964 raft_consensus.cc:504] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.025645 22964 leader_election.cc:288] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:47.026559 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:47.027375 22735 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.027364 22850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a6164b6930d4ad08cc26b898fe3837b" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:47.028033 22850 raft_consensus.cc:3032] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:47.028364 22850 raft_consensus.cc:729] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:47.029085 22850 consensus_queue.cc:260] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.029606 22737 raft_consensus.cc:504] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.030618 22850 raft_consensus.cc:3037] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.032925 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:47.033174 22735 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:47.033473 22854 raft_consensus.cc:3032] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:16:47.033785 22854 raft_consensus.cc:729] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:47.034515 22854 consensus_queue.cc:260] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.035593 22854 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.035684 23295 raft_consensus.cc:979] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.035632 22914 leader_election.cc:302] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:47.036371 23291 raft_consensus.cc:1067] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.036675 23243 raft_consensus.cc:2781] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:47.038036 23243 raft_consensus.cc:686] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:47.038112 22850 raft_consensus.cc:2454] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:47.039026 23243 consensus_queue.cc:237] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.040361 22735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48102
I20260809 23:16:47.040925 22735 raft_consensus.cc:482] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:47.041280 22735 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.041739 22854 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 5.
I20260809 23:16:47.042411 22737 leader_election.cc:288] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 5 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:47.043596 22686 leader_election.cc:302] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:47.043855 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33d3436d82c047b1b742bbb14c7cf58e" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:47.044406 22964 raft_consensus.cc:3037] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.048753 22735 raft_consensus.cc:504] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.051522 23298 raft_consensus.cc:2781] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:16:47.052075 22965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:47.052330 22964 raft_consensus.cc:2454] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 5.
I20260809 23:16:47.052788 22965 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.054879 22735 leader_election.cc:288] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 5 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:47.054920 22584 catalog_manager.cc:5495] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 2 to 3, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 3 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:47.056102 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2831a340c2641648f0d13fc637782d8" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:47.056876 22854 raft_consensus.cc:3032] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:16:47.057276 22854 raft_consensus.cc:729] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:47.058297 22854 consensus_queue.cc:260] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.060154 22854 raft_consensus.cc:3037] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:16:47.063221 23298 raft_consensus.cc:686] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 5 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:47.064467 22965 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 5.
I20260809 23:16:47.064659 23298 consensus_queue.cc:237] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.066416 22688 leader_election.cc:302] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, e609d1a6e91147bb978f92fcd441ef58; no voters: 
I20260809 23:16:47.067487 22854 raft_consensus.cc:2454] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 5.
I20260809 23:16:47.067863 23299 raft_consensus.cc:2781] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:16:47.070911 23299 raft_consensus.cc:686] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 5 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:47.071837 23299 consensus_queue.cc:237] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.083002 22577 catalog_manager.cc:5495] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 4 to 5, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 5 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:47.096156 22577 catalog_manager.cc:5495] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 4 to 5, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 5 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:47.202301 23292 raft_consensus.cc:979] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.202768 23291 raft_consensus.cc:1067] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Signalling peer e609d1a6e91147bb978f92fcd441ef58 to start an election
I20260809 23:16:47.204483 22735 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d"
dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
 from {username='slave'} at 127.0.0.1:48102
I20260809 23:16:47.205044 22735 raft_consensus.cc:482] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:47.205358 22735 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:47.211401 22735 raft_consensus.cc:504] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.213088 22735 leader_election.cc:288] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Requested vote from peers f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785), 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3:35565)
I20260809 23:16:47.214221 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:47.214462 22964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "416c561d8e8e4f109692dd3f0063319d" candidate_uuid: "e609d1a6e91147bb978f92fcd441ef58" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
I20260809 23:16:47.214756 22854 raft_consensus.cc:3032] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:47.215020 22964 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:47.215116 22854 raft_consensus.cc:729] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:47.215359 23291 raft_consensus.cc:979] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: : Instructing follower 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:47.215816 23291 raft_consensus.cc:1067] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Signalling peer 0a36468db59a4d328c8c7a060916e4bd to start an election
I20260809 23:16:47.215888 22854 consensus_queue.cc:260] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.217084 22854 raft_consensus.cc:3037] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:47.217439 22965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee"
dest_uuid: "0a36468db59a4d328c8c7a060916e4bd"
 from {username='slave'} at 127.0.0.1:53770
I20260809 23:16:47.217938 22965 raft_consensus.cc:482] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:47.218259 22965 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.221145 22854 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:47.221246 22964 raft_consensus.cc:2454] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e609d1a6e91147bb978f92fcd441ef58 in term 2.
I20260809 23:16:47.222143 22686 leader_election.cc:302] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e609d1a6e91147bb978f92fcd441ef58, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:47.222883 23299 raft_consensus.cc:2781] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:47.223347 23299 raft_consensus.cc:686] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Becoming Leader. State: Replica: e609d1a6e91147bb978f92fcd441ef58, State: Running, Role: LEADER
I20260809 23:16:47.223886 22965 raft_consensus.cc:504] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.224098 23299 consensus_queue.cc:237] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.225447 22965 leader_election.cc:288] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Requested vote from peers e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667), f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785)
I20260809 23:16:47.227084 22735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e609d1a6e91147bb978f92fcd441ef58"
I20260809 23:16:47.227653 22735 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.229130 22854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0bfd567ce714705b5383659e80041ee" candidate_uuid: "0a36468db59a4d328c8c7a060916e4bd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf"
I20260809 23:16:47.229851 22854 raft_consensus.cc:3032] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:16:47.230172 22854 raft_consensus.cc:729] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6d3e3e4c8424c91adee4f5c26e134bf, State: Running, Role: LEADER
I20260809 23:16:47.230965 22854 consensus_queue.cc:260] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.232007 22854 raft_consensus.cc:3037] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:16:47.234475 22735 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:47.233846 22577 catalog_manager.cc:5495] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 reported cstate change: term changed from 1 to 2, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "e609d1a6e91147bb978f92fcd441ef58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:47.236636 22854 raft_consensus.cc:2454] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a36468db59a4d328c8c7a060916e4bd in term 3.
I20260809 23:16:47.237883 22913 leader_election.cc:302] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a36468db59a4d328c8c7a060916e4bd, f6d3e3e4c8424c91adee4f5c26e134bf; no voters: 
I20260809 23:16:47.238582 23243 raft_consensus.cc:2781] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:16:47.239111 23243 raft_consensus.cc:686] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Becoming Leader. State: Replica: 0a36468db59a4d328c8c7a060916e4bd, State: Running, Role: LEADER
I20260809 23:16:47.239917 23243 consensus_queue.cc:237] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } }
I20260809 23:16:47.247588 22577 catalog_manager.cc:5495] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd reported cstate change: term changed from 2 to 3, leader changed from f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2) to 0a36468db59a4d328c8c7a060916e4bd (127.21.255.3). New cstate: current_term: 3 leader_uuid: "0a36468db59a4d328c8c7a060916e4bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:47.433548 22854 raft_consensus.cc:1261] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:47.434826 23243 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:47.445623 22735 raft_consensus.cc:1261] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:47.447583 23296 consensus_queue.cc:1035] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:16:47.476827 22965 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:16:47.478070 23299 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:16:47.487005 22854 raft_consensus.cc:1261] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:16:47.490881 23298 consensus_queue.cc:1035] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:16:47.548477 22965 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:16:47.549772 23298 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:16:47.559116 22854 raft_consensus.cc:1261] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:16:47.561841 23304 consensus_queue.cc:1035] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:16:47.580852 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080019f5a0 after lost signal to thread 22536
W20260809 23:16:47.581601 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080003f3a0 after lost signal to thread 22636
W20260809 23:16:47.582324 22885 debug-util.cc:398] Leaking SignalData structure 0x7b080012f180 after lost signal to thread 22779
W20260809 23:16:47.583035 22885 debug-util.cc:398] Leaking SignalData structure 0x7b0800085d00 after lost signal to thread 22888
W20260809 23:16:47.583674 22885 debug-util.cc:398] Leaking SignalData structure 0x7b08001de760 after lost signal to thread 22997
I20260809 23:16:47.629308 22737 raft_consensus.cc:1261] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:47.630657 23316 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e609d1a6e91147bb978f92fcd441ef58" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:47.641628 22854 raft_consensus.cc:1261] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Refusing update from remote peer 0a36468db59a4d328c8c7a060916e4bd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:16:47.643316 23316 consensus_queue.cc:1035] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:16:47.702131 22965 raft_consensus.cc:1261] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:47.703869 23298 consensus_queue.cc:1035] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a36468db59a4d328c8c7a060916e4bd" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 35565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:47.713016 22854 raft_consensus.cc:1261] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Refusing update from remote peer e609d1a6e91147bb978f92fcd441ef58: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:16:47.714700 23299 consensus_queue.cc:1035] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3e3e4c8424c91adee4f5c26e134bf" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:16:47.780378 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:47.789333 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:47.789903 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:48.790795 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:48.796694 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:48.797251 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:49.798390 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:49.803670 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:49.804299 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:50.805192 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:50.810771 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:50.811183 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:51.812000 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:51.817513 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:51.818095 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:52.819355 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:52.825536 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:52.826037 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:53.826825 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:53.831733 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:53.832062 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:54.832778 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:54.837922 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:54.838248 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:55.839784 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:16:55.847501 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:16:55.847950 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:16:56.880010 22524 auto_leader_rebalancer-test.cc:314] The leader distribution is 
I20260809 23:16:56.880626 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
I20260809 23:16:56.925302 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:16:56.926069 22524 tablet_replica.cc:330] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.927130 22524 raft_consensus.cc:2229] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 5 LEADER]: Raft consensus shutting down.
I20260809 23:16:56.928747 22524 raft_consensus.cc:2258] T f2831a340c2641648f0d13fc637782d8 P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.932446 22524 tablet_replica.cc:330] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.933271 22524 raft_consensus.cc:2229] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.933844 22524 raft_consensus.cc:2258] T 1a1aad2b54974018a9e3a15521b84c2c P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.936219 22524 tablet_replica.cc:330] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.936901 22524 raft_consensus.cc:2229] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:16:56.937948 22524 raft_consensus.cc:2258] T 416c561d8e8e4f109692dd3f0063319d P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.940143 22524 tablet_replica.cc:330] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.940786 22524 raft_consensus.cc:2229] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.941378 22524 raft_consensus.cc:2258] T 03850407c41840899265114271b654b5 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.943656 22524 tablet_replica.cc:330] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.944381 22524 raft_consensus.cc:2229] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.944873 22524 raft_consensus.cc:2258] T 3a6164b6930d4ad08cc26b898fe3837b P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:16:56.946168 22914 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:16:56.947393 22524 tablet_replica.cc:330] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.947970 22524 raft_consensus.cc:2229] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.948472 22524 raft_consensus.cc:2258] T 981102a03e6d4b0192c3bf28d0631104 P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:16:56.949587 22914 consensus_peers.cc:487] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd -> Peer e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667): Couldn't send request to peer e609d1a6e91147bb978f92fcd441ef58. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:16:56.950448 22524 tablet_replica.cc:330] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.950994 22524 raft_consensus.cc:2229] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.951427 22524 raft_consensus.cc:2258] T d0bfd567ce714705b5383659e80041ee P e609d1a6e91147bb978f92fcd441ef58 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.953011 22524 tablet_replica.cc:330] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.953454 22524 raft_consensus.cc:2229] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:56.953869 22524 raft_consensus.cc:2258] T 3d4debc6d6aa47d181f4f8d655f5aced P e609d1a6e91147bb978f92fcd441ef58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:56.956202 22524 tablet_replica.cc:330] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58: stopping tablet replica
I20260809 23:16:56.956861 22524 raft_consensus.cc:2229] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 5 LEADER]: Raft consensus shutting down.
I20260809 23:16:56.957926 22524 raft_consensus.cc:2258] T 33d3436d82c047b1b742bbb14c7cf58e P e609d1a6e91147bb978f92fcd441ef58 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260809 23:16:57.109923 22914 consensus_peers.cc:487] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd -> Peer e609d1a6e91147bb978f92fcd441ef58 (127.21.255.1:43667): Couldn't send request to peer e609d1a6e91147bb978f92fcd441ef58. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:43667: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:16:57.110494 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
I20260809 23:16:57.133662 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
I20260809 23:16:57.168907 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:16:57.169657 22524 tablet_replica.cc:330] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.170759 22524 raft_consensus.cc:2229] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.171566 22524 raft_consensus.cc:2258] T 416c561d8e8e4f109692dd3f0063319d P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.173627 22524 tablet_replica.cc:330] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.174172 22524 raft_consensus.cc:2229] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.174612 22524 raft_consensus.cc:2258] T 3a6164b6930d4ad08cc26b898fe3837b P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.177070 22524 tablet_replica.cc:330] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.177798 22524 raft_consensus.cc:2229] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.178563 22524 raft_consensus.cc:2258] T 33d3436d82c047b1b742bbb14c7cf58e P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.180806 22524 tablet_replica.cc:330] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.181378 22524 raft_consensus.cc:2229] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.182381 22524 raft_consensus.cc:2258] T 03850407c41840899265114271b654b5 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.184924 22524 tablet_replica.cc:330] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.185524 22524 raft_consensus.cc:2229] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.186091 22524 raft_consensus.cc:2258] T 3d4debc6d6aa47d181f4f8d655f5aced P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.188005 22524 tablet_replica.cc:330] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.188608 22524 raft_consensus.cc:2229] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.189080 22524 raft_consensus.cc:2258] T f2831a340c2641648f0d13fc637782d8 P f6d3e3e4c8424c91adee4f5c26e134bf [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.190714 22524 tablet_replica.cc:330] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.191242 22524 raft_consensus.cc:2229] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.192039 22524 raft_consensus.cc:2258] T 1a1aad2b54974018a9e3a15521b84c2c P f6d3e3e4c8424c91adee4f5c26e134bf [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.194049 22524 tablet_replica.cc:330] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.194697 22524 raft_consensus.cc:2229] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.195165 22524 raft_consensus.cc:2258] T d0bfd567ce714705b5383659e80041ee P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.197386 22524 tablet_replica.cc:330] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf: stopping tablet replica
I20260809 23:16:57.197970 22524 raft_consensus.cc:2229] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.198882 22524 raft_consensus.cc:2258] T 981102a03e6d4b0192c3bf28d0631104 P f6d3e3e4c8424c91adee4f5c26e134bf [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:16:57.233500 22913 consensus_peers.cc:487] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd -> Peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Couldn't send request to peer f6d3e3e4c8424c91adee4f5c26e134bf. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:45785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:16:57.253049 22913 consensus_peers.cc:487] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd -> Peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Couldn't send request to peer f6d3e3e4c8424c91adee4f5c26e134bf. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:45785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:16:57.255264 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
W20260809 23:16:57.260635 22913 consensus_peers.cc:487] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd -> Peer f6d3e3e4c8424c91adee4f5c26e134bf (127.21.255.2:45785): Couldn't send request to peer f6d3e3e4c8424c91adee4f5c26e134bf. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:45785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:16:57.274238 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
I20260809 23:16:57.316516 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:16:57.317237 22524 tablet_replica.cc:330] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.318205 22524 raft_consensus.cc:2229] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.319024 22524 raft_consensus.cc:2258] T 1a1aad2b54974018a9e3a15521b84c2c P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.320998 22524 tablet_replica.cc:330] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.321944 22524 raft_consensus.cc:2229] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.322371 22524 raft_consensus.cc:2258] T f2831a340c2641648f0d13fc637782d8 P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.324039 22524 tablet_replica.cc:330] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.324476 22524 raft_consensus.cc:2229] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.324819 22524 raft_consensus.cc:2258] T 416c561d8e8e4f109692dd3f0063319d P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.326156 22524 tablet_replica.cc:330] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.326565 22524 raft_consensus.cc:2229] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.327039 22524 raft_consensus.cc:2258] T 33d3436d82c047b1b742bbb14c7cf58e P 0a36468db59a4d328c8c7a060916e4bd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.328473 22524 tablet_replica.cc:330] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.328895 22524 raft_consensus.cc:2229] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.329259 22524 raft_consensus.cc:2258] T 03850407c41840899265114271b654b5 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.330691 22524 tablet_replica.cc:330] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.331259 22524 raft_consensus.cc:2229] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.332023 22524 raft_consensus.cc:2258] T d0bfd567ce714705b5383659e80041ee P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.333484 22524 tablet_replica.cc:330] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.333875 22524 raft_consensus.cc:2229] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:16:57.334234 22524 raft_consensus.cc:2258] T 981102a03e6d4b0192c3bf28d0631104 P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.335934 22524 tablet_replica.cc:330] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.336653 22524 raft_consensus.cc:2229] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.337689 22524 raft_consensus.cc:2258] T 3d4debc6d6aa47d181f4f8d655f5aced P 0a36468db59a4d328c8c7a060916e4bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.340330 22524 tablet_replica.cc:330] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd: stopping tablet replica
I20260809 23:16:57.341063 22524 raft_consensus.cc:2229] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.342094 22524 raft_consensus.cc:2258] T 3a6164b6930d4ad08cc26b898fe3837b P 0a36468db59a4d328c8c7a060916e4bd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.374315 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:16:57.394820 22524 master.cc:537] Master@127.21.255.62:41299 shutting down...
I20260809 23:16:57.747169 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:16:57.747802 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:16:57.748190 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P d639e2db02b14b668b28ae8a4cfe5f8f: stopping tablet replica
I20260809 23:16:57.769110 22524 master.cc:552] Master@127.21.255.62:41299 shutdown complete.
0a36468db59a4d328c8c7a060916e4bd  1
e609d1a6e91147bb978f92fcd441ef58  4
f6d3e3e4c8424c91adee4f5c26e134bf  4
0a36468db59a4d328c8c7a060916e4bd  3
e609d1a6e91147bb978f92fcd441ef58  3
f6d3e3e4c8424c91adee4f5c26e134bf  3
0a36468db59a4d328c8c7a060916e4bd  1
e609d1a6e91147bb978f92fcd441ef58  0
f6d3e3e4c8424c91adee4f5c26e134bf  8
0a36468db59a4d328c8c7a060916e4bd  3
e609d1a6e91147bb978f92fcd441ef58  3
f6d3e3e4c8424c91adee4f5c26e134bf  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (32872 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260809 23:16:57.815712 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:37439
I20260809 23:16:57.816834 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:57.824301 23360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:57.825006 23361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:57.826485 23363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:57.827867 22524 server_base.cc:1034] running on GCE node
I20260809 23:16:57.829066 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:57.829253 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:57.829373 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317417829364 us; error 0 us; skew 500 ppm
I20260809 23:16:57.829834 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:57.832410 22524 webserver.cc:460] Webserver started at http://127.21.255.62:39633/ using document root <none> and password file <none>
I20260809 23:16:57.832866 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:57.833020 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:57.833240 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:57.834390 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "2e910adb349d488cbb05cd295f3987b1"
format_stamp: "Formatted at 2026-08-09 23:16:57 on dist-test-slave-drl0"
I20260809 23:16:57.839570 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:16:57.844393 23370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:57.845311 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260809 23:16:57.845623 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "2e910adb349d488cbb05cd295f3987b1"
format_stamp: "Formatted at 2026-08-09 23:16:57 on dist-test-slave-drl0"
I20260809 23:16:57.845932 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:57.876153 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:57.877472 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:57.923316 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:37439
I20260809 23:16:57.923461 23450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:37439 every 8 connection(s)
I20260809 23:16:57.927315 23452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:57.938357 23452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1: Bootstrap starting.
I20260809 23:16:57.944990 23452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:57.950562 23452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1: No bootstrap required, opened a new log
I20260809 23:16:57.952742 23452 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER }
I20260809 23:16:57.953214 23452 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:57.953500 23452 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e910adb349d488cbb05cd295f3987b1, State: Initialized, Role: FOLLOWER
I20260809 23:16:57.954195 23452 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER }
I20260809 23:16:57.954658 23452 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:16:57.954916 23452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:16:57.955379 23452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:57.960232 23452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER }
I20260809 23:16:57.960770 23452 leader_election.cc:302] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e910adb349d488cbb05cd295f3987b1; no voters: 
I20260809 23:16:57.962194 23452 leader_election.cc:288] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:16:57.962604 23460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:57.964104 23460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 1 LEADER]: Becoming Leader. State: Replica: 2e910adb349d488cbb05cd295f3987b1, State: Running, Role: LEADER
I20260809 23:16:57.965385 23452 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:16:57.964994 23460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER }
I20260809 23:16:57.971733 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e910adb349d488cbb05cd295f3987b1. Latest consensus state: current_term: 1 leader_uuid: "2e910adb349d488cbb05cd295f3987b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER } }
I20260809 23:16:57.972651 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [sys.catalog]: This master's current role is: LEADER
I20260809 23:16:57.972349 23461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e910adb349d488cbb05cd295f3987b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e910adb349d488cbb05cd295f3987b1" member_type: VOTER } }
I20260809 23:16:57.973240 23461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [sys.catalog]: This master's current role is: LEADER
I20260809 23:16:57.977265 23471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:16:57.986150 23471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:16:57.987315 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:16:57.995632 23471 catalog_manager.cc:1347] Generated new cluster ID: 798d27bbf01647c88d9f12fca9b357d0
I20260809 23:16:57.996054 23471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:16:58.014305 23471 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:16:58.015938 23471 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:16:58.030431 23471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1: Generated new TSK 0
I20260809 23:16:58.031138 23471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:16:58.053857 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:58.061019 23482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:58.061749 23483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:58.064507 23487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:58.064970 22524 server_base.cc:1034] running on GCE node
I20260809 23:16:58.066110 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:58.066319 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:58.066496 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317418066482 us; error 0 us; skew 500 ppm
I20260809 23:16:58.067240 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:58.069854 22524 webserver.cc:460] Webserver started at http://127.21.255.1:44055/ using document root <none> and password file <none>
I20260809 23:16:58.070402 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:58.070618 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:58.070894 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:58.072182 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "7958adffbd364c45ac3e90fcadf23355"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.077179 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260809 23:16:58.080930 23493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.081882 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260809 23:16:58.082126 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "7958adffbd364c45ac3e90fcadf23355"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.082384 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:58.099565 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:58.101133 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:58.102998 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:58.106103 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:58.106434 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.106695 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:58.106878 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.156690 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:45725
I20260809 23:16:58.156819 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:45725 every 8 connection(s)
I20260809 23:16:58.161886 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:58.169409 23601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:58.171124 23602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:58.175294 23604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:58.175726 22524 server_base.cc:1034] running on GCE node
I20260809 23:16:58.178273 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:58.178624 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:58.178860 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317418178837 us; error 0 us; skew 500 ppm
I20260809 23:16:58.179862 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:58.181488 23592 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37439
I20260809 23:16:58.182157 23592 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:58.183360 23592 heartbeater.cc:511] Master 127.21.255.62:37439 requested a full tablet report, sending...
I20260809 23:16:58.183887 22524 webserver.cc:460] Webserver started at http://127.21.255.2:39939/ using document root <none> and password file <none>
I20260809 23:16:58.184715 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:58.184988 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:58.185322 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:58.186169 23399 ts_manager.cc:194] Registered new tserver with Master: 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725)
I20260809 23:16:58.187032 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "68fa31cbc36c4524afcab09fe96b2be6"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.188985 23399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57566
I20260809 23:16:58.193501 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.001s
I20260809 23:16:58.197235 23614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.198118 22524 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260809 23:16:58.198437 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "68fa31cbc36c4524afcab09fe96b2be6"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.198698 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:58.220227 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:58.221904 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:58.223805 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:58.226627 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:58.226913 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.227243 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:58.227421 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.277577 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:32899
I20260809 23:16:58.277689 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:32899 every 8 connection(s)
I20260809 23:16:58.282281 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:16:58.290388 23706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:16:58.292165 23708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:58.294667 22524 server_base.cc:1034] running on GCE node
W20260809 23:16:58.295555 23710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:16:58.296602 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:16:58.296880 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:16:58.297086 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317418297068 us; error 0 us; skew 500 ppm
I20260809 23:16:58.297777 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:16:58.300333 23701 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37439
I20260809 23:16:58.300925 23701 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:58.301575 22524 webserver.cc:460] Webserver started at http://127.21.255.3:40973/ using document root <none> and password file <none>
I20260809 23:16:58.301921 23701 heartbeater.cc:511] Master 127.21.255.62:37439 requested a full tablet report, sending...
I20260809 23:16:58.302217 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:16:58.302484 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:16:58.302790 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:16:58.304610 23399 ts_manager.cc:194] Registered new tserver with Master: 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:58.304713 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "5684fadd273849a5a6a7e1e281cf60b0"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.306831 23399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57574
I20260809 23:16:58.310639 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20260809 23:16:58.314126 23719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.315062 22524 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260809 23:16:58.315301 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "5684fadd273849a5a6a7e1e281cf60b0"
format_stamp: "Formatted at 2026-08-09 23:16:58 on dist-test-slave-drl0"
I20260809 23:16:58.315546 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:16:58.345297 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:16:58.346464 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:16:58.347864 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:16:58.350310 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:16:58.350513 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.350771 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:16:58.350919 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:16:58.399365 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:33395
I20260809 23:16:58.399461 23822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:33395 every 8 connection(s)
I20260809 23:16:58.419332 23825 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37439
I20260809 23:16:58.419727 23825 heartbeater.cc:464] Registering TS with master...
I20260809 23:16:58.420462 23825 heartbeater.cc:511] Master 127.21.255.62:37439 requested a full tablet report, sending...
I20260809 23:16:58.422529 23399 ts_manager.cc:194] Registered new tserver with Master: 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:58.423555 22524 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020231204s
I20260809 23:16:58.424145 23399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57588
I20260809 23:16:58.426075 22524 test_util.cc:268] Using random seed: 1865313987
I20260809 23:16:58.456820 23399 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57594:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:16:58.460704 23399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:16:58.546581 23647 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5bc3f286944be4bddba0edaeec36b9 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 23:16:58.548072 23647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5bc3f286944be4bddba0edaeec36b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.551299 23645 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff367fda1d4949a94460835c37867a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 23:16:58.551463 23642 tablet_service.cc:1467] Processing CreateTablet for tablet 422b11fb9ebc495aa825f4c1658b93ff (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 23:16:58.549157 23643 tablet_service.cc:1467] Processing CreateTablet for tablet fef90253eba44d7bbe54c20ad4b9b399 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 23:16:58.552795 23645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff367fda1d4949a94460835c37867a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.553418 23643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef90253eba44d7bbe54c20ad4b9b399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.554912 23644 tablet_service.cc:1467] Processing CreateTablet for tablet ac19379b45ba4438a63d37c3fb7d6f0a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 23:16:58.556348 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac19379b45ba4438a63d37c3fb7d6f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.556679 23646 tablet_service.cc:1467] Processing CreateTablet for tablet 098773334a6148729ab7f54cc0cb7084 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 23:16:58.557889 23646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098773334a6148729ab7f54cc0cb7084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.559463 23642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b11fb9ebc495aa825f4c1658b93ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.564366 23641 tablet_service.cc:1467] Processing CreateTablet for tablet aeceab82bfe741b98138f9204f27e7b2 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 23:16:58.565657 23641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeceab82bfe741b98138f9204f27e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.566087 23639 tablet_service.cc:1467] Processing CreateTablet for tablet 2005b0f7b5e449a0a1f4c66bacae226f (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 23:16:58.566430 23638 tablet_service.cc:1467] Processing CreateTablet for tablet 21430435848b4603896c5e59b1edf664 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 23:16:58.567391 23639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005b0f7b5e449a0a1f4c66bacae226f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.568585 23637 tablet_service.cc:1467] Processing CreateTablet for tablet d142bafc563b40aab31a0eb10e1e8ff5 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 23:16:58.570252 23637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d142bafc563b40aab31a0eb10e1e8ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.571367 23638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21430435848b4603896c5e59b1edf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.593814 23843 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.603056 23538 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5bc3f286944be4bddba0edaeec36b9 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 23:16:58.604462 23538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5bc3f286944be4bddba0edaeec36b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.609777 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 098773334a6148729ab7f54cc0cb7084 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 23:16:58.611137 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098773334a6148729ab7f54cc0cb7084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.613482 23534 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff367fda1d4949a94460835c37867a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 23:16:58.614853 23534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff367fda1d4949a94460835c37867a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.618119 23761 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5bc3f286944be4bddba0edaeec36b9 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260809 23:16:58.619452 23761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5bc3f286944be4bddba0edaeec36b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.619406 23753 tablet_service.cc:1467] Processing CreateTablet for tablet fef90253eba44d7bbe54c20ad4b9b399 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 23:16:58.620803 23753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef90253eba44d7bbe54c20ad4b9b399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.620545 23749 tablet_service.cc:1467] Processing CreateTablet for tablet 2005b0f7b5e449a0a1f4c66bacae226f (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 23:16:58.621878 23749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005b0f7b5e449a0a1f4c66bacae226f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.624250 23760 tablet_service.cc:1467] Processing CreateTablet for tablet 098773334a6148729ab7f54cc0cb7084 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260809 23:16:58.624750 23533 tablet_service.cc:1467] Processing CreateTablet for tablet ac19379b45ba4438a63d37c3fb7d6f0a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 23:16:58.625677 23760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 098773334a6148729ab7f54cc0cb7084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.626015 23533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac19379b45ba4438a63d37c3fb7d6f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.628152 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 6eff367fda1d4949a94460835c37867a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260809 23:16:58.628525 23530 tablet_service.cc:1467] Processing CreateTablet for tablet fef90253eba44d7bbe54c20ad4b9b399 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260809 23:16:58.628592 23748 tablet_service.cc:1467] Processing CreateTablet for tablet 21430435848b4603896c5e59b1edf664 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 23:16:58.629861 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6eff367fda1d4949a94460835c37867a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.630695 23748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21430435848b4603896c5e59b1edf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.631609 23528 tablet_service.cc:1467] Processing CreateTablet for tablet 422b11fb9ebc495aa825f4c1658b93ff (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 23:16:58.632421 23758 tablet_service.cc:1467] Processing CreateTablet for tablet ac19379b45ba4438a63d37c3fb7d6f0a (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260809 23:16:58.634196 23758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac19379b45ba4438a63d37c3fb7d6f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.634565 23845 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.635205 23527 tablet_service.cc:1467] Processing CreateTablet for tablet aeceab82bfe741b98138f9204f27e7b2 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 23:16:58.636135 23751 tablet_service.cc:1467] Processing CreateTablet for tablet 422b11fb9ebc495aa825f4c1658b93ff (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260809 23:16:58.636521 23747 tablet_service.cc:1467] Processing CreateTablet for tablet d142bafc563b40aab31a0eb10e1e8ff5 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 23:16:58.637439 23751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b11fb9ebc495aa825f4c1658b93ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.637269 23526 tablet_service.cc:1467] Processing CreateTablet for tablet 2005b0f7b5e449a0a1f4c66bacae226f (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260809 23:16:58.638986 23530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef90253eba44d7bbe54c20ad4b9b399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.640362 23750 tablet_service.cc:1467] Processing CreateTablet for tablet aeceab82bfe741b98138f9204f27e7b2 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260809 23:16:58.641178 23747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d142bafc563b40aab31a0eb10e1e8ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.641378 23526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2005b0f7b5e449a0a1f4c66bacae226f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.642827 23750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeceab82bfe741b98138f9204f27e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.642984 23527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeceab82bfe741b98138f9204f27e7b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.647397 23528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422b11fb9ebc495aa825f4c1658b93ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.653466 23523 tablet_service.cc:1467] Processing CreateTablet for tablet d142bafc563b40aab31a0eb10e1e8ff5 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260809 23:16:58.655587 23523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d142bafc563b40aab31a0eb10e1e8ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.676352 23537 tablet_service.cc:1467] Processing CreateTablet for tablet 21430435848b4603896c5e59b1edf664 (DEFAULT_TABLE table=test-workload [id=0d715dd4cc4c4de398b51ea11a641a81]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260809 23:16:58.682281 23537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21430435848b4603896c5e59b1edf664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:16:58.694495 23845 tablet_bootstrap.cc:654] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.707357 23843 tablet_bootstrap.cc:654] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.732527 23843 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.732999 23843 ts_tablet_manager.cc:1397] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.140s	user 0.004s	sys 0.008s
I20260809 23:16:58.736445 23843 raft_consensus.cc:348] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.737315 23843 raft_consensus.cc:374] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.737599 23843 raft_consensus.cc:729] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.739562 23843 consensus_queue.cc:260] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.753343 23847 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:58.760462 23845 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.760975 23845 ts_tablet_manager.cc:1397] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.127s	user 0.018s	sys 0.008s
I20260809 23:16:58.763718 23845 raft_consensus.cc:348] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.764655 23845 raft_consensus.cc:374] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.764977 23845 raft_consensus.cc:729] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.765775 23845 consensus_queue.cc:260] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.768147 23701 heartbeater.cc:503] Master 127.21.255.62:37439 was elected leader, sending a full tablet report...
I20260809 23:16:58.768617 23843 ts_tablet_manager.cc:1428] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.006s
I20260809 23:16:58.768658 23847 tablet_bootstrap.cc:654] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.769606 23843 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.775981 23845 ts_tablet_manager.cc:1428] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.007s
I20260809 23:16:58.776700 23843 tablet_bootstrap.cc:654] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.777423 23592 heartbeater.cc:503] Master 127.21.255.62:37439 was elected leader, sending a full tablet report...
W20260809 23:16:58.781929 23702 tablet.cc:2369] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 23:16:58.782029 23845 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.790817 23847 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:58.791572 23845 tablet_bootstrap.cc:654] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.792595 23847 ts_tablet_manager.cc:1397] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.040s	user 0.008s	sys 0.012s
I20260809 23:16:58.793717 23843 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.794190 23843 ts_tablet_manager.cc:1397] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.014s
I20260809 23:16:58.795691 23847 raft_consensus.cc:348] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.796567 23847 raft_consensus.cc:374] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.796914 23847 raft_consensus.cc:729] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.797363 23843 raft_consensus.cc:348] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.798345 23843 raft_consensus.cc:374] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.797814 23847 consensus_queue.cc:260] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.798745 23843 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.799732 23843 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.802744 23843 ts_tablet_manager.cc:1428] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:16:58.803946 23843 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.812160 23843 tablet_bootstrap.cc:654] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.824839 23825 heartbeater.cc:503] Master 127.21.255.62:37439 was elected leader, sending a full tablet report...
I20260809 23:16:58.825520 23847 ts_tablet_manager.cc:1428] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.033s	user 0.014s	sys 0.012s
I20260809 23:16:58.825817 23845 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.826470 23845 ts_tablet_manager.cc:1397] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.045s	user 0.012s	sys 0.027s
I20260809 23:16:58.826642 23847 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:58.830859 23845 raft_consensus.cc:348] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.832265 23845 raft_consensus.cc:374] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.832653 23845 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.833475 23847 tablet_bootstrap.cc:654] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.833875 23845 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.837291 23845 ts_tablet_manager.cc:1428] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.004s
I20260809 23:16:58.838585 23845 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.844256 23843 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.844727 23843 ts_tablet_manager.cc:1397] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.041s	user 0.029s	sys 0.009s
I20260809 23:16:58.844827 23845 tablet_bootstrap.cc:654] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.847976 23843 raft_consensus.cc:348] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.848908 23843 raft_consensus.cc:374] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.849295 23843 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.850338 23843 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.857137 23843 ts_tablet_manager.cc:1428] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.003s
I20260809 23:16:58.858371 23843 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.867390 23843 tablet_bootstrap.cc:654] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.868450 23845 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.868940 23845 ts_tablet_manager.cc:1397] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.014s
I20260809 23:16:58.871837 23845 raft_consensus.cc:348] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.872411 23847 tablet_bootstrap.cc:492] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:58.872563 23845 raft_consensus.cc:374] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.873080 23847 ts_tablet_manager.cc:1397] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.047s	user 0.018s	sys 0.016s
I20260809 23:16:58.873168 23845 raft_consensus.cc:729] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.874752 23845 consensus_queue.cc:260] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.876529 23843 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.877170 23843 ts_tablet_manager.cc:1397] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.004s
I20260809 23:16:58.877689 23845 ts_tablet_manager.cc:1428] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.008s	user 0.010s	sys 0.000s
I20260809 23:16:58.878715 23845 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.879990 23843 raft_consensus.cc:348] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.880733 23843 raft_consensus.cc:374] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.881042 23843 raft_consensus.cc:729] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.881753 23843 consensus_queue.cc:260] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.884179 23843 ts_tablet_manager.cc:1428] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260809 23:16:58.885291 23843 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.885437 23845 tablet_bootstrap.cc:654] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.875516 23847 raft_consensus.cc:348] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.890986 23847 raft_consensus.cc:374] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.892033 23850 raft_consensus.cc:482] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:58.892498 23850 raft_consensus.cc:504] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.894452 23843 tablet_bootstrap.cc:654] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.891323 23847 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.897122 23850 leader_election.cc:288] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:58.896292 23847 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.907667 23847 ts_tablet_manager.cc:1428] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.000s
I20260809 23:16:58.910457 23845 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.911176 23845 ts_tablet_manager.cc:1397] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.033s	user 0.017s	sys 0.002s
I20260809 23:16:58.915655 23847 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:58.922284 23847 tablet_bootstrap.cc:654] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.914211 23845 raft_consensus.cc:348] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.928776 23845 raft_consensus.cc:374] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.929116 23845 raft_consensus.cc:729] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.929942 23845 consensus_queue.cc:260] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.931672 23843 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.932451 23843 ts_tablet_manager.cc:1397] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.047s	user 0.025s	sys 0.011s
I20260809 23:16:58.932875 23845 ts_tablet_manager.cc:1428] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
I20260809 23:16:58.933878 23845 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.935678 23843 raft_consensus.cc:348] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.936470 23843 raft_consensus.cc:374] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.936807 23843 raft_consensus.cc:729] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.937635 23843 consensus_queue.cc:260] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.939769 23845 tablet_bootstrap.cc:654] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.941953 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:58.945255 23847 tablet_bootstrap.cc:492] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:58.945858 23847 ts_tablet_manager.cc:1397] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.030s	user 0.003s	sys 0.006s
I20260809 23:16:58.947319 23843 ts_tablet_manager.cc:1428] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
W20260809 23:16:58.948055 23501 leader_election.cc:341] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:58.951862 23847 raft_consensus.cc:348] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.953415 23847 raft_consensus.cc:374] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.953815 23847 raft_consensus.cc:729] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.955683 23843 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.956514 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:58.954891 23847 consensus_queue.cc:260] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.958591 23847 ts_tablet_manager.cc:1428] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260809 23:16:58.958791 23845 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.959319 23845 ts_tablet_manager.cc:1397] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.016s
I20260809 23:16:58.959592 23847 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:58.962155 23845 raft_consensus.cc:348] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.962706 23843 tablet_bootstrap.cc:654] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.962889 23845 raft_consensus.cc:374] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.963269 23845 raft_consensus.cc:729] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.963984 23845 consensus_queue.cc:260] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
W20260809 23:16:58.959358 23499 leader_election.cc:341] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:58.965189 23499 leader_election.cc:302] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7958adffbd364c45ac3e90fcadf23355; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:58.966351 23850 raft_consensus.cc:2726] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:58.966737 23845 ts_tablet_manager.cc:1428] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:16:58.968230 23845 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:58.968500 23847 tablet_bootstrap.cc:654] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:58.976538 23845 tablet_bootstrap.cc:654] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
W20260809 23:16:58.978456 23472 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:16:58.979244 23847 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:58.979705 23847 ts_tablet_manager.cc:1397] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.005s
I20260809 23:16:58.981263 23843 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:58.982721 23849 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:58.982370 23847 raft_consensus.cc:348] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.983448 23847 raft_consensus.cc:374] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.983896 23847 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.983392 23849 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.981925 23843 ts_tablet_manager.cc:1397] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.006s
I20260809 23:16:58.984795 23847 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.987747 23843 raft_consensus.cc:348] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.989912 23849 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395), 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725)
I20260809 23:16:58.990600 23843 raft_consensus.cc:374] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.990967 23843 raft_consensus.cc:729] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:58.991672 23843 consensus_queue.cc:260] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:58.993944 23843 ts_tablet_manager.cc:1428] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260809 23:16:58.995009 23843 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:58.995863 23845 tablet_bootstrap.cc:492] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:58.996351 23845 ts_tablet_manager.cc:1397] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.011s
I20260809 23:16:58.990267 23847 ts_tablet_manager.cc:1428] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20260809 23:16:58.998920 23845 raft_consensus.cc:348] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:58.999688 23845 raft_consensus.cc:374] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:58.999969 23845 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.000869 23847 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:59.000725 23845 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.002074 23843 tablet_bootstrap.cc:654] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.006067 23845 ts_tablet_manager.cc:1428] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260809 23:16:59.007030 23845 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:59.014539 23845 tablet_bootstrap.cc:654] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.016115 23849 raft_consensus.cc:482] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.016991 23849 raft_consensus.cc:504] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.019538 23849 leader_election.cc:288] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.027985 23847 tablet_bootstrap.cc:654] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.032856 23843 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:59.033330 23843 ts_tablet_manager.cc:1397] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.016s
I20260809 23:16:59.033854 23847 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.034303 23847 ts_tablet_manager.cc:1397] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.004s
I20260809 23:16:59.036906 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.037091 23847 raft_consensus.cc:348] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.037549 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.038141 23777 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 0.
W20260809 23:16:59.038616 23618 leader_election.cc:341] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.038919 23847 raft_consensus.cc:374] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.039278 23847 raft_consensus.cc:729] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.039719 23845 tablet_bootstrap.cc:492] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:59.039631 23618 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.040226 23845 ts_tablet_manager.cc:1397] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.004s
I20260809 23:16:59.041165 23849 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.036141 23843 raft_consensus.cc:348] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.041002 23847 consensus_queue.cc:260] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.041532 23849 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.041612 23843 raft_consensus.cc:374] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.041917 23849 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.043332 23847 ts_tablet_manager.cc:1428] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260809 23:16:59.043022 23845 raft_consensus.cc:348] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.043761 23845 raft_consensus.cc:374] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.044070 23845 raft_consensus.cc:729] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.044286 23847 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:59.044807 23843 raft_consensus.cc:729] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.044780 23845 consensus_queue.cc:260] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.045776 23843 consensus_queue.cc:260] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.047554 23845 ts_tablet_manager.cc:1428] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.007s
I20260809 23:16:59.048480 23845 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:59.054576 23845 tablet_bootstrap.cc:654] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.055028 23849 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.058833 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.059623 23777 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.065946 23850 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.066430 23850 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.068543 23850 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.069657 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.070163 23849 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Requested vote from peers 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395), 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725)
I20260809 23:16:59.070986 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.071743 23665 raft_consensus.cc:2379] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7958adffbd364c45ac3e90fcadf23355 in current term 1: Already voted for candidate 68fa31cbc36c4524afcab09fe96b2be6 in this term.
I20260809 23:16:59.073633 23501 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7958adffbd364c45ac3e90fcadf23355; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.071130 23847 tablet_bootstrap.cc:654] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.074994 23777 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 1.
I20260809 23:16:59.075125 23850 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.083856 23850 raft_consensus.cc:2726] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.084306 23893 raft_consensus.cc:482] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.084698 23850 raft_consensus.cc:482] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.084918 23893 raft_consensus.cc:504] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.086968 23618 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.088107 23849 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.088750 23849 raft_consensus.cc:686] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.092182 23849 consensus_queue.cc:237] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.085490 23850 raft_consensus.cc:504] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.094730 23847 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.095403 23847 ts_tablet_manager.cc:1397] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.051s	user 0.013s	sys 0.006s
I20260809 23:16:59.095551 23843 ts_tablet_manager.cc:1428] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.062s	user 0.014s	sys 0.020s
I20260809 23:16:59.096645 23850 leader_election.cc:288] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.096990 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.097601 23665 raft_consensus.cc:2454] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.098816 23499 leader_election.cc:302] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.100009 23893 leader_election.cc:288] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.100766 23893 raft_consensus.cc:2781] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.101197 23893 raft_consensus.cc:482] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.101265 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.101780 23893 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.102166 23665 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.098876 23847 raft_consensus.cc:348] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.103440 23847 raft_consensus.cc:374] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.103840 23847 raft_consensus.cc:729] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.104276 23499 leader_election.cc:302] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.105566 23856 raft_consensus.cc:482] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.106145 23856 raft_consensus.cc:504] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.107326 23847 consensus_queue.cc:260] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.110234 23856 leader_election.cc:288] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.112980 23850 raft_consensus.cc:2781] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.113437 23850 raft_consensus.cc:482] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.113828 23850 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.124753 23893 raft_consensus.cc:504] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.128551 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.129338 23665 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.133867 23847 ts_tablet_manager.cc:1428] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.003s
I20260809 23:16:59.135694 23847 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:59.136855 23665 raft_consensus.cc:2454] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.138034 23499 leader_election.cc:302] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.139500 23843 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:59.142303 23896 raft_consensus.cc:2781] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.144125 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.144949 23847 tablet_bootstrap.cc:654] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.146174 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.146816 23777 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.146698 23893 leader_election.cc:288] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.149006 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
W20260809 23:16:59.151587 23501 leader_election.cc:341] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.152307 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
W20260809 23:16:59.153553 23501 leader_election.cc:341] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.151554 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.134199 23885 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.158593 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.162366 23885 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
W20260809 23:16:59.161947 23618 leader_election.cc:341] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.159603 23850 raft_consensus.cc:504] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.163282 23618 leader_election.cc:302] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.165879 23849 raft_consensus.cc:2726] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.167377 23850 leader_election.cc:288] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.172958 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.176895 23885 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.175122 23665 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.179683 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.180298 23777 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.179503 23397 catalog_manager.cc:5495] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.184350 23845 tablet_bootstrap.cc:492] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:59.185422 23845 ts_tablet_manager.cc:1397] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.137s	user 0.042s	sys 0.060s
I20260809 23:16:59.186556 23546 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 1.
I20260809 23:16:59.188922 23845 raft_consensus.cc:348] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.189843 23845 raft_consensus.cc:374] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.190173 23845 raft_consensus.cc:729] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.190958 23845 consensus_queue.cc:260] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.192637 23901 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.193099 23901 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.194661 23665 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.198024 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.198770 23550 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 0.
I20260809 23:16:59.199146 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.199424 23499 leader_election.cc:302] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.199750 23551 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 0.
I20260809 23:16:59.200127 23724 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.195434 23845 ts_tablet_manager.cc:1428] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260809 23:16:59.196406 23901 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.197257 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.200718 23847 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.200675 23850 raft_consensus.cc:2781] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.201043 23856 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.201357 23847 ts_tablet_manager.cc:1397] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.066s	user 0.018s	sys 0.022s
I20260809 23:16:59.201501 23856 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.201843 23845 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:59.201922 23856 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.202647 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.196941 23777 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.204058 23847 raft_consensus.cc:348] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.204737 23885 raft_consensus.cc:482] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.204806 23847 raft_consensus.cc:374] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.205268 23847 raft_consensus.cc:729] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.205308 23885 raft_consensus.cc:504] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.206147 23847 consensus_queue.cc:260] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.207360 23885 leader_election.cc:288] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.208730 23847 ts_tablet_manager.cc:1428] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:16:59.209004 23618 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.209671 23847 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:59.209759 23885 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.210737 23885 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.211151 23885 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.210144 23850 raft_consensus.cc:686] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.212110 23850 consensus_queue.cc:237] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.209414 23856 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.214594 23776 raft_consensus.cc:2379] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68fa31cbc36c4524afcab09fe96b2be6 in current term 1: Already voted for candidate 5684fadd273849a5a6a7e1e281cf60b0 in this term.
I20260809 23:16:59.215610 23856 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.218775 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.197854 23843 tablet_bootstrap.cc:654] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.220780 23845 tablet_bootstrap.cc:654] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.216581 23847 tablet_bootstrap.cc:654] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.219558 23551 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.223047 23896 raft_consensus.cc:686] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.223968 23896 consensus_queue.cc:237] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.228396 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.234025 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.234726 23776 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 0.
I20260809 23:16:59.243911 23843 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:59.244395 23843 ts_tablet_manager.cc:1397] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.105s	user 0.011s	sys 0.002s
I20260809 23:16:59.248404 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.249255 23665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.249603 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.250085 23665 raft_consensus.cc:2379] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5684fadd273849a5a6a7e1e281cf60b0 in current term 1: Already voted for candidate 7958adffbd364c45ac3e90fcadf23355 in this term.
I20260809 23:16:59.248641 23618 leader_election.cc:302] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.251432 23905 raft_consensus.cc:2781] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.251765 23723 leader_election.cc:302] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0; no voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355
W20260809 23:16:59.252317 23618 leader_election.cc:341] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.253898 23856 raft_consensus.cc:2726] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.253660 23885 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.251813 23905 raft_consensus.cc:482] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.255116 23905 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.255421 23664 raft_consensus.cc:2379] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5684fadd273849a5a6a7e1e281cf60b0 in current term 1: Already voted for candidate 68fa31cbc36c4524afcab09fe96b2be6 in this term.
I20260809 23:16:59.261920 23843 raft_consensus.cc:348] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.262945 23843 raft_consensus.cc:374] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.263230 23843 raft_consensus.cc:729] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.263814 23843 consensus_queue.cc:260] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.264310 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.266286 23843 ts_tablet_manager.cc:1428] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260809 23:16:59.267031 23885 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.267545 23843 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:59.251775 23551 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 1.
I20260809 23:16:59.271117 23850 raft_consensus.cc:482] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.271433 23724 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.272203 23847 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.271641 23850 raft_consensus.cc:504] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.269280 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.272635 23856 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.273140 23856 raft_consensus.cc:686] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:16:59.275739 23850 leader_election.cc:288] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.275686 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.276294 23845 tablet_bootstrap.cc:492] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:59.275846 23856 consensus_queue.cc:237] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.276564 23847 ts_tablet_manager.cc:1397] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.067s	user 0.034s	sys 0.020s
I20260809 23:16:59.276763 23845 ts_tablet_manager.cc:1397] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.075s	user 0.009s	sys 0.012s
I20260809 23:16:59.277755 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.278329 23777 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.279258 23847 raft_consensus.cc:348] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.279726 23501 leader_election.cc:302] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.280006 23847 raft_consensus.cc:374] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.280349 23847 raft_consensus.cc:729] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
W20260809 23:16:59.280455 23499 leader_election.cc:341] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.280730 23850 raft_consensus.cc:2781] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.281015 23847 consensus_queue.cc:260] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.281237 23850 raft_consensus.cc:482] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.281795 23850 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.286657 23843 tablet_bootstrap.cc:654] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.288853 23850 raft_consensus.cc:504] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.290511 23399 catalog_manager.cc:5495] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.291101 23905 raft_consensus.cc:504] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.291692 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.292120 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.292671 23777 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:16:59.294390 23499 leader_election.cc:341] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.296391 23618 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.297623 23885 raft_consensus.cc:2726] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.298569 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.297643 23856 raft_consensus.cc:482] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:16:59.299795 23618 leader_election.cc:341] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.300328 23856 raft_consensus.cc:504] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.301023 23905 leader_election.cc:288] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.299906 23845 raft_consensus.cc:348] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.302211 23845 raft_consensus.cc:374] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.302512 23845 raft_consensus.cc:729] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.302970 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.303697 23776 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.304939 23845 consensus_queue.cc:260] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.308506 23847 ts_tablet_manager.cc:1428] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260809 23:16:59.309474 23843 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:59.309441 23856 leader_election.cc:288] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.309949 23845 ts_tablet_manager.cc:1428] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.033s	user 0.003s	sys 0.003s
I20260809 23:16:59.310014 23843 ts_tablet_manager.cc:1397] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.043s	user 0.010s	sys 0.000s
I20260809 23:16:59.310693 23847 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
I20260809 23:16:59.310699 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.311488 23664 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 0.
I20260809 23:16:59.311522 23777 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.310729 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.312510 23845 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: Bootstrap starting.
I20260809 23:16:59.312651 23723 leader_election.cc:302] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.312870 23551 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 0.
I20260809 23:16:59.313449 23856 raft_consensus.cc:2781] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.313784 23856 raft_consensus.cc:482] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.314157 23776 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 1.
I20260809 23:16:59.314108 23856 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.315526 23618 leader_election.cc:302] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.316433 23905 raft_consensus.cc:2781] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.317221 23905 raft_consensus.cc:686] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.318264 23905 consensus_queue.cc:237] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.319772 23847 tablet_bootstrap.cc:654] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.319890 23843 raft_consensus.cc:348] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.320591 23843 raft_consensus.cc:374] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.320854 23843 raft_consensus.cc:729] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.323649 23501 leader_election.cc:302] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.324540 23896 raft_consensus.cc:2781] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.325110 23896 raft_consensus.cc:686] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.324539 23843 consensus_queue.cc:260] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.325989 23896 consensus_queue.cc:237] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.327111 23845 tablet_bootstrap.cc:654] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.328037 23843 ts_tablet_manager.cc:1428] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260809 23:16:59.329051 23843 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: Bootstrap starting.
I20260809 23:16:59.325098 23850 leader_election.cc:288] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.335989 23399 catalog_manager.cc:5495] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.339851 23856 raft_consensus.cc:504] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.338124 23397 catalog_manager.cc:5495] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:59.345834 23847 tablet_bootstrap.cc:492] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.346287 23847 ts_tablet_manager.cc:1397] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.036s	user 0.011s	sys 0.005s
I20260809 23:16:59.347236 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.348114 23551 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.351060 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.351320 23847 raft_consensus.cc:348] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.351661 23664 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.352016 23847 raft_consensus.cc:374] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.352312 23847 raft_consensus.cc:729] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.352779 23843 tablet_bootstrap.cc:654] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.353047 23847 consensus_queue.cc:260] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.346594 23396 catalog_manager.cc:5495] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.353427 23856 leader_election.cc:288] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.355755 23847 ts_tablet_manager.cc:1428] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:16:59.356329 23856 raft_consensus.cc:482] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.356745 23856 raft_consensus.cc:504] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.359386 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d142bafc563b40aab31a0eb10e1e8ff5" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.359666 23847 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: Bootstrap starting.
W20260809 23:16:59.360694 23724 leader_election.cc:341] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:16:59.362948 23551 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 1.
I20260809 23:16:59.363344 23856 leader_election.cc:288] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.363619 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d142bafc563b40aab31a0eb10e1e8ff5" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.363981 23724 leader_election.cc:302] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.364256 23662 raft_consensus.cc:2454] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 0.
I20260809 23:16:59.360687 23845 tablet_bootstrap.cc:492] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: No bootstrap required, opened a new log
I20260809 23:16:59.365427 23723 leader_election.cc:302] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.365337 23845 ts_tablet_manager.cc:1397] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: Time spent bootstrapping tablet: real 0.053s	user 0.005s	sys 0.012s
I20260809 23:16:59.366191 23901 raft_consensus.cc:2781] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.369240 23845 raft_consensus.cc:348] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.369895 23845 raft_consensus.cc:374] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.370162 23845 raft_consensus.cc:729] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.370847 23845 consensus_queue.cc:260] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.372117 23664 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 1.
I20260809 23:16:59.373004 23845 ts_tablet_manager.cc:1428] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.001s
I20260809 23:16:59.375415 23847 tablet_bootstrap.cc:654] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: Neither blocks nor log segments found. Creating new log.
I20260809 23:16:59.375825 23856 raft_consensus.cc:2781] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.376194 23856 raft_consensus.cc:482] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.376484 23905 raft_consensus.cc:482] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.376710 23856 raft_consensus.cc:3037] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.376210 23399 catalog_manager.cc:5495] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.376981 23905 raft_consensus.cc:504] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.379187 23905 leader_election.cc:288] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.380015 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.380473 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.380631 23546 raft_consensus.cc:2454] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 0.
I20260809 23:16:59.381099 23777 raft_consensus.cc:2454] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 0.
I20260809 23:16:59.382917 23901 raft_consensus.cc:686] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:16:59.383747 23847 tablet_bootstrap.cc:492] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: No bootstrap required, opened a new log
I20260809 23:16:59.384220 23847 ts_tablet_manager.cc:1397] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.004s
I20260809 23:16:59.384449 23618 leader_election.cc:302] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.387043 23905 raft_consensus.cc:2781] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.387385 23905 raft_consensus.cc:482] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.387692 23905 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.383777 23901 consensus_queue.cc:237] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.393317 23847 raft_consensus.cc:348] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.393769 23847 raft_consensus.cc:374] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.393939 23847 raft_consensus.cc:729] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.394615 23847 consensus_queue.cc:260] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.397065 23847 ts_tablet_manager.cc:1428] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260809 23:16:59.400358 23905 raft_consensus.cc:504] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.404942 23843 tablet_bootstrap.cc:492] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: No bootstrap required, opened a new log
I20260809 23:16:59.405594 23843 ts_tablet_manager.cc:1397] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: Time spent bootstrapping tablet: real 0.077s	user 0.010s	sys 0.000s
I20260809 23:16:59.407109 23905 leader_election.cc:288] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.408315 23843 raft_consensus.cc:348] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.408715 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.408969 23843 raft_consensus.cc:374] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:16:59.409251 23843 raft_consensus.cc:729] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Initialized, Role: FOLLOWER
I20260809 23:16:59.409291 23546 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.409516 23856 raft_consensus.cc:504] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.410032 23843 consensus_queue.cc:260] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.412248 23885 raft_consensus.cc:482] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.412590 23856 leader_election.cc:288] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:16:59.412694 23885 raft_consensus.cc:504] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.413286 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.413671 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d142bafc563b40aab31a0eb10e1e8ff5" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.414047 23777 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.414538 23551 raft_consensus.cc:3037] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.415247 23885 leader_election.cc:288] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.420068 23550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7958adffbd364c45ac3e90fcadf23355" is_pre_election: true
I20260809 23:16:59.420181 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d142bafc563b40aab31a0eb10e1e8ff5" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.421252 23664 raft_consensus.cc:3037] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.421311 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.422119 23776 raft_consensus.cc:2379] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68fa31cbc36c4524afcab09fe96b2be6 in current term 1: Already voted for candidate 7958adffbd364c45ac3e90fcadf23355 in this term.
I20260809 23:16:59.422189 23843 ts_tablet_manager.cc:1428] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.000s
I20260809 23:16:59.426606 23546 raft_consensus.cc:2454] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 1.
I20260809 23:16:59.427731 23618 leader_election.cc:302] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.428881 23905 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.429659 23618 leader_election.cc:302] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.430617 23885 raft_consensus.cc:2781] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.431370 23885 raft_consensus.cc:686] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.431495 23777 raft_consensus.cc:2454] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 1.
I20260809 23:16:59.432530 23885 consensus_queue.cc:237] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.426709 23664 raft_consensus.cc:2454] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 1.
I20260809 23:16:59.434329 23723 leader_election.cc:302] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.433892 23397 catalog_manager.cc:5495] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:59.435474 23901 raft_consensus.cc:2781] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.435952 23901 raft_consensus.cc:686] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:16:59.436723 23901 consensus_queue.cc:237] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.438158 23905 raft_consensus.cc:2726] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.421274 23551 raft_consensus.cc:2454] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 1.
I20260809 23:16:59.452085 23397 catalog_manager.cc:5495] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.452713 23396 catalog_manager.cc:5495] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:59.461113 23896 raft_consensus.cc:482] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.461601 23896 raft_consensus.cc:504] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.463763 23896 leader_election.cc:288] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.464546 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.465066 23664 raft_consensus.cc:2454] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.464880 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.465855 23777 raft_consensus.cc:2454] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 0.
I20260809 23:16:59.466178 23499 leader_election.cc:302] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.466972 23896 raft_consensus.cc:2781] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:16:59.467341 23896 raft_consensus.cc:482] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:16:59.467640 23896 raft_consensus.cc:3037] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.472127 23896 raft_consensus.cc:504] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.473723 23896 leader_election.cc:288] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.474539 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.475072 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef90253eba44d7bbe54c20ad4b9b399" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.475544 23664 raft_consensus.cc:3037] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.475805 23777 raft_consensus.cc:3037] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.480352 23664 raft_consensus.cc:2454] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.480705 23777 raft_consensus.cc:2454] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 1.
I20260809 23:16:59.481307 23499 leader_election.cc:302] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.481981 23896 raft_consensus.cc:2781] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:16:59.482414 23896 raft_consensus.cc:686] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.483210 23896 consensus_queue.cc:237] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.490705 23896 raft_consensus.cc:482] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:16:59.491132 23896 raft_consensus.cc:504] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.492779 23896 leader_election.cc:288] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.492245 23396 catalog_manager.cc:5495] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.493558 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6" is_pre_election: true
I20260809 23:16:59.494132 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0" is_pre_election: true
I20260809 23:16:59.495107 23777 raft_consensus.cc:2379] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7958adffbd364c45ac3e90fcadf23355 in current term 1: Already voted for candidate 68fa31cbc36c4524afcab09fe96b2be6 in this term.
I20260809 23:16:59.496114 23501 leader_election.cc:302] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7958adffbd364c45ac3e90fcadf23355; no voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.496776 23896 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.501878 23896 raft_consensus.cc:2726] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:16:59.514124 23885 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.528650 23849 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.612514 23896 consensus_queue.cc:1035] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.623062 23777 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:16:59.631219 23893 consensus_queue.cc:1035] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.643914 23664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:38188
I20260809 23:16:59.644538 23664 raft_consensus.cc:595] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.650763 23664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:38188
I20260809 23:16:59.651278 23664 raft_consensus.cc:595] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.674257 23777 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
mode: GRACEFUL
new_leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:55798
I20260809 23:16:59.674808 23777 raft_consensus.cc:595] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Received request to transfer leadership to TS 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.690770 23551 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:56730
I20260809 23:16:59.691336 23551 raft_consensus.cc:595] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Received request to transfer leadership to TS 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.696605 23893 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.701349 23777 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21430435848b4603896c5e59b1edf664"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
mode: GRACEFUL
new_leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:55798
I20260809 23:16:59.702131 23777 raft_consensus.cc:595] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Received request to transfer leadership to TS 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:16:59.705324 23908 raft_consensus.cc:979] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.705761 23908 raft_consensus.cc:1067] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Signalling peer 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.708299 23662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:38174
I20260809 23:16:59.708869 23662 raft_consensus.cc:482] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:59.709204 23662 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.709954 23850 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.717108 23850 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.717893 23662 raft_consensus.cc:504] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.721248 23664 raft_consensus.cc:1226] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Rejecting Update request from peer 7958adffbd364c45ac3e90fcadf23355 for earlier term 1. Current term is 2. Ops: []
I20260809 23:16:59.722677 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.723299 23551 raft_consensus.cc:3032] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:59.723524 23662 leader_election.cc:288] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.723625 23551 raft_consensus.cc:729] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.724068 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.724898 23776 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.724439 23551 consensus_queue.cc:260] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.725435 23856 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.725827 23551 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.735563 23776 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:16:59.737010 23618 leader_election.cc:302] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:16:59.737294 23551 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:16:59.738065 23849 raft_consensus.cc:2781] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:59.738591 23849 raft_consensus.cc:686] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.739593 23849 consensus_queue.cc:237] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.744273 23901 raft_consensus.cc:979] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: : Instructing follower 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.744796 23901 raft_consensus.cc:1067] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Signalling peer 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.749482 23905 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.749974 23662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:38180
I20260809 23:16:59.750564 23662 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:59.751031 23662 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.751754 23933 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:16:59.758952 23662 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.765650 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.770162 23662 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.772327 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.772923 23850 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.772933 23777 raft_consensus.cc:3032] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:59.774724 23777 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:16:59.775063 23397 catalog_manager.cc:5495] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 1 to 2, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.775678 23777 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.777994 23777 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.788877 23885 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.790512 23777 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:16:59.795979 23618 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 7958adffbd364c45ac3e90fcadf23355
I20260809 23:16:59.797035 23885 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:59.797526 23885 raft_consensus.cc:686] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.798286 23885 consensus_queue.cc:237] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.812883 23399 catalog_manager.cc:5495] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 1 to 2, leader changed from 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3) to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.835675 23901 consensus_queue.cc:1035] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.845641 23901 consensus_queue.cc:1035] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.853183 23856 raft_consensus.cc:979] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: : Instructing follower 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.853619 23901 raft_consensus.cc:1067] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Signalling peer 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:16:59.855151 23662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21430435848b4603896c5e59b1edf664"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:38180
I20260809 23:16:59.855705 23662 raft_consensus.cc:482] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:59.856052 23662 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.862358 23662 raft_consensus.cc:504] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.863998 23662 leader_election.cc:288] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.864784 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:16:59.865033 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21430435848b4603896c5e59b1edf664" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.865367 23551 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.865650 23777 raft_consensus.cc:3032] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:59.865988 23777 raft_consensus.cc:729] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:16:59.866710 23777 consensus_queue.cc:260] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.867791 23777 raft_consensus.cc:3037] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.871052 23551 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:16:59.871968 23618 leader_election.cc:302] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.872699 23777 raft_consensus.cc:2454] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:16:59.873003 23885 raft_consensus.cc:2781] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:59.873668 23885 raft_consensus.cc:686] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.874629 23885 consensus_queue.cc:237] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.881922 23399 catalog_manager.cc:5495] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 1 to 2, leader changed from 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3) to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:16:59.928345 23901 consensus_queue.cc:1035] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.935187 23905 raft_consensus.cc:979] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:16:59.935652 23905 raft_consensus.cc:1067] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:16:59.938596 23546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56712
I20260809 23:16:59.939379 23546 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:59.939759 23546 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.939970 23933 consensus_queue.cc:1035] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.945963 23905 consensus_queue.cc:1035] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.951457 23546 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.953982 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.954435 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.954747 23662 raft_consensus.cc:3032] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:59.955346 23546 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.955232 23662 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.955271 23777 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.956662 23662 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.960481 23905 raft_consensus.cc:979] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:16:59.961009 23849 raft_consensus.cc:1067] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:16:59.959996 23662 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.965360 23777 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 2.
I20260809 23:16:59.965487 23546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56712
I20260809 23:16:59.966109 23546 raft_consensus.cc:482] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:16:59.966470 23546 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.966434 23501 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:16:59.967353 23850 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:16:59.967823 23850 raft_consensus.cc:686] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:16:59.967949 23662 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 2.
I20260809 23:16:59.968660 23850 consensus_queue.cc:237] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:16:59.970883 23885 consensus_queue.cc:1035] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.974851 23546 raft_consensus.cc:504] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.977509 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:16:59.978097 23662 raft_consensus.cc:3032] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:16:59.978008 23776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "098773334a6148729ab7f54cc0cb7084" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:16:59.978308 23397 catalog_manager.cc:5495] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 1 to 2, leader changed from 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } }
I20260809 23:16:59.979771 23662 raft_consensus.cc:729] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:16:59.980633 23662 consensus_queue.cc:260] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:16:59.981041 23551 raft_consensus.cc:1226] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Rejecting Update request from peer 68fa31cbc36c4524afcab09fe96b2be6 for earlier term 1. Current term is 2. Ops: []
I20260809 23:16:59.981356 23850 consensus_queue.cc:1035] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:16:59.982162 23546 leader_election.cc:288] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:16:59.981596 23662 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:16:59.994586 23662 raft_consensus.cc:2454] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 2.
I20260809 23:17:00.002673 23499 leader_election.cc:302] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 5684fadd273849a5a6a7e1e281cf60b0
I20260809 23:17:00.004509 23896 raft_consensus.cc:2781] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:17:00.005048 23896 raft_consensus.cc:686] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:00.005465 23850 consensus_queue.cc:1035] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:00.006016 23896 consensus_queue.cc:237] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:00.020882 23399 catalog_manager.cc:5495] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 1 to 2, leader changed from 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:00.227715 23774 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.229434 23885 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.236801 23546 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:00.244819 23546 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.246178 23885 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.279989 23774 raft_consensus.cc:1261] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.281262 23885 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.289958 23546 raft_consensus.cc:1261] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.291443 23885 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.312601 23546 raft_consensus.cc:1261] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.313740 23885 consensus_queue.cc:1035] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.320932 23774 raft_consensus.cc:1261] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.322256 23885 consensus_queue.cc:1035] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.394485 23774 raft_consensus.cc:3037] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:00.401155 23774 raft_consensus.cc:1261] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.402558 23850 consensus_queue.cc:1035] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.412048 23664 raft_consensus.cc:1261] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.413318 23850 consensus_queue.cc:1035] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.421505 23664 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.423377 23893 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:00.430963 23774 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:00.432488 23981 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:02.748027 22524 auto_leader_rebalancer-test.cc:339] The leader distribution is 
I20260809 23:17:02.748751 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:02.768867 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:02.769387 23546 raft_consensus.cc:595] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Received request to transfer leadership to TS 5684fadd273849a5a6a7e1e281cf60b0
I20260809 23:17:02.771771 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eff367fda1d4949a94460835c37867a"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:02.772172 23546 raft_consensus.cc:595] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Received request to transfer leadership to TS 5684fadd273849a5a6a7e1e281cf60b0
I20260809 23:17:02.773074 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:17:02.773538 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:02.879117 23989 raft_consensus.cc:979] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:02.879731 23989 raft_consensus.cc:1067] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Signalling peer 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:02.881882 23774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6eff367fda1d4949a94460835c37867a"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:55778
I20260809 23:17:02.882455 23774 raft_consensus.cc:482] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:02.882848 23774 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:02.888016 23774 raft_consensus.cc:504] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:02.889710 23774 leader_election.cc:288] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 2 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:17:02.890719 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:02.890779 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:02.891500 23662 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:02.891494 23546 raft_consensus.cc:3032] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:17:02.892023 23546 raft_consensus.cc:729] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:02.892797 23546 consensus_queue.cc:260] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:02.893996 23546 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:02.898077 23662 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 2.
I20260809 23:17:02.899230 23723 leader_election.cc:302] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:02.900034 23983 raft_consensus.cc:2781] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:17:02.900975 23546 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 2.
I20260809 23:17:02.903359 23983 raft_consensus.cc:686] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:02.905649 23981 raft_consensus.cc:979] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:02.905297 23983 consensus_queue.cc:237] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:02.906138 23989 raft_consensus.cc:1067] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Signalling peer 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:02.907760 23774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:55778
I20260809 23:17:02.908217 23774 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:02.908540 23774 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:02.913668 23399 catalog_manager.cc:5495] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 1 to 2, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:02.914237 23774 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:02.917309 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:02.917928 23546 raft_consensus.cc:3032] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:17:02.917867 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:02.918309 23546 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:02.918537 23662 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:02.919133 23546 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:02.919971 23546 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:02.923219 23774 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 3 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725)
I20260809 23:17:02.926415 23662 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 3.
I20260809 23:17:02.927554 23546 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 3.
I20260809 23:17:02.927841 23723 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:02.928848 23983 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:17:02.929378 23983 raft_consensus.cc:686] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:02.930265 23983 consensus_queue.cc:237] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:02.941773 23399 catalog_manager.cc:5495] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 2 to 3, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:03.292565 23546 raft_consensus.cc:1261] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:03.294101 23983 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:03.308233 23662 raft_consensus.cc:1261] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:03.310227 23994 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:03.464685 23546 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:03.466547 23994 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:03.475689 23662 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:03.477468 24005 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:03.774528 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:03.780509 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:03.781230 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:04.782483 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:04.789223 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:04.789680 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:05.790735 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:05.797096 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:05.797544 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:06.798708 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:06.805546 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:06.806129 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:07.807446 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:07.813627 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:07.814122 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:08.815107 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:08.821277 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:08.821743 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:09.822973 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:09.828918 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:09.829414 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:10.830250 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:10.835731 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:10.836155 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:11.837108 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:11.843186 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:11.843760 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:12.878439 22524 auto_leader_rebalancer-test.cc:358] The leader distribution is 
I20260809 23:17:12.884514 23774 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:55798
I20260809 23:17:12.885668 23774 raft_consensus.cc:595] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:17:12.894569 23774 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6eff367fda1d4949a94460835c37867a"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:55798
I20260809 23:17:12.895267 23774 raft_consensus.cc:595] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:17:12.906715 23664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:38188
I20260809 23:17:12.907262 23664 raft_consensus.cc:595] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:17:12.912587 23664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:38188
I20260809 23:17:12.913003 23664 raft_consensus.cc:595] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:17:12.918591 23664 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
mode: GRACEFUL
new_leader_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:38188
I20260809 23:17:12.919317 23664 raft_consensus.cc:595] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Received request to transfer leadership to TS 7958adffbd364c45ac3e90fcadf23355
I20260809 23:17:13.283541 24036 raft_consensus.cc:979] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.284210 24036 raft_consensus.cc:1067] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.286145 23551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56712
I20260809 23:17:13.286725 23551 raft_consensus.cc:482] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:13.287124 23551 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.293128 23551 raft_consensus.cc:504] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.294929 23551 leader_election.cc:288] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:13.295923 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:13.295995 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeceab82bfe741b98138f9204f27e7b2" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:13.296720 23664 raft_consensus.cc:3032] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:17:13.296777 23774 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.297001 23664 raft_consensus.cc:729] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:17:13.297823 23664 consensus_queue.cc:260] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.298892 23664 raft_consensus.cc:3037] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.302163 23774 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.303445 23501 leader_election.cc:302] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:17:13.304297 24034 raft_consensus.cc:2781] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:17:13.304737 23664 raft_consensus.cc:2454] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.305310 24034 raft_consensus.cc:686] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:13.306305 24034 consensus_queue.cc:237] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.313957 23399 catalog_manager.cc:5495] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 2 to 3, leader changed from 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:13.332906 24037 raft_consensus.cc:979] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.333454 24036 raft_consensus.cc:1067] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.334959 23551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56712
I20260809 23:17:13.335402 23551 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:13.335758 23551 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.342056 23551 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.344772 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:13.345263 23551 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:13.345292 23664 raft_consensus.cc:3032] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:17:13.346184 23664 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:17:13.346745 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:13.347380 23774 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.346961 23664 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.348383 23664 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.356690 23774 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.357654 23664 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.358230 23501 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:17:13.359145 24034 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:17:13.359602 24048 raft_consensus.cc:979] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.359654 24034 raft_consensus.cc:686] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:13.360069 24048 raft_consensus.cc:1067] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.360970 24034 consensus_queue.cc:237] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.361615 23551 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6eff367fda1d4949a94460835c37867a"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56722
I20260809 23:17:13.362200 23551 raft_consensus.cc:482] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:13.362991 23551 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.363900 24047 raft_consensus.cc:979] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.364537 24048 raft_consensus.cc:1067] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.366542 23546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56722
I20260809 23:17:13.367089 23546 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:13.367416 23546 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:13.370694 23399 catalog_manager.cc:5495] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 2 to 3, leader changed from 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:13.371369 23551 raft_consensus.cc:504] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.373781 23551 leader_election.cc:288] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:13.373960 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:13.374191 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6eff367fda1d4949a94460835c37867a" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:13.374581 23664 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.374676 23774 raft_consensus.cc:3032] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:17:13.372947 23546 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:13.375025 23774 raft_consensus.cc:729] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:13.375862 23774 consensus_queue.cc:260] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.377543 23774 raft_consensus.cc:3037] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:13.378290 23546 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 4 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:13.378330 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:13.379624 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:13.379953 23662 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:13.380241 23777 raft_consensus.cc:3032] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:17:13.380589 23777 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:13.381496 23664 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.381320 23777 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:13.382350 23777 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:13.384176 23499 leader_election.cc:302] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:17:13.385572 24034 raft_consensus.cc:2781] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:17:13.386651 24034 raft_consensus.cc:686] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:13.387481 24034 consensus_queue.cc:237] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.388357 23662 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 4.
I20260809 23:17:13.389374 23499 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68fa31cbc36c4524afcab09fe96b2be6, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:17:13.389569 23777 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 4.
I20260809 23:17:13.390419 24039 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:17:13.391156 23774 raft_consensus.cc:2454] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 3.
I20260809 23:17:13.392488 24039 raft_consensus.cc:686] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:13.393474 24039 consensus_queue.cc:237] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:13.396994 23399 catalog_manager.cc:5495] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 2 to 3, leader changed from 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:13.410648 23399 catalog_manager.cc:5495] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 3 to 4, leader changed from 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 4 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:13.468482 24037 raft_consensus.cc:979] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: : Instructing follower 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.468930 24036 raft_consensus.cc:1067] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Signalling peer 7958adffbd364c45ac3e90fcadf23355 to start an election
I20260809 23:17:13.470670 23546 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
 from {username='slave'} at 127.0.0.1:56712
I20260809 23:17:13.471161 23546 raft_consensus.cc:482] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:13.471455 23546 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:13.475800 23546 raft_consensus.cc:504] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.477135 23546 leader_election.cc:288] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:13.478191 23777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:13.478648 23777 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:13.478394 23662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "7958adffbd364c45ac3e90fcadf23355" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:13.479293 23662 raft_consensus.cc:3032] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:17:13.479586 23662 raft_consensus.cc:729] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:17:13.480413 23662 consensus_queue.cc:260] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.481546 23662 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:13.484753 23777 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 2.
I20260809 23:17:13.485850 23501 leader_election.cc:302] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 7958adffbd364c45ac3e90fcadf23355; no voters: 
I20260809 23:17:13.486730 24039 raft_consensus.cc:2781] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:17:13.487321 24039 raft_consensus.cc:686] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Becoming Leader. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:13.488153 24039 consensus_queue.cc:237] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:13.488353 23662 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7958adffbd364c45ac3e90fcadf23355 in term 2.
I20260809 23:17:13.495695 23399 catalog_manager.cc:5495] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 reported cstate change: term changed from 1 to 2, leader changed from 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2) to 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "7958adffbd364c45ac3e90fcadf23355" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:13.768226 23777 raft_consensus.cc:1261] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.769510 24039 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:13.779165 23662 raft_consensus.cc:1261] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.781121 24034 consensus_queue.cc:1035] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:13.784878 23777 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.786510 24039 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:13.789891 23664 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:17:13.791405 24034 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:13.803884 23664 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.805514 24051 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:13.810822 23774 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:17:13.813671 24039 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:13.891615 23777 raft_consensus.cc:1261] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.892791 24039 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:13.902120 23662 raft_consensus.cc:1261] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:13.903846 24051 consensus_queue.cc:1035] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:14.000049 23662 raft_consensus.cc:1261] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:14.001734 24051 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:14.011291 23777 raft_consensus.cc:1261] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 7958adffbd364c45ac3e90fcadf23355: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:14.012511 24072 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:15.964237 22524 auto_leader_rebalancer-test.cc:373] The leader distribution is 
I20260809 23:17:15.965302 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:15.974040 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:15.974762 23546 raft_consensus.cc:595] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Received request to transfer leadership to TS 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:17:15.977963 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:15.978747 23546 raft_consensus.cc:595] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 LEADER]: Received request to transfer leadership to TS 5684fadd273849a5a6a7e1e281cf60b0
I20260809 23:17:15.981667 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:15.982167 23546 raft_consensus.cc:595] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Received request to transfer leadership to TS 5684fadd273849a5a6a7e1e281cf60b0
I20260809 23:17:15.985213 23546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a"
dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
mode: GRACEFUL
new_leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:56736
I20260809 23:17:15.985783 23546 raft_consensus.cc:595] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Received request to transfer leadership to TS 68fa31cbc36c4524afcab09fe96b2be6
I20260809 23:17:15.987241 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:17:15.987757 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:16.034794 24039 raft_consensus.cc:979] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:16.035388 24039 raft_consensus.cc:1067] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Signalling peer 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:16.037307 23774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:55778
I20260809 23:17:16.037784 23774 raft_consensus.cc:482] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:16.038107 23774 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:16.045650 24086 raft_consensus.cc:979] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:17:16.046166 24039 raft_consensus.cc:1067] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Signalling peer 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:17:16.047439 23774 raft_consensus.cc:504] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.048393 23662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:38174
I20260809 23:17:16.048852 23662 raft_consensus.cc:482] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:16.049119 23662 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:16.050020 23774 leader_election.cc:288] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 4 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899)
I20260809 23:17:16.050982 23546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:16.051285 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422b11fb9ebc495aa825f4c1658b93ff" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:16.051723 23546 raft_consensus.cc:3032] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:17:16.051869 23664 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:16.052052 23546 raft_consensus.cc:729] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:16.052831 23546 consensus_queue.cc:260] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.054244 23546 raft_consensus.cc:3037] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:17:16.055763 23662 raft_consensus.cc:504] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.057729 23662 leader_election.cc:288] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 3 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:16.058758 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:16.059602 23774 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:16.060330 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2005b0f7b5e449a0a1f4c66bacae226f" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:16.060693 23664 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 4.
I20260809 23:17:16.061327 23551 raft_consensus.cc:3032] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:17:16.061741 23551 raft_consensus.cc:729] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:16.062740 23723 leader_election.cc:302] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:16.063541 23546 raft_consensus.cc:2454] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 4.
I20260809 23:17:16.063074 23551 consensus_queue.cc:260] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.064133 24048 raft_consensus.cc:2781] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:17:16.065546 23551 raft_consensus.cc:3037] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:17:16.065840 24048 raft_consensus.cc:686] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:16.066989 24048 consensus_queue.cc:237] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.069455 23774 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 3.
I20260809 23:17:16.071202 23618 leader_election.cc:302] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:16.072285 24036 raft_consensus.cc:2781] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:17:16.074726 23551 raft_consensus.cc:2454] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 3.
I20260809 23:17:16.075588 23399 catalog_manager.cc:5495] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 3 to 4, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:16.076502 24036 raft_consensus.cc:686] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:17:16.078246 24036 consensus_queue.cc:237] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.089010 23397 catalog_manager.cc:5495] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 2 to 3, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 3 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:16.097208 24039 raft_consensus.cc:979] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:16.097589 24039 raft_consensus.cc:1067] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 LEADER]: Signalling peer 5684fadd273849a5a6a7e1e281cf60b0 to start an election
I20260809 23:17:16.099493 23774 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9"
dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
 from {username='slave'} at 127.0.0.1:55778
I20260809 23:17:16.100004 23774 raft_consensus.cc:482] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:16.100281 23774 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:17:16.106140 23774 raft_consensus.cc:504] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:16.108211 23774 leader_election.cc:288] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 5 election: Requested vote from peers 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2:32899), 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725)
I20260809 23:17:16.109357 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:16.109723 23664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5bc3f286944be4bddba0edaeec36b9" candidate_uuid: "5684fadd273849a5a6a7e1e281cf60b0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
I20260809 23:17:16.109948 23551 raft_consensus.cc:3032] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:17:16.110291 23664 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:17:16.110309 23551 raft_consensus.cc:729] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:16.111204 23551 consensus_queue.cc:260] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:16.112263 23551 raft_consensus.cc:3037] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:17:16.114954 23664 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 5.
I20260809 23:17:16.115928 23723 leader_election.cc:302] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:16.116712 24048 raft_consensus.cc:2781] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:17:16.117102 24048 raft_consensus.cc:686] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 5 LEADER]: Becoming Leader. State: Replica: 5684fadd273849a5a6a7e1e281cf60b0, State: Running, Role: LEADER
I20260809 23:17:16.117221 23551 raft_consensus.cc:2454] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5684fadd273849a5a6a7e1e281cf60b0 in term 5.
I20260809 23:17:16.118101 24048 consensus_queue.cc:237] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } }
I20260809 23:17:16.125530 23397 catalog_manager.cc:5495] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 reported cstate change: term changed from 4 to 5, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3). New cstate: current_term: 5 leader_uuid: "5684fadd273849a5a6a7e1e281cf60b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:16.411650 24039 raft_consensus.cc:979] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: : Instructing follower 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:17:16.412222 24039 raft_consensus.cc:1067] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Signalling peer 68fa31cbc36c4524afcab09fe96b2be6 to start an election
I20260809 23:17:16.413854 23664 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a"
dest_uuid: "68fa31cbc36c4524afcab09fe96b2be6"
 from {username='slave'} at 127.0.0.1:38174
I20260809 23:17:16.414413 23664 raft_consensus.cc:482] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:17:16.414662 23664 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:16.421939 23664 raft_consensus.cc:504] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.423516 23664 leader_election.cc:288] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Requested vote from peers 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725), 5684fadd273849a5a6a7e1e281cf60b0 (127.21.255.3:33395)
I20260809 23:17:16.424304 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7958adffbd364c45ac3e90fcadf23355"
I20260809 23:17:16.424638 23774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac19379b45ba4438a63d37c3fb7d6f0a" candidate_uuid: "68fa31cbc36c4524afcab09fe96b2be6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5684fadd273849a5a6a7e1e281cf60b0"
I20260809 23:17:16.425014 23551 raft_consensus.cc:3032] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:17:16.425422 23774 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:16.425446 23551 raft_consensus.cc:729] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7958adffbd364c45ac3e90fcadf23355, State: Running, Role: LEADER
I20260809 23:17:16.426540 23551 consensus_queue.cc:260] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.428643 23551 raft_consensus.cc:3037] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:17:16.432245 23774 raft_consensus.cc:2454] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:17:16.433281 23618 leader_election.cc:302] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5684fadd273849a5a6a7e1e281cf60b0, 68fa31cbc36c4524afcab09fe96b2be6; no voters: 
I20260809 23:17:16.434113 24036 raft_consensus.cc:2781] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:17:16.434669 24036 raft_consensus.cc:686] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Becoming Leader. State: Replica: 68fa31cbc36c4524afcab09fe96b2be6, State: Running, Role: LEADER
I20260809 23:17:16.435784 24036 consensus_queue.cc:237] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } }
I20260809 23:17:16.437260 23551 raft_consensus.cc:2454] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fa31cbc36c4524afcab09fe96b2be6 in term 2.
I20260809 23:17:16.443933 23397 catalog_manager.cc:5495] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 reported cstate change: term changed from 1 to 2, leader changed from 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1) to 68fa31cbc36c4524afcab09fe96b2be6 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "68fa31cbc36c4524afcab09fe96b2be6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:16.496791 23551 raft_consensus.cc:1261] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:16.498297 24036 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:16.509387 23774 raft_consensus.cc:1261] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:17:16.511402 24104 consensus_queue.cc:1035] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:16.555434 23551 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 5 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:17:16.556703 24048 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:17:16.570549 23664 raft_consensus.cc:1261] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 5 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:17:16.572455 24048 consensus_queue.cc:1035] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:17:16.625375 23551 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:17:16.627081 24048 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:16.636389 23664 raft_consensus.cc:1261] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Refusing update from remote peer 5684fadd273849a5a6a7e1e281cf60b0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:17:16.637884 24111 consensus_queue.cc:1035] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fa31cbc36c4524afcab09fe96b2be6" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 32899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:16.892992 23774 raft_consensus.cc:1261] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:16.894274 24104 consensus_queue.cc:1035] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5684fadd273849a5a6a7e1e281cf60b0" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:16.908262 23551 raft_consensus.cc:1261] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Refusing update from remote peer 68fa31cbc36c4524afcab09fe96b2be6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:17:16.909534 24036 consensus_queue.cc:1035] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7958adffbd364c45ac3e90fcadf23355" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 45725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:16.988620 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:16.994009 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:16.994505 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:17.995400 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:18.001341 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:18.001672 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:19.002486 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:19.007978 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:19.008343 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:20.009229 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:20.014705 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:20.015130 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:21.015987 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:21.022603 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:21.023283 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:22.024158 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:22.033367 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:22.034080 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:23.034904 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:23.041324 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:23.041770 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:24.042685 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:24.049878 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:24.050329 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:25.051216 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:17:25.056619 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:17:25.057050 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:17:26.091357 22524 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260809 23:17:26.092047 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
I20260809 23:17:26.125870 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:17:26.126995 22524 tablet_replica.cc:330] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.128212 22524 raft_consensus.cc:2229] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.129422 22524 raft_consensus.cc:2258] T d142bafc563b40aab31a0eb10e1e8ff5 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.131969 22524 tablet_replica.cc:330] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.132536 22524 raft_consensus.cc:2229] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.132994 22524 raft_consensus.cc:2258] T 2005b0f7b5e449a0a1f4c66bacae226f P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.134750 22524 tablet_replica.cc:330] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.135342 22524 raft_consensus.cc:2229] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.135782 22524 raft_consensus.cc:2258] T 21430435848b4603896c5e59b1edf664 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.137434 22524 tablet_replica.cc:330] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.138135 22524 raft_consensus.cc:2229] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.138787 22524 raft_consensus.cc:2258] T 2d5bc3f286944be4bddba0edaeec36b9 P 7958adffbd364c45ac3e90fcadf23355 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.141448 22524 tablet_replica.cc:330] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.142084 22524 raft_consensus.cc:2229] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.142581 22524 raft_consensus.cc:2258] T ac19379b45ba4438a63d37c3fb7d6f0a P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.144747 22524 tablet_replica.cc:330] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.145341 22524 raft_consensus.cc:2229] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.146317 22524 raft_consensus.cc:2258] T aeceab82bfe741b98138f9204f27e7b2 P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.148969 22524 tablet_replica.cc:330] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.149538 22524 raft_consensus.cc:2229] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.150420 22524 raft_consensus.cc:2258] T fef90253eba44d7bbe54c20ad4b9b399 P 7958adffbd364c45ac3e90fcadf23355 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.152195 22524 tablet_replica.cc:330] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.152719 22524 raft_consensus.cc:2229] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.153832 22524 raft_consensus.cc:2258] T 6eff367fda1d4949a94460835c37867a P 7958adffbd364c45ac3e90fcadf23355 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.155695 22524 tablet_replica.cc:330] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.156175 22524 raft_consensus.cc:2229] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.156616 22524 raft_consensus.cc:2258] T 422b11fb9ebc495aa825f4c1658b93ff P 7958adffbd364c45ac3e90fcadf23355 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.158394 22524 tablet_replica.cc:330] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355: stopping tablet replica
I20260809 23:17:26.158845 22524 raft_consensus.cc:2229] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.159678 22524 raft_consensus.cc:2258] T 098773334a6148729ab7f54cc0cb7084 P 7958adffbd364c45ac3e90fcadf23355 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:17:26.166031 23618 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20260809 23:17:26.169505 23618 consensus_peers.cc:487] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 -> Peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Couldn't send request to peer 7958adffbd364c45ac3e90fcadf23355. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:17:26.176378 23618 consensus_peers.cc:487] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 -> Peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Couldn't send request to peer 7958adffbd364c45ac3e90fcadf23355. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:26.188824 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
I20260809 23:17:26.211062 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
I20260809 23:17:26.245554 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:17:26.246567 22524 tablet_replica.cc:330] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.247593 22524 raft_consensus.cc:2229] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.248172 22524 raft_consensus.cc:2258] T d142bafc563b40aab31a0eb10e1e8ff5 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.250103 22524 tablet_replica.cc:330] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.250629 22524 raft_consensus.cc:2229] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.251153 22524 raft_consensus.cc:2258] T 098773334a6148729ab7f54cc0cb7084 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.252959 22524 tablet_replica.cc:330] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.253526 22524 raft_consensus.cc:2229] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.254348 22524 raft_consensus.cc:2258] T ac19379b45ba4438a63d37c3fb7d6f0a P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.256075 22524 tablet_replica.cc:330] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.256573 22524 raft_consensus.cc:2229] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.257344 22524 raft_consensus.cc:2258] T 21430435848b4603896c5e59b1edf664 P 68fa31cbc36c4524afcab09fe96b2be6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.259873 22524 tablet_replica.cc:330] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.260516 22524 raft_consensus.cc:2229] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.261031 22524 raft_consensus.cc:2258] T 6eff367fda1d4949a94460835c37867a P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.263103 22524 tablet_replica.cc:330] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.263684 22524 raft_consensus.cc:2229] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.264226 22524 raft_consensus.cc:2258] T fef90253eba44d7bbe54c20ad4b9b399 P 68fa31cbc36c4524afcab09fe96b2be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.266458 22524 tablet_replica.cc:330] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.267115 22524 raft_consensus.cc:2229] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.267671 22524 raft_consensus.cc:2258] T 2d5bc3f286944be4bddba0edaeec36b9 P 68fa31cbc36c4524afcab09fe96b2be6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.270051 22524 tablet_replica.cc:330] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.270792 22524 raft_consensus.cc:2229] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.271440 22524 raft_consensus.cc:2258] T 422b11fb9ebc495aa825f4c1658b93ff P 68fa31cbc36c4524afcab09fe96b2be6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.273335 22524 tablet_replica.cc:330] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.273821 22524 raft_consensus.cc:2229] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.274684 22524 raft_consensus.cc:2258] T 2005b0f7b5e449a0a1f4c66bacae226f P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.276306 22524 tablet_replica.cc:330] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6: stopping tablet replica
I20260809 23:17:26.276930 22524 raft_consensus.cc:2229] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.277348 22524 raft_consensus.cc:2258] T aeceab82bfe741b98138f9204f27e7b2 P 68fa31cbc36c4524afcab09fe96b2be6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:17:26.300745 23724 consensus_peers.cc:487] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 -> Peer 7958adffbd364c45ac3e90fcadf23355 (127.21.255.1:45725): Couldn't send request to peer 7958adffbd364c45ac3e90fcadf23355. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:45725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:26.309859 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
I20260809 23:17:26.334139 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
I20260809 23:17:26.370038 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:17:26.371024 22524 tablet_replica.cc:330] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.371978 22524 raft_consensus.cc:2229] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.372714 22524 raft_consensus.cc:2258] T ac19379b45ba4438a63d37c3fb7d6f0a P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.374662 22524 tablet_replica.cc:330] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.375311 22524 raft_consensus.cc:2229] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.375802 22524 raft_consensus.cc:2258] T 098773334a6148729ab7f54cc0cb7084 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.377589 22524 tablet_replica.cc:330] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.378057 22524 raft_consensus.cc:2229] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.379103 22524 raft_consensus.cc:2258] T d142bafc563b40aab31a0eb10e1e8ff5 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.380697 22524 tablet_replica.cc:330] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.381186 22524 raft_consensus.cc:2229] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.381624 22524 raft_consensus.cc:2258] T 21430435848b4603896c5e59b1edf664 P 5684fadd273849a5a6a7e1e281cf60b0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.383247 22524 tablet_replica.cc:330] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.383713 22524 raft_consensus.cc:2229] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.384158 22524 raft_consensus.cc:2258] T 6eff367fda1d4949a94460835c37867a P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.385628 22524 tablet_replica.cc:330] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.386080 22524 raft_consensus.cc:2229] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.386482 22524 raft_consensus.cc:2258] T fef90253eba44d7bbe54c20ad4b9b399 P 5684fadd273849a5a6a7e1e281cf60b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.388052 22524 tablet_replica.cc:330] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.388533 22524 raft_consensus.cc:2229] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 5 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.389339 22524 raft_consensus.cc:2258] T 2d5bc3f286944be4bddba0edaeec36b9 P 5684fadd273849a5a6a7e1e281cf60b0 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.390913 22524 tablet_replica.cc:330] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.391392 22524 raft_consensus.cc:2229] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.392127 22524 raft_consensus.cc:2258] T 422b11fb9ebc495aa825f4c1658b93ff P 5684fadd273849a5a6a7e1e281cf60b0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.393723 22524 tablet_replica.cc:330] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.394161 22524 raft_consensus.cc:2229] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.394584 22524 raft_consensus.cc:2258] T aeceab82bfe741b98138f9204f27e7b2 P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.396134 22524 tablet_replica.cc:330] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0: stopping tablet replica
I20260809 23:17:26.396570 22524 raft_consensus.cc:2229] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:26.397001 22524 raft_consensus.cc:2258] T 2005b0f7b5e449a0a1f4c66bacae226f P 5684fadd273849a5a6a7e1e281cf60b0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.429849 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:17:26.457899 22524 master.cc:537] Master@127.21.255.62:37439 shutting down...
I20260809 23:17:26.495030 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:17:26.495615 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:26.495978 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2e910adb349d488cbb05cd295f3987b1: stopping tablet replica
I20260809 23:17:26.515100 22524 master.cc:552] Master@127.21.255.62:37439 shutdown complete.
5684fadd273849a5a6a7e1e281cf60b0  1
68fa31cbc36c4524afcab09fe96b2be6  4
7958adffbd364c45ac3e90fcadf23355  5
5684fadd273849a5a6a7e1e281cf60b0  3
68fa31cbc36c4524afcab09fe96b2be6  4
7958adffbd364c45ac3e90fcadf23355  3
5684fadd273849a5a6a7e1e281cf60b0  1
68fa31cbc36c4524afcab09fe96b2be6  1
7958adffbd364c45ac3e90fcadf23355  8
5684fadd273849a5a6a7e1e281cf60b0  3
68fa31cbc36c4524afcab09fe96b2be6  3
7958adffbd364c45ac3e90fcadf23355  4
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28746 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260809 23:17:26.562243 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:37921
I20260809 23:17:26.563439 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:17:26.569618 24133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.569625 24134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.571009 24136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:26.571420 22524 server_base.cc:1034] running on GCE node
I20260809 23:17:26.572402 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:17:26.572619 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:17:26.572794 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317446572767 us; error 0 us; skew 500 ppm
I20260809 23:17:26.573309 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:17:26.576063 22524 webserver.cc:460] Webserver started at http://127.21.255.62:33691/ using document root <none> and password file <none>
I20260809 23:17:26.576552 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:17:26.576761 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:17:26.577072 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:17:26.578307 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "e3357d0fe4ef46db9a4739044fa527d5"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.582818 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260809 23:17:26.586341 24143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.587216 22524 fs_manager.cc:725] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260809 23:17:26.587677 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "e3357d0fe4ef46db9a4739044fa527d5"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.588004 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:17:26.604683 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:17:26.607522 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:17:26.653895 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:37921
I20260809 23:17:26.654016 24220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:37921 every 8 connection(s)
I20260809 23:17:26.658890 24221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:26.668677 24221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Bootstrap starting.
I20260809 23:17:26.673089 24221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:26.677314 24221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: No bootstrap required, opened a new log
I20260809 23:17:26.680310 24221 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER }
I20260809 23:17:26.680799 24221 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:26.681093 24221 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3357d0fe4ef46db9a4739044fa527d5, State: Initialized, Role: FOLLOWER
I20260809 23:17:26.681713 24221 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER }
I20260809 23:17:26.682165 24221 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:17:26.682394 24221 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:17:26.682699 24221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:26.686491 24221 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER }
I20260809 23:17:26.687039 24221 leader_election.cc:302] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3357d0fe4ef46db9a4739044fa527d5; no voters: 
I20260809 23:17:26.688370 24221 leader_election.cc:288] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:17:26.688611 24230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:26.690294 24230 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 LEADER]: Becoming Leader. State: Replica: e3357d0fe4ef46db9a4739044fa527d5, State: Running, Role: LEADER
I20260809 23:17:26.691078 24230 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER }
I20260809 23:17:26.691987 24221 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:17:26.695958 24232 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3357d0fe4ef46db9a4739044fa527d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER } }
I20260809 23:17:26.696429 24232 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [sys.catalog]: This master's current role is: LEADER
I20260809 23:17:26.696075 24234 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3357d0fe4ef46db9a4739044fa527d5. Latest consensus state: current_term: 1 leader_uuid: "e3357d0fe4ef46db9a4739044fa527d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3357d0fe4ef46db9a4739044fa527d5" member_type: VOTER } }
I20260809 23:17:26.696769 24234 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [sys.catalog]: This master's current role is: LEADER
I20260809 23:17:26.699774 24238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:17:26.706409 24238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:17:26.708786 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:17:26.715929 24238 catalog_manager.cc:1347] Generated new cluster ID: 20e0ed93d21f41f78545910de72bcf7e
I20260809 23:17:26.716208 24238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:17:26.733425 24238 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:17:26.735010 24238 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:17:26.753041 24238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Generated new TSK 0
I20260809 23:17:26.753906 24238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:17:26.776276 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:17:26.782351 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.783766 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.785312 24257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:26.786753 22524 server_base.cc:1034] running on GCE node
I20260809 23:17:26.787808 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:17:26.788029 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:17:26.788205 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317446788190 us; error 0 us; skew 500 ppm
I20260809 23:17:26.788743 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:17:26.791591 22524 webserver.cc:460] Webserver started at http://127.21.255.1:33743/ using document root <none> and password file <none>
I20260809 23:17:26.792147 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:17:26.792358 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:17:26.792631 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:17:26.793859 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "60d5c69728134c8bb918a900eb6a8c21"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.798709 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20260809 23:17:26.802394 24262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.803623 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:17:26.804688 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "60d5c69728134c8bb918a900eb6a8c21"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.805034 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:17:26.821635 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:17:26.822809 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:17:26.824409 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:17:26.826597 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:17:26.826822 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.827085 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:17:26.827237 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.877305 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:36681
I20260809 23:17:26.877411 24355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:36681 every 8 connection(s)
I20260809 23:17:26.882336 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:17:26.890429 24364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.893435 24365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:26.895447 24367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:26.895680 22524 server_base.cc:1034] running on GCE node
I20260809 23:17:26.896872 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:17:26.897059 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:17:26.897217 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317446897201 us; error 0 us; skew 500 ppm
I20260809 23:17:26.898002 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:17:26.900638 24356 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37921
I20260809 23:17:26.901140 24356 heartbeater.cc:464] Registering TS with master...
I20260809 23:17:26.901168 22524 webserver.cc:460] Webserver started at http://127.21.255.2:42835/ using document root <none> and password file <none>
I20260809 23:17:26.901824 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:17:26.902038 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:17:26.901913 24356 heartbeater.cc:511] Master 127.21.255.62:37921 requested a full tablet report, sending...
I20260809 23:17:26.902377 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:17:26.903791 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.904435 24171 ts_manager.cc:194] Registered new tserver with Master: 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:26.907423 24171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35202
I20260809 23:17:26.910812 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:17:26.914549 24375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.915735 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:17:26.916059 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
format_stamp: "Formatted at 2026-08-09 23:17:26 on dist-test-slave-drl0"
I20260809 23:17:26.916353 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:17:26.933727 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:17:26.935160 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:17:26.936718 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:17:26.938984 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:17:26.939201 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.939461 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:17:26.939628 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:26.987720 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:42099
I20260809 23:17:26.987808 24469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:42099 every 8 connection(s)
I20260809 23:17:26.993119 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:17:27.001365 24474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:27.004128 24475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:27.005527 22524 server_base.cc:1034] running on GCE node
W20260809 23:17:27.007560 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:27.008911 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:17:27.009249 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:17:27.009473 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317447009454 us; error 0 us; skew 500 ppm
I20260809 23:17:27.010290 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:17:27.011581 24470 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37921
I20260809 23:17:27.012110 24470 heartbeater.cc:464] Registering TS with master...
I20260809 23:17:27.012905 24470 heartbeater.cc:511] Master 127.21.255.62:37921 requested a full tablet report, sending...
I20260809 23:17:27.013738 22524 webserver.cc:460] Webserver started at http://127.21.255.3:41253/ using document root <none> and password file <none>
I20260809 23:17:27.014544 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:17:27.014837 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:17:27.015324 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:17:27.015470 24171 ts_manager.cc:194] Registered new tserver with Master: ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:27.017453 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "808fed3ed3f04882ad757a3db9abfdd9"
format_stamp: "Formatted at 2026-08-09 23:17:27 on dist-test-slave-drl0"
I20260809 23:17:27.018179 24171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35216
I20260809 23:17:27.023943 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.005s
I20260809 23:17:27.027827 24490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:27.028795 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260809 23:17:27.029114 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "808fed3ed3f04882ad757a3db9abfdd9"
format_stamp: "Formatted at 2026-08-09 23:17:27 on dist-test-slave-drl0"
I20260809 23:17:27.029367 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:17:27.050421 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:17:27.051671 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:17:27.053102 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:17:27.055392 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:17:27.055861 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:27.056380 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:17:27.056630 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:27.103417 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:37765
I20260809 23:17:27.103538 24579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:37765 every 8 connection(s)
I20260809 23:17:27.120622 24580 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37921
I20260809 23:17:27.121049 24580 heartbeater.cc:464] Registering TS with master...
I20260809 23:17:27.121742 24580 heartbeater.cc:511] Master 127.21.255.62:37921 requested a full tablet report, sending...
I20260809 23:17:27.123698 24171 ts_manager.cc:194] Registered new tserver with Master: 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:27.124259 22524 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017631848s
I20260809 23:17:27.125366 24171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35222
I20260809 23:17:27.127411 22524 test_util.cc:268] Using random seed: 1894015323
I20260809 23:17:27.156490 24171 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35234:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:17:27.162756 24171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:17:27.388332 24420 tablet_service.cc:1467] Processing CreateTablet for tablet 48af4d51aa8f41a486c4fe6e7541700e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:17:27.388423 24419 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2662351f0e4b14bede1be0deaba6c8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:17:27.390604 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48af4d51aa8f41a486c4fe6e7541700e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.391409 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2662351f0e4b14bede1be0deaba6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.393157 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 90dd68b3876b4b75aea408edfa68d5e5 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:17:27.394642 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dd68b3876b4b75aea408edfa68d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.396131 24412 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcb08685ecd486c8f1d951336f39982 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:17:27.399178 24414 tablet_service.cc:1467] Processing CreateTablet for tablet 974355bb93274291bb6be768b656cc67 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:17:27.401788 24409 tablet_service.cc:1467] Processing CreateTablet for tablet ba62e1cbbfd84b86a817f9118be39b00 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:17:27.408267 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2662351f0e4b14bede1be0deaba6c8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:17:27.409554 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2662351f0e4b14bede1be0deaba6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.412026 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 48af4d51aa8f41a486c4fe6e7541700e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:17:27.413231 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48af4d51aa8f41a486c4fe6e7541700e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.414028 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcb08685ecd486c8f1d951336f39982 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:17:27.416594 24526 tablet_service.cc:1467] Processing CreateTablet for tablet bc822924176247c7abbc4a4756daf3a6 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:17:27.391031 24417 tablet_service.cc:1467] Processing CreateTablet for tablet cd2b577e2c3e4021bc8ebdfa3462437d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:17:27.418712 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 974355bb93274291bb6be768b656cc67 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:17:27.420725 24527 tablet_service.cc:1467] Processing CreateTablet for tablet ba62e1cbbfd84b86a817f9118be39b00 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:17:27.423144 24531 tablet_service.cc:1467] Processing CreateTablet for tablet cd2b577e2c3e4021bc8ebdfa3462437d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:17:27.415078 24524 tablet_service.cc:1467] Processing CreateTablet for tablet 9429f7311dd744bea4301956e143e9a7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:17:27.422706 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 90dd68b3876b4b75aea408edfa68d5e5 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:17:27.427305 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcb08685ecd486c8f1d951336f39982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.427538 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcb08685ecd486c8f1d951336f39982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.428416 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9429f7311dd744bea4301956e143e9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.430809 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba62e1cbbfd84b86a817f9118be39b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.431192 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc822924176247c7abbc4a4756daf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.435348 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba62e1cbbfd84b86a817f9118be39b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.435514 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974355bb93274291bb6be768b656cc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.436759 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974355bb93274291bb6be768b656cc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.436951 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2b577e2c3e4021bc8ebdfa3462437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.438277 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2b577e2c3e4021bc8ebdfa3462437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.440546 24407 tablet_service.cc:1467] Processing CreateTablet for tablet bc822924176247c7abbc4a4756daf3a6 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:17:27.442257 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc822924176247c7abbc4a4756daf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.444626 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dd68b3876b4b75aea408edfa68d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.459856 24405 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:17:27.461429 24405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.464089 24404 tablet_service.cc:1467] Processing CreateTablet for tablet b1115ecb68bd4c399da85a680700bb7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:17:27.465368 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1115ecb68bd4c399da85a680700bb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.480629 24522 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:17:27.482050 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.483523 24419 tablet_service.cc:1467] Processing CreateTablet for tablet 9429f7311dd744bea4301956e143e9a7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:17:27.484750 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9429f7311dd744bea4301956e143e9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.500116 24306 tablet_service.cc:1467] Processing CreateTablet for tablet 48af4d51aa8f41a486c4fe6e7541700e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:17:27.501484 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48af4d51aa8f41a486c4fe6e7541700e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.504081 24303 tablet_service.cc:1467] Processing CreateTablet for tablet cd2b577e2c3e4021bc8ebdfa3462437d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:17:27.505363 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2b577e2c3e4021bc8ebdfa3462437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.508062 24302 tablet_service.cc:1467] Processing CreateTablet for tablet 90dd68b3876b4b75aea408edfa68d5e5 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:17:27.509318 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dd68b3876b4b75aea408edfa68d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.517885 24298 tablet_service.cc:1467] Processing CreateTablet for tablet bc822924176247c7abbc4a4756daf3a6 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:17:27.519536 24301 tablet_service.cc:1467] Processing CreateTablet for tablet 974355bb93274291bb6be768b656cc67 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:17:27.523758 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc822924176247c7abbc4a4756daf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.533382 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974355bb93274291bb6be768b656cc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.533624 24300 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcb08685ecd486c8f1d951336f39982 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:17:27.534906 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcb08685ecd486c8f1d951336f39982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.539049 24305 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2662351f0e4b14bede1be0deaba6c8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:17:27.540414 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2662351f0e4b14bede1be0deaba6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.541198 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57d179ede43aebb4ed26fdaf60001 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:17:27.542477 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57d179ede43aebb4ed26fdaf60001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.543777 24602 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:27.548182 24420 tablet_service.cc:1467] Processing CreateTablet for tablet cec68a6991264e72a23a8750a420795d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 23:17:27.549377 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec68a6991264e72a23a8750a420795d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.534397 24522 tablet_service.cc:1467] Processing CreateTablet for tablet b1115ecb68bd4c399da85a680700bb7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:17:27.555938 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1115ecb68bd4c399da85a680700bb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.561801 24602 tablet_bootstrap.cc:654] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:27.564246 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 9429f7311dd744bea4301956e143e9a7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:17:27.565584 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9429f7311dd744bea4301956e143e9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.568506 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:17:27.569448 24407 tablet_service.cc:1467] Processing CreateTablet for tablet c37e848050ec44069438f4c88a76f9ef (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:17:27.570353 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3f3d2a169145ffad7fb65adf8b7d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.570654 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37e848050ec44069438f4c88a76f9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.571790 24299 tablet_service.cc:1467] Processing CreateTablet for tablet ba62e1cbbfd84b86a817f9118be39b00 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:17:27.575590 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba62e1cbbfd84b86a817f9118be39b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.579469 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57d179ede43aebb4ed26fdaf60001 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:17:27.580715 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57d179ede43aebb4ed26fdaf60001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.584610 24404 tablet_service.cc:1467] Processing CreateTablet for tablet 3913bd5adc0247eb847943b7e9a3ceb7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:17:27.589339 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3913bd5adc0247eb847943b7e9a3ceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.615512 24603 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:27.654125 24298 tablet_service.cc:1467] Processing CreateTablet for tablet b1115ecb68bd4c399da85a680700bb7f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:17:27.667343 24300 tablet_service.cc:1467] Processing CreateTablet for tablet 07c57d179ede43aebb4ed26fdaf60001 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:17:27.668677 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 1a28a5ad332b42c2ba471779e78f3e25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:17:27.669600 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a28a5ad332b42c2ba471779e78f3e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.680572 24414 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad5d0ab888b41229481d32d207c7466 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:17:27.687208 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad5d0ab888b41229481d32d207c7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.684620 24527 tablet_service.cc:1467] Processing CreateTablet for tablet cec68a6991264e72a23a8750a420795d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 23:17:27.688414 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec68a6991264e72a23a8750a420795d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.694190 24524 tablet_service.cc:1467] Processing CreateTablet for tablet c66997a772ef46a5ae07d530d26ab8a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:17:27.695438 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66997a772ef46a5ae07d530d26ab8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.685842 24529 tablet_service.cc:1467] Processing CreateTablet for tablet c37e848050ec44069438f4c88a76f9ef (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:17:27.683921 24409 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:17:27.699880 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.701097 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 1a28a5ad332b42c2ba471779e78f3e25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:17:27.702675 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a28a5ad332b42c2ba471779e78f3e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.703955 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1115ecb68bd4c399da85a680700bb7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.714617 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 3913bd5adc0247eb847943b7e9a3ceb7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:17:27.705093 24603 tablet_bootstrap.cc:654] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:27.715919 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3913bd5adc0247eb847943b7e9a3ceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.610914 24417 tablet_service.cc:1467] Processing CreateTablet for tablet c66997a772ef46a5ae07d530d26ab8a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:17:27.707104 24302 tablet_service.cc:1467] Processing CreateTablet for tablet cec68a6991264e72a23a8750a420795d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
W20260809 23:17:27.705005 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:27.719611 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c57d179ede43aebb4ed26fdaf60001. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.720046 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66997a772ef46a5ae07d530d26ab8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.720288 24306 tablet_service.cc:1467] Processing CreateTablet for tablet c66997a772ef46a5ae07d530d26ab8a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:17:27.712636 24303 tablet_service.cc:1467] Processing CreateTablet for tablet c37e848050ec44069438f4c88a76f9ef (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:17:27.721848 24605 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:27.728987 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66997a772ef46a5ae07d530d26ab8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.731549 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec68a6991264e72a23a8750a420795d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.690712 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 3913bd5adc0247eb847943b7e9a3ceb7 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:17:27.714475 24405 tablet_service.cc:1467] Processing CreateTablet for tablet b8a29ab049db4236a3198e8545cafb9c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:17:27.734084 24419 tablet_service.cc:1467] Processing CreateTablet for tablet d273006c869f4749852291bcee467158 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:17:27.736943 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d273006c869f4749852291bcee467158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.739333 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37e848050ec44069438f4c88a76f9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.739071 24412 tablet_service.cc:1467] Processing CreateTablet for tablet eba754600fd94d2588497bb7f43a92a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:17:27.740304 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba754600fd94d2588497bb7f43a92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.759948 24420 tablet_service.cc:1467] Processing CreateTablet for tablet a682e0d089354a44a680bd8ad9b4b789 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:17:27.761617 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a682e0d089354a44a680bd8ad9b4b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.764043 24301 tablet_service.cc:1467] Processing CreateTablet for tablet 1a28a5ad332b42c2ba471779e78f3e25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:17:27.765295 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a28a5ad332b42c2ba471779e78f3e25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.766167 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 65d375db793a4542808e1cff72fb6617 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:17:27.767442 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d375db793a4542808e1cff72fb6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.768890 24603 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:27.769361 24603 ts_tablet_manager.cc:1397] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.154s	user 0.021s	sys 0.000s
I20260809 23:17:27.772099 24603 raft_consensus.cc:348] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:27.772830 24603 raft_consensus.cc:374] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:27.773113 24603 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:27.773281 24405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a29ab049db4236a3198e8545cafb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.773875 24603 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:27.779541 24605 tablet_bootstrap.cc:654] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:27.780088 24299 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad5d0ab888b41229481d32d207c7466 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:17:27.781360 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad5d0ab888b41229481d32d207c7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.807703 24407 tablet_service.cc:1467] Processing CreateTablet for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:17:27.808982 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fc12c55ab24490a1ad12ed4b4b5c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.813163 24300 tablet_service.cc:1467] Processing CreateTablet for tablet b8a29ab049db4236a3198e8545cafb9c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:17:27.799934 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3913bd5adc0247eb847943b7e9a3ceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.762626 24522 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad5d0ab888b41229481d32d207c7466 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:17:27.828127 24602 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:27.837808 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a29ab049db4236a3198e8545cafb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.842073 24605 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:27.844858 24602 ts_tablet_manager.cc:1397] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.301s	user 0.030s	sys 0.022s
I20260809 23:17:27.851641 24605 ts_tablet_manager.cc:1397] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.130s	user 0.021s	sys 0.000s
I20260809 23:17:27.854720 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37e848050ec44069438f4c88a76f9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.856832 24602 raft_consensus.cc:348] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:27.857564 24602 raft_consensus.cc:374] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:27.858486 24602 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:27.859496 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad5d0ab888b41229481d32d207c7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.859666 24602 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:27.862354 24603 ts_tablet_manager.cc:1428] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.093s	user 0.027s	sys 0.019s
I20260809 23:17:27.862757 24305 tablet_service.cc:1467] Processing CreateTablet for tablet eba754600fd94d2588497bb7f43a92a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:17:27.863133 24580 heartbeater.cc:503] Master 127.21.255.62:37921 was elected leader, sending a full tablet report...
I20260809 23:17:27.863374 24603 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:27.864235 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba754600fd94d2588497bb7f43a92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.867965 24407 tablet_service.cc:1467] Processing CreateTablet for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:17:27.869186 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.869620 24605 raft_consensus.cc:348] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:27.870381 24605 raft_consensus.cc:374] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:27.870675 24605 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:27.871523 24605 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:27.799384 24531 tablet_service.cc:1467] Processing CreateTablet for tablet b8a29ab049db4236a3198e8545cafb9c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:17:27.874305 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a29ab049db4236a3198e8545cafb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.852509 24298 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:17:27.887943 24532 tablet_service.cc:1467] Processing CreateTablet for tablet d273006c869f4749852291bcee467158 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:17:27.889372 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d273006c869f4749852291bcee467158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.890071 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.889403 24412 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef03e5e7d0443d69a2228dd175bd010 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:17:27.892526 24526 tablet_service.cc:1467] Processing CreateTablet for tablet a682e0d089354a44a680bd8ad9b4b789 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:17:27.893513 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef03e5e7d0443d69a2228dd175bd010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.894018 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a682e0d089354a44a680bd8ad9b4b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.899827 24524 tablet_service.cc:1467] Processing CreateTablet for tablet eba754600fd94d2588497bb7f43a92a0 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:17:27.905737 24528 tablet_service.cc:1467] Processing CreateTablet for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:17:27.907016 24603 tablet_bootstrap.cc:654] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:27.908388 24404 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed27208ddea495a9e5ab4132e38adbb (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:17:27.906512 24407 tablet_service.cc:1467] Processing CreateTablet for tablet ca74ab4e31434c79a52a994e74129545 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:17:27.911589 24356 heartbeater.cc:503] Master 127.21.255.62:37921 was elected leader, sending a full tablet report...
I20260809 23:17:27.927536 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed27208ddea495a9e5ab4132e38adbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.931468 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddd2e2b2ca834a8a967d719c1ce9f6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.931747 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74ab4e31434c79a52a994e74129545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.920571 24414 tablet_service.cc:1467] Processing CreateTablet for tablet c9b804168f1e458cae6cd72cbdb5370e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:17:27.886407 24302 tablet_service.cc:1467] Processing CreateTablet for tablet d273006c869f4749852291bcee467158 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:17:27.951051 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba754600fd94d2588497bb7f43a92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.963503 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 65d375db793a4542808e1cff72fb6617 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:17:27.965418 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d375db793a4542808e1cff72fb6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.957662 24611 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:27.967516 24303 tablet_service.cc:1467] Processing CreateTablet for tablet a682e0d089354a44a680bd8ad9b4b789 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:17:27.967454 24611 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:27.968912 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a682e0d089354a44a680bd8ad9b4b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.971410 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d273006c869f4749852291bcee467158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.974107 24405 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c69723b934fd5869e2f079439b2f9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 23:17:27.976013 24605 ts_tablet_manager.cc:1428] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.124s	user 0.009s	sys 0.000s
I20260809 23:17:27.976974 24605 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:27.977186 24306 tablet_service.cc:1467] Processing CreateTablet for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:17:27.978396 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fc12c55ab24490a1ad12ed4b4b5c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.981621 24419 tablet_service.cc:1467] Processing CreateTablet for tablet bcf691f757af44e6a7cc16228b7424e3 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:17:27.982350 24417 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 23:17:27.983155 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b804168f1e458cae6cd72cbdb5370e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.983721 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a400f2643da95b8f744ebab6f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:27.959098 24412 tablet_service.cc:1467] Processing CreateTablet for tablet 54afb87e83f14564b7c4ec761e92304a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:17:27.985431 24527 tablet_service.cc:1467] Processing CreateTablet for tablet 65d375db793a4542808e1cff72fb6617 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:17:28.001737 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d375db793a4542808e1cff72fb6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.002242 24470 heartbeater.cc:503] Master 127.21.255.62:37921 was elected leader, sending a full tablet report...
I20260809 23:17:28.002091 24614 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.003724 24614 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:28.006177 24614 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:27.998870 24405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c69723b934fd5869e2f079439b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.031754 24611 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:28.053880 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54afb87e83f14564b7c4ec761e92304a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.064296 24603 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:28.064818 24603 ts_tablet_manager.cc:1397] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.202s	user 0.011s	sys 0.016s
I20260809 23:17:28.081283 24407 tablet_service.cc:1467] Processing CreateTablet for tablet f20a6f5f014e45d3a917a197db7a0998 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:17:28.084090 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 8e43c87eb09044548379ab2c35cce3c1 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:17:28.087750 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef03e5e7d0443d69a2228dd175bd010 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:17:28.091692 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef03e5e7d0443d69a2228dd175bd010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.096365 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:28.098685 24498 leader_election.cc:341] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.099411 24420 tablet_service.cc:1467] Processing CreateTablet for tablet c831643e28e34208b6edf775607155e4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:17:28.100059 24532 tablet_service.cc:1467] Processing CreateTablet for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:17:28.103292 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fc12c55ab24490a1ad12ed4b4b5c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.103657 24301 tablet_service.cc:1467] Processing CreateTablet for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:17:28.105172 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.111318 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e43c87eb09044548379ab2c35cce3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.078374 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 34970da619114abc942b839d0862a300 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:17:28.129418 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf691f757af44e6a7cc16228b7424e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.142119 24603 raft_consensus.cc:348] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.143191 24603 raft_consensus.cc:374] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.143544 24603 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.144452 24603 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.140235 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef03e5e7d0443d69a2228dd175bd010 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:17:28.146546 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef03e5e7d0443d69a2228dd175bd010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.157426 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed27208ddea495a9e5ab4132e38adbb (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:17:28.147068 24602 ts_tablet_manager.cc:1428] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.295s	user 0.019s	sys 0.019s
I20260809 23:17:28.168817 24404 tablet_service.cc:1467] Processing CreateTablet for tablet af1ef8fe019645f3b99232df6916cb2c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:17:28.173609 24605 tablet_bootstrap.cc:654] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.146907 24603 ts_tablet_manager.cc:1428] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.082s	user 0.004s	sys 0.002s
I20260809 23:17:28.179615 24602 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:28.179315 24299 tablet_service.cc:1467] Processing CreateTablet for tablet c9b804168f1e458cae6cd72cbdb5370e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:17:28.150264 24300 tablet_service.cc:1467] Processing CreateTablet for tablet ca74ab4e31434c79a52a994e74129545 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:17:28.120612 24531 tablet_service.cc:1467] Processing CreateTablet for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:17:28.191344 24616 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.201519 24611 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.201516 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed27208ddea495a9e5ab4132e38adbb (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:17:28.202176 24611 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.203099 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed27208ddea495a9e5ab4132e38adbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.203071 24306 tablet_service.cc:1467] Processing CreateTablet for tablet 54afb87e83f14564b7c4ec761e92304a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:17:28.204347 24611 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:28.205070 24616 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.200335 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20a6f5f014e45d3a917a197db7a0998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.149063 24522 tablet_service.cc:1467] Processing CreateTablet for tablet ca74ab4e31434c79a52a994e74129545 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:17:28.224699 24305 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 23:17:28.217234 24303 tablet_service.cc:1467] Processing CreateTablet for tablet bcf691f757af44e6a7cc16228b7424e3 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:17:28.215938 24616 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:28.229941 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74ab4e31434c79a52a994e74129545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.231320 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:28.233382 24380 leader_election.cc:341] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.233884 24603 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:28.200886 24530 tablet_service.cc:1467] Processing CreateTablet for tablet c9b804168f1e458cae6cd72cbdb5370e (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:17:28.239981 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b804168f1e458cae6cd72cbdb5370e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.243257 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.243872 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1ef8fe019645f3b99232df6916cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.245079 24526 tablet_service.cc:1467] Processing CreateTablet for tablet 54afb87e83f14564b7c4ec761e92304a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:17:28.246282 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54afb87e83f14564b7c4ec761e92304a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.233407 24414 tablet_service.cc:1467] Processing CreateTablet for tablet ef1955f952024cb2815f7065f861cf25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:17:28.248533 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1955f952024cb2815f7065f861cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.238814 24527 tablet_service.cc:1467] Processing CreateTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 23:17:28.258168 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a400f2643da95b8f744ebab6f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.268656 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34970da619114abc942b839d0862a300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.278074 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c831643e28e34208b6edf775607155e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.292593 24605 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:28.293100 24605 ts_tablet_manager.cc:1397] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.316s	user 0.013s	sys 0.017s
I20260809 23:17:28.295878 24605 raft_consensus.cc:348] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.297329 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed27208ddea495a9e5ab4132e38adbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.308149 24409 tablet_service.cc:1467] Processing CreateTablet for tablet cadfc1bde6cb4d6997f3fb215dc184ec (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 23:17:28.309433 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadfc1bde6cb4d6997f3fb215dc184ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.309933 24405 tablet_service.cc:1467] Processing CreateTablet for tablet b3063532c2cd49f8b7075e771a2e818a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:17:28.314275 24602 tablet_bootstrap.cc:654] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.316221 24532 tablet_service.cc:1467] Processing CreateTablet for tablet bcf691f757af44e6a7cc16228b7424e3 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:17:28.317485 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf691f757af44e6a7cc16228b7424e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.320042 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:28.320683 24323 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:28.322062 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:28.322160 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c69723b934fd5869e2f079439b2f9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260809 23:17:28.323719 24378 leader_election.cc:341] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.324157 24378 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:28.324345 24298 tablet_service.cc:1467] Processing CreateTablet for tablet 4a4c69723b934fd5869e2f079439b2f9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 23:17:28.325549 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c69723b934fd5869e2f079439b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.325544 24618 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:28.296607 24605 raft_consensus.cc:374] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.335078 24405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3063532c2cd49f8b7075e771a2e818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.337392 24302 tablet_service.cc:1467] Processing CreateTablet for tablet c831643e28e34208b6edf775607155e4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:17:28.338912 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c831643e28e34208b6edf775607155e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.339123 24603 tablet_bootstrap.cc:700] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent opening tablet: real 0.100s	user 0.005s	sys 0.000s
I20260809 23:17:28.378340 24603 tablet_bootstrap.cc:654] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.357990 24524 tablet_service.cc:1467] Processing CreateTablet for tablet c831643e28e34208b6edf775607155e4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:17:28.378912 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74ab4e31434c79a52a994e74129545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.357817 24301 tablet_service.cc:1467] Processing CreateTablet for tablet 8e43c87eb09044548379ab2c35cce3c1 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:17:28.380285 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e43c87eb09044548379ab2c35cce3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.380853 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54afb87e83f14564b7c4ec761e92304a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.327739 24605 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.360668 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 8e43c87eb09044548379ab2c35cce3c1 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:17:28.365003 24529 tablet_service.cc:1467] Processing CreateTablet for tablet f20a6f5f014e45d3a917a197db7a0998 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:17:28.369547 24294 tablet_service.cc:1467] Processing CreateTablet for tablet f20a6f5f014e45d3a917a197db7a0998 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:17:28.374861 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c69723b934fd5869e2f079439b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.376266 24498 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:28.388046 24611 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:28.388407 24611 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:28.388741 24611 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:28.389012 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf691f757af44e6a7cc16228b7424e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.391115 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20a6f5f014e45d3a917a197db7a0998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.392768 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b804168f1e458cae6cd72cbdb5370e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.387852 24419 tablet_service.cc:1467] Processing CreateTablet for tablet ac3c46a49aea44a2b2721eb2a750f055 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:17:28.394095 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3c46a49aea44a2b2721eb2a750f055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.384121 24605 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.404100 24522 tablet_service.cc:1467] Processing CreateTablet for tablet 34970da619114abc942b839d0862a300 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:17:28.404891 24605 ts_tablet_manager.cc:1428] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.111s	user 0.008s	sys 0.001s
I20260809 23:17:28.405830 24605 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:28.394819 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e43c87eb09044548379ab2c35cce3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.395264 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a400f2643da95b8f744ebab6f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.413800 24611 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.415484 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c831643e28e34208b6edf775607155e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.334765 24412 tablet_service.cc:1467] Processing CreateTablet for tablet 7f974a8c0944480688bfdf1cf728da89 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:17:28.431686 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f974a8c0944480688bfdf1cf728da89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.435045 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:28.435905 24546 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:28.440963 24414 tablet_service.cc:1467] Processing CreateTablet for tablet 9932f35c1d8a431f9c035a848f63cb0b (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:17:28.441712 24602 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:28.447455 24602 ts_tablet_manager.cc:1397] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.268s	user 0.019s	sys 0.004s
I20260809 23:17:28.450508 24602 raft_consensus.cc:348] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.451274 24602 raft_consensus.cc:374] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.451536 24602 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.452260 24602 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.455096 24531 tablet_service.cc:1467] Processing CreateTablet for tablet af1ef8fe019645f3b99232df6916cb2c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:17:28.458977 24602 ts_tablet_manager.cc:1428] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260809 23:17:28.459894 24602 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:28.444820 24298 tablet_service.cc:1467] Processing CreateTablet for tablet 34970da619114abc942b839d0862a300 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:17:28.460896 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34970da619114abc942b839d0862a300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.460350 24297 tablet_service.cc:1467] Processing CreateTablet for tablet af1ef8fe019645f3b99232df6916cb2c (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:17:28.462858 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1ef8fe019645f3b99232df6916cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.464831 24611 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:28.465107 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:28.465693 24321 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:28.469043 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 904621b566f64064bc1a7411ab27c677 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:17:28.460427 24527 tablet_service.cc:1467] Processing CreateTablet for tablet cadfc1bde6cb4d6997f3fb215dc184ec (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 23:17:28.472872 24321 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:28.473893 24498 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:28.437652 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34970da619114abc942b839d0862a300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.480149 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1ef8fe019645f3b99232df6916cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.481860 24526 tablet_service.cc:1467] Processing CreateTablet for tablet ef1955f952024cb2815f7065f861cf25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:17:28.463266 24605 tablet_bootstrap.cc:654] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.494136 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9932f35c1d8a431f9c035a848f63cb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.494793 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:28.496513 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:28.497798 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904621b566f64064bc1a7411ab27c677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.497900 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:28.499007 24611 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:28.499372 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:17:28.502293 24266 leader_election.cc:341] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.502882 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:28.503857 24616 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:28.507045 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1955f952024cb2815f7065f861cf25. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:17:28.513043 24496 leader_election.cc:341] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.513315 24530 tablet_service.cc:1467] Processing CreateTablet for tablet b3063532c2cd49f8b7075e771a2e818a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:17:28.513689 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:28.516530 24603 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:28.517071 24603 ts_tablet_manager.cc:1397] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.295s	user 0.022s	sys 0.008s
I20260809 23:17:28.514043 24602 tablet_bootstrap.cc:654] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.522861 24603 raft_consensus.cc:348] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.523638 24603 raft_consensus.cc:374] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.523708 24611 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:28.523969 24603 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.488853 24417 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad562edaec4eebaa047293c085c0c4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:17:28.530916 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad562edaec4eebaa047293c085c0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.532857 24305 tablet_service.cc:1467] Processing CreateTablet for tablet cadfc1bde6cb4d6997f3fb215dc184ec (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 23:17:28.534085 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadfc1bde6cb4d6997f3fb215dc184ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.543538 24647 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:28.544677 24420 tablet_service.cc:1467] Processing CreateTablet for tablet 7be87796c0684430b07c73073572ee50 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:17:28.546242 24420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be87796c0684430b07c73073572ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.546010 24412 tablet_service.cc:1467] Processing CreateTablet for tablet 3962e082dc274290a1f20e2fa64877a9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260809 23:17:28.548661 24496 leader_election.cc:341] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:28.551386 24496 leader_election.cc:341] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.552001 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3063532c2cd49f8b7075e771a2e818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.549409 24404 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a92891fb40e8911ac15230653aab (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:17:28.557304 24412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3962e082dc274290a1f20e2fa64877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.554466 24405 tablet_service.cc:1467] Processing CreateTablet for tablet deaa6cc935614b7bbb64f5ee8faf4c74 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:17:28.524713 24603 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.561174 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a92891fb40e8911ac15230653aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.561733 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20a6f5f014e45d3a917a197db7a0998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.565297 24532 tablet_service.cc:1467] Processing CreateTablet for tablet 7f974a8c0944480688bfdf1cf728da89 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:17:28.573017 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f974a8c0944480688bfdf1cf728da89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.567415 24605 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:28.586877 24605 ts_tablet_manager.cc:1397] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.181s	user 0.012s	sys 0.011s
I20260809 23:17:28.586704 24301 tablet_service.cc:1467] Processing CreateTablet for tablet ef1955f952024cb2815f7065f861cf25 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:17:28.587961 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1955f952024cb2815f7065f861cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.590127 24405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaa6cc935614b7bbb64f5ee8faf4c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.589967 24605 raft_consensus.cc:348] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.590876 24605 raft_consensus.cc:374] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.603334 24605 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.580265 24603 ts_tablet_manager.cc:1428] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.000s
I20260809 23:17:28.573172 24407 tablet_service.cc:1467] Processing CreateTablet for tablet 827f95d8d9f946f6b673db8272386c5f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:17:28.594285 24527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadfc1bde6cb4d6997f3fb215dc184ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.576938 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 9932f35c1d8a431f9c035a848f63cb0b (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:17:28.603950 24302 tablet_service.cc:1467] Processing CreateTablet for tablet 904621b566f64064bc1a7411ab27c677 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:17:28.593498 24299 tablet_service.cc:1467] Processing CreateTablet for tablet b3063532c2cd49f8b7075e771a2e818a (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:17:28.612314 24407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827f95d8d9f946f6b673db8272386c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.612689 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3063532c2cd49f8b7075e771a2e818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.617044 24529 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 36) took 1018 ms. Trace:
I20260809 23:17:28.597229 24303 tablet_service.cc:1467] Processing CreateTablet for tablet 7f974a8c0944480688bfdf1cf728da89 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:17:28.598361 24300 tablet_service.cc:1467] Processing CreateTablet for tablet ac3c46a49aea44a2b2721eb2a750f055 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:17:28.605705 24298 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad562edaec4eebaa047293c085c0c4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:17:28.619375 24603 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:28.608029 24297 tablet_service.cc:1467] Processing CreateTablet for tablet c3876fa8158d49ec9c1128fbebe6f994 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:17:28.524919 24611 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.617872 24529 rpcz_store.cc:276] 0809 23:17:27.598266 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.364134 (+765868us) service_pool.cc:227] Handling call
0809 23:17:28.425649 (+ 61515us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.434718 (+  9069us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.581667 (+146949us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.616529 (+ 34862us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.616973 (+   444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.621513 24602 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:28.619702 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f974a8c0944480688bfdf1cf728da89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.624759 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad562edaec4eebaa047293c085c0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.625378 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3c46a49aea44a2b2721eb2a750f055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.635214 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3876fa8158d49ec9c1128fbebe6f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.636299 24414 tablet_service.cc:1467] Processing CreateTablet for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:17:28.638046 24414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.638171 24605 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.637161 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9932f35c1d8a431f9c035a848f63cb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.622886 24602 ts_tablet_manager.cc:1397] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.163s	user 0.012s	sys 0.005s
I20260809 23:17:28.645560 24602 raft_consensus.cc:348] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.646368 24602 raft_consensus.cc:374] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.646703 24602 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.647516 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904621b566f64064bc1a7411ab27c677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.639462 24531 tablet_service.cc:1467] Processing CreateTablet for tablet ac3c46a49aea44a2b2721eb2a750f055 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:17:28.651428 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3c46a49aea44a2b2721eb2a750f055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.653733 24533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 33) took 1057 ms. Trace:
I20260809 23:17:28.654188 24533 rpcz_store.cc:276] 0809 23:17:27.596725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.321247 (+724522us) service_pool.cc:227] Handling call
0809 23:17:28.322918 (+  1671us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.347430 (+ 24512us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.473285 (+125855us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.618950 (+145665us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.653671 (+ 34721us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22280}
I20260809 23:17:28.655211 24306 tablet_service.cc:1467] Processing CreateTablet for tablet 9932f35c1d8a431f9c035a848f63cb0b (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:17:28.656992 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9932f35c1d8a431f9c035a848f63cb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.657835 24533 tablet_service.cc:1467] Processing CreateTablet for tablet 7be87796c0684430b07c73073572ee50 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:17:28.658967 24530 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 41) took 1022 ms. Trace:
I20260809 23:17:28.640920 24529 tablet_service.cc:1467] Processing CreateTablet for tablet 9dad562edaec4eebaa047293c085c0c4 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:17:28.653740 24404 tablet_service.cc:1467] Processing CreateTablet for tablet 137c4d66939a48e59f84422216ede16f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:17:28.668497 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c4d66939a48e59f84422216ede16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.671777 24409 tablet_service.cc:1467] Processing CreateTablet for tablet b3538d14b2274bf99cd72e3872fda1fa (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:17:28.672497 24524 tablet_service.cc:1467] Processing CreateTablet for tablet 904621b566f64064bc1a7411ab27c677 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:17:28.675014 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be87796c0684430b07c73073572ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.675711 24294 tablet_service.cc:1467] Processing CreateTablet for tablet 7be87796c0684430b07c73073572ee50 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:17:28.673019 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3538d14b2274bf99cd72e3872fda1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.662431 24530 rpcz_store.cc:276] 0809 23:17:27.636441 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.488912 (+852471us) service_pool.cc:227] Handling call
0809 23:17:28.514447 (+ 25535us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.514561 (+   114us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.598083 (+ 83522us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.630761 (+ 32678us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.658892 (+ 28131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.679874 24417 tablet_service.cc:1467] Processing CreateTablet for tablet fb0467d784e841868c4a1188ee0af4bc (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:17:28.647529 24602 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.689875 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0467d784e841868c4a1188ee0af4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.690857 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 37) took 1092 ms. Trace:
I20260809 23:17:28.692430 24522 rpcz_store.cc:276] 0809 23:17:27.598675 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.403053 (+804378us) service_pool.cc:227] Handling call
0809 23:17:28.404901 (+  1848us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.404993 (+    92us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.582749 (+177756us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.674013 (+ 91264us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.690783 (+ 16770us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47872}
I20260809 23:17:28.694840 24522 tablet_service.cc:1467] Processing CreateTablet for tablet 827f95d8d9f946f6b673db8272386c5f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:17:28.695508 24416 tablet_service.cc:1467] Processing CreateTablet for tablet 8542be4415fd4e648a44a76fdbc3e176 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:17:28.696767 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8542be4415fd4e648a44a76fdbc3e176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.697151 24170 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:28.703794 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad562edaec4eebaa047293c085c0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.703850 24618 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.704383 24618 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.706863 24618 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:28.707875 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904621b566f64064bc1a7411ab27c677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.708285 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:28.709736 24380 leader_election.cc:341] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.712175 24602 ts_tablet_manager.cc:1428] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.069s	user 0.005s	sys 0.001s
I20260809 23:17:28.713164 24602 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:28.715406 24294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be87796c0684430b07c73073572ee50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.715193 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:28.691737 24530 tablet_service.cc:1467] Processing CreateTablet for tablet 3962e082dc274290a1f20e2fa64877a9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 23:17:28.716111 24603 tablet_bootstrap.cc:654] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.716439 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3962e082dc274290a1f20e2fa64877a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:17:28.716866 24378 leader_election.cc:341] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.717315 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:28.719635 24618 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:28.720638 24305 tablet_service.cc:1467] Processing CreateTablet for tablet 3962e082dc274290a1f20e2fa64877a9 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260809 23:17:28.721903 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:28.726540 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3962e082dc274290a1f20e2fa64877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.727564 24420 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 47) took 1065 ms. Trace:
I20260809 23:17:28.724215 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827f95d8d9f946f6b673db8272386c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.723776 24605 ts_tablet_manager.cc:1428] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.136s	user 0.005s	sys 0.003s
I20260809 23:17:28.735363 24419 tablet_service.cc:1467] Processing CreateTablet for tablet c3876fa8158d49ec9c1128fbebe6f994 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:17:28.736706 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3876fa8158d49ec9c1128fbebe6f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.736917 24605 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:28.746766 24300 tablet_service.cc:1467] Processing CreateTablet for tablet deaa6cc935614b7bbb64f5ee8faf4c74 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:17:28.748028 24300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaa6cc935614b7bbb64f5ee8faf4c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.749258 24526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 40) took 1113 ms. Trace:
I20260809 23:17:28.749601 24526 rpcz_store.cc:276] 0809 23:17:27.636027 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.481000 (+844973us) service_pool.cc:227] Handling call
0809 23:17:28.482649 (+  1649us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.482737 (+    88us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.675995 (+193258us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.748788 (+ 72793us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.749201 (+   413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.752153 24526 tablet_service.cc:1467] Processing CreateTablet for tablet fb0467d784e841868c4a1188ee0af4bc (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:17:28.752483 24404 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 55) took 1071 ms. Trace:
I20260809 23:17:28.752936 24404 rpcz_store.cc:276] 0809 23:17:27.680815 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.652775 (+971960us) service_pool.cc:227] Handling call
0809 23:17:28.665343 (+ 12568us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.666134 (+   791us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.683395 (+ 17261us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.751985 (+ 68590us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.752427 (+   442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.754076 24526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0467d784e841868c4a1188ee0af4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.759791 24298 tablet_service.cc:1467] Processing CreateTablet for tablet 827f95d8d9f946f6b673db8272386c5f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:17:28.760075 24299 tablet_service.cc:1467] Processing CreateTablet for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:17:28.761016 24298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827f95d8d9f946f6b673db8272386c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.761946 24616 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.762358 24299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.762374 24616 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.769423 24616 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:28.770793 24532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 42) took 1134 ms. Trace:
I20260809 23:17:28.771245 24532 rpcz_store.cc:276] 0809 23:17:27.636839 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.564450 (+927611us) service_pool.cc:227] Handling call
0809 23:17:28.572429 (+  7979us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.572552 (+   123us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.637227 (+ 64675us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.765077 (+127850us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.770732 (+  5655us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.771428 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:28.772689 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:28.773305 24270 leader_election.cc:341] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.773319 24435 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:28.774693 24405 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 50) took 1102 ms. Trace:
I20260809 23:17:28.775125 24266 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:28.775079 24405 rpcz_store.cc:276] 0809 23:17:27.672167 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.552001 (+879834us) service_pool.cc:227] Handling call
0809 23:17:28.589670 (+ 37669us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.589763 (+    93us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.767853 (+178090us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.774243 (+  6390us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.774639 (+   396us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.776432 24616 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:28.776835 24616 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:28.777235 24616 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:28.777948 24412 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 48) took 1106 ms. Trace:
I20260809 23:17:28.777835 24306 tablet_service.cc:1467] Processing CreateTablet for tablet fb0467d784e841868c4a1188ee0af4bc (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:17:28.781790 24528 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 44) took 1144 ms. Trace:
I20260809 23:17:28.782155 24528 rpcz_store.cc:276] 0809 23:17:27.637501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.575815 (+938314us) service_pool.cc:227] Handling call
0809 23:17:28.618600 (+ 42785us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.618698 (+    98us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.680227 (+ 61529us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.781331 (+101104us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.781736 (+   405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.785121 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 137c4d66939a48e59f84422216ede16f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:17:28.786283 24605 tablet_bootstrap.cc:654] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.778316 24412 rpcz_store.cc:276] 0809 23:17:27.671513 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.545112 (+873599us) service_pool.cc:227] Handling call
0809 23:17:28.556724 (+ 11612us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.556846 (+   122us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.604778 (+ 47932us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.777469 (+172691us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.777893 (+   424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.789705 24528 tablet_service.cc:1467] Processing CreateTablet for tablet 137c4d66939a48e59f84422216ede16f (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:17:28.782793 24602 tablet_bootstrap.cc:654] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.796905 24301 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 40) took 1012 ms. Trace:
I20260809 23:17:28.797326 24301 rpcz_store.cc:276] 0809 23:17:27.784913 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.585880 (+800967us) service_pool.cc:227] Handling call
0809 23:17:28.587518 (+  1638us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.587625 (+   107us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.682056 (+ 94431us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.796418 (+114362us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.796845 (+   427us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.796923 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c4d66939a48e59f84422216ede16f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:17:28.798414 24472 tablet.cc:2369] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 23:17:28.801694 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0467d784e841868c4a1188ee0af4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.734807 24303 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a92891fb40e8911ac15230653aab (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:17:28.817730 24524 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 45) took 1180 ms. Trace:
I20260809 23:17:28.818131 24524 rpcz_store.cc:276] 0809 23:17:27.637827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.594325 (+956498us) service_pool.cc:227] Handling call
0809 23:17:28.691540 (+ 97215us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.691695 (+   155us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.794014 (+102319us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.817185 (+ 23171us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.817671 (+   486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.741794 24420 rpcz_store.cc:276] 0809 23:17:27.671144 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.543460 (+872316us) service_pool.cc:227] Handling call
0809 23:17:28.545678 (+  2218us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.545788 (+   110us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.600889 (+ 55101us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.727078 (+126189us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.727501 (+   423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.823097 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 51) took 1094 ms. Trace:
I20260809 23:17:28.821980 24532 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a92891fb40e8911ac15230653aab (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:17:28.828750 24414 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 53) took 1155 ms. Trace:
I20260809 23:17:28.829120 24414 rpcz_store.cc:276] 0809 23:17:27.673089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.635432 (+962343us) service_pool.cc:227] Handling call
0809 23:17:28.637574 (+  2142us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.637684 (+   110us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.714394 (+ 76710us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.828253 (+113859us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.828693 (+   440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.823479 24522 rpcz_store.cc:276] 0809 23:17:27.728960 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.694018 (+965058us) service_pool.cc:227] Handling call
0809 23:17:28.695737 (+  1719us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.695839 (+   102us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.788154 (+ 92315us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.822611 (+ 34457us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.823043 (+   432us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.831413 24522 tablet_service.cc:1467] Processing CreateTablet for tablet deaa6cc935614b7bbb64f5ee8faf4c74 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:17:28.831768 24301 tablet_service.cc:1467] Processing CreateTablet for tablet b3538d14b2274bf99cd72e3872fda1fa (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:17:28.833060 24524 tablet_service.cc:1467] Processing CreateTablet for tablet c3876fa8158d49ec9c1128fbebe6f994 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:17:28.839349 24603 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:28.840022 24603 ts_tablet_manager.cc:1397] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.222s	user 0.013s	sys 0.005s
I20260809 23:17:28.845824 24603 raft_consensus.cc:348] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.846843 24603 raft_consensus.cc:374] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.847241 24603 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.848331 24603 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.869324 24301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3538d14b2274bf99cd72e3872fda1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.870019 24303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a92891fb40e8911ac15230653aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.870915 24603 ts_tablet_manager.cc:1428] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.030s	user 0.010s	sys 0.000s
I20260809 23:17:28.877597 24532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a92891fb40e8911ac15230653aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.882865 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaa6cc935614b7bbb64f5ee8faf4c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.882144 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c4d66939a48e59f84422216ede16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.884109 24605 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:28.884562 24605 ts_tablet_manager.cc:1397] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.148s	user 0.017s	sys 0.005s
I20260809 23:17:28.886173 24524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3876fa8158d49ec9c1128fbebe6f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.887246 24605 raft_consensus.cc:348] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.888003 24605 raft_consensus.cc:374] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.888288 24605 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.889108 24605 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.890625 24404 tablet_service.cc:1467] Processing CreateTablet for tablet b2c4cf897ac64cb8981b99dda67dbc92 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 23:17:28.878209 24305 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 48) took 1049 ms. Trace:
I20260809 23:17:28.892758 24305 rpcz_store.cc:276] 0809 23:17:27.829279 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.719799 (+890520us) service_pool.cc:227] Handling call
0809 23:17:28.725988 (+  6189us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.726191 (+   203us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.784413 (+ 58222us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.829373 (+ 44960us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.877281 (+ 47908us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.895803 24603 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:28.896447 24407 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 51) took 1224 ms. Trace:
I20260809 23:17:28.896901 24407 rpcz_store.cc:276] 0809 23:17:27.672478 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.572256 (+899778us) service_pool.cc:227] Handling call
0809 23:17:28.611867 (+ 39611us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.611966 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.720276 (+108310us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.868311 (+148035us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.896387 (+ 28076us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.899190 24404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c4cf897ac64cb8981b99dda67dbc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.899240 24305 tablet_service.cc:1467] Processing CreateTablet for tablet 8542be4415fd4e648a44a76fdbc3e176 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:17:28.900544 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8542be4415fd4e648a44a76fdbc3e176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.881006 24533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 47) took 1241 ms. Trace:
I20260809 23:17:28.915637 24533 rpcz_store.cc:276] 0809 23:17:27.638564 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.656907 (+1018343us) service_pool.cc:227] Handling call
0809 23:17:28.660889 (+  3982us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.661025 (+   136us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.790249 (+129224us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.839286 (+ 49037us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.880222 (+ 40936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.916536 24611 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.917124 24611 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.919685 24611 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:28.920756 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:28.922314 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:28.923379 24305 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 57) took 1022 ms. Trace:
W20260809 23:17:28.923808 24496 leader_election.cc:341] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.923763 24305 rpcz_store.cc:276] 0809 23:17:27.900530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.898239 (+997709us) service_pool.cc:227] Handling call
0809 23:17:28.900086 (+  1847us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.900178 (+    92us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.916727 (+ 16549us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.922845 (+  6118us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.923324 (+   479us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:17:28.924520 24498 leader_election.cc:341] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.925063 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:28.926313 24611 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:28.926710 24305 tablet_service.cc:1467] Processing CreateTablet for tablet b2c4cf897ac64cb8981b99dda67dbc92 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 23:17:28.928006 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c4cf897ac64cb8981b99dda67dbc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:28.940655 24616 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.906956 24528 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 55) took 1175 ms. Trace:
I20260809 23:17:28.918530 24533 tablet_service.cc:1467] Processing CreateTablet for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:17:28.954919 24302 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 45) took 1168 ms. Trace:
I20260809 23:17:28.955122 24602 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:28.957541 24602 ts_tablet_manager.cc:1397] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.245s	user 0.020s	sys 0.050s
I20260809 23:17:28.957324 24302 rpcz_store.cc:276] 0809 23:17:27.786504 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.603121 (+816617us) service_pool.cc:227] Handling call
0809 23:17:28.647042 (+ 43921us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.647145 (+   103us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.786476 (+139331us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.954423 (+167947us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.954861 (+   438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.960359 24602 raft_consensus.cc:348] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.961076 24602 raft_consensus.cc:374] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:28.961490 24602 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:28.962276 24602 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.965487 24419 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 52) took 1292 ms. Trace:
I20260809 23:17:28.965893 24419 rpcz_store.cc:276] 0809 23:17:27.672810 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.639434 (+966624us) service_pool.cc:227] Handling call
0809 23:17:28.736191 (+ 96757us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.736285 (+    94us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.921534 (+185249us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.964781 (+ 43247us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.965426 (+   645us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.970178 24602 ts_tablet_manager.cc:1428] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260809 23:17:28.970515 24603 tablet_bootstrap.cc:654] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:28.971149 24659 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:28.971588 24659 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:28.974017 24659 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:28.974790 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:28.975513 24435 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:28.976099 24300 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 50) took 1146 ms. Trace:
I20260809 23:17:28.976643 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:28.976475 24300 rpcz_store.cc:276] 0809 23:17:27.830158 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.740672 (+910514us) service_pool.cc:227] Handling call
0809 23:17:28.747587 (+  6915us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.747686 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.867242 (+119556us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.975646 (+108404us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.976047 (+   401us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:28.983982 24530 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 48) took 1345 ms. Trace:
I20260809 23:17:28.987938 24661 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:28.989504 24661 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:28.989837 24661 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:28.987131 24605 ts_tablet_manager.cc:1428] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.102s	user 0.023s	sys 0.034s
I20260809 23:17:28.992899 24602 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:28.985270 24531 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 43) took 1348 ms. Trace:
I20260809 23:17:28.948294 24528 rpcz_store.cc:276] 0809 23:17:27.730245 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.788801 (+1058556us) service_pool.cc:227] Handling call
0809 23:17:28.796467 (+  7666us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.796564 (+    97us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.852508 (+ 55944us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.897313 (+ 44805us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.905615 (+  8302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.006346 24531 rpcz_store.cc:276] 0809 23:17:27.637170 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.566230 (+929060us) service_pool.cc:227] Handling call
0809 23:17:28.650838 (+ 84608us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.651088 (+   250us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.708378 (+ 57290us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.967016 (+258638us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.984941 (+ 17925us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:17:28.985198 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39396224}
I20260809 23:17:29.007786 24528 tablet_service.cc:1467] Processing CreateTablet for tablet b3538d14b2274bf99cd72e3872fda1fa (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:17:29.008175 24531 tablet_service.cc:1467] Processing CreateTablet for tablet 8542be4415fd4e648a44a76fdbc3e176 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:17:28.986452 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:28.987125 24527 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 39) took 1351 ms. Trace:
I20260809 23:17:28.986421 24529 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 46) took 1348 ms. Trace:
I20260809 23:17:28.988468 24530 rpcz_store.cc:276] 0809 23:17:27.638911 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.690591 (+1051680us) service_pool.cc:227] Handling call
0809 23:17:28.716002 (+ 25411us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.716099 (+    97us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.844848 (+128749us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.977616 (+132768us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.983555 (+  5939us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:17:28.983924 (+   369us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10896768}
I20260809 23:17:29.009397 24527 rpcz_store.cc:276] 0809 23:17:27.635555 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.459616 (+824061us) service_pool.cc:227] Handling call
0809 23:17:28.471908 (+ 12292us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.471998 (+    90us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.670020 (+198022us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.956852 (+286832us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.986771 (+ 29919us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:17:28.987066 (+   295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65596672}
I20260809 23:17:28.993678 24605 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:28.997545 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.011276 24435 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.011103 24529 rpcz_store.cc:276] 0809 23:17:27.638156 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.640076 (+1001920us) service_pool.cc:227] Handling call
0809 23:17:28.668100 (+ 28024us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.684935 (+ 16835us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.813452 (+128517us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.952831 (+139379us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.986088 (+ 33257us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:17:28.986366 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36486,"spinlock_wait_cycles":66374400}
I20260809 23:17:29.010986 24530 tablet_service.cc:1467] Processing CreateTablet for tablet b2c4cf897ac64cb8981b99dda67dbc92 (DEFAULT_TABLE table=test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260809 23:17:29.013933 24270 leader_election.cc:341] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:28.998360 24616 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:28.989050 24417 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 54) took 1315 ms. Trace:
I20260809 23:17:28.999797 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:28.999923 24416 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 57) took 1312 ms. Trace:
I20260809 23:17:28.999979 24294 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 47) took 1176 ms. Trace:
I20260809 23:17:28.984195 24522 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 50) took 1255 ms. Trace:
I20260809 23:17:29.000494 24611 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.000809 24533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e16e58b6ff4df4bcb9bb7ca0300f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:29.017171 24299 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 53) took 1111 ms. Trace:
I20260809 23:17:29.019605 24522 rpcz_store.cc:276] 0809 23:17:27.728618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.830493 (+1101875us) service_pool.cc:227] Handling call
0809 23:17:28.869171 (+ 38678us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900b7c 00007fac2090073b 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:17:28.879891 (+ 10720us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.880002 (+   111us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.909844 (+ 29842us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.983704 (+ 73860us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.984127 (+   423us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2606208}
I20260809 23:17:29.026366 24409 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 56) took 1342 ms. Trace:
I20260809 23:17:29.021175 24611 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:29.022773 24270 leader_election.cc:341] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.027626 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8542be4415fd4e648a44a76fdbc3e176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:29.028707 24611 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.017585 24417 rpcz_store.cc:276] 0809 23:17:27.673389 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.634504 (+961115us) service_pool.cc:227] Handling call
0809 23:17:28.689303 (+ 54799us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.689405 (+   102us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.944895 (+255490us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.988594 (+ 43699us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.988994 (+   400us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.029822 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c4cf897ac64cb8981b99dda67dbc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:29.030165 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.018795 24294 rpcz_store.cc:276] 0809 23:17:27.826516 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.674784 (+848268us) service_pool.cc:227] Handling call
0809 23:17:28.688984 (+ 14200us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.715038 (+ 26054us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.963632 (+248594us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.999438 (+ 35806us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.999927 (+   489us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:17:29.032526 24496 leader_election.cc:341] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.027380 24299 rpcz_store.cc:276] 0809 23:17:27.899186 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.759221 (+860035us) service_pool.cc:227] Handling call
0809 23:17:28.760829 (+  1608us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.760938 (+   109us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.967782 (+206844us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.010382 (+ 42600us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.010873 (+   491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.018029 24416 rpcz_store.cc:276] 0809 23:17:27.687200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.666689 (+979489us) service_pool.cc:227] Handling call
0809 23:17:28.696265 (+ 29576us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.696363 (+    98us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.796188 (+ 99825us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:28.999399 (+203211us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:28.999867 (+   468us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.023697 24605 tablet_bootstrap.cc:654] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.025822 24435 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.038496 24266 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.020289 24526 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 54) took 1289 ms. Trace:
I20260809 23:17:29.039316 24616 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.039903 24616 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:29.039767 24526 rpcz_store.cc:276] 0809 23:17:27.729933 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.751302 (+1021369us) service_pool.cc:227] Handling call
0809 23:17:28.752960 (+  1658us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.753166 (+   206us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.965341 (+212175us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.008531 (+ 43190us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.019499 (+ 10968us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.040849 24616 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.032502 24603 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.046692 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.047067 24603 ts_tablet_manager.cc:1397] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.152s	user 0.012s	sys 0.006s
W20260809 23:17:29.048641 24498 leader_election.cc:341] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.049070 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:29.051036 24306 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 54) took 1151 ms. Trace:
I20260809 23:17:29.054044 24305 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 58) took 1153 ms. Trace:
I20260809 23:17:29.051528 24306 rpcz_store.cc:276] 0809 23:17:27.899568 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.776893 (+877325us) service_pool.cc:227] Handling call
0809 23:17:28.778719 (+  1826us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.778821 (+   102us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.857692 (+ 78871us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.017548 (+159856us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.050953 (+ 33405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.054549 24305 rpcz_store.cc:276] 0809 23:17:27.901039 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.925749 (+1024710us) service_pool.cc:227] Handling call
0809 23:17:28.927482 (+  1733us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.927571 (+    89us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.994500 (+ 66929us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.018126 (+ 23626us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.053976 (+ 35850us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.026753 24409 rpcz_store.cc:276] 0809 23:17:27.683826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.664039 (+980213us) service_pool.cc:227] Handling call
0809 23:17:28.672576 (+  8537us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.672679 (+   103us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.832559 (+159880us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.025837 (+193278us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.026310 (+   473us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.053319 24301 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 56) took 1152 ms. Trace:
I20260809 23:17:29.056739 24301 rpcz_store.cc:276] 0809 23:17:27.900228 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.800801 (+900573us) service_pool.cc:227] Handling call
0809 23:17:28.868826 (+ 68025us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.868938 (+   112us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.972720 (+103782us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.028433 (+ 55713us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.052580 (+ 24147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.057976 24298 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 51) took 1227 ms. Trace:
I20260809 23:17:29.058385 24298 rpcz_store.cc:276] 0809 23:17:27.830636 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.755423 (+924787us) service_pool.cc:227] Handling call
0809 23:17:28.760550 (+  5127us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.760642 (+    92us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.002596 (+241954us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.042235 (+ 39639us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.057916 (+ 15681us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.060093 24297 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 55) took 1160 ms. Trace:
I20260809 23:17:29.060004 24524 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 52) took 1330 ms. Trace:
I20260809 23:17:29.061182 24603 raft_consensus.cc:348] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.076048 24603 raft_consensus.cc:374] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.075080 24297 rpcz_store.cc:276] 0809 23:17:27.899904 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.784235 (+884331us) service_pool.cc:227] Handling call
0809 23:17:28.868965 (+ 84730us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.869346 (+   381us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.980219 (+110873us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.059642 (+ 79423us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.060041 (+   399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.076627 24603 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.076879 24532 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 49) took 1348 ms. Trace:
I20260809 23:17:29.077283 24532 rpcz_store.cc:276] 0809 23:17:27.728224 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.820171 (+1091947us) service_pool.cc:227] Handling call
0809 23:17:28.833597 (+ 13426us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.860069 (+ 26472us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.067120 (+207051us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.076133 (+  9013us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.076819 (+   686us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.077510 24603 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.062203 24404 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:34762 (request call id 58) took 1371 ms. Trace:
I20260809 23:17:29.079115 24404 rpcz_store.cc:276] 0809 23:17:27.690885 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.791452 (+1100567us) service_pool.cc:227] Handling call
0809 23:17:28.897811 (+106359us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.898769 (+   958us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.958190 (+ 59421us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.061737 (+103547us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.062142 (+   405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.053539 24611 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.026607 24661 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.080555 24616 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.057274 24602 tablet_bootstrap.cc:654] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.081166 24616 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.084337 24603 ts_tablet_manager.cc:1428] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.002s
I20260809 23:17:29.085822 24603 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.062891 24303 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40854 (request call id 49) took 1233 ms. Trace:
I20260809 23:17:29.087363 24303 rpcz_store.cc:276] 0809 23:17:27.829774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.733811 (+904037us) service_pool.cc:227] Handling call
0809 23:17:28.869101 (+135290us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.869185 (+    84us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.994458 (+125273us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.045174 (+ 50716us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.062830 (+ 17656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.065894 24605 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.071075 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3538d14b2274bf99cd72e3872fda1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:29.088606 24605 ts_tablet_manager.cc:1397] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.095s	user 0.018s	sys 0.013s
I20260809 23:17:29.088899 24531 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 57) took 1358 ms. Trace:
I20260809 23:17:29.089480 24171 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:29.092018 24605 raft_consensus.cc:348] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.075192 24524 rpcz_store.cc:276] 0809 23:17:27.729326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.832229 (+1102903us) service_pool.cc:227] Handling call
0809 23:17:28.877475 (+ 45246us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:28.879866 (+  2391us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:28.980317 (+100451us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.059295 (+ 78978us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.059930 (+   635us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.089887 24531 rpcz_store.cc:276] 0809 23:17:27.730968 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:29.007396 (+1276428us) service_pool.cc:227] Handling call
0809 23:17:29.009174 (+  1778us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:29.009282 (+   108us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.037594 (+ 28312us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.074482 (+ 36888us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.088783 (+ 14301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.092876 24605 raft_consensus.cc:374] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.093313 24605 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.090785 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.093982 24605 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.062891 24530 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 58) took 1331 ms. Trace:
I20260809 23:17:29.083027 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.082425 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.095579 24435 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.081460 24661 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.096212 24605 ts_tablet_manager.cc:1428] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260809 23:17:29.096225 24270 leader_election.cc:341] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.103879 24616 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.105357 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.114231 24533 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 53) took 1384 ms. Trace:
I20260809 23:17:29.114780 24533 rpcz_store.cc:276] 0809 23:17:27.729615 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:28.917504 (+1187889us) service_pool.cc:227] Handling call
0809 23:17:29.000302 (+ 82798us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:29.000408 (+   106us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.065436 (+ 65028us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.113733 (+ 48297us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.114175 (+   442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.097193 24605 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.097175 24530 rpcz_store.cc:276] 0809 23:17:27.731338 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:29.010029 (+1278691us) service_pool.cc:227] Handling call
0809 23:17:29.013769 (+  3740us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:29.013860 (+    91us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.037679 (+ 23819us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.062424 (+ 24745us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.062830 (+   406us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:17:29.099148 24266 leader_election.cc:341] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:29.120081 24270 leader_election.cc:341] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.106688 24435 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.122390 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:29.126462 24616 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.126629 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.127694 24616 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.128171 24616 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:29.128944 24616 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.131332 24603 tablet_bootstrap.cc:654] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:29.139451 24496 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:29.148626 24611 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:29.149879 24528 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38080 (request call id 56) took 1419 ms. Trace:
I20260809 23:17:29.150277 24528 rpcz_store.cc:276] 0809 23:17:27.730564 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:17:29.006759 (+1276195us) service_pool.cc:227] Handling call
0809 23:17:29.008593 (+  1834us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:17:29.008688 (+    95us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:17:29.124943 (+116255us) tablet_metadata.cc:641] Metadata flushed
0809 23:17:29.149399 (+ 24456us) tablet_replica.cc:205] Creating consensus instance
0809 23:17:29.149820 (+   421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:17:29.154228 24171 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:29.155442 24602 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.156069 24602 ts_tablet_manager.cc:1397] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.163s	user 0.011s	sys 0.011s
I20260809 23:17:29.160336 24605 tablet_bootstrap.cc:654] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.171471 24602 raft_consensus.cc:348] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.172300 24602 raft_consensus.cc:374] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.172593 24602 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.173270 24602 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.180410 24602 ts_tablet_manager.cc:1428] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260809 23:17:29.181417 24602 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.187100 24603 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.187691 24603 ts_tablet_manager.cc:1397] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.102s	user 0.026s	sys 0.019s
I20260809 23:17:29.190009 24602 tablet_bootstrap.cc:654] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.192068 24603 raft_consensus.cc:348] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.193208 24603 raft_consensus.cc:374] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.193641 24603 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.194460 24603 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.197613 24602 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.197829 24603 ts_tablet_manager.cc:1428] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:17:29.198122 24602 ts_tablet_manager.cc:1397] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.008s
I20260809 23:17:29.199169 24603 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.201192 24602 raft_consensus.cc:348] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.202003 24602 raft_consensus.cc:374] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.202276 24602 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.203039 24602 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.206388 24605 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.206852 24605 ts_tablet_manager.cc:1397] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.110s	user 0.005s	sys 0.017s
I20260809 23:17:29.207702 24602 ts_tablet_manager.cc:1428] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:17:29.209992 24602 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.210314 24605 raft_consensus.cc:348] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.211126 24605 raft_consensus.cc:374] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.211433 24605 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.212500 24605 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.217279 24602 tablet_bootstrap.cc:654] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.217626 24605 ts_tablet_manager.cc:1428] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260809 23:17:29.218652 24605 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.224813 24605 tablet_bootstrap.cc:654] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.227278 24602 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.227872 24603 tablet_bootstrap.cc:654] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.227870 24602 ts_tablet_manager.cc:1397] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260809 23:17:29.232422 24602 raft_consensus.cc:348] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.233155 24605 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.233350 24602 raft_consensus.cc:374] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.233454 24603 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.233603 24605 ts_tablet_manager.cc:1397] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.007s
I20260809 23:17:29.233800 24602 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.234102 24603 ts_tablet_manager.cc:1397] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.035s	user 0.009s	sys 0.004s
I20260809 23:17:29.234788 24602 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.236086 24605 raft_consensus.cc:348] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.236810 24605 raft_consensus.cc:374] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.237076 24605 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.236968 24603 raft_consensus.cc:348] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.237568 24602 ts_tablet_manager.cc:1428] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260809 23:17:29.238219 24603 raft_consensus.cc:374] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.237803 24605 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.239140 24602 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.240413 24605 ts_tablet_manager.cc:1428] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:17:29.241592 24605 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.245819 24602 tablet_bootstrap.cc:654] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.247198 24605 tablet_bootstrap.cc:654] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.238664 24603 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.251973 24603 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.254901 24603 ts_tablet_manager.cc:1428] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.004s
I20260809 23:17:29.255446 24605 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.255844 24605 ts_tablet_manager.cc:1397] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.014s	user 0.013s	sys 0.000s
I20260809 23:17:29.256181 24603 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.257917 24649 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.258219 24605 raft_consensus.cc:348] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.258697 24649 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.258915 24605 raft_consensus.cc:374] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.261565 24602 tablet_bootstrap.cc:492] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.261993 24602 ts_tablet_manager.cc:1397] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.023s	user 0.003s	sys 0.015s
I20260809 23:17:29.261824 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.262450 24321 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.263027 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.263651 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:17:29.264406 24496 leader_election.cc:341] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.261827 24649 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.264470 24602 raft_consensus.cc:348] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.265314 24602 raft_consensus.cc:374] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.264454 24611 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.265627 24602 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.265688 24611 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.266075 24611 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.266405 24602 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.266984 24603 tablet_bootstrap.cc:654] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.271157 24618 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.271509 24602 ts_tablet_manager.cc:1428] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:17:29.271595 24618 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.272400 24602 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.273540 24618 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.275727 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:29.277221 24380 leader_election.cc:341] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.277065 24611 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.278132 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.278376 24603 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.278891 24603 ts_tablet_manager.cc:1397] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.009s
I20260809 23:17:29.279255 24602 tablet_bootstrap.cc:654] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.260454 24605 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
W20260809 23:17:29.279633 24378 leader_election.cc:341] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.280049 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.280035 24605 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.281461 24603 raft_consensus.cc:348] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.282092 24603 raft_consensus.cc:374] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.282347 24603 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.283097 24603 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.285285 24603 ts_tablet_manager.cc:1428] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:29.286208 24618 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.286334 24611 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.287533 24603 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.287902 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.288661 24605 ts_tablet_manager.cc:1428] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
W20260809 23:17:29.289407 24496 leader_election.cc:341] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.290199 24605 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.295679 24603 tablet_bootstrap.cc:654] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.296029 24605 tablet_bootstrap.cc:654] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.295683 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.296826 24602 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.296808 24321 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.297509 24602 ts_tablet_manager.cc:1397] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.005s
I20260809 23:17:29.300531 24602 raft_consensus.cc:348] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.301453 24602 raft_consensus.cc:374] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.305794 24602 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.306722 24602 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.308147 24603 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.308588 24603 ts_tablet_manager.cc:1397] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260809 23:17:29.309803 24602 ts_tablet_manager.cc:1428] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260809 23:17:29.311098 24602 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.310967 24603 raft_consensus.cc:348] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.311685 24603 raft_consensus.cc:374] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.311959 24603 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.312745 24603 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.314968 24603 ts_tablet_manager.cc:1428] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.004s
I20260809 23:17:29.315817 24603 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.317968 24616 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.318506 24616 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.319146 24602 tablet_bootstrap.cc:654] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.320501 24616 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.320840 24603 tablet_bootstrap.cc:654] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.322553 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.322912 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.323241 24435 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:29.323552 24545 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:29.324426 24266 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.325887 24605 tablet_bootstrap.cc:492] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.325929 24616 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.326315 24602 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.326388 24616 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.326766 24602 ts_tablet_manager.cc:1397] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.004s
I20260809 23:17:29.326800 24616 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.329255 24602 raft_consensus.cc:348] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.329910 24602 raft_consensus.cc:374] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.330168 24602 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.330861 24602 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.333149 24602 ts_tablet_manager.cc:1428] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:29.333999 24602 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.335588 24611 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.335712 24616 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.336108 24611 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.338150 24611 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.338532 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.339212 24435 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.339982 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.340299 24602 tablet_bootstrap.cc:654] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:29.341327 24498 leader_election.cc:341] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.342633 24603 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.343082 24603 ts_tablet_manager.cc:1397] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.003s
I20260809 23:17:29.343847 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.344717 24431 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.346232 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:29.347716 24611 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.348191 24611 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.348702 24618 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.348771 24611 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.349395 24618 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.353493 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:29.355831 24378 leader_election.cc:341] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.358399 24603 raft_consensus.cc:348] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.359220 24603 raft_consensus.cc:374] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.359501 24603 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.360172 24603 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.360489 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.361141 24545 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:29.363066 24603 ts_tablet_manager.cc:1428] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.020s	user 0.014s	sys 0.003s
I20260809 23:17:29.362857 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:29.363947 24603 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.364003 24618 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.364655 24618 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.365093 24618 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.365772 24618 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.326388 24605 ts_tablet_manager.cc:1397] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.037s	user 0.019s	sys 0.015s
I20260809 23:17:29.370770 24603 tablet_bootstrap.cc:654] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.370424 24605 raft_consensus.cc:348] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.372426 24605 raft_consensus.cc:374] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.372785 24605 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.373566 24605 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.375502 24435 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.378904 24266 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.380079 24661 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.380470 24616 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.380877 24661 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:29.383551 24611 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.384536 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.385313 24545 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.382272 24661 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } }
I20260809 23:17:29.387102 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.389667 24323 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.376286 24605 ts_tablet_manager.cc:1428] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260809 23:17:29.400018 24605 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.401929 24611 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.403362 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.403838 24602 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.404048 24435 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.404310 24602 ts_tablet_manager.cc:1397] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.070s	user 0.036s	sys 0.031s
I20260809 23:17:29.406195 24323 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.407003 24602 raft_consensus.cc:348] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.407735 24602 raft_consensus.cc:374] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.407995 24602 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.408419 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:29.408706 24602 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.409324 24611 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.410043 24611 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:29.410677 24602 ts_tablet_manager.cc:1428] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:29.410992 24611 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.411526 24602 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.414098 24435 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.417745 24602 tablet_bootstrap.cc:654] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.417860 24603 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.418187 24605 tablet_bootstrap.cc:654] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.425187 24545 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.418309 24603 ts_tablet_manager.cc:1397] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.055s	user 0.018s	sys 0.016s
I20260809 23:17:29.422338 24321 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.423425 24602 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.428656 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:17:29.428750 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66184.4us
I20260809 23:17:29.429067 24602 ts_tablet_manager.cc:1397] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.005s
I20260809 23:17:29.429832 24649 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.430361 24603 raft_consensus.cc:348] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.431470 24603 raft_consensus.cc:374] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.431689 24603 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.432251 24603 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.432622 24618 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.438344 24618 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.434136 24603 ts_tablet_manager.cc:1428] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.001s
I20260809 23:17:29.440552 24603 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.443400 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.434854 24602 raft_consensus.cc:348] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.443768 24649 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:29.444041 24602 raft_consensus.cc:374] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.443146 24165 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:29.444357 24602 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.444809 24649 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.445068 24602 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.447305 24602 ts_tablet_manager.cc:1428] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260809 23:17:29.447714 24545 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.450804 24602 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.455173 24545 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:29.457026 24603 tablet_bootstrap.cc:654] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.456866 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.458664 24702 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.459676 24661 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.459443 24164 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:29.460141 24661 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.460474 24712 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.462137 24661 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.462424 24602 tablet_bootstrap.cc:654] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.462972 24702 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:29.463289 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.463919 24546 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:29.463826 24702 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.466300 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.467092 24435 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:29.468582 24270 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:29.470997 24661 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.471511 24661 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.471989 24661 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.477118 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.461140 24712 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:29.478416 24378 leader_election.cc:341] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.480674 24712 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.481011 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:29.482889 24378 leader_election.cc:341] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.481684 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:29.484645 24380 leader_election.cc:341] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.485100 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.486196 24712 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.484856 24661 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.489940 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.490557 24602 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.490518 24435 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.491743 24602 ts_tablet_manager.cc:1397] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.041s	user 0.013s	sys 0.009s
I20260809 23:17:29.492551 24661 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.492852 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.499588 24545 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.499653 24649 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.500146 24649 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.502281 24649 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.497345 24168 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:29.503436 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.508003 24435 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.510891 24545 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:29.503786 24602 raft_consensus.cc:348] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.510694 24170 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:29.511628 24602 raft_consensus.cc:374] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.511951 24602 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.512570 24602 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.514026 24270 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:29.514436 24602 ts_tablet_manager.cc:1428] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.000s
I20260809 23:17:29.514916 24661 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.515319 24602 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.504242 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.515576 24661 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:29.515978 24431 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 1: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:17:29.516414 24661 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.516937 24496 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:29.518049 24649 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.506098 24603 tablet_bootstrap.cc:492] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.520653 24605 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.520606 24710 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.521311 24605 ts_tablet_manager.cc:1397] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.122s	user 0.013s	sys 0.003s
I20260809 23:17:29.521278 24710 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.520684 24603 ts_tablet_manager.cc:1397] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.080s	user 0.013s	sys 0.011s
I20260809 23:17:29.522722 24602 tablet_bootstrap.cc:654] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.523545 24710 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.524266 24605 raft_consensus.cc:348] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.524994 24605 raft_consensus.cc:374] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.524868 24603 raft_consensus.cc:348] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.525403 24605 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.525763 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.525583 24603 raft_consensus.cc:374] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.526379 24603 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.526628 24605 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.527428 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.529655 24603 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.531656 24321 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.531865 24605 ts_tablet_manager.cc:1428] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260809 23:17:29.533473 24605 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.534324 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:17:29.535006 24496 leader_election.cc:341] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.535427 24611 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.535784 24611 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.536108 24611 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.538192 24603 ts_tablet_manager.cc:1428] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260809 23:17:29.541239 24649 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.543149 24603 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.550494 24611 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.552663 24611 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.554054 24602 tablet_bootstrap.cc:492] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.553781 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.554226 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.554515 24602 ts_tablet_manager.cc:1397] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.039s	user 0.022s	sys 0.013s
I20260809 23:17:29.554564 24321 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:29.555751 24496 leader_election.cc:341] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.555934 24605 tablet_bootstrap.cc:654] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.557044 24602 raft_consensus.cc:348] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.557754 24602 raft_consensus.cc:374] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.558020 24602 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.558630 24602 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.560560 24602 ts_tablet_manager.cc:1428] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:17:29.562335 24602 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.570576 24712 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.571081 24712 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.573028 24712 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.574051 24603 tablet_bootstrap.cc:654] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.573738 24170 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:29.574051 24712 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.576207 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.576445 24712 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.576849 24323 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:29.577215 24605 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.577626 24605 ts_tablet_manager.cc:1397] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.044s	user 0.004s	sys 0.017s
I20260809 23:17:29.578181 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.578373 24712 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.574077 24321 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.578974 24702 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.579249 24618 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.579376 24702 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.579754 24702 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.579748 24618 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.580821 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.581467 24546 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:29.583951 24618 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.580029 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.586345 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.585888 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:29.587674 24611 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.588135 24611 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:29.588850 24605 raft_consensus.cc:348] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.588944 24611 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.589596 24605 raft_consensus.cc:374] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.589438 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.589939 24605 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.590698 24602 tablet_bootstrap.cc:654] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.590560 24605 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.586627 24545 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:29.593381 24605 ts_tablet_manager.cc:1428] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
W20260809 23:17:29.593348 24380 leader_election.cc:341] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.593932 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.594764 24605 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.595466 24618 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.595858 24618 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:17:29.597380 24378 leader_election.cc:341] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:29.597908 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.598861 24712 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.599164 24618 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.599526 24659 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:29.605780 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.606506 24321 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:29.607467 24702 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.609629 24702 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.614568 24702 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.614357 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.615111 24546 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.619630 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.620432 24321 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.624270 24546 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:29.625623 24605 tablet_bootstrap.cc:654] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.625411 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.627354 24712 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.639154 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:29.640188 24659 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:29.628726 24702 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.641840 24712 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:29.644831 24712 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.680641 24659 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:29.684010 24170 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:29.685644 24603 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.686213 24603 ts_tablet_manager.cc:1397] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.143s	user 0.012s	sys 0.017s
I20260809 23:17:29.686244 24618 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.689525 24603 raft_consensus.cc:348] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.690451 24603 raft_consensus.cc:374] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.690600 24618 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.690740 24603 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.691028 24546 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.691936 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:29.692708 24544 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.695402 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.696154 24323 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.702461 24603 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.704676 24603 ts_tablet_manager.cc:1428] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.003s
I20260809 23:17:29.705816 24603 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.710101 24611 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.710659 24611 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.713943 24611 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.714461 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.715366 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.717507 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:29.719544 24611 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:17:29.721948 24602 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.722496 24602 ts_tablet_manager.cc:1397] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.160s	user 0.015s	sys 0.011s
I20260809 23:17:29.723218 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:29.724017 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.724543 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.725754 24659 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:29.727181 24168 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:17:29.729311 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:29.730400 24730 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.731130 24730 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:29.721804 24323 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:29.732618 24603 tablet_bootstrap.cc:654] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.732019 24730 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.736382 24602 raft_consensus.cc:348] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.737068 24602 raft_consensus.cc:374] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.737350 24602 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
W20260809 23:17:29.738133 24728 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.055s	user 0.003s	sys 0.009s
I20260809 23:17:29.739418 24712 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.740054 24712 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.741935 24321 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:29.741706 24602 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.747450 24712 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.748001 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.748611 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.749575 24321 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:29.750810 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.752312 24730 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.755395 24430 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.759469 24605 tablet_bootstrap.cc:492] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.759903 24605 ts_tablet_manager.cc:1397] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.165s	user 0.016s	sys 0.019s
I20260809 23:17:29.758838 24602 ts_tablet_manager.cc:1428] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.036s	user 0.004s	sys 0.005s
I20260809 23:17:29.761369 24602 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.763840 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:29.765507 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:29.769179 24611 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.769796 24611 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.770846 24605 raft_consensus.cc:348] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.771772 24605 raft_consensus.cc:374] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.772001 24605 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.769114 24735 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.762945 24702 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:29.780985 24605 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.783066 24605 ts_tablet_manager.cc:1428] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.002s
I20260809 23:17:29.785796 24611 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.787225 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.788111 24321 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.789732 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:29.787250 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.791673 24429 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.792781 24649 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.793170 24649 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.793510 24649 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.807488 24735 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.787662 24605 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.809582 24649 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.791026 24730 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:29.820498 24649 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.820365 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:29.820749 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:29.821112 24321 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.821424 24429 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.811766 24602 tablet_bootstrap.cc:654] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.826246 24661 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:29.838567 24605 tablet_bootstrap.cc:654] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.841020 24321 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.843271 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:29.841743 24649 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:17:29.844121 24611 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:29.845582 24611 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:29.846482 24611 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.847308 24603 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:29.847730 24603 ts_tablet_manager.cc:1397] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.142s	user 0.027s	sys 0.015s
I20260809 23:17:29.851990 24429 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:29.850275 24603 raft_consensus.cc:348] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.854250 24603 raft_consensus.cc:374] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.854590 24603 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.855301 24603 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.857402 24605 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:29.857928 24605 ts_tablet_manager.cc:1397] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.070s	user 0.011s	sys 0.011s
I20260809 23:17:29.862439 24603 ts_tablet_manager.cc:1428] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260809 23:17:29.864117 24603 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:29.865732 24170 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:29.870434 24605 raft_consensus.cc:348] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.871131 24605 raft_consensus.cc:374] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.871412 24605 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.872265 24605 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.874425 24602 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:29.874881 24602 ts_tablet_manager.cc:1397] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.114s	user 0.023s	sys 0.010s
I20260809 23:17:29.878559 24605 ts_tablet_manager.cc:1428] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
I20260809 23:17:29.879689 24605 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:29.879226 24164 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:29.880029 24602 raft_consensus.cc:348] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.880663 24602 raft_consensus.cc:374] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:29.880923 24602 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:29.881515 24602 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.891294 24659 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:29.894089 24603 tablet_bootstrap.cc:654] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.898509 24649 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.899099 24649 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.897353 24730 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:17:29.909777 24605 tablet_bootstrap.cc:654] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:29.911706 24602 ts_tablet_manager.cc:1428] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20260809 23:17:29.913036 24649 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.918079 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.918869 24602 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:29.918768 24435 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.921132 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:29.922472 24649 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:29.923009 24649 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:29.923445 24649 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.899278 24611 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260809 23:17:29.931226 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:29.932022 24321 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:29.933265 24611 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:29.934217 24611 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.944617 24659 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:29.953004 24611 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:29.969416 24321 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:29.951370 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:29.956575 24649 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:29.984575 24602 tablet_bootstrap.cc:654] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:29.991463 24496 leader_election.cc:341] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:30.001682 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61019.7us
I20260809 23:17:30.012343 24760 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260809 23:17:30.015455 24649 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.017469 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.018438 24431 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.020264 24605 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.020987 24605 ts_tablet_manager.cc:1397] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.142s	user 0.020s	sys 0.022s
I20260809 23:17:30.029189 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.038208 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:30.038834 24323 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.041793 24602 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.042243 24602 ts_tablet_manager.cc:1397] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.124s	user 0.014s	sys 0.027s
I20260809 23:17:30.043109 24661 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.043449 24710 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.043612 24661 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.049075 24603 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.049535 24603 ts_tablet_manager.cc:1397] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.186s	user 0.018s	sys 0.022s
I20260809 23:17:30.046720 24431 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:30.057204 24605 raft_consensus.cc:348] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.058179 24605 raft_consensus.cc:374] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.058511 24605 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.059631 24605 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.062744 24605 ts_tablet_manager.cc:1428] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.003s
I20260809 23:17:30.064399 24605 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
W20260809 23:17:30.048991 24498 leader_election.cc:341] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.067266 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.066354 24603 raft_consensus.cc:348] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.070566 24603 raft_consensus.cc:374] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.070969 24603 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.073680 24661 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.081081 24603 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.085445 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.088810 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.091460 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.092288 24659 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 23:17:30.093508 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44528.3us
I20260809 23:17:30.093600 24616 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.096773 24734 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.101575 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.102597 24760 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.102967 24649 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.103421 24760 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.104044 24760 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:30.106287 24760 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.112805 24323 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:30.117455 24603 ts_tablet_manager.cc:1428] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.068s	user 0.010s	sys 0.000s
I20260809 23:17:30.119117 24603 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.122150 24734 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.142546 24605 tablet_bootstrap.cc:654] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.146875 24602 raft_consensus.cc:348] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.147688 24602 raft_consensus.cc:374] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.147969 24602 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.148720 24164 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:30.151147 24602 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.155104 24602 ts_tablet_manager.cc:1428] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.113s	user 0.009s	sys 0.000s
I20260809 23:17:30.156024 24602 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.162323 24603 tablet_bootstrap.cc:654] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.158818 24730 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.163672 24730 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.175781 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.176072 24734 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.177318 24323 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:30.177618 24734 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.179167 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.180342 24702 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.180773 24702 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.181152 24702 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.182507 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.183657 24735 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.184212 24735 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:30.184607 24380 leader_election.cc:341] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.196120 24730 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.180789 24734 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.198117 24735 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.216804 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.218729 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.226918 24765 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:17:30.230036 24602 tablet_bootstrap.cc:654] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.234026 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.234972 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.237521 24760 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.237982 24760 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.239413 24734 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.240696 24760 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.242717 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:30.238114 24380 leader_election.cc:341] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:30.243803 24380 leader_election.cc:341] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:30.244570 24496 leader_election.cc:341] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.245803 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:30.247095 24498 leader_election.cc:341] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.247502 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.248543 24765 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:17:30.256254 24378 leader_election.cc:341] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.256912 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:30.261890 24734 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.262390 24702 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.263454 24735 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:17:30.265123 24378 leader_election.cc:341] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.265563 24378 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:30.270965 24735 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.289079 24603 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.289525 24603 ts_tablet_manager.cc:1397] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.171s	user 0.014s	sys 0.012s
I20260809 23:17:30.293848 24603 raft_consensus.cc:348] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.294863 24603 raft_consensus.cc:374] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.295274 24603 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.296216 24603 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.298607 24603 ts_tablet_manager.cc:1428] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.002s
I20260809 23:17:30.299549 24603 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.304028 24602 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.304522 24602 ts_tablet_manager.cc:1397] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.149s	user 0.017s	sys 0.030s
I20260809 23:17:30.306849 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:30.307650 24320 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.310832 24603 tablet_bootstrap.cc:654] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.313243 24702 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.330049 24320 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:30.334424 24602 raft_consensus.cc:348] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.335233 24602 raft_consensus.cc:374] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.335465 24602 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.336365 24602 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.337010 24605 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.338567 24605 ts_tablet_manager.cc:1397] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.275s	user 0.032s	sys 0.032s
I20260809 23:17:30.340584 24602 ts_tablet_manager.cc:1428] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.036s	user 0.010s	sys 0.000s
I20260809 23:17:30.341519 24602 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.341370 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.342602 24797 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.344484 24797 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:30.345429 24797 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.347288 24605 raft_consensus.cc:348] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.347937 24605 raft_consensus.cc:374] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.348188 24605 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.349521 24605 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.351667 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.356395 24605 ts_tablet_manager.cc:1428] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260809 23:17:30.357357 24605 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.359038 24602 tablet_bootstrap.cc:654] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.379817 24605 tablet_bootstrap.cc:654] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:30.371908 24380 leader_election.cc:341] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.381351 24797 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.390393 24603 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.391167 24603 ts_tablet_manager.cc:1397] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.092s	user 0.013s	sys 0.003s
I20260809 23:17:30.395637 24603 raft_consensus.cc:348] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.396559 24603 raft_consensus.cc:374] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.396853 24603 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.397598 24603 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.400504 24603 ts_tablet_manager.cc:1428] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260809 23:17:30.401547 24603 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.414211 24603 tablet_bootstrap.cc:654] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.413854 24765 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.415455 24760 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.410622 24602 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.422644 24602 ts_tablet_manager.cc:1397] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.081s	user 0.011s	sys 0.012s
I20260809 23:17:30.425930 24602 raft_consensus.cc:348] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.429145 24603 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.429579 24603 ts_tablet_manager.cc:1397] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.028s	user 0.007s	sys 0.009s
I20260809 23:17:30.426733 24602 raft_consensus.cc:374] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.435645 24602 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.432698 24603 raft_consensus.cc:348] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.436324 24603 raft_consensus.cc:374] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.436595 24603 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.436688 24170 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:30.437459 24603 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.438380 24602 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.439610 24605 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.440042 24605 ts_tablet_manager.cc:1397] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.083s	user 0.004s	sys 0.019s
I20260809 23:17:30.440419 24603 ts_tablet_manager.cc:1428] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260809 23:17:30.441877 24602 ts_tablet_manager.cc:1428] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260809 23:17:30.442749 24602 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.442447 24605 raft_consensus.cc:348] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.443140 24605 raft_consensus.cc:374] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.443423 24605 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.443976 24605 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.446493 24605 ts_tablet_manager.cc:1428] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260809 23:17:30.447891 24605 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.448956 24603 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.452481 24765 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.452952 24765 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.453841 24605 tablet_bootstrap.cc:654] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.455808 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.456384 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:30.457830 24496 leader_election.cc:341] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.458370 24765 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.461598 24603 tablet_bootstrap.cc:654] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.465725 24602 tablet_bootstrap.cc:654] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:30.461369 24498 leader_election.cc:341] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.466874 24498 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.468780 24760 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.475239 24603 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.475775 24603 ts_tablet_manager.cc:1397] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.027s	user 0.007s	sys 0.003s
I20260809 23:17:30.478847 24603 raft_consensus.cc:348] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.479729 24603 raft_consensus.cc:374] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.480028 24603 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.480819 24603 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.483928 24603 ts_tablet_manager.cc:1428] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260809 23:17:30.484998 24603 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.492744 24603 tablet_bootstrap.cc:654] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.497334 24602 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.498082 24602 ts_tablet_manager.cc:1397] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.055s	user 0.023s	sys 0.010s
I20260809 23:17:30.501564 24605 tablet_bootstrap.cc:492] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.502023 24605 ts_tablet_manager.cc:1397] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.054s	user 0.021s	sys 0.024s
I20260809 23:17:30.503022 24797 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.503518 24797 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.503130 24602 raft_consensus.cc:348] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.504572 24602 raft_consensus.cc:374] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.504727 24603 tablet_bootstrap.cc:492] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.504441 24605 raft_consensus.cc:348] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.504971 24602 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.505175 24605 raft_consensus.cc:374] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.505261 24603 ts_tablet_manager.cc:1397] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.005s
I20260809 23:17:30.505622 24797 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.506693 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.505546 24605 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.506824 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.507823 24765 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.507539 24605 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:30.508272 24380 leader_election.cc:341] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.508345 24765 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:30.508882 24378 leader_election.cc:341] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.505940 24602 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.509318 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:30.510344 24765 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.511195 24603 raft_consensus.cc:348] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.511412 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.511858 24603 raft_consensus.cc:374] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.512130 24603 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.512420 24797 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.512810 24603 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.513085 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:30.514043 24496 leader_election.cc:341] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.514515 24605 ts_tablet_manager.cc:1428] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260809 23:17:30.514972 24603 ts_tablet_manager.cc:1428] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.001s
W20260809 23:17:30.515103 24498 leader_election.cc:341] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.515456 24605 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.515542 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.515812 24603 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.516609 24765 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.522249 24602 ts_tablet_manager.cc:1428] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.001s
I20260809 23:17:30.522493 24603 tablet_bootstrap.cc:654] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.523283 24602 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.523707 24605 tablet_bootstrap.cc:654] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.530979 24602 tablet_bootstrap.cc:654] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.533303 24603 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.533746 24603 ts_tablet_manager.cc:1397] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.007s
I20260809 23:17:30.536815 24603 raft_consensus.cc:348] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.537678 24603 raft_consensus.cc:374] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.537992 24603 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.540410 24603 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.541672 24605 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.542106 24605 ts_tablet_manager.cc:1397] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.027s	user 0.013s	sys 0.010s
I20260809 23:17:30.542969 24603 ts_tablet_manager.cc:1428] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260809 23:17:30.543805 24603 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.544893 24602 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.545351 24602 ts_tablet_manager.cc:1397] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.022s	user 0.006s	sys 0.012s
I20260809 23:17:30.546167 24765 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:30.547169 24760 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.547609 24760 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.549639 24760 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.550462 24602 raft_consensus.cc:348] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.551446 24602 raft_consensus.cc:374] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.551874 24602 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.552706 24602 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.554971 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.555568 24431 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:30.555915 24605 raft_consensus.cc:348] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.556571 24605 raft_consensus.cc:374] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.556761 24797 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.556864 24605 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.557236 24797 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.557581 24605 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.558684 24603 tablet_bootstrap.cc:654] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.559487 24602 ts_tablet_manager.cc:1428] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260809 23:17:30.559633 24605 ts_tablet_manager.cc:1428] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260809 23:17:30.560559 24602 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.560297 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:30.562116 24378 leader_election.cc:341] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.563288 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.564857 24797 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.565160 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.565838 24317 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:30.566840 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.567473 24546 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:30.568442 24760 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.568819 24760 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.569015 24602 tablet_bootstrap.cc:654] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.569233 24380 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:30.572247 24760 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.577688 24797 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.578159 24797 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.578591 24797 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.578886 24760 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.583045 24603 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.560708 24605 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.583503 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:30.584110 24317 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.584913 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.585486 24431 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.587612 24797 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.601294 24730 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.601778 24730 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.603703 24730 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.597337 24760 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.597570 24765 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.603744 24797 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.605103 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.606217 24605 tablet_bootstrap.cc:654] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.606422 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.598006 24431 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:30.607069 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.607918 24545 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.608281 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.609097 24649 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.609560 24649 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:30.583500 24603 ts_tablet_manager.cc:1397] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.040s	user 0.015s	sys 0.004s
I20260809 23:17:30.605109 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:30.611649 24378 leader_election.cc:341] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.610736 24649 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.612277 24317 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
W20260809 23:17:30.613559 24380 leader_election.cc:341] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:30.613833 24378 leader_election.cc:341] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.614027 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:30.615095 24797 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.619406 24661 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.620007 24661 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.620930 24603 raft_consensus.cc:348] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.621891 24603 raft_consensus.cc:374] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.617432 24545 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:30.622193 24603 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.623344 24661 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.623008 24603 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.623281 24380 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:30.624732 24797 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.625600 24797 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:30.628937 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.629243 24797 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.629804 24430 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.632236 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.633297 24661 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.633715 24661 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.634104 24661 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.642107 24602 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.642563 24602 ts_tablet_manager.cc:1397] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.082s	user 0.012s	sys 0.002s
I20260809 23:17:30.645068 24602 raft_consensus.cc:348] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.645833 24602 raft_consensus.cc:374] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.646140 24602 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.646957 24602 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.649304 24602 ts_tablet_manager.cc:1428] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:17:30.642339 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.650422 24545 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.654069 24602 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.661823 24603 ts_tablet_manager.cc:1428] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.002s
I20260809 23:17:30.663007 24603 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.645062 24661 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.674638 24661 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.677505 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.679836 24545 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.682256 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.683027 24431 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.684355 24165 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:30.689018 24611 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.689543 24611 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.691301 24611 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.692781 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.693069 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.693872 24317 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
W20260809 23:17:30.694214 24496 leader_election.cc:341] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.694974 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.695643 24611 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.696128 24611 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.695791 24164 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:30.696470 24611 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.700676 24616 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.701476 24616 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.708757 24605 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.709219 24605 ts_tablet_manager.cc:1397] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.149s	user 0.014s	sys 0.015s
I20260809 23:17:30.709631 24602 tablet_bootstrap.cc:654] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.711321 24603 tablet_bootstrap.cc:654] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.714814 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.715699 24546 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.709290 24616 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.716974 24605 raft_consensus.cc:348] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.717753 24605 raft_consensus.cc:374] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.718345 24605 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.719054 24605 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.718591 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.720175 24430 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.717753 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:30.721367 24605 ts_tablet_manager.cc:1428] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260809 23:17:30.722275 24605 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.723659 24616 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.724071 24616 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.724462 24616 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.717972 24649 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.725104 24649 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.716976 24545 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.729446 24431 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.728319 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:30.734321 24616 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.735407 24649 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.737211 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:30.736403 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.738819 24661 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.739362 24661 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:17:30.739573 24496 leader_election.cc:341] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.740206 24661 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:30.741425 24498 leader_election.cc:341] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.741863 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.747030 24649 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.749940 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.751925 24431 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.753245 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.754285 24545 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.758813 24605 tablet_bootstrap.cc:654] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.753883 24616 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.764524 24602 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.765014 24602 ts_tablet_manager.cc:1397] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.111s	user 0.017s	sys 0.007s
I20260809 23:17:30.767714 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.766708 24165 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:30.769068 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:30.769837 24616 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.770294 24616 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:30.768687 24602 raft_consensus.cc:348] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.771528 24602 raft_consensus.cc:374] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.771916 24602 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.771571 24616 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.772521 24602 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.775008 24602 ts_tablet_manager.cc:1428] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260809 23:17:30.776319 24431 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.777009 24603 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.777416 24603 ts_tablet_manager.cc:1397] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.115s	user 0.012s	sys 0.004s
I20260809 23:17:30.777611 24602 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.779932 24603 raft_consensus.cc:348] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.780507 24603 raft_consensus.cc:374] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.780726 24603 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.781412 24603 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.783741 24611 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.786511 24603 ts_tablet_manager.cc:1428] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260809 23:17:30.788748 24603 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.816721 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.828394 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:30.829614 24496 leader_election.cc:341] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.830106 24616 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.793960 24602 tablet_bootstrap.cc:654] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.831939 24616 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.816742 24611 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:30.813657 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:30.826678 24603 tablet_bootstrap.cc:654] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.831233 24605 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.838413 24605 ts_tablet_manager.cc:1397] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.116s	user 0.014s	sys 0.012s
I20260809 23:17:30.829056 24317 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.840999 24605 raft_consensus.cc:348] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.842491 24605 raft_consensus.cc:374] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.841804 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.842804 24605 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.843269 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.843947 24545 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.843595 24605 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.845261 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:30.845465 24317 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:30.846053 24659 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.846237 24605 ts_tablet_manager.cc:1428] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:17:30.843274 24431 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.846478 24659 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.846544 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.846956 24659 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.847570 24605 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.847752 24611 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.848212 24611 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:30.848989 24611 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.850098 24165 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:30.855392 24605 tablet_bootstrap.cc:654] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.855989 24659 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.858160 24659 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.842084 24616 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.860468 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.861114 24431 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.862442 24602 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.863046 24602 ts_tablet_manager.cc:1397] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.086s	user 0.015s	sys 0.000s
I20260809 23:17:30.866205 24602 raft_consensus.cc:348] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.866552 24431 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.868287 24266 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.869673 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.869957 24605 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.870262 24545 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.870301 24661 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.870853 24605 ts_tablet_manager.cc:1397] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260809 23:17:30.874125 24605 raft_consensus.cc:348] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.874842 24605 raft_consensus.cc:374] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.875141 24605 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.875725 24605 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.877801 24605 ts_tablet_manager.cc:1428] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:17:30.878621 24605 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:30.880921 24603 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.881417 24603 ts_tablet_manager.cc:1397] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.094s	user 0.009s	sys 0.005s
I20260809 23:17:30.881824 24602 raft_consensus.cc:374] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.882061 24602 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.882676 24602 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.884717 24602 ts_tablet_manager.cc:1428] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260809 23:17:30.885073 24603 raft_consensus.cc:348] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:30.886821 24380 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:30.887058 24603 raft_consensus.cc:374] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.887548 24603 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.888460 24611 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.888249 24603 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.891924 24603 ts_tablet_manager.cc:1428] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260809 23:17:30.888885 24611 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.893425 24603 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.892503 24165 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:30.894630 24545 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.895725 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:30.896898 24498 leader_election.cc:341] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.898025 24611 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:30.892396 24605 tablet_bootstrap.cc:654] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.900317 24603 tablet_bootstrap.cc:654] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.900846 24730 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:30.901765 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.902351 24602 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
W20260809 23:17:30.903168 24496 leader_election.cc:341] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:30.903612 24496 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:30.904706 24611 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:30.910921 24602 tablet_bootstrap.cc:654] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.912496 24661 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:30.913362 24661 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.928177 24603 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.928812 24603 ts_tablet_manager.cc:1397] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.036s	user 0.018s	sys 0.011s
I20260809 23:17:30.939599 24603 raft_consensus.cc:348] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.940316 24603 raft_consensus.cc:374] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.940555 24603 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.942097 24603 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.944389 24603 ts_tablet_manager.cc:1428] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.002s
I20260809 23:17:30.945297 24603 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.947124 24659 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:30.947566 24659 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.949863 24659 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.951064 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:30.951498 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:30.951676 24545 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.952085 24430 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:30.952716 24603 tablet_bootstrap.cc:654] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.953402 24266 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:30.955271 24659 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:30.955685 24659 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:30.956055 24659 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.958578 24603 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.959088 24603 ts_tablet_manager.cc:1397] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.014s	user 0.012s	sys 0.000s
I20260809 23:17:30.960460 24602 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:30.960939 24602 ts_tablet_manager.cc:1397] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.059s	user 0.011s	sys 0.002s
I20260809 23:17:30.961508 24603 raft_consensus.cc:348] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.962224 24603 raft_consensus.cc:374] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.962481 24603 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.963120 24603 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.964095 24659 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.965397 24603 ts_tablet_manager.cc:1428] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:30.965478 24602 raft_consensus.cc:348] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.966357 24603 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.965422 24605 tablet_bootstrap.cc:492] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:30.966475 24602 raft_consensus.cc:374] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.966687 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:30.967489 24659 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:30.968700 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:30.969396 24545 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.930840 24170 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:30.973845 24603 tablet_bootstrap.cc:654] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:30.967521 24430 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:30.981240 24603 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:30.981685 24603 ts_tablet_manager.cc:1397] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.007s
I20260809 23:17:30.983846 24545 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.984066 24603 raft_consensus.cc:348] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.986604 24603 raft_consensus.cc:374] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.986902 24603 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.986729 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:30.985319 24430 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:30.988233 24659 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:30.988777 24659 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:30.989233 24603 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.989619 24659 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.985704 24602 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:30.992301 24603 ts_tablet_manager.cc:1428] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.010s	user 0.001s	sys 0.006s
I20260809 23:17:30.992645 24602 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.993945 24603 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:30.966886 24605 ts_tablet_manager.cc:1397] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.088s	user 0.020s	sys 0.004s
I20260809 23:17:30.995589 24602 ts_tablet_manager.cc:1428] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.002s
I20260809 23:17:30.996471 24602 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:30.997488 24605 raft_consensus.cc:348] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:30.998273 24605 raft_consensus.cc:374] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:30.998605 24605 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.001796 24603 tablet_bootstrap.cc:654] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.003526 24605 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.007180 24602 tablet_bootstrap.cc:654] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.008657 24605 ts_tablet_manager.cc:1428] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.002s
I20260809 23:17:31.009953 24605 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:31.018649 24602 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:31.019184 24602 ts_tablet_manager.cc:1397] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.011s
I20260809 23:17:31.021593 24602 raft_consensus.cc:348] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.022347 24605 tablet_bootstrap.cc:654] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.022401 24602 raft_consensus.cc:374] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.023005 24602 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.023754 24602 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.035234 24605 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:31.035734 24605 ts_tablet_manager.cc:1397] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.007s
I20260809 23:17:31.036243 24603 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:31.036788 24603 ts_tablet_manager.cc:1397] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.043s	user 0.021s	sys 0.018s
I20260809 23:17:31.036015 24170 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:31.026182 24602 ts_tablet_manager.cc:1428] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260809 23:17:31.039870 24602 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:31.041682 24605 raft_consensus.cc:348] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.042464 24605 raft_consensus.cc:374] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.043054 24605 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.043999 24605 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.044265 24603 raft_consensus.cc:348] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.045145 24603 raft_consensus.cc:374] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.045469 24603 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.046533 24605 ts_tablet_manager.cc:1428] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260809 23:17:31.046141 24603 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.047506 24605 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:31.048310 24603 ts_tablet_manager.cc:1428] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260809 23:17:31.051265 24603 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:31.053565 24602 tablet_bootstrap.cc:654] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.057371 24603 tablet_bootstrap.cc:654] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.060593 24317 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.069523 24603 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:31.070205 24603 ts_tablet_manager.cc:1397] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.003s
I20260809 23:17:31.069607 24702 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260809 23:17:31.074091 24603 raft_consensus.cc:348] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.074849 24603 raft_consensus.cc:374] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.075155 24603 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.060626 24605 tablet_bootstrap.cc:654] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.075906 24603 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.078110 24659 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.078831 24603 ts_tablet_manager.cc:1428] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.004s
I20260809 23:17:31.078621 24659 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.079792 24603 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:31.081749 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.082470 24545 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:31.083030 24659 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.084162 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:31.075232 24760 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.088912 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.089547 24430 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:31.090291 24661 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.090834 24661 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.091301 24661 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.095471 24760 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.095685 24603 tablet_bootstrap.cc:654] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.095963 24760 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.097756 24702 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.098299 24702 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.098806 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.100270 24702 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.103080 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.104089 24760 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:31.105031 24378 leader_election.cc:341] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.106418 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.107091 24545 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.108352 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.106366 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.109107 24702 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.110042 24702 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.110455 24702 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.110638 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:31.111754 24765 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:31.114981 24661 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.116578 24760 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:31.117794 24702 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.122402 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:31.126960 24603 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:31.127494 24603 ts_tablet_manager.cc:1397] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.048s	user 0.006s	sys 0.011s
W20260809 23:17:31.128226 24378 leader_election.cc:341] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.130688 24603 raft_consensus.cc:348] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.131644 24603 raft_consensus.cc:374] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.131947 24603 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.132500 24797 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:31.132714 24603 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.134137 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:31.134692 24431 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.134895 24603 ts_tablet_manager.cc:1428] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260809 23:17:31.140662 24603 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:31.145246 24602 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:31.145879 24602 ts_tablet_manager.cc:1397] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.106s	user 0.014s	sys 0.006s
I20260809 23:17:31.151988 24702 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.153579 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:31.154660 24546 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.148221 24659 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.173452 24760 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.176456 24649 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.187781 24605 tablet_bootstrap.cc:492] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:31.188256 24605 ts_tablet_manager.cc:1397] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.141s	user 0.024s	sys 0.059s
I20260809 23:17:31.188585 24765 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.189039 24765 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.190593 24605 raft_consensus.cc:348] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.191181 24605 raft_consensus.cc:374] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.191397 24605 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.191556 24616 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.192157 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.192004 24605 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.174700 24659 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.179678 24760 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.193581 24765 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.194350 24605 ts_tablet_manager.cc:1428] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20260809 23:17:31.242522 24659 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.197935 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:31.198508 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.261605 24605 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:31.209975 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:31.262228 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.262513 24544 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.212409 24431 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:31.206059 24603 tablet_bootstrap.cc:654] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.263499 24797 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.196063 24602 raft_consensus.cc:348] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.267215 24602 raft_consensus.cc:374] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.267307 24797 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:31.267503 24602 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.268170 24602 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.268400 24797 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.269492 24602 ts_tablet_manager.cc:1428] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.123s	user 0.004s	sys 0.003s
I20260809 23:17:31.270149 24602 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:31.208570 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.271224 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.271842 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:31.182174 24649 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.273602 24649 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.274729 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.274961 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.220865 24661 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.275848 24765 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.276052 24661 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.276208 24765 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.276525 24661 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:31.276580 24765 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.277459 24661 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.261798 24429 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:31.279630 24852 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:17:31.263890 24498 leader_election.cc:341] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.201375 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.264868 24659 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.266564 24730 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.213915 24760 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.285806 24730 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.286206 24496 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.287911 24730 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.288846 24611 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.289206 24611 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.289535 24611 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.287487 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.292276 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.292965 24543 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.294456 24380 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.295451 24730 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.295841 24730 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.296149 24730 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.292579 24605 tablet_bootstrap.cc:654] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:31.302466 24496 leader_election.cc:341] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.305718 24611 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.306538 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.307247 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.308641 24605 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:31.309073 24605 ts_tablet_manager.cc:1397] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.048s	user 0.010s	sys 0.000s
I20260809 23:17:31.305684 24730 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.335835 24611 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.335947 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.336820 24543 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:31.338843 24602 tablet_bootstrap.cc:654] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.332633 24605 raft_consensus.cc:348] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.341919 24605 raft_consensus.cc:374] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.342190 24605 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.342850 24605 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.345414 24605 ts_tablet_manager.cc:1428] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260809 23:17:31.351593 24168 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:31.357602 24797 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.359170 24427 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:17:31.358984 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:31.359720 24317 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.365816 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.366859 24730 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.367374 24730 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.367851 24730 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:31.375985 24498 leader_election.cc:341] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:31.362675 24496 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:17:31.377691 24498 leader_election.cc:341] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.378244 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:31.378132 24760 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.377534 24765 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.370239 24544 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:31.372525 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:31.382174 24611 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:31.374776 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:31.383068 24765 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.383497 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:31.384418 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:31.384598 24427 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.384845 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:31.385004 24543 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.385232 24544 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.385776 24435 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.390321 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.391777 24798 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.392297 24798 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:31.392539 24270 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:31.393095 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.393193 24798 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.393750 24852 raft_consensus.cc:2741] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 23:17:31.393798 24321 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.396775 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.397652 24702 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.397363 24852 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:31.398022 24702 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.398386 24702 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.403303 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:31.391908 24498 leader_election.cc:341] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.417806 24605 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:31.359004 24797 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.428200 24730 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.422356 24702 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.446322 24797 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.449826 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.451771 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.444022 24765 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:17:31.456291 24498 leader_election.cc:341] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:31.462824 24378 leader_election.cc:341] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.462179 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.463614 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.470480 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:31.457677 24765 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.465988 24851 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.061s
I20260809 23:17:31.471724 24543 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:31.473876 24765 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.474049 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:31.474645 24542 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:31.474812 24702 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.477350 24427 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:31.477763 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.479133 24428 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:31.477877 24320 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.480275 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.482684 24603 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:31.483157 24603 ts_tablet_manager.cc:1397] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.343s	user 0.024s	sys 0.027s
I20260809 23:17:31.485821 24170 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:31.487332 24602 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:31.487986 24602 ts_tablet_manager.cc:1397] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.218s	user 0.032s	sys 0.051s
I20260809 23:17:31.487031 24611 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:31.489713 24611 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:31.490136 24611 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.489110 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:31.493320 24540 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.500384 24611 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.502547 24611 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.511046 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.511775 24602 raft_consensus.cc:348] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.512447 24602 raft_consensus.cc:374] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.512429 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.512872 24605 tablet_bootstrap.cc:654] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.513432 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:31.512858 24602 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.513444 24611 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.516898 24611 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:31.517779 24611 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.516074 24602 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.521986 24602 ts_tablet_manager.cc:1428] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.002s
I20260809 23:17:31.521718 24603 raft_consensus.cc:348] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.522418 24603 raft_consensus.cc:374] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.522787 24603 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
W20260809 23:17:31.524612 24498 leader_election.cc:341] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.524133 24603 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.515559 24168 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:31.527928 24602 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:31.527982 24603 ts_tablet_manager.cc:1428] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260809 23:17:31.515434 24659 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.534333 24603 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:31.537544 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:31.538252 24427 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.544050 24798 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.536485 24540 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:31.544587 24798 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.546120 24320 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:31.547410 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:31.548465 24702 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.535238 24659 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.549049 24702 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:31.550508 24702 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.550668 24659 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.734326 24798 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:31.794828 24605 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:31.583324 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.738619 24603 tablet_bootstrap.cc:654] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.795346 24605 ts_tablet_manager.cc:1397] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.384s	user 0.020s	sys 0.022s
I20260809 23:17:31.797830 24605 raft_consensus.cc:348] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.798523 24605 raft_consensus.cc:374] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.798808 24605 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.737169 24602 tablet_bootstrap.cc:654] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.535573 24496 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.803151 24649 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.803678 24649 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:31.804605 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.804491 24649 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.806337 24320 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.810721 24603 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:31.811193 24603 ts_tablet_manager.cc:1397] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.277s	user 0.021s	sys 0.047s
I20260809 23:17:31.743572 24851 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260809 23:17:31.542786 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.826337 24851 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:17:31.827220 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 292186us
I20260809 23:17:31.830120 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.792505 24427 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
W20260809 23:17:31.803067 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 271227us
I20260809 23:17:31.831147 24798 raft_consensus.cc:2741] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20260809 23:17:31.534248 24498 leader_election.cc:341] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:31.833088 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 301392us
I20260809 23:17:31.800132 24605 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.835726 24605 ts_tablet_manager.cc:1428] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.000s
I20260809 23:17:31.836090 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:31.836400 24649 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.836604 24605 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:31.836864 24649 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.837651 24430 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:31.816485 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:31.841830 24541 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:17:31.819006 24603 raft_consensus.cc:348] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.842202 24603 raft_consensus.cc:374] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.819247 24852 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:31.842502 24603 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.820796 24602 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:31.843165 24602 ts_tablet_manager.cc:1397] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.315s	user 0.009s	sys 0.003s
I20260809 23:17:31.843199 24603 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.844617 24649 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.847294 24661 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260809 23:17:31.853842 24730 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:31.855665 24797 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.858670 24603 ts_tablet_manager.cc:1428] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.000s
I20260809 23:17:31.859875 24603 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:31.860116 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:31.864790 24661 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:31.866290 24852 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:31.867583 24605 tablet_bootstrap.cc:654] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:31.878504 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:31.880414 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:31.889290 24602 raft_consensus.cc:348] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.890002 24602 raft_consensus.cc:374] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:31.890249 24602 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:31.892530 24602 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.883742 24710 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:31.884661 24760 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:31.886855 24168 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:31.897213 24760 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.899714 24710 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:31.903458 24602 ts_tablet_manager.cc:1428] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.060s	user 0.005s	sys 0.002s
I20260809 23:17:31.904410 24602 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:31.904209 24168 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:17:31.898172 24496 leader_election.cc:341] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:31.900072 24760 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:31.900588 24710 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:31.935223 24702 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:17:31.944667 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260809 23:17:32.480705 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.003s	sys 0.003s
I20260809 23:17:32.494630 24603 tablet_bootstrap.cc:654] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:32.371749 24919 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260809 23:17:32.405838 24760 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:17:32.479655 24710 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.009s	sys 0.003s
I20260809 23:17:32.248756 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:31.999552 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:32.214319 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50410.8us
W20260809 23:17:32.490345 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.004s	sys 0.007s
W20260809 23:17:32.098959 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44623.1us
I20260809 23:17:32.508074 24605 tablet_bootstrap.cc:492] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
W20260809 23:17:32.507005 24920 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260809 23:17:32.492336 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:32.516544 24605 ts_tablet_manager.cc:1397] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.680s	user 0.024s	sys 0.067s
W20260809 23:17:32.506321 24710 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.010s	sys 0.003s
I20260809 23:17:32.507953 24760 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.519456 24798 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:32.535462 24934 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:32.536154 24934 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.578222 24605 raft_consensus.cc:348] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.579088 24605 raft_consensus.cc:374] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:32.579355 24605 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:32.579927 24605 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:32.577579 24923 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
W20260809 23:17:32.599757 24496 leader_election.cc:341] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:32.598913 24937 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260809 23:17:32.605288 24605 ts_tablet_manager.cc:1428] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.086s	user 0.007s	sys 0.003s
I20260809 23:17:32.611002 24851 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:32.618193 24602 tablet_bootstrap.cc:654] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:32.617851 24851 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.620060 24851 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:32.623734 24934 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:32.598156 24730 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260809 23:17:32.639629 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:32.640336 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:32.610049 24760 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:32.639667 24929 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260809 23:17:32.606472 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63450.6us
I20260809 23:17:32.646225 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:32.648288 24730 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:32.648764 24730 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:32.649163 24730 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:32.651355 24611 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:32.656231 24798 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:32.674649 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:32.698272 24603 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:32.698693 24603 ts_tablet_manager.cc:1397] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.839s	user 0.015s	sys 0.021s
I20260809 23:17:32.701666 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:32.707650 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:32.720346 24605 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:32.718786 24603 raft_consensus.cc:348] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.724581 24603 raft_consensus.cc:374] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:32.724893 24603 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:32.725400 24603 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.732295 24943 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:17:32.735085 24603 ts_tablet_manager.cc:1428] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260809 23:17:32.687660 24730 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:32.737586 24498 leader_election.cc:341] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:32.746789 24496 leader_election.cc:341] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:32.766784 24602 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:32.767431 24602 ts_tablet_manager.cc:1397] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.863s	user 0.019s	sys 0.013s
I20260809 23:17:32.770170 24602 raft_consensus.cc:348] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.745226 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:32.770924 24602 raft_consensus.cc:374] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:32.777590 24602 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:32.779088 24602 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.792287 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:32.797228 24602 ts_tablet_manager.cc:1428] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260809 23:17:32.798494 24602 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:32.749486 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:17:32.813300 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79272.7us
W20260809 23:17:32.814368 24498 leader_election.cc:341] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:32.814886 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:32.815817 24611 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:32.816327 24611 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:17:32.820220 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87140.7us
I20260809 23:17:32.821452 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:32.823585 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:32.817355 24498 consensus_peers.cc:487] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:32.840134 24321 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:32.842861 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:32.844467 24603 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:32.823607 24730 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:17:32.857913 24498 leader_election.cc:341] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:32.858343 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:32.859606 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:32.863907 24661 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:32.865715 24760 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:32.854746 24544 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:32.867990 24602 tablet_bootstrap.cc:654] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:32.873065 24603 tablet_bootstrap.cc:654] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:32.875212 24605 tablet_bootstrap.cc:654] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:32.877007 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:32.883322 24165 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:32.885398 24947 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:32.886330 24947 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:32.887317 24317 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:32.890058 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:32.891430 24380 leader_election.cc:341] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:32.891960 24947 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:32.892160 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:32.894982 24730 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:32.895586 24730 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:32.896004 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:32.896427 24730 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:32.897317 24378 leader_election.cc:341] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:32.897717 24378 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:32.898731 24943 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:32.900070 24544 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
W20260809 23:17:32.938344 24972 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260809 23:17:32.939327 24941 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260809 23:17:32.944880 24966 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260809 23:17:32.940006 24498 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:32.951032 24611 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:32.957768 24941 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:32.971539 24603 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:32.972224 24603 ts_tablet_manager.cc:1397] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.128s	user 0.014s	sys 0.000s
I20260809 23:17:32.985535 24605 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:32.986024 24605 ts_tablet_manager.cc:1397] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.266s	user 0.031s	sys 0.007s
I20260809 23:17:32.995136 24602 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:32.995669 24602 ts_tablet_manager.cc:1397] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.197s	user 0.017s	sys 0.012s
I20260809 23:17:33.003190 24605 raft_consensus.cc:348] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.005334 24605 raft_consensus.cc:374] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.005722 24605 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.008816 24602 raft_consensus.cc:348] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.009490 24602 raft_consensus.cc:374] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.009750 24602 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.010413 24602 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.006502 24605 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.030283 24605 ts_tablet_manager.cc:1428] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260809 23:17:33.032184 24602 ts_tablet_manager.cc:1428] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.001s
I20260809 23:17:33.046830 24602 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.047643 24605 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
W20260809 23:17:33.067277 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:33.067114 24170 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.071962 24603 raft_consensus.cc:348] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.073112 24603 raft_consensus.cc:374] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.073382 24603 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.074397 24603 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.082170 24603 ts_tablet_manager.cc:1428] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.110s	user 0.008s	sys 0.000s
I20260809 23:17:33.083079 24603 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.084502 24760 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.084980 24760 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.086175 24605 tablet_bootstrap.cc:654] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.088013 24760 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.088773 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:33.090363 24496 leader_election.cc:341] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.093997 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:33.095471 24498 leader_election.cc:341] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.095897 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.097599 24602 tablet_bootstrap.cc:654] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.101424 24603 tablet_bootstrap.cc:654] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.097872 24760 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.110491 24603 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.111703 24603 ts_tablet_manager.cc:1397] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.001s
I20260809 23:17:33.113685 24602 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.114131 24602 ts_tablet_manager.cc:1397] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.000s
I20260809 23:17:33.116556 24602 raft_consensus.cc:348] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.117287 24602 raft_consensus.cc:374] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.117151 24603 raft_consensus.cc:348] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.117964 24603 raft_consensus.cc:374] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.118283 24603 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.117619 24602 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.119375 24602 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.121636 24602 ts_tablet_manager.cc:1428] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260809 23:17:33.122493 24602 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.123116 24603 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.125731 24603 ts_tablet_manager.cc:1428] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260809 23:17:33.126825 24603 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.127364 24165 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:33.129046 24602 tablet_bootstrap.cc:654] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.139428 24603 tablet_bootstrap.cc:654] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.143155 24605 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.143602 24605 ts_tablet_manager.cc:1397] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.112s	user 0.019s	sys 0.002s
I20260809 23:17:33.145972 24605 raft_consensus.cc:348] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.146647 24605 raft_consensus.cc:374] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.146962 24605 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.147783 24605 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.149091 24602 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.149518 24602 ts_tablet_manager.cc:1397] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.003s
I20260809 23:17:33.150076 24605 ts_tablet_manager.cc:1428] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:17:33.151043 24605 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.153013 24602 raft_consensus.cc:348] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.153667 24602 raft_consensus.cc:374] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.153940 24602 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.154623 24602 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.159370 24603 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.159890 24603 ts_tablet_manager.cc:1397] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.011s
I20260809 23:17:33.160928 24602 ts_tablet_manager.cc:1428] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:17:33.161654 24605 tablet_bootstrap.cc:654] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.164744 24602 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.172005 24943 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.172461 24943 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.174520 24943 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.174085 24603 raft_consensus.cc:348] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.175319 24943 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.175549 24603 raft_consensus.cc:374] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.176024 24603 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.176795 24603 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.177347 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:33.178918 24378 leader_election.cc:341] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.179632 24603 ts_tablet_manager.cc:1428] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.001s
I20260809 23:17:33.175810 24943 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.182911 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.183300 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.183586 24320 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:33.183982 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:33.185990 24605 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.186050 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:33.186486 24605 ts_tablet_manager.cc:1397] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.009s
I20260809 23:17:33.187955 24797 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.188283 24797 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.188608 24797 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.188874 24605 raft_consensus.cc:348] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.189716 24605 raft_consensus.cc:374] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.189975 24605 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.190668 24605 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.193264 24605 ts_tablet_manager.cc:1428] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:17:33.193734 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.194068 24605 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.197399 24798 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.197770 24798 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.198086 24798 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.199045 24943 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.199532 24605 tablet_bootstrap.cc:654] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.199966 24971 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.200642 24971 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.180866 24603 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.207907 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.209693 24797 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.210635 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.212973 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:33.214179 24378 leader_election.cc:341] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.218411 24971 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.218672 24603 tablet_bootstrap.cc:654] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.223701 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:33.219947 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.224452 24540 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.219740 24798 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.225745 24270 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.227077 24971 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.227487 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:33.223802 24380 leader_election.cc:341] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.228075 24320 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.231233 24540 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:33.221276 24797 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.235188 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:33.236251 24943 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.236714 24943 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:33.236846 24320 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:33.237867 24943 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.238389 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.241725 24730 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.242195 24730 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:33.243155 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:33.242995 24730 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:33.244383 24380 leader_election.cc:341] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.257292 24603 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.258163 24603 ts_tablet_manager.cc:1397] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.077s	user 0.009s	sys 0.004s
I20260809 23:17:33.246604 24798 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.267650 24603 raft_consensus.cc:348] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.268867 24603 raft_consensus.cc:374] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.269127 24603 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.271050 24603 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.276364 24602 tablet_bootstrap.cc:654] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.275383 24168 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.278869 24605 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.282186 24605 ts_tablet_manager.cc:1397] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.088s	user 0.029s	sys 0.040s
I20260809 23:17:33.285019 24605 raft_consensus.cc:348] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.285706 24605 raft_consensus.cc:374] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.285992 24605 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.286618 24605 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.291638 24605 ts_tablet_manager.cc:1428] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260809 23:17:33.292287 24941 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.292897 24941 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.293908 24605 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.301290 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.301963 24320 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:33.302834 24941 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.303089 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:33.303995 24941 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.304363 24941 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.304685 24941 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.306293 24603 ts_tablet_manager.cc:1428] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.000s
I20260809 23:17:33.306704 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.307174 24603 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.307331 24435 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:33.313351 24603 tablet_bootstrap.cc:654] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.312693 24168 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.313361 24941 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.318315 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:33.318884 24321 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.322284 24605 tablet_bootstrap.cc:654] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.326543 24602 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.327883 24602 ts_tablet_manager.cc:1397] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.164s	user 0.012s	sys 0.006s
I20260809 23:17:33.328401 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:33.329035 24435 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.331079 24602 raft_consensus.cc:348] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.331910 24602 raft_consensus.cc:374] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.332401 24602 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.333189 24603 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.333105 24602 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.327327 24941 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.335628 24602 ts_tablet_manager.cc:1428] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:17:33.336835 24602 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.342314 24602 tablet_bootstrap.cc:654] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.348255 24605 tablet_bootstrap.cc:492] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.348673 24605 ts_tablet_manager.cc:1397] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.055s	user 0.009s	sys 0.000s
I20260809 23:17:33.326606 24321 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:33.351375 24605 raft_consensus.cc:348] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.352262 24605 raft_consensus.cc:374] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.351986 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:33.352586 24605 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.353268 24605 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.353617 24938 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.354530 24938 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:33.356067 24938 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.333705 24603 ts_tablet_manager.cc:1397] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.003s
I20260809 23:17:33.360790 24603 raft_consensus.cc:348] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.361492 24603 raft_consensus.cc:374] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.361752 24603 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.362341 24603 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.342391 24435 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:33.364598 24603 ts_tablet_manager.cc:1428] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:17:33.364797 24605 ts_tablet_manager.cc:1428] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.001s
I20260809 23:17:33.365481 24603 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.366979 24605 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.376050 24603 tablet_bootstrap.cc:654] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.377137 24602 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.377633 24602 ts_tablet_manager.cc:1397] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.017s
I20260809 23:17:33.379776 24602 raft_consensus.cc:348] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.380451 24602 raft_consensus.cc:374] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.380688 24602 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.381356 24602 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.383283 24602 ts_tablet_manager.cc:1428] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.004s
I20260809 23:17:33.384534 24602 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.387204 24605 tablet_bootstrap.cc:654] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.390187 24602 tablet_bootstrap.cc:654] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.382227 24164 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:33.416391 24943 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:33.442348 24938 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.443017 24938 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.445667 24938 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.447847 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.448809 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:33.449527 24498 leader_election.cc:341] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:33.450074 24496 leader_election.cc:341] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.450467 24496 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.451504 24760 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.454437 24947 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:33.456214 24321 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.460566 24730 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.461702 24730 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.463613 24730 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.466275 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.467715 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:33.467834 24378 leader_election.cc:341] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.468369 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:33.469861 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:33.475261 24797 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.475623 24797 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.475967 24797 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.480588 24602 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.481002 24602 ts_tablet_manager.cc:1397] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.097s	user 0.022s	sys 0.036s
I20260809 23:17:33.482239 24605 tablet_bootstrap.cc:492] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.482620 24605 ts_tablet_manager.cc:1397] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.116s	user 0.013s	sys 0.021s
I20260809 23:17:33.485981 24602 raft_consensus.cc:348] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.486814 24602 raft_consensus.cc:374] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.487123 24602 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.487936 24602 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.488466 24603 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.488878 24603 ts_tablet_manager.cc:1397] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.124s	user 0.009s	sys 0.001s
I20260809 23:17:33.490031 24602 ts_tablet_manager.cc:1428] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260809 23:17:33.491194 24603 raft_consensus.cc:348] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.492185 24603 raft_consensus.cc:374] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.492523 24603 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.490841 24602 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.486358 24797 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.497864 24605 raft_consensus.cc:348] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.498594 24603 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.500402 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:33.498589 24605 raft_consensus.cc:374] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.503443 24605 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.501443 24797 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.504251 24605 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.504822 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:33.505405 24540 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.506903 24605 ts_tablet_manager.cc:1428] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.002s
I20260809 23:17:33.507897 24605 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.511420 24317 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.516849 24605 tablet_bootstrap.cc:654] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.519625 24602 tablet_bootstrap.cc:654] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.521350 24603 ts_tablet_manager.cc:1428] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.003s
I20260809 23:17:33.522295 24603 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.530059 24797 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.530520 24797 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.531327 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:33.532771 24797 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.532897 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.534801 24797 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.534642 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.535317 24797 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:17:33.536481 24378 leader_election.cc:341] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.534657 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:33.539458 24380 leader_election.cc:341] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.539973 24380 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:33.540921 24947 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.536239 24797 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.548584 24317 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:33.555992 24603 tablet_bootstrap.cc:654] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.557916 24938 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.558395 24938 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.560401 24938 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.561863 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.562848 24603 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
W20260809 23:17:33.563126 24496 leader_election.cc:341] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.563324 24603 ts_tablet_manager.cc:1397] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.000s
I20260809 23:17:33.556339 24602 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.563967 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.565074 24602 ts_tablet_manager.cc:1397] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.074s	user 0.010s	sys 0.001s
I20260809 23:17:33.565932 24603 raft_consensus.cc:348] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:33.566603 24498 leader_election.cc:341] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.566690 24603 raft_consensus.cc:374] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.567180 24603 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.567471 24602 raft_consensus.cc:348] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.568141 24602 raft_consensus.cc:374] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.568405 24602 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.567124 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.570310 24938 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.571274 24602 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.580955 24603 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.583456 24603 ts_tablet_manager.cc:1428] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260809 23:17:33.584611 24602 ts_tablet_manager.cc:1428] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.001s
I20260809 23:17:33.584765 24603 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.587582 24602 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.591075 24603 tablet_bootstrap.cc:654] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.590497 24164 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.595019 24605 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.595468 24605 ts_tablet_manager.cc:1397] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.088s	user 0.015s	sys 0.000s
I20260809 23:17:33.597702 24605 raft_consensus.cc:348] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.598359 24605 raft_consensus.cc:374] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.598611 24605 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.599066 24797 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.599538 24797 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.601418 24797 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.603570 24605 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.605664 24605 ts_tablet_manager.cc:1428] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260809 23:17:33.608372 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.607568 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:33.611106 24378 leader_election.cc:341] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.611231 24851 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.608678 24605 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.611688 24851 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.613543 24851 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.615844 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.616500 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:33.618131 24605 tablet_bootstrap.cc:654] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.618160 24942 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.619148 24942 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.620570 24603 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.609099 24602 tablet_bootstrap.cc:654] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.621037 24603 ts_tablet_manager.cc:1397] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.036s	user 0.006s	sys 0.007s
W20260809 23:17:33.611752 24380 leader_election.cc:341] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.606135 24797 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.621968 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:33.622243 24797 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.623538 24730 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.621045 24942 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.624177 24797 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.623553 24603 raft_consensus.cc:348] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.625028 24603 raft_consensus.cc:374] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.625088 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.625337 24603 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.625656 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.626246 24540 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:33.626003 24603 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.626803 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.627130 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.628021 24661 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.628252 24603 ts_tablet_manager.cc:1428] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:17:33.628561 24317 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.628505 24661 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.629047 24603 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.629267 24851 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.629622 24851 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.629933 24851 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.632702 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.630828 24661 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:17:33.633929 24378 leader_election.cc:341] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.634622 24797 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.635037 24603 tablet_bootstrap.cc:654] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.635043 24797 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.636137 24797 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.641649 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.643328 24947 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:33.645951 24797 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.649575 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:33.650864 24378 leader_election.cc:341] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.656750 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:33.657478 24540 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.657897 24797 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.669251 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.669970 24544 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:33.671859 24603 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.671622 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.672310 24603 ts_tablet_manager.cc:1397] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.043s	user 0.025s	sys 0.008s
I20260809 23:17:33.672442 24546 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:17:33.675295 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.676035 24605 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.676517 24605 ts_tablet_manager.cc:1397] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.008s
I20260809 23:17:33.680982 24603 raft_consensus.cc:348] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.681636 24603 raft_consensus.cc:374] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.681883 24603 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.681897 24798 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:33.682579 24603 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.684091 24941 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.684803 24603 ts_tablet_manager.cc:1428] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260809 23:17:33.684520 24941 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.685672 24603 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.686451 24941 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.687883 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.688113 24605 raft_consensus.cc:348] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.689033 24605 raft_consensus.cc:374] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.689350 24605 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.689714 24851 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.690336 24605 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.691536 24603 tablet_bootstrap.cc:654] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:33.691443 24496 leader_election.cc:341] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.692533 24851 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.692560 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:33.693266 24435 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.693732 24605 ts_tablet_manager.cc:1428] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.002s
I20260809 23:17:33.695032 24605 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.697741 24603 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.698074 24603 ts_tablet_manager.cc:1397] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20260809 23:17:33.699875 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.700176 24603 raft_consensus.cc:348] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.700824 24603 raft_consensus.cc:374] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.701107 24603 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.701102 24270 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.702015 24971 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.701933 24603 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.702781 24971 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.703997 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:33.704581 24546 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.705140 24603 ts_tablet_manager.cc:1428] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:17:33.707114 24605 tablet_bootstrap.cc:654] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.709414 24603 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.712419 24602 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:33.712836 24602 ts_tablet_manager.cc:1397] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.125s	user 0.008s	sys 0.007s
I20260809 23:17:33.718410 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.721319 24540 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
W20260809 23:17:33.722522 24498 leader_election.cc:341] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.723186 24498 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.714632 24602 raft_consensus.cc:348] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.724191 24602 raft_consensus.cc:374] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.724577 24602 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.725428 24760 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.723758 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:33.726713 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:33.728576 24798 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.729182 24798 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:33.731361 24602 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.730093 24798 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.734824 24546 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:33.735630 24941 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.736665 24270 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:33.738044 24851 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:33.736253 24941 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.738646 24851 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:33.745005 24602 ts_tablet_manager.cc:1428] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.005s
I20260809 23:17:33.745026 24851 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.745862 24602 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:33.748131 24941 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.751312 24760 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.751775 24760 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.754231 24760 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.754993 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.758164 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.758328 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.759090 24321 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:17:33.760149 24498 leader_election.cc:341] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.760653 24498 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.762815 24760 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.764569 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.761034 24170 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.767509 24941 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.767966 24941 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.769865 24941 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:33.770792 24496 leader_election.cc:341] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.771704 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:33.772814 24941 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.780223 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.787809 24435 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:33.788986 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.790395 24603 tablet_bootstrap.cc:654] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.791288 24941 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.791669 24941 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.792564 24938 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:33.793040 24941 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.798415 24170 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:33.787209 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:33.808816 24498 leader_election.cc:341] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.806167 24941 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.844529 24942 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.853196 24602 tablet_bootstrap.cc:654] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.852336 24942 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.856719 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:33.860610 24435 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.862527 24323 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.855309 24760 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:33.855074 24941 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.867964 24603 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:33.869414 24603 ts_tablet_manager.cc:1397] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.160s	user 0.027s	sys 0.024s
I20260809 23:17:33.871229 24941 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:17:33.872604 25033 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260809 23:17:33.869947 24938 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.877481 24605 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:33.884507 24605 ts_tablet_manager.cc:1397] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.190s	user 0.029s	sys 0.017s
I20260809 23:17:33.884749 24938 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.888134 24765 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:33.888929 24938 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.891306 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.892120 24425 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:33.877812 24942 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.876375 24941 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.883111 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.876561 24798 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260809 23:17:33.867341 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:33.895381 24605 raft_consensus.cc:348] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.896139 24605 raft_consensus.cc:374] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.896481 24605 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.897140 24605 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.898804 24605 ts_tablet_manager.cc:1428] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260809 23:17:33.892859 24435 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:33.900074 24941 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.900493 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.907909 24320 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:33.897449 24546 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:33.910727 24605 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:33.902431 24603 raft_consensus.cc:348] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.911276 24603 raft_consensus.cc:374] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:33.911516 24603 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:33.915758 24603 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.919204 24603 ts_tablet_manager.cc:1428] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.000s
I20260809 23:17:33.920311 24603 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:33.912042 24266 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:17:33.922590 24498 leader_election.cc:341] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.925176 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.926487 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:33.929155 24605 tablet_bootstrap.cc:654] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.924748 24942 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.932490 24765 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.943938 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.943969 24765 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.944666 24435 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:33.945439 24947 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.948163 24947 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.946599 24730 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:17:33.950173 24947 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:33.954811 24603 tablet_bootstrap.cc:654] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:33.954878 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:33.956878 24545 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:33.957885 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:33.958515 24317 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
W20260809 23:17:33.960259 24498 leader_election.cc:341] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:33.960845 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:33.946822 24765 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.961988 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:33.940096 24942 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:33.964031 24942 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:33.972985 24730 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:33.982019 25041 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:33.987866 25041 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:33.989868 25041 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:33.984930 24730 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.007885 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:34.009654 24602 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.010411 24602 ts_tablet_manager.cc:1397] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.265s	user 0.027s	sys 0.026s
I20260809 23:17:34.010727 24938 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.011284 24938 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:34.012126 24938 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.013875 24605 tablet_bootstrap.cc:492] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.014320 24605 ts_tablet_manager.cc:1397] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.104s	user 0.025s	sys 0.008s
I20260809 23:17:34.016222 24602 raft_consensus.cc:348] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.016705 24602 raft_consensus.cc:374] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.016937 24602 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.017391 24602 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.005332 24730 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.025776 24605 raft_consensus.cc:348] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.026829 24605 raft_consensus.cc:374] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.027263 24605 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.028110 24605 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.030504 24605 ts_tablet_manager.cc:1428] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260809 23:17:34.033043 24605 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
W20260809 23:17:34.044808 24496 leader_election.cc:341] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.045460 24496 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.053407 24602 ts_tablet_manager.cc:1428] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.043s	user 0.008s	sys 0.000s
I20260809 23:17:34.054533 24602 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.070021 24943 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.076113 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.082274 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.082105 24942 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.084096 24942 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.086896 24164 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:34.095116 24602 tablet_bootstrap.cc:654] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.086910 24938 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:17:34.091429 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52370.1us
I20260809 23:17:34.107285 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:17:34.108278 24496 leader_election.cc:341] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.110001 24765 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.111229 24540 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:34.114398 24765 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:34.117132 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:34.126135 24425 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.139333 24603 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.139927 24603 ts_tablet_manager.cc:1397] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.220s	user 0.023s	sys 0.018s
I20260809 23:17:34.140591 24605 tablet_bootstrap.cc:654] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:34.132061 24498 leader_election.cc:341] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.142591 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.144421 24797 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.146291 24730 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.147405 24941 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.148440 24730 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.151467 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:34.151717 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:34.152799 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:34.153461 24546 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.153537 24321 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 1: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:17:34.159795 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:34.160457 24321 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.155470 24425 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:34.165035 24496 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:34.165783 24546 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:34.166877 24380 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:34.167400 24321 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:34.167495 24266 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.168522 24797 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.168910 24971 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.169001 24797 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:34.171183 25051 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.171380 24797 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.172320 25051 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:34.173211 25051 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.177729 24603 raft_consensus.cc:348] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.178256 24603 raft_consensus.cc:374] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.178459 24603 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.180402 24603 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.183923 24603 ts_tablet_manager.cc:1428] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.001s
I20260809 23:17:34.184813 24323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.189734 24603 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:34.206058 24765 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.206565 24765 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.211407 24765 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:34.213382 24168 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:34.214282 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:34.216099 24498 leader_election.cc:341] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.220956 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:34.231720 24496 leader_election.cc:341] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.239461 24496 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.245508 24765 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.246553 24730 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260809 23:17:34.261118 24605 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.261657 24605 ts_tablet_manager.cc:1397] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.229s	user 0.009s	sys 0.018s
I20260809 23:17:34.272851 24971 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.273571 24603 tablet_bootstrap.cc:654] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.283834 24602 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.284317 24602 ts_tablet_manager.cc:1397] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.230s	user 0.017s	sys 0.009s
I20260809 23:17:34.284441 24605 raft_consensus.cc:348] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.285310 24605 raft_consensus.cc:374] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.285620 24605 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.289256 24730 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.294250 24602 raft_consensus.cc:348] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.294904 24602 raft_consensus.cc:374] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.295166 24602 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.295748 24602 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.296686 24661 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.300509 24605 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.302855 24605 ts_tablet_manager.cc:1428] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.041s	user 0.005s	sys 0.001s
I20260809 23:17:34.306332 24605 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.314157 24164 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:34.315295 24661 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.316412 24602 ts_tablet_manager.cc:1428] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.027s	user 0.004s	sys 0.002s
I20260809 23:17:34.316164 24661 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.324035 24661 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.325618 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.326378 24544 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:34.327752 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:34.328693 24661 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:34.329113 24661 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.329561 24661 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.334249 24602 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.337244 24605 tablet_bootstrap.cc:654] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.344393 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.345019 24430 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:34.357095 24602 tablet_bootstrap.cc:654] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.370163 24603 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.370632 24603 ts_tablet_manager.cc:1397] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.181s	user 0.008s	sys 0.025s
I20260809 23:17:34.371327 24605 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.371768 24605 ts_tablet_manager.cc:1397] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.066s	user 0.013s	sys 0.000s
I20260809 23:17:34.373700 24603 raft_consensus.cc:348] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.374059 24605 raft_consensus.cc:348] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.374646 24603 raft_consensus.cc:374] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.375041 24603 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.375355 24730 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.375784 24730 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.374868 24605 raft_consensus.cc:374] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.377246 24605 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.377988 24602 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.377925 24730 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.377833 24605 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.378491 24602 ts_tablet_manager.cc:1397] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.000s
I20260809 23:17:34.380040 24605 ts_tablet_manager.cc:1428] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:17:34.380350 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.381356 24605 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.381246 24544 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:34.380992 24602 raft_consensus.cc:348] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.383319 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:34.383534 24602 raft_consensus.cc:374] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.383860 24602 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.384395 24798 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:34.384930 24798 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.384680 24602 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.385304 24798 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.386953 24602 ts_tablet_manager.cc:1428] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.004s
I20260809 23:17:34.387789 24602 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
W20260809 23:17:34.388818 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:34.375699 24603 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.391378 24603 ts_tablet_manager.cc:1428] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260809 23:17:34.392252 24603 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:34.398309 24603 tablet_bootstrap.cc:654] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.398420 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:34.403646 24378 leader_election.cc:341] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.343216 24661 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.404994 24798 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.404956 24602 tablet_bootstrap.cc:654] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.408195 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:34.409574 24378 leader_election.cc:341] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.413261 25051 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.413913 25051 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.417591 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:34.419229 24498 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.421156 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:34.421857 24430 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.422804 24661 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.446132 24798 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.451478 24798 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.486852 24605 tablet_bootstrap.cc:654] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.493660 24798 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.491951 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:34.495577 24546 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.501605 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.505661 24546 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:34.490201 25051 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:34.506628 24430 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:34.507025 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:17:34.507730 24496 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.507968 24730 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.508162 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.508435 24730 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:34.509135 25051 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.509227 24730 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.512043 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.519783 24266 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:34.520669 24971 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.521162 24971 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:34.522167 24971 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:34.524305 24378 leader_election.cc:341] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.526660 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:34.534653 24545 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.539371 24942 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.539988 24942 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.554566 24603 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.555051 24603 ts_tablet_manager.cc:1397] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.163s	user 0.021s	sys 0.047s
I20260809 23:17:34.542227 24798 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.551052 24545 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:34.562474 24942 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.572706 24603 raft_consensus.cc:348] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.573372 24603 raft_consensus.cc:374] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.573539 24603 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.573926 24603 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.576198 24603 ts_tablet_manager.cc:1428] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260809 23:17:34.577978 24603 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:34.578393 24605 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.578814 24605 ts_tablet_manager.cc:1397] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.198s	user 0.011s	sys 0.005s
I20260809 23:17:34.581943 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.582140 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.582787 24546 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
W20260809 23:17:34.587082 24266 leader_election.cc:341] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:34.578791 24498 consensus_peers.cc:487] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:34.587980 24380 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:34.590476 24798 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:34.590844 24798 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.591190 24798 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.587159 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.595602 24540 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:34.596918 24270 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.597995 24851 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:34.598088 24941 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.598373 24851 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.598798 24851 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.602386 24798 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.605815 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:34.608021 24378 leader_election.cc:341] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.608891 24851 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.610997 24851 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.612335 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:34.613651 24603 tablet_bootstrap.cc:654] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:34.614398 24266 leader_election.cc:341] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.616070 24168 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:34.624212 24605 raft_consensus.cc:348] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.624853 24605 raft_consensus.cc:374] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.625114 24605 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.625710 24605 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.627369 24798 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.628810 25041 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.630630 24164 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:34.628419 24602 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.631930 24602 ts_tablet_manager.cc:1397] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.244s	user 0.031s	sys 0.066s
I20260809 23:17:34.633160 25041 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.634588 24605 ts_tablet_manager.cc:1428] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.054s	user 0.006s	sys 0.001s
I20260809 23:17:34.635419 25041 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:34.637718 24605 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.651099 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.653143 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:34.653988 24496 leader_election.cc:341] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:34.654762 24498 leader_election.cc:341] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.655076 24730 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.655260 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.656635 24941 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.658172 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:34.658811 24545 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.675038 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:34.675752 24546 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.689595 24605 tablet_bootstrap.cc:654] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.684937 24545 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:34.695850 24602 raft_consensus.cc:348] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.696556 24602 raft_consensus.cc:374] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.696762 24602 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.697324 24602 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.701108 24602 ts_tablet_manager.cc:1428] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.000s
I20260809 23:17:34.701562 24730 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.701977 24602 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.706368 24603 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.706802 24603 ts_tablet_manager.cc:1397] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.129s	user 0.009s	sys 0.006s
I20260809 23:17:34.714298 24603 raft_consensus.cc:348] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.715817 24603 raft_consensus.cc:374] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.716234 24603 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.715027 24730 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.717694 24730 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.718569 24270 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.718915 24603 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.719408 24661 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.719879 24661 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:34.722776 24661 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.723686 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.724431 24545 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 1: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:17:34.725631 24760 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.726249 24603 ts_tablet_manager.cc:1428] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260809 23:17:34.727170 24603 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:34.720435 24730 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.727123 24546 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:34.734392 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.775045 24380 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:34.779839 24603 tablet_bootstrap.cc:654] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.781549 24602 tablet_bootstrap.cc:654] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.780480 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:34.783324 25051 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:34.783849 24798 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.784925 25067 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:34.785390 25067 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:34.786175 25067 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.793926 24164 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:34.809142 24798 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.832239 24605 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.834395 24602 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.834856 24602 ts_tablet_manager.cc:1397] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.133s	user 0.007s	sys 0.010s
I20260809 23:17:34.832708 24605 ts_tablet_manager.cc:1397] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.195s	user 0.008s	sys 0.011s
I20260809 23:17:34.840348 24605 raft_consensus.cc:348] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.841030 24605 raft_consensus.cc:374] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.841153 24603 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.841322 24605 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.841854 24603 ts_tablet_manager.cc:1397] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.115s	user 0.011s	sys 0.007s
I20260809 23:17:34.841954 24605 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.842363 24602 raft_consensus.cc:348] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.843114 24602 raft_consensus.cc:374] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.843374 24602 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.844197 24605 ts_tablet_manager.cc:1428] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.002s
I20260809 23:17:34.844003 24602 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.845016 24605 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.845140 24603 raft_consensus.cc:348] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.845866 24603 raft_consensus.cc:374] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.846156 24603 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.846985 24798 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.846879 24603 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.848542 24602 ts_tablet_manager.cc:1428] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260809 23:17:34.849351 24602 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.847450 24798 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.853861 24798 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.855532 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:17:34.856972 24380 leader_election.cc:341] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.859156 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.859973 24602 tablet_bootstrap.cc:654] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.852972 24164 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:34.866595 24603 ts_tablet_manager.cc:1428] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260809 23:17:34.868530 24605 tablet_bootstrap.cc:654] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.867755 24603 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
W20260809 23:17:34.871873 24378 leader_election.cc:341] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.872390 24378 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:34.874766 24798 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.873977 24602 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.877274 24602 ts_tablet_manager.cc:1397] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.004s
I20260809 23:17:34.879496 24603 tablet_bootstrap.cc:654] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.880702 24602 raft_consensus.cc:348] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.881453 24602 raft_consensus.cc:374] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.881791 24602 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.882666 24602 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.889634 24605 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.890079 24605 ts_tablet_manager.cc:1397] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.045s	user 0.007s	sys 0.003s
I20260809 23:17:34.890316 24602 ts_tablet_manager.cc:1428] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260809 23:17:34.891158 24602 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.892326 24605 raft_consensus.cc:348] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.893684 24605 raft_consensus.cc:374] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.894071 24605 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.894681 24605 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.896724 24605 ts_tablet_manager.cc:1428] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:17:34.898214 24605 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.904626 24605 tablet_bootstrap.cc:654] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.906450 24602 tablet_bootstrap.cc:654] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.910817 24603 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.911281 24603 ts_tablet_manager.cc:1397] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.044s	user 0.013s	sys 0.015s
I20260809 23:17:34.917771 24603 raft_consensus.cc:348] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.918437 24603 raft_consensus.cc:374] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.918761 24603 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.919421 24603 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.923323 24605 tablet_bootstrap.cc:492] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.923851 24605 ts_tablet_manager.cc:1397] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.002s
I20260809 23:17:34.926450 24605 raft_consensus.cc:348] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.927042 24603 ts_tablet_manager.cc:1428] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260809 23:17:34.927383 24605 raft_consensus.cc:374] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.927758 24605 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.928656 24605 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.929360 24603 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:34.936211 24605 ts_tablet_manager.cc:1428] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260809 23:17:34.937084 24605 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.942507 24603 tablet_bootstrap.cc:654] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.947007 24851 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.947603 24851 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.949829 24851 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.951997 24605 tablet_bootstrap.cc:654] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.951678 24851 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:34.956035 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.956444 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.957367 24798 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.957845 24798 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:34.957785 24270 leader_election.cc:341] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:34.962590 24266 leader_election.cc:341] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.963249 24602 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:34.963426 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.963701 24602 ts_tablet_manager.cc:1397] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.073s	user 0.008s	sys 0.016s
W20260809 23:17:34.964709 24378 leader_election.cc:341] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.965054 24266 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.965705 24798 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:34.966120 24602 raft_consensus.cc:348] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.967288 24602 raft_consensus.cc:374] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.967633 24602 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.967337 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:34.967816 24661 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.969550 24546 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:34.968327 24602 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.972226 24602 ts_tablet_manager.cc:1428] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260809 23:17:34.973800 24380 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:34.974121 24602 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:34.974017 24605 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:34.978442 24605 ts_tablet_manager.cc:1397] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.004s
I20260809 23:17:34.979149 24798 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:34.979501 24798 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:34.979808 24798 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:34.980526 25051 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.980997 25051 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.983192 24605 raft_consensus.cc:348] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.983868 24605 raft_consensus.cc:374] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:34.984037 24605 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:34.985208 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:34.985229 24605 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.986024 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:34.986837 25051 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:34.987140 24496 leader_election.cc:341] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.987500 24605 ts_tablet_manager.cc:1428] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
W20260809 23:17:34.987555 24498 leader_election.cc:341] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:34.988071 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:34.988458 24605 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:34.989286 24602 tablet_bootstrap.cc:654] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:34.990451 24603 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:34.990885 24603 ts_tablet_manager.cc:1397] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.062s	user 0.012s	sys 0.000s
I20260809 23:17:34.991487 24765 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:34.997268 24798 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:34.998319 24661 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:34.999989 24603 raft_consensus.cc:348] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.002766 24661 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.003597 24603 raft_consensus.cc:374] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.005189 24661 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.003970 24603 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.006285 24603 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.007012 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.007599 24430 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:35.009080 24266 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:35.009807 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:35.010257 24605 tablet_bootstrap.cc:654] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.010462 24544 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:35.009820 24661 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:35.011623 24661 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:35.011950 24661 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.013607 24851 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:35.014557 24603 ts_tablet_manager.cc:1428] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260809 23:17:35.015774 24603 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:35.030062 24603 tablet_bootstrap.cc:654] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.030427 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:35.032191 24378 leader_election.cc:341] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.030122 24661 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.036628 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:35.037153 24544 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.037842 25051 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.038280 25051 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.039554 25067 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:35.041029 24605 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:35.041628 24605 ts_tablet_manager.cc:1397] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.000s
I20260809 23:17:35.043915 24544 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:35.044957 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:35.045241 25051 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.045652 24661 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.046895 25067 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.046720 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.047386 25067 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:35.048540 24496 leader_election.cc:341] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.049368 24603 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:35.050239 24603 ts_tablet_manager.cc:1397] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.004s
I20260809 23:17:35.051371 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.052028 24316 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:35.052313 24602 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:35.052673 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:35.053030 24602 ts_tablet_manager.cc:1397] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.079s	user 0.014s	sys 0.049s
I20260809 23:17:35.053112 24603 raft_consensus.cc:348] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.053861 24603 raft_consensus.cc:374] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.054108 24603 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.054699 24603 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.055368 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:35.055486 24602 raft_consensus.cc:348] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.056157 24730 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:35.056360 24661 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:35.056576 24730 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:35.056880 24661 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:35.056977 24730 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.057091 24605 raft_consensus.cc:348] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.058010 24605 raft_consensus.cc:374] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.058266 24605 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.058277 24661 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.058965 24605 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.056178 24602 raft_consensus.cc:374] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.061035 24605 ts_tablet_manager.cc:1428] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260809 23:17:35.061168 24602 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.061884 24605 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:35.061720 24602 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.053411 24544 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:35.062821 24798 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.075618 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:35.076503 24430 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.075595 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:35.078260 24544 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.081701 24603 ts_tablet_manager.cc:1428] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260809 23:17:35.082660 24603 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:35.084717 24316 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.088485 25067 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.089890 24605 tablet_bootstrap.cc:654] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.090679 24730 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.093806 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:35.094413 24314 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.095186 24730 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.096127 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:35.096391 24603 tablet_bootstrap.cc:654] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.102408 24546 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.101747 24544 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:35.104760 24605 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:35.105201 24605 ts_tablet_manager.cc:1397] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.043s	user 0.008s	sys 0.005s
I20260809 23:17:35.105110 24380 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:35.105070 24602 ts_tablet_manager.cc:1428] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.000s
I20260809 23:17:35.108407 25121 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:35.108630 24602 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:35.109004 25121 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:35.110136 24546 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:35.109887 25121 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.111377 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:35.112545 25067 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:35.113040 25067 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:35.113085 24314 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:35.114027 25067 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.107717 24605 raft_consensus.cc:348] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.122012 24605 raft_consensus.cc:374] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.122314 24605 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.125201 24605 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.126636 24602 tablet_bootstrap.cc:654] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.130040 24605 ts_tablet_manager.cc:1428] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260809 23:17:35.140148 24605 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:35.142916 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:35.144686 24602 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:35.145155 24602 ts_tablet_manager.cc:1397] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.037s	user 0.010s	sys 0.008s
I20260809 23:17:35.145759 24603 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:35.146322 24603 ts_tablet_manager.cc:1397] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.002s
I20260809 23:17:35.149015 24603 raft_consensus.cc:348] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.149549 24430 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:35.149966 24603 raft_consensus.cc:374] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.153947 24602 raft_consensus.cc:348] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.156889 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.158419 24602 raft_consensus.cc:374] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.150244 24603 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.158761 24602 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.159518 24602 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.159436 24603 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.161863 24602 ts_tablet_manager.cc:1428] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.002s
I20260809 23:17:35.202972 24760 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.203125 24603 ts_tablet_manager.cc:1428] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.057s	user 0.005s	sys 0.004s
I20260809 23:17:35.203522 24760 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.204511 24603 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:35.205468 24760 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.207491 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.207785 24605 tablet_bootstrap.cc:654] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.208105 24435 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:35.209100 24496 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:35.210590 24765 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:35.210959 24765 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:35.211299 24765 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.213977 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.214619 24323 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:35.215145 24321 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:35.219868 24498 leader_election.cc:341] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.220258 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:35.220994 25053 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.228247 24602 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:35.229223 25053 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.229777 25053 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.231706 25053 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.235965 24947 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.243435 24765 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.244937 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.251226 24765 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.259311 24603 tablet_bootstrap.cc:654] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.260581 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:35.261200 24430 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:35.259443 24496 leader_election.cc:341] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.263336 24730 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:35.269512 24942 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:35.270092 24797 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.274544 24602 tablet_bootstrap.cc:654] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.317073 24605 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:35.317248 24603 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:35.317564 24605 ts_tablet_manager.cc:1397] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.178s	user 0.016s	sys 0.012s
I20260809 23:17:35.317766 24603 ts_tablet_manager.cc:1397] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.113s	user 0.020s	sys 0.012s
I20260809 23:17:35.364646 24170 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:35.369613 24170 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:35.421062 24971 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.432586 24605 raft_consensus.cc:348] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.433774 24605 raft_consensus.cc:374] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.434124 24605 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.441882 24971 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.469929 24603 raft_consensus.cc:348] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.471560 24603 raft_consensus.cc:374] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.471949 24603 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.472896 24603 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.445768 24605 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:35.460700 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53807.4us
I20260809 23:17:35.490547 24971 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.490840 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.492300 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:35.493010 24314 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.493311 24539 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:17:35.495493 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:35.496675 24321 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.508708 24605 ts_tablet_manager.cc:1428] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.191s	user 0.007s	sys 0.006s
I20260809 23:17:35.516585 25145 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.521286 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.517622 24605 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
W20260809 23:17:35.527542 24498 leader_election.cc:341] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.528162 24498 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:35.528554 25051 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.529261 24765 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.529147 25051 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.532372 25051 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.535578 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.537482 24602 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:35.537786 24797 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:17:35.538190 24496 leader_election.cc:341] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.541392 24430 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:35.534739 24603 ts_tablet_manager.cc:1428] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.148s	user 0.005s	sys 0.008s
I20260809 23:17:35.538125 24602 ts_tablet_manager.cc:1397] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.310s	user 0.012s	sys 0.016s
I20260809 23:17:35.561959 24496 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:35.563112 25053 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:35.563488 25145 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:35.563714 25053 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:35.562203 24603 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:35.565012 25053 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.585150 24321 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:35.603302 24947 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.604955 24603 tablet_bootstrap.cc:654] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.613157 24971 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.613937 24971 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.615159 24605 tablet_bootstrap.cc:654] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.616043 24971 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.603945 24947 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.629308 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:35.630090 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:35.637856 24730 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.639655 24313 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.640148 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:35.645562 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:35.648857 25146 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:35.649135 24947 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.649238 25146 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:35.649645 25146 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.650558 24602 raft_consensus.cc:348] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.656873 24602 raft_consensus.cc:374] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.657167 24602 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.657804 24602 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.666837 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.667852 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:35.671191 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260809 23:17:35.683610 24498 leader_election.cc:341] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.684724 24498 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:17:35.685357 24266 leader_election.cc:341] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.659797 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:35.694568 25146 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.700191 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:17:35.701718 24266 leader_election.cc:341] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.705891 24661 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.705862 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.708359 25146 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.719146 24760 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.727589 24603 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:35.735515 24603 ts_tablet_manager.cc:1397] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.174s	user 0.008s	sys 0.008s
I20260809 23:17:35.728581 24602 ts_tablet_manager.cc:1428] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.184s	user 0.006s	sys 0.002s
W20260809 23:17:35.733196 24380 leader_election.cc:341] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.753894 24168 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:35.740504 24971 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.767887 24603 raft_consensus.cc:348] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.768711 24603 raft_consensus.cc:374] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.769016 24603 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.769708 24603 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.770845 25123 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:35.771595 24602 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
W20260809 23:17:35.777861 24378 leader_election.cc:341] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.779444 24603 ts_tablet_manager.cc:1428] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.000s
I20260809 23:17:35.793164 24378 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:17:35.794123 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66752.8us
I20260809 23:17:35.794353 24605 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:35.794787 24605 ts_tablet_manager.cc:1397] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.279s	user 0.026s	sys 0.057s
I20260809 23:17:35.797662 24603 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:35.799268 25067 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.833564 25053 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.839222 25053 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.841833 25053 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.922714 24765 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.923399 24765 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.926600 24765 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:35.935638 24602 tablet_bootstrap.cc:700] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent opening tablet: real 0.139s	user 0.002s	sys 0.001s
I20260809 23:17:35.936811 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:35.938364 24498 leader_election.cc:341] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.939436 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.940263 24430 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:35.953460 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:35.958655 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:35.959290 24540 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:35.961992 24498 leader_election.cc:341] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.962946 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:35.966279 24605 raft_consensus.cc:348] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.967020 24605 raft_consensus.cc:374] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:35.966854 24540 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:35.967331 24605 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:35.967976 24605 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.968142 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:35.969005 25146 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:35.969480 25146 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:35.970068 24605 ts_tablet_manager.cc:1428] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.175s	user 0.021s	sys 0.027s
I20260809 23:17:35.969880 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:35.970695 25053 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:35.970300 25146 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.971155 25053 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:35.971594 25053 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:35.971066 24605 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
W20260809 23:17:35.982571 24496 leader_election.cc:341] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:35.936522 24602 tablet_bootstrap.cc:654] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.983028 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:35.986963 24765 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:35.988029 24603 tablet_bootstrap.cc:654] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:35.988989 24661 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:35.989424 24661 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:35.991175 24661 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:35.994623 24605 tablet_bootstrap.cc:654] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.007616 24602 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.008174 24602 ts_tablet_manager.cc:1397] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.237s	user 0.010s	sys 0.003s
I20260809 23:17:36.003093 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.025650 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.026576 24539 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:36.003468 25123 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
W20260809 23:17:36.032882 24266 leader_election.cc:341] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.044502 24602 raft_consensus.cc:348] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.045542 24602 raft_consensus.cc:374] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.045853 24602 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.046623 24602 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.049837 24602 ts_tablet_manager.cc:1428] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260809 23:17:36.051142 24602 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.043030 24603 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:36.058770 24603 ts_tablet_manager.cc:1397] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.261s	user 0.027s	sys 0.022s
I20260809 23:17:36.063673 25051 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.067565 24164 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:36.069685 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.074007 25146 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.074862 25146 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.075245 25146 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.087368 25166 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:36.128607 25053 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.131649 25053 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.133869 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:36.133720 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:36.134681 24425 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:36.135991 24498 leader_election.cc:341] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.139865 25067 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.140318 25067 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.143062 25067 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.143294 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.144136 24539 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:36.145578 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:36.148363 24602 tablet_bootstrap.cc:654] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.156852 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.148363 25067 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.159350 25067 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.160513 25067 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:36.169476 24378 leader_election.cc:341] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.171185 24603 raft_consensus.cc:348] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.175840 24603 raft_consensus.cc:374] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.176075 24603 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.176707 24603 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.178766 25150 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.179279 25150 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.182071 25150 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.183200 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.183607 25146 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:36.185688 24496 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.185844 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:36.207850 24498 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.208266 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.200934 25150 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.215973 24760 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.216320 25150 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.219594 25150 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.220544 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.220808 24425 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:36.221760 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.222133 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.222411 24430 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:36.225626 24496 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:36.226416 25051 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.226869 25051 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:17:36.227771 24498 leader_election.cc:341] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.228830 25051 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.231163 25166 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.231545 25166 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.231869 25166 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.232620 24603 ts_tablet_manager.cc:1428] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.173s	user 0.005s	sys 0.003s
I20260809 23:17:36.236481 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:36.241264 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:36.241829 24545 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:36.243754 24266 leader_election.cc:341] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.250671 24605 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:36.253485 24605 ts_tablet_manager.cc:1397] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.283s	user 0.018s	sys 0.022s
I20260809 23:17:36.255119 24545 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:36.239493 25146 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.246191 25166 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.261776 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.262737 25166 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.257256 24603 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:36.257447 25067 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.263664 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:36.291088 24430 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.290791 24605 raft_consensus.cc:348] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.291307 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:36.293249 24498 leader_election.cc:341] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.294903 24602 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.296623 24602 ts_tablet_manager.cc:1397] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.246s	user 0.019s	sys 0.005s
I20260809 23:17:36.291563 24605 raft_consensus.cc:374] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.299314 24605 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.264189 25146 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.300168 24605 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.311633 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:36.311887 24603 tablet_bootstrap.cc:654] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.311848 25146 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:17:36.312928 24378 leader_election.cc:341] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.313737 24602 raft_consensus.cc:348] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.314431 24602 raft_consensus.cc:374] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.312786 25067 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.314286 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:36.314883 24602 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.315856 24543 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.316295 24602 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.318146 24605 ts_tablet_manager.cc:1428] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.060s	user 0.007s	sys 0.000s
I20260809 23:17:36.319240 24605 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:36.317672 25146 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.325258 24543 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:36.326344 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.327492 24851 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.327662 25067 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.327965 24851 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.328349 25067 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:36.329717 25067 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.330356 24851 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.334057 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.334550 24602 ts_tablet_manager.cc:1428] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.000s
I20260809 23:17:36.335475 24602 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
W20260809 23:17:36.335623 24266 leader_election.cc:341] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.324625 24605 tablet_bootstrap.cc:654] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.324862 24430 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:36.339397 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.340474 24496 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.341694 25051 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.342181 25051 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:36.342993 25051 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.348892 24602 tablet_bootstrap.cc:654] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.348601 24164 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:36.359989 24168 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:17:36.375977 24270 leader_election.cc:341] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.376416 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.377416 24851 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.385968 24602 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.386662 24602 ts_tablet_manager.cc:1397] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.005s
I20260809 23:17:36.390873 24602 raft_consensus.cc:348] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.394945 24164 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:36.407370 24603 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:36.407905 24603 ts_tablet_manager.cc:1397] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.151s	user 0.026s	sys 0.020s
I20260809 23:17:36.410324 24603 raft_consensus.cc:348] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.411084 24603 raft_consensus.cc:374] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.411404 24603 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.412034 24603 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.415680 24605 tablet_bootstrap.cc:492] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:36.416260 24605 ts_tablet_manager.cc:1397] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.097s	user 0.024s	sys 0.012s
I20260809 23:17:36.415975 24165 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:36.420579 24603 ts_tablet_manager.cc:1428] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.003s
I20260809 23:17:36.391896 24602 raft_consensus.cc:374] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.421125 24602 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.421551 24603 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:36.421748 24602 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.423972 24602 ts_tablet_manager.cc:1428] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20260809 23:17:36.426235 24605 raft_consensus.cc:348] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.426947 24605 raft_consensus.cc:374] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.427210 24605 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.427448 24602 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.427819 24605 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.430500 24605 ts_tablet_manager.cc:1428] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260809 23:17:36.431221 25146 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.432648 25146 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.433730 24605 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:36.435165 25146 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.435787 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.436564 24430 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:36.438364 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.439795 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.441566 24603 tablet_bootstrap.cc:654] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.444958 25146 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.448779 24602 tablet_bootstrap.cc:654] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.451840 24605 tablet_bootstrap.cc:654] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.452775 25146 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.457301 24851 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:36.464129 24603 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:36.464615 24603 ts_tablet_manager.cc:1397] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.043s	user 0.002s	sys 0.008s
I20260809 23:17:36.466547 24605 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:36.467001 24605 ts_tablet_manager.cc:1397] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.001s
I20260809 23:17:36.469784 24605 raft_consensus.cc:348] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.470443 24605 raft_consensus.cc:374] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.470692 24605 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.471369 24605 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.473451 24605 ts_tablet_manager.cc:1428] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:36.474381 24605 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:36.480057 24605 tablet_bootstrap.cc:654] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.466990 24603 raft_consensus.cc:348] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.481315 24603 raft_consensus.cc:374] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.481583 24603 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.482191 24603 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:36.482615 24266 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:36.483654 25145 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.484275 24603 ts_tablet_manager.cc:1428] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.019s	user 0.000s	sys 0.006s
I20260809 23:17:36.484086 25145 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.485150 24603 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Bootstrap starting.
I20260809 23:17:36.486016 25145 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.487799 24602 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.488263 24602 ts_tablet_manager.cc:1397] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.061s	user 0.006s	sys 0.004s
I20260809 23:17:36.488412 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.489082 24430 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:36.492211 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:36.492187 24602 raft_consensus.cc:348] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.493198 24603 tablet_bootstrap.cc:654] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.493384 24602 raft_consensus.cc:374] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.493706 24602 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.493638 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.494338 24543 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 0.
I20260809 23:17:36.495919 25145 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.496284 25145 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.496600 25145 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.505044 24603 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: No bootstrap required, opened a new log
I20260809 23:17:36.505527 24603 ts_tablet_manager.cc:1397] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260809 23:17:36.514637 25145 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.521343 24603 raft_consensus.cc:348] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.522050 24603 raft_consensus.cc:374] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.523960 24603 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.524863 25145 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.525390 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:36.525631 24603 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.526101 24543 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.528452 24603 ts_tablet_manager.cc:1428] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.001s
I20260809 23:17:36.494490 24602 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.530642 24602 ts_tablet_manager.cc:1428] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260809 23:17:36.531450 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:36.531688 24602 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.531960 24435 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.537941 24602 tablet_bootstrap.cc:654] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.538198 24543 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:36.539256 24270 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:36.540127 25145 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.540668 25145 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:17:36.542353 25145 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.552340 25051 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.552821 25051 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.555886 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.556526 24314 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:36.561031 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:17:36.562572 25051 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.563278 25051 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.565759 25174 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.567042 25174 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.569324 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.576148 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.576978 24545 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
W20260809 23:17:36.578692 24496 leader_election.cc:341] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.578775 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.579938 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:36.576145 25174 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:17:36.581571 24378 leader_election.cc:341] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.582082 24798 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.584054 24798 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.584399 24798 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.592726 24164 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:36.595542 25051 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.595924 25051 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.605978 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:17:36.608374 25051 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.616613 24798 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.619089 24798 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.620589 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:36.621294 24543 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.624099 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:36.624791 24313 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.625046 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:36.626379 24378 leader_election.cc:341] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.628636 24602 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.629096 24602 ts_tablet_manager.cc:1397] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.098s	user 0.030s	sys 0.055s
I20260809 23:17:36.630116 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:36.631119 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.632090 25174 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.632558 25174 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:36.633365 25174 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.636265 24602 raft_consensus.cc:348] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.636972 24602 raft_consensus.cc:374] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.637228 24602 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.637830 24602 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.638581 25051 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.640691 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:17:36.642484 24496 leader_election.cc:341] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.642463 24602 ts_tablet_manager.cc:1428] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260809 23:17:36.643815 24602 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.644654 24313 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:36.648895 24602 tablet_bootstrap.cc:654] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.652375 25051 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.654166 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.655719 25051 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.656472 25051 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:36.657512 25051 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.661078 24602 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.661516 24602 ts_tablet_manager.cc:1397] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
I20260809 23:17:36.661942 24605 tablet_bootstrap.cc:492] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:36.662456 24605 ts_tablet_manager.cc:1397] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.188s	user 0.008s	sys 0.002s
I20260809 23:17:36.663867 24602 raft_consensus.cc:348] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.664552 24602 raft_consensus.cc:374] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.664809 24602 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.665659 24602 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.666774 25067 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.667258 25067 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.669446 25067 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.670306 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.670583 24605 raft_consensus.cc:348] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.672111 24602 ts_tablet_manager.cc:1428] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260809 23:17:36.674170 24602 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.672047 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.671639 24543 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 1: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:17:36.672340 24605 raft_consensus.cc:374] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.681221 24605 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.682020 24605 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.684324 24378 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:36.684970 24605 ts_tablet_manager.cc:1428] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.006s
I20260809 23:17:36.685635 25067 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.686092 24605 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:36.689852 24164 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:36.691201 24602 tablet_bootstrap.cc:654] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.693454 25067 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.703950 24165 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:36.709496 24760 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.727866 24602 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.728331 24602 ts_tablet_manager.cc:1397] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.054s	user 0.019s	sys 0.008s
I20260809 23:17:36.730641 24971 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.730746 24602 raft_consensus.cc:348] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.731503 24602 raft_consensus.cc:374] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.731774 24602 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.732383 24602 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.740190 25166 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.740947 25166 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.742555 24602 ts_tablet_manager.cc:1428] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260809 23:17:36.744622 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.746955 25166 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:17:36.750706 24498 leader_election.cc:341] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:17:36.773597 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:17:36.776624 24605 tablet_bootstrap.cc:654] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.776546 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:17:36.779878 24496 leader_election.cc:341] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.780453 24496 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:36.781942 25166 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.794883 25051 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.795342 25051 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.798198 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:36.799677 24498 leader_election.cc:341] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.800758 25051 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:36.802798 24851 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.804324 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:36.805872 24430 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:36.807976 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.810719 25051 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.811513 25051 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.816015 25051 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.844273 24798 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.844903 24798 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.848212 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:36.848487 25067 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:36.849718 24602 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
W20260809 23:17:36.852217 24498 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:36.856626 24798 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.858121 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.859470 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:36.860411 24798 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:36.866349 24602 tablet_bootstrap.cc:654] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.869745 24798 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:36.870675 24798 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.875144 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:17:36.877560 24378 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.878896 24798 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.879791 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:36.880461 24545 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:36.882200 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:17:36.887243 25067 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:36.887629 25067 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:36.887984 25067 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.896313 25067 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.898608 24605 tablet_bootstrap.cc:492] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:36.899085 24605 ts_tablet_manager.cc:1397] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.213s	user 0.042s	sys 0.043s
I20260809 23:17:36.899811 24602 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.900228 24602 ts_tablet_manager.cc:1397] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.051s	user 0.011s	sys 0.007s
I20260809 23:17:36.901607 24605 raft_consensus.cc:348] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.902279 24605 raft_consensus.cc:374] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.902590 24605 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.903813 24602 raft_consensus.cc:348] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.904768 24605 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.907502 24605 ts_tablet_manager.cc:1428] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260809 23:17:36.904816 24602 raft_consensus.cc:374] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.908222 24602 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.913170 24605 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:36.917371 24760 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:36.920950 24605 tablet_bootstrap.cc:654] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.921181 24314 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.908902 24602 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.925087 24602 ts_tablet_manager.cc:1428] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260809 23:17:36.926115 24602 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:36.933486 24602 tablet_bootstrap.cc:654] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:36.933560 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:36.934288 24543 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:36.946548 24543 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:36.948945 25067 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:36.949489 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:36.955623 25067 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:36.964478 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:36.964599 24425 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:36.970094 24378 leader_election.cc:341] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:36.978507 24602 tablet_bootstrap.cc:492] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:36.979144 24602 ts_tablet_manager.cc:1397] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.053s	user 0.012s	sys 0.002s
I20260809 23:17:36.996138 24602 raft_consensus.cc:348] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:36.972571 25067 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:36.997098 24602 raft_consensus.cc:374] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:36.997604 24602 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:36.998636 24602 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.000254 25146 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.007992 25067 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.009958 24605 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:37.010408 24605 ts_tablet_manager.cc:1397] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.097s	user 0.028s	sys 0.036s
I20260809 23:17:37.016848 24602 ts_tablet_manager.cc:1428] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.002s
I20260809 23:17:37.011816 25051 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.024902 24602 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:37.027680 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:37.029337 24435 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.031271 25145 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.031939 24605 raft_consensus.cc:348] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.032610 24605 raft_consensus.cc:374] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:37.032884 24605 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:37.033561 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:37.033500 24605 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.034147 24320 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.037037 24435 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:37.040688 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.043668 24605 ts_tablet_manager.cc:1428] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.033s	user 0.001s	sys 0.005s
I20260809 23:17:37.038435 25051 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:37.043473 25166 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:37.072693 24605 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:37.083324 25226 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.120751 25226 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:37.180109 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56681.8us
I20260809 23:17:37.189262 24164 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:37.175227 25166 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:37.201377 25166 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.201932 24320 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:37.226073 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:37.227773 24798 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.228179 24798 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.229825 24798 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.218693 25226 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.240489 24605 tablet_bootstrap.cc:654] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:37.251026 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:37.259027 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:37.254778 24545 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:37.265170 24602 tablet_bootstrap.cc:654] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:37.269872 24380 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.276671 24797 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.266752 24313 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:37.287968 24798 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 23:17:37.293743 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:17:37.299907 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56329.7us
I20260809 23:17:37.283392 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:37.277057 25240 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260809 23:17:37.323115 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55260.3us
I20260809 23:17:37.306810 25240 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:37.330603 25240 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.331928 24314 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 1: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:17:37.345884 24602 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:37.346488 24602 ts_tablet_manager.cc:1397] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.327s	user 0.014s	sys 0.007s
I20260809 23:17:37.327375 24971 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.362408 24971 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.364910 24971 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.365863 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:37.372362 24428 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:17:37.398357 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57521.2us
I20260809 23:17:37.403168 25240 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.427253 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:37.427897 24543 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.438894 24543 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:37.439100 25053 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:17:37.423393 25240 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.443421 24380 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.443606 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:17:37.444293 24798 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:37.444980 25266 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:37.441200 24605 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:37.445647 25266 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:37.445849 24605 ts_tablet_manager.cc:1397] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.392s	user 0.011s	sys 0.014s
I20260809 23:17:37.439581 24602 raft_consensus.cc:348] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.447567 24602 raft_consensus.cc:374] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:37.447803 24602 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:37.449717 24605 raft_consensus.cc:348] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.450381 24605 raft_consensus.cc:374] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:37.450625 24605 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:37.453948 25053 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.457412 24605 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.448383 24602 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.468233 24602 ts_tablet_manager.cc:1428] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.121s	user 0.006s	sys 0.000s
I20260809 23:17:37.469170 24602 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Bootstrap starting.
I20260809 23:17:37.470992 24168 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:37.464857 25266 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.476307 24602 tablet_bootstrap.cc:654] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:37.476626 24605 ts_tablet_manager.cc:1428] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.001s
I20260809 23:17:37.482905 24605 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:37.484715 25267 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.539436 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:37.540118 24320 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.572376 24605 tablet_bootstrap.cc:654] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:37.575122 25268 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:17:37.629559 25226 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.630055 25226 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:17:37.637781 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71705.7us
W20260809 23:17:37.652192 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70844.6us
I20260809 23:17:37.709741 24320 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:37.715864 24602 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: No bootstrap required, opened a new log
I20260809 23:17:37.716472 24602 ts_tablet_manager.cc:1397] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent bootstrapping tablet: real 0.247s	user 0.013s	sys 0.023s
I20260809 23:17:37.725185 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:37.725993 24545 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 0.
I20260809 23:17:37.728258 24380 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.718223 25226 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.731045 24602 raft_consensus.cc:348] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.737885 24602 raft_consensus.cc:374] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:37.738250 24602 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Initialized, Role: FOLLOWER
I20260809 23:17:37.739024 24602 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.729692 24168 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:37.744267 25240 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:37.746893 25240 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:37.747262 25240 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.751257 25051 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.751713 25051 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.753583 25051 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:37.753981 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:37.758246 25255 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:37.742800 24602 ts_tablet_manager.cc:1428] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260809 23:17:37.772583 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:37.773200 24435 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 0.
I20260809 23:17:37.773581 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:37.837281 25240 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.838316 24605 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:37.838826 24605 ts_tablet_manager.cc:1397] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.358s	user 0.032s	sys 0.036s
W20260809 23:17:37.845208 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 98074.7us
I20260809 23:17:37.848073 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.851279 25240 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.851397 24851 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:37.852012 24851 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.853953 24851 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:37.856415 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:17:37.857976 24498 leader_election.cc:341] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:17:37.857187 24314 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.863025 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:37.863942 25235 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:17:37.863687 24605 raft_consensus.cc:348] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.864279 25235 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:17:37.864938 24605 raft_consensus.cc:374] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:37.865202 24605 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:37.864632 25235 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.865767 24605 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.871811 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:37.881114 24605 ts_tablet_manager.cc:1428] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.042s	user 0.006s	sys 0.000s
I20260809 23:17:37.873165 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:17:37.888319 24539 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.890196 24605 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:37.882718 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:37.890779 25239 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:37.892771 24427 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 1: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:17:37.894145 25280 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:17:37.899220 24539 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:37.903779 24380 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.879549 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:17:37.905323 25240 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:37.905983 25240 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:17:37.906817 25240 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.914757 24270 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:37.921515 24797 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:37.903573 25235 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.927474 25235 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:37.929560 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:17:37.930749 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:17:37.931141 24427 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.931432 24320 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:37.954797 24427 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:37.956250 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:17:37.957041 25239 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:17:37.957481 25239 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:17:37.958258 25239 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:37.981498 24320 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:17:37.982682 24314 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:17:37.983220 25145 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:17:37.987861 24244 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260809 23:17:37.987951 24378 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:38.015201 25266 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:38.029335 24605 tablet_bootstrap.cc:654] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
W20260809 23:17:38.035064 25284 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260809 23:17:38.065090 24605 tablet_bootstrap.cc:492] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:38.065558 24605 ts_tablet_manager.cc:1397] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.176s	user 0.019s	sys 0.007s
I20260809 23:17:38.068058 24605 raft_consensus.cc:348] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.069694 24605 raft_consensus.cc:374] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:38.069955 24605 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:38.070633 24605 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.072777 24605 ts_tablet_manager.cc:1428] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:17:38.082823 24605 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:38.087416 24164 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:17:38.088742 24605 tablet_bootstrap.cc:654] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:38.096733 24605 tablet_bootstrap.cc:492] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:38.097153 24605 ts_tablet_manager.cc:1397] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.004s
I20260809 23:17:38.099195 24605 raft_consensus.cc:348] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.099732 24605 raft_consensus.cc:374] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:38.100072 24605 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:38.100708 24605 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.102461 24605 ts_tablet_manager.cc:1428] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:17:38.102401 24168 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 0 to 1, leader changed from <none> to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:38.103502 24605 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Bootstrap starting.
I20260809 23:17:38.109021 24605 tablet_bootstrap.cc:654] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Neither blocks nor log segments found. Creating new log.
I20260809 23:17:38.115999 24605 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: No bootstrap required, opened a new log
I20260809 23:17:38.116869 24605 ts_tablet_manager.cc:1397] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.001s
I20260809 23:17:38.119562 24605 raft_consensus.cc:348] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.123085 24605 raft_consensus.cc:374] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:17:38.123412 24605 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Initialized, Role: FOLLOWER
I20260809 23:17:38.124053 24605 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.125330 25280 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:38.125749 25280 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.126246 24605 ts_tablet_manager.cc:1428] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260809 23:17:38.127754 25280 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:38.129094 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:38.135526 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:38.137154 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:38.137857 25145 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:38.358146 25289 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:38.376911 25267 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:17:38.384390 24851 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:17:38.384860 24851 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.385495 25051 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:17:38.387750 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:38.389418 24851 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:38.392860 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:38.395524 24266 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:38.396203 25280 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:17:38.402889 25051 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:17:38.455660 24317 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:38.540398 24320 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:38.571123 24320 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:17:38.581902 25145 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:17:38.582406 25145 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:17:38.586673 25145 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:38.586648 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:17:38.587393 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:17:38.588658 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:17:38.590058 25145 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:17:39.780493 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:17:39.815016 25310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:17:39.815689 25312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:39.816011 22524 server_base.cc:1034] running on GCE node
W20260809 23:17:39.819446 25314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:17:39.821137 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:17:39.821447 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:17:39.821672 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317459821647 us; error 0 us; skew 500 ppm
I20260809 23:17:39.822466 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:17:39.830408 22524 webserver.cc:460] Webserver started at http://127.21.255.4:42651/ using document root <none> and password file <none>
I20260809 23:17:39.831377 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:17:39.831664 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:17:39.832115 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:17:39.833916 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-3-root/instance:
uuid: "b9b54436d0e94a5595cdded73fe33da9"
format_stamp: "Formatted at 2026-08-09 23:17:39 on dist-test-slave-drl0"
I20260809 23:17:39.840178 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.003s	sys 0.005s
I20260809 23:17:39.844367 25319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:39.845374 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260809 23:17:39.845716 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
uuid: "b9b54436d0e94a5595cdded73fe33da9"
format_stamp: "Formatted at 2026-08-09 23:17:39 on dist-test-slave-drl0"
I20260809 23:17:39.846005 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:17:39.867141 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:17:39.868595 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:17:39.870476 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:17:39.874233 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:17:39.874491 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:17:39.874815 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:17:39.875065 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260809 23:17:39.947759 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.4:38713
I20260809 23:17:39.947873 25410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.4:38713 every 8 connection(s)
I20260809 23:17:39.952534 22524 auto_leader_rebalancer-test.cc:413] add a tserver: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:39.968851 25411 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37921
I20260809 23:17:39.969316 25411 heartbeater.cc:464] Registering TS with master...
I20260809 23:17:39.970101 25411 heartbeater.cc:511] Master 127.21.255.62:37921 requested a full tablet report, sending...
I20260809 23:17:39.972280 24168 ts_manager.cc:194] Registered new tserver with Master: b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:17:39.974066 24168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49570
I20260809 23:17:40.147300 24535 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.157272 24425 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.159245 25051 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.166690 24320 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.172662 25051 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.175551 25307 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.177316 24425 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:17:40.181821 24498 consensus_peers.cc:487] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ef03e5e7d0443d69a2228dd175bd010. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:40.186352 24320 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.195608 24168 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:40.234746 24425 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.243497 24544 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.247952 25267 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.257721 24320 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.259666 25308 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:40.270025 25308 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.274537 24320 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:17:40.282652 24378 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb0467d784e841868c4a1188ee0af4bc. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:40.300415 24165 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:40.306418 24317 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.306080 24544 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.340986 24544 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.343091 24425 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.342801 25332 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:40.345088 25145 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.359054 25280 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.363310 24425 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.383893 24320 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.388652 24164 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:40.410284 24539 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.422281 24544 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.423969 25454 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.426304 24435 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:40.431274 25145 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:40.450404 25280 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.452121 24425 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:17:40.456683 24269 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90dd68b3876b4b75aea408edfa68d5e5. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:17:40.457444 24269 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3962e082dc274290a1f20e2fa64877a9. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:40.461035 24535 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:40.483901 24164 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:40.644372 25459 ts_tablet_manager.cc:927] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:40.646767 25459 tablet_copy_client.cc:316] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:40.666105 25464 ts_tablet_manager.cc:927] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:40.671830 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 2ef03e5e7d0443d69a2228dd175bd010 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:40.672555 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 2ef03e5e7d0443d69a2228dd175bd010 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-2ef03e5e7d0443d69a2228dd175bd010
I20260809 23:17:40.677362 25464 tablet_copy_client.cc:316] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:40.682873 24557 tablet_copy_source_session.cc:205] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:40.687156 25459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ef03e5e7d0443d69a2228dd175bd010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:40.704406 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet fb0467d784e841868c4a1188ee0af4bc from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:40.705051 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet fb0467d784e841868c4a1188ee0af4bc from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-fb0467d784e841868c4a1188ee0af4bc
I20260809 23:17:40.705756 25459 tablet_copy_client.cc:791] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:40.706524 25459 tablet_copy_client.cc:655] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:40.711666 25459 tablet_copy_client.cc:531] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:40.719766 24450 tablet_copy_source_session.cc:205] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:40.723312 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb0467d784e841868c4a1188ee0af4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:40.724151 25459 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:40.742074 25464 tablet_copy_client.cc:791] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:40.743034 25464 tablet_copy_client.cc:655] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:40.747202 25464 tablet_copy_client.cc:531] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:40.752409 25459 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:40.753525 25459 tablet_bootstrap.cc:492] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:40.754308 25459 ts_tablet_manager.cc:1397] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.030s	user 0.016s	sys 0.007s
I20260809 23:17:40.757742 25459 raft_consensus.cc:348] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.758867 25459 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:40.759518 25459 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.763490 25411 heartbeater.cc:503] Master 127.21.255.62:37921 was elected leader, sending a full tablet report...
I20260809 23:17:40.764058 25459 ts_tablet_manager.cc:1428] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.009s	user 0.011s	sys 0.001s
I20260809 23:17:40.767867 25464 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:40.773026 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-2ef03e5e7d0443d69a2228dd175bd010 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:40.773695 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-2ef03e5e7d0443d69a2228dd175bd010 on tablet 2ef03e5e7d0443d69a2228dd175bd010 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:40.797904 25464 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:40.798714 25464 tablet_bootstrap.cc:492] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:40.799381 25464 ts_tablet_manager.cc:1397] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.011s
I20260809 23:17:40.802040 25464 raft_consensus.cc:348] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.802881 25464 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:40.804911 25464 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.808466 25464 ts_tablet_manager.cc:1428] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260809 23:17:40.810480 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-fb0467d784e841868c4a1188ee0af4bc received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:40.810940 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-fb0467d784e841868c4a1188ee0af4bc on tablet fb0467d784e841868c4a1188ee0af4bc with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:40.814739 25464 ts_tablet_manager.cc:927] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:40.822145 25464 tablet_copy_client.cc:316] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:40.839535 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 3962e082dc274290a1f20e2fa64877a9 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:40.840134 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 3962e082dc274290a1f20e2fa64877a9 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-3962e082dc274290a1f20e2fa64877a9
I20260809 23:17:40.848224 24334 tablet_copy_source_session.cc:205] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:40.853255 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3962e082dc274290a1f20e2fa64877a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:40.870325 25464 tablet_copy_client.cc:791] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:40.871037 25464 tablet_copy_client.cc:655] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:40.875933 25464 tablet_copy_client.cc:531] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:40.883352 25464 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:40.904069 25464 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:40.904767 25464 tablet_bootstrap.cc:492] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:40.905364 25464 ts_tablet_manager.cc:1397] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260809 23:17:40.908010 25464 raft_consensus.cc:348] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.908771 25464 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:40.909423 25464 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:40.911686 25464 ts_tablet_manager.cc:1428] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:17:40.913903 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-3962e082dc274290a1f20e2fa64877a9 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:40.914301 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-3962e082dc274290a1f20e2fa64877a9 on tablet 3962e082dc274290a1f20e2fa64877a9 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:40.948617 25464 ts_tablet_manager.cc:927] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:40.950387 25464 tablet_copy_client.cc:316] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:40.951853 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 90dd68b3876b4b75aea408edfa68d5e5 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:40.952340 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 90dd68b3876b4b75aea408edfa68d5e5 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-90dd68b3876b4b75aea408edfa68d5e5
I20260809 23:17:40.959064 24334 tablet_copy_source_session.cc:205] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:40.963127 25464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dd68b3876b4b75aea408edfa68d5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:40.981403 25464 tablet_copy_client.cc:791] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:40.982044 25464 tablet_copy_client.cc:655] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:40.988830 25464 tablet_copy_client.cc:531] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:40.999083 25464 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:41.017928 25464 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:41.018667 25464 tablet_bootstrap.cc:492] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:41.019277 25464 ts_tablet_manager.cc:1397] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.020s	user 0.019s	sys 0.004s
I20260809 23:17:41.021430 25464 raft_consensus.cc:348] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:41.022075 25464 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:41.022540 25464 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:41.024366 25464 ts_tablet_manager.cc:1428] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:17:41.027381 25373 raft_consensus.cc:1203] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:41.030228 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-90dd68b3876b4b75aea408edfa68d5e5 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:41.030702 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-90dd68b3876b4b75aea408edfa68d5e5 on tablet 90dd68b3876b4b75aea408edfa68d5e5 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:41.164925 25373 raft_consensus.cc:1203] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:41.308934 25373 raft_consensus.cc:1203] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:41.418203 25373 raft_consensus.cc:1203] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:41.453924 25235 raft_consensus.cc:1050] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:41.456357 25235 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.462512 24425 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.463107 25373 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.464572 25235 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.466442 24317 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.467941 25051 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.470878 25307 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.489379 25307 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.491268 24425 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.494194 24317 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.509891 25373 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.524757 24165 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.565583 24535 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.570231 24425 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.571744 25235 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:41.571770 25373 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.574131 25051 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:41.583498 25480 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.584755 24435 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.591341 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2ef03e5e7d0443d69a2228dd175bd010 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:41.589672 25373 raft_consensus.cc:2932] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.601336 24168 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.619015 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ef03e5e7d0443d69a2228dd175bd010 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:41.622313 25490 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:41.623355 25490 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:41.624985 25490 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:41.628773 25490 ts_tablet_manager.cc:1905] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:41.644954 25490 ts_tablet_manager.cc:1918] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:41.645411 25490 log.cc:1196] T 2ef03e5e7d0443d69a2228dd175bd010 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/2ef03e5e7d0443d69a2228dd175bd010
I20260809 23:17:41.648329 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 2ef03e5e7d0443d69a2228dd175bd010 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:41.648717 25442 raft_consensus.cc:1050] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:41.652360 25442 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.666882 25373 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.667665 24535 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.669108 25267 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.671263 25267 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.677927 24321 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.684811 25308 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.699717 25267 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.700992 24544 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.701372 24320 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.722373 25373 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.727603 24170 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.745258 24425 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.752408 24321 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.754029 25267 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:41.754714 25373 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.756608 25442 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:41.760036 25442 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.765175 25373 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.772850 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fb0467d784e841868c4a1188ee0af4bc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:41.776607 24314 raft_consensus.cc:2932] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.779948 24170 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.794169 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet fb0467d784e841868c4a1188ee0af4bc with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:41.796924 25501 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:41.797957 25501 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:41.798683 25501 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:41.801748 25501 ts_tablet_manager.cc:1905] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:41.825151 25280 raft_consensus.cc:1050] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:41.825716 25501 ts_tablet_manager.cc:1918] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:41.826113 25501 log.cc:1196] T fb0467d784e841868c4a1188ee0af4bc P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/fb0467d784e841868c4a1188ee0af4bc
I20260809 23:17:41.827798 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet fb0467d784e841868c4a1188ee0af4bc (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:41.827804 25280 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.837625 25373 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.838713 24544 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.839241 24425 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:41.839875 25280 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260809 23:17:41.842504 25453 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:41.843372 25332 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260809 23:17:41.885532 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45187.5us
I20260809 23:17:41.910230 25506 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.912086 24427 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.915151 25373 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.929844 24535 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.929890 24164 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:41.986989 24314 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:41.991638 25373 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.993680 25512 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:41.994246 24535 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:41.995646 25512 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:42.001643 25453 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.008119 24544 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.011555 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3962e082dc274290a1f20e2fa64877a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:42.012346 25373 raft_consensus.cc:2932] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.028512 25512 raft_consensus.cc:1050] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:42.030349 24168 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:42.031359 25512 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:42.045632 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet 3962e082dc274290a1f20e2fa64877a9 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:42.048923 25375 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:42.045711 24544 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:42.050715 25506 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:42.052366 25516 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:42.053439 25516 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:42.053385 25453 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:42.056115 24427 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:42.058161 25332 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:42.061959 25516 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:42.079165 25516 ts_tablet_manager.cc:1905] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:42.091671 25453 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.095170 25516 ts_tablet_manager.cc:1918] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:42.095649 25516 log.cc:1196] T 3962e082dc274290a1f20e2fa64877a9 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/3962e082dc274290a1f20e2fa64877a9
I20260809 23:17:42.098664 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet 3962e082dc274290a1f20e2fa64877a9 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:42.098053 24425 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.104416 24544 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.103824 25373 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.110800 24170 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:42.126484 24320 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:42.132932 24544 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:42.135620 25332 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:42.135859 25375 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:42.139266 25332 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:42.142567 25507 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.145817 24535 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.153043 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 90dd68b3876b4b75aea408edfa68d5e5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:42.153064 25375 raft_consensus.cc:2932] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:42.164865 24170 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:42.175463 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet 90dd68b3876b4b75aea408edfa68d5e5 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:42.176530 25516 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:42.177533 25516 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:42.178139 25516 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:42.181464 25516 ts_tablet_manager.cc:1905] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:42.198518 25516 ts_tablet_manager.cc:1918] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:42.198884 25516 log.cc:1196] T 90dd68b3876b4b75aea408edfa68d5e5 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/90dd68b3876b4b75aea408edfa68d5e5
I20260809 23:17:42.200205 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet 90dd68b3876b4b75aea408edfa68d5e5 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:43.234921 24425 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.241290 24535 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:43.243799 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9932f35c1d8a431f9c035a848f63cb0b. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:43.244810 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:43.247651 24314 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.250556 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:43.256664 25308 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.257763 24535 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.260880 24313 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.271145 24544 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.276499 24168 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.297150 24425 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.299567 25480 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:43.309665 24313 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:43.310179 24498 consensus_peers.cc:487] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec68a6991264e72a23a8750a420795d. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:43.313446 25525 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:43.322418 25051 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.327775 24425 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.341149 24435 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.352778 24164 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:43.357630 24544 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.359513 25524 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:17:43.363883 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ad5d0ab888b41229481d32d207c7466. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:43.364630 24313 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.365969 25308 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:43.366711 24313 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.391577 24314 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.388331 25308 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.427932 24314 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.434185 24164 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:17:43.442618 24269 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2fc12c55ab24490a1ad12ed4b4b5c35. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:43.444351 24539 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.443602 24535 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.444670 24425 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:43.446837 25507 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:43.447921 25332 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:43.461804 25332 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.465287 24539 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.469491 24425 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.485462 24170 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:43.723865 25558 ts_tablet_manager.cc:927] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:43.726150 25558 tablet_copy_client.cc:316] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:43.728091 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet cec68a6991264e72a23a8750a420795d from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:43.728664 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet cec68a6991264e72a23a8750a420795d from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-cec68a6991264e72a23a8750a420795d
I20260809 23:17:43.737149 24557 tablet_copy_source_session.cc:205] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:43.739948 25558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec68a6991264e72a23a8750a420795d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:43.744519 25560 ts_tablet_manager.cc:927] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:43.747689 25560 tablet_copy_client.cc:316] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:43.749312 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet 9932f35c1d8a431f9c035a848f63cb0b from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:43.750021 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet 9932f35c1d8a431f9c035a848f63cb0b from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-9932f35c1d8a431f9c035a848f63cb0b
I20260809 23:17:43.756603 24450 tablet_copy_source_session.cc:205] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:43.760665 25560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9932f35c1d8a431f9c035a848f63cb0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:43.768810 25558 tablet_copy_client.cc:791] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:43.769420 25558 tablet_copy_client.cc:655] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:43.781930 25560 tablet_copy_client.cc:791] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:43.782553 25560 tablet_copy_client.cc:655] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:43.783789 25558 tablet_copy_client.cc:531] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:43.786087 25560 tablet_copy_client.cc:531] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:43.797714 25560 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:43.803889 25558 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:43.820773 25563 ts_tablet_manager.cc:927] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:43.838579 25563 tablet_copy_client.cc:316] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:43.840659 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet 3ad5d0ab888b41229481d32d207c7466 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:43.841419 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet 3ad5d0ab888b41229481d32d207c7466 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-3ad5d0ab888b41229481d32d207c7466
I20260809 23:17:43.847232 25560 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:43.848291 25560 tablet_bootstrap.cc:492] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:43.848970 25560 ts_tablet_manager.cc:1397] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.051s	user 0.022s	sys 0.008s
I20260809 23:17:43.849538 24450 tablet_copy_source_session.cc:205] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:43.851950 25560 raft_consensus.cc:348] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.852876 25560 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:43.840579 25558 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:43.853617 25560 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.854828 25558 tablet_bootstrap.cc:492] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:43.854295 25563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad5d0ab888b41229481d32d207c7466. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:43.859189 25558 ts_tablet_manager.cc:1397] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.055s	user 0.014s	sys 0.012s
I20260809 23:17:43.859673 25560 ts_tablet_manager.cc:1428] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260809 23:17:43.864782 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-9932f35c1d8a431f9c035a848f63cb0b received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:43.865166 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-9932f35c1d8a431f9c035a848f63cb0b on tablet 9932f35c1d8a431f9c035a848f63cb0b with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:43.867197 25558 raft_consensus.cc:348] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.871093 25558 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:43.871706 25558 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.873950 25558 ts_tablet_manager.cc:1428] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260809 23:17:43.886351 25563 tablet_copy_client.cc:791] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:43.887017 25563 tablet_copy_client.cc:655] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:43.887274 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-cec68a6991264e72a23a8750a420795d received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:43.887693 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-cec68a6991264e72a23a8750a420795d on tablet cec68a6991264e72a23a8750a420795d with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:43.892159 25563 tablet_copy_client.cc:531] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:43.907006 25563 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:43.915401 25558 ts_tablet_manager.cc:927] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:43.920658 25558 tablet_copy_client.cc:316] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:43.922845 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:43.924012 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet f2fc12c55ab24490a1ad12ed4b4b5c35 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-f2fc12c55ab24490a1ad12ed4b4b5c35
I20260809 23:17:43.930437 24334 tablet_copy_source_session.cc:205] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:43.931659 25563 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:43.932513 25563 tablet_bootstrap.cc:492] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:43.933169 25563 ts_tablet_manager.cc:1397] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.026s	user 0.021s	sys 0.004s
I20260809 23:17:43.936518 25563 raft_consensus.cc:348] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.937618 25563 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:43.938331 25563 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:43.940680 25563 ts_tablet_manager.cc:1428] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:17:43.944044 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-3ad5d0ab888b41229481d32d207c7466 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:43.944427 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-3ad5d0ab888b41229481d32d207c7466 on tablet 3ad5d0ab888b41229481d32d207c7466 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:43.948540 25558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2fc12c55ab24490a1ad12ed4b4b5c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:43.982550 25558 tablet_copy_client.cc:791] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:43.983160 25558 tablet_copy_client.cc:655] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:43.988205 25558 tablet_copy_client.cc:531] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:43.996968 25558 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:44.015290 25558 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:44.016023 25558 tablet_bootstrap.cc:492] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:44.016675 25558 ts_tablet_manager.cc:1397] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.020s	user 0.016s	sys 0.004s
I20260809 23:17:44.018914 25558 raft_consensus.cc:348] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:44.019786 25558 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:44.020440 25558 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:44.022985 25558 ts_tablet_manager.cc:1428] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.001s
I20260809 23:17:44.025642 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-f2fc12c55ab24490a1ad12ed4b4b5c35 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:44.026022 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-f2fc12c55ab24490a1ad12ed4b4b5c35 on tablet f2fc12c55ab24490a1ad12ed4b4b5c35 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:44.152201 25373 raft_consensus.cc:1203] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:44.209101 25373 raft_consensus.cc:1203] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:44.355984 25373 raft_consensus.cc:1203] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:44.425340 25373 raft_consensus.cc:1203] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:44.698050 25308 raft_consensus.cc:1050] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:44.700594 25308 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.708607 24535 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.708907 25373 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.710101 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.711858 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.717360 24314 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.718914 25544 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.723862 25524 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.729104 24313 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.734493 25373 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.743948 24164 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.749775 24543 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.781159 25051 raft_consensus.cc:1050] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:44.783456 25051 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.787607 24427 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.809685 25373 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:44.811030 24320 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:44.810884 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:44.812440 25524 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:44.820420 24320 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.821456 25373 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.822641 25375 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.824980 25524 raft_consensus.cc:2932] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.827363 25550 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.835985 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9932f35c1d8a431f9c035a848f63cb0b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:44.846407 24427 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.844367 24165 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.849114 25589 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.860461 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 9932f35c1d8a431f9c035a848f63cb0b with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:44.881045 24314 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.882467 25051 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.906031 25592 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:44.910476 25592 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:44.911243 25592 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:44.913488 24425 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.915634 25051 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.927907 25592 ts_tablet_manager.cc:1905] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:44.927188 25375 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.939354 25576 raft_consensus.cc:1050] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:44.945554 24168 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:44.941973 25576 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.950338 25592 ts_tablet_manager.cc:1918] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:44.950774 25592 log.cc:1196] T 9932f35c1d8a431f9c035a848f63cb0b P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/9932f35c1d8a431f9c035a848f63cb0b
I20260809 23:17:44.952479 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet 9932f35c1d8a431f9c035a848f63cb0b (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:44.958317 24313 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:44.963445 24314 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.964296 25373 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.965830 25544 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.967156 25576 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:44.968129 24543 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:44.977752 25576 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:44.979779 24543 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.983217 25332 raft_consensus.cc:1050] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:44.990049 25332 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:44.993870 25373 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:44.998528 24425 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:45.003875 25590 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:45.008970 25308 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.013602 25589 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.012377 24539 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.019814 25051 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:45.023911 24320 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.026363 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cec68a6991264e72a23a8750a420795d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:45.025938 25373 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.039857 24425 raft_consensus.cc:2932] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.040648 24165 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.051432 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet cec68a6991264e72a23a8750a420795d with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
W20260809 23:17:45.057354 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86663.5us
I20260809 23:17:45.026230 24168 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:45.067201 25372 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.079882 24540 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:45.086542 24427 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:45.088016 25332 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:45.080089 25373 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:45.098089 25606 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:45.099211 25604 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:45.102408 25601 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:45.113545 25606 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:45.115900 25606 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:45.120463 25606 ts_tablet_manager.cc:1905] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:45.137768 25507 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.140933 24535 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.144666 24427 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:45.148211 25373 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.151610 25372 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:45.152907 25524 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:45.158092 24313 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:45.161224 25524 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:45.160140 24427 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.165658 24170 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:45.172039 25606 ts_tablet_manager.cc:1918] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:45.172457 25606 log.cc:1196] T cec68a6991264e72a23a8750a420795d P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/cec68a6991264e72a23a8750a420795d
I20260809 23:17:45.174642 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet cec68a6991264e72a23a8750a420795d (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:45.179647 25308 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.191641 24320 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:45.198534 25373 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:45.198593 24539 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:45.200727 25604 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:45.202762 25507 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:45.207435 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3ad5d0ab888b41229481d32d207c7466 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:45.210604 24313 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.211511 25373 raft_consensus.cc:2932] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.220424 25372 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.221082 24535 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.221988 24170 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:45.218111 25604 raft_consensus.cc:2932] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.237443 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ad5d0ab888b41229481d32d207c7466 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:45.238646 25592 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:45.239760 25592 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:45.240630 25592 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:45.245731 25592 ts_tablet_manager.cc:1905] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:45.249500 24165 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:45.265008 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:45.266306 25592 ts_tablet_manager.cc:1918] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:45.266742 25592 log.cc:1196] T 3ad5d0ab888b41229481d32d207c7466 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/3ad5d0ab888b41229481d32d207c7466
I20260809 23:17:45.268214 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet 3ad5d0ab888b41229481d32d207c7466 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:45.268994 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f2fc12c55ab24490a1ad12ed4b4b5c35 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:45.275437 25611 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:45.276446 25611 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:45.277036 25611 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:45.281210 25611 ts_tablet_manager.cc:1905] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:45.294502 25611 ts_tablet_manager.cc:1918] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:45.294876 25611 log.cc:1196] T f2fc12c55ab24490a1ad12ed4b4b5c35 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/f2fc12c55ab24490a1ad12ed4b4b5c35
I20260809 23:17:45.296290 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet f2fc12c55ab24490a1ad12ed4b4b5c35 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:46.345855 24425 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.353343 24544 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:46.354151 24313 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:46.354159 24378 consensus_peers.cc:487] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d273006c869f4749852291bcee467158. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:46.355048 25544 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:46.356971 25308 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:46.365680 25308 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.370085 24544 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.380532 24320 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.391891 24165 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.397485 24314 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.412815 24425 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:46.414969 24544 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:46.415725 24269 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: deaa6cc935614b7bbb64f5ee8faf4c74. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:46.420094 25620 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:46.422132 25604 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:46.434569 25332 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.439546 24544 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.441586 24435 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.461119 24535 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.476631 24435 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:46.476797 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:46.478708 25590 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:17:46.482632 24498 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34970da619114abc942b839d0862a300. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:46.483557 25590 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:46.492870 25590 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.497056 24170 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.508124 24535 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.509977 24435 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.520735 24168 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:46.527767 24425 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:46.530301 24498 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca74ab4e31434c79a52a994e74129545. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:46.531138 25633 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:46.536985 24313 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:46.538462 25633 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:46.555507 24314 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.560441 25590 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.571748 24320 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.596253 24435 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.609163 24165 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:46.806049 25644 ts_tablet_manager.cc:927] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:46.808221 25644 tablet_copy_client.cc:316] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:46.810086 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet d273006c869f4749852291bcee467158 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:46.810560 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet d273006c869f4749852291bcee467158 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-d273006c869f4749852291bcee467158
I20260809 23:17:46.818710 24450 tablet_copy_source_session.cc:205] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:46.822989 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d273006c869f4749852291bcee467158. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:46.839108 25644 tablet_copy_client.cc:791] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:46.839838 25644 tablet_copy_client.cc:655] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:46.846431 25644 tablet_copy_client.cc:531] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:46.855254 25644 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:46.878109 25644 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:46.879056 25644 tablet_bootstrap.cc:492] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:46.879807 25644 ts_tablet_manager.cc:1397] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.004s
I20260809 23:17:46.882468 25644 raft_consensus.cc:348] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.883301 25644 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:46.883991 25644 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:46.894280 25644 ts_tablet_manager.cc:1428] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.003s
I20260809 23:17:46.896133 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-d273006c869f4749852291bcee467158 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:46.896493 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-d273006c869f4749852291bcee467158 on tablet d273006c869f4749852291bcee467158 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:46.941015 25644 ts_tablet_manager.cc:927] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:46.943567 25644 tablet_copy_client.cc:316] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:46.947796 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 34970da619114abc942b839d0862a300 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:46.948418 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 34970da619114abc942b839d0862a300 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-34970da619114abc942b839d0862a300
I20260809 23:17:46.963508 25653 ts_tablet_manager.cc:927] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:46.982159 24557 tablet_copy_source_session.cc:205] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:46.985749 25644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34970da619114abc942b839d0862a300. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:46.984747 25653 tablet_copy_client.cc:316] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:47.003451 25644 tablet_copy_client.cc:791] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:47.004128 25644 tablet_copy_client.cc:655] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:47.006131 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet deaa6cc935614b7bbb64f5ee8faf4c74 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:47.006806 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet deaa6cc935614b7bbb64f5ee8faf4c74 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-deaa6cc935614b7bbb64f5ee8faf4c74
I20260809 23:17:47.014729 25644 tablet_copy_client.cc:531] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:47.019006 24334 tablet_copy_source_session.cc:205] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:47.022544 25653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deaa6cc935614b7bbb64f5ee8faf4c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:47.024375 25656 ts_tablet_manager.cc:927] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:47.027434 25644 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:47.046829 25653 tablet_copy_client.cc:791] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:47.047385 25653 tablet_copy_client.cc:655] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:47.055276 25653 tablet_copy_client.cc:531] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:47.057602 25656 tablet_copy_client.cc:316] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:47.064291 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet ca74ab4e31434c79a52a994e74129545 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:47.064731 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet ca74ab4e31434c79a52a994e74129545 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-ca74ab4e31434c79a52a994e74129545
I20260809 23:17:47.064859 25653 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:47.067093 25644 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:47.069176 25644 tablet_bootstrap.cc:492] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:47.069898 25644 ts_tablet_manager.cc:1397] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.043s	user 0.015s	sys 0.015s
I20260809 23:17:47.072957 24557 tablet_copy_source_session.cc:205] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:47.073462 25644 raft_consensus.cc:348] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.074342 25644 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:47.075064 25644 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.075923 25656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca74ab4e31434c79a52a994e74129545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:47.077340 25644 ts_tablet_manager.cc:1428] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260809 23:17:47.082849 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-34970da619114abc942b839d0862a300 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:47.083331 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-34970da619114abc942b839d0862a300 on tablet 34970da619114abc942b839d0862a300 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:47.093961 25656 tablet_copy_client.cc:791] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:47.094036 25653 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:47.094652 25656 tablet_copy_client.cc:655] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:47.096932 25653 tablet_bootstrap.cc:492] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:47.097652 25653 ts_tablet_manager.cc:1397] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.033s	user 0.021s	sys 0.002s
I20260809 23:17:47.099284 25656 tablet_copy_client.cc:531] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:47.100592 25653 raft_consensus.cc:348] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.101493 25653 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:47.102181 25653 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.104679 25653 ts_tablet_manager.cc:1428] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:17:47.109191 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-deaa6cc935614b7bbb64f5ee8faf4c74 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:47.109584 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-deaa6cc935614b7bbb64f5ee8faf4c74 on tablet deaa6cc935614b7bbb64f5ee8faf4c74 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:47.114003 25656 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:47.145262 25656 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:47.146062 25656 tablet_bootstrap.cc:492] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:47.146708 25656 ts_tablet_manager.cc:1397] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.011s
I20260809 23:17:47.149866 25656 raft_consensus.cc:348] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.150650 25656 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:47.151315 25656 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:47.153874 25656 ts_tablet_manager.cc:1428] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.004s
I20260809 23:17:47.155647 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-ca74ab4e31434c79a52a994e74129545 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:47.156034 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-ca74ab4e31434c79a52a994e74129545 on tablet ca74ab4e31434c79a52a994e74129545 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:47.357264 25372 raft_consensus.cc:1203] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:47.412919 25373 raft_consensus.cc:1203] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:47.539565 25373 raft_consensus.cc:1203] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:47.621773 25373 raft_consensus.cc:1203] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:47.829998 25662 raft_consensus.cc:1050] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:47.832475 25662 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:47.845424 24435 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.845741 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.846158 25373 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.847234 25051 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:47.848161 25633 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:47.852367 25662 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:47.864152 25633 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:47.874580 25373 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:47.875411 24427 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:47.882439 25524 raft_consensus.cc:1050] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:47.884683 25524 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:47.892349 24540 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.889315 24168 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:47.896310 25308 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:47.906121 24314 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:47.922034 25373 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.923362 24314 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:47.931488 25544 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:47.932487 25524 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:48.027335 25308 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.047048 25604 raft_consensus.cc:1050] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:48.049404 25604 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.049244 24320 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.055603 24540 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.070669 25372 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:48.071316 24544 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:48.072582 25620 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:48.077292 25620 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:48.049093 25373 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.082494 24535 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.090281 25375 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.091095 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.093719 25633 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:48.098230 25633 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:48.112568 25590 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.113541 25620 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.116506 25370 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.098646 24165 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.122637 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 34970da619114abc942b839d0862a300 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:48.140583 24425 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:48.152351 25662 raft_consensus.cc:1050] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:48.155019 24168 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.155970 25662 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.150961 24314 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.168001 25604 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:48.177080 25373 raft_consensus.cc:2932] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.180658 24425 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.209936 24540 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.205621 24168 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:48.224969 24427 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:48.228614 24314 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:48.229496 25373 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260809 23:17:48.230842 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55016.5us
I20260809 23:17:48.233071 25590 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:48.235913 25051 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:17:48.244681 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 100256us
I20260809 23:17:48.245431 25370 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.253441 25308 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:48.257941 25705 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:48.288213 25676 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:48.314167 25524 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.324296 24535 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.327943 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet 34970da619114abc942b839d0862a300 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:48.330621 25712 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:48.341893 24425 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.356765 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d273006c869f4749852291bcee467158 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:48.363261 25662 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.365281 25712 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:48.366091 25712 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:48.370358 24429 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.368002 25370 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.401499 24321 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.404605 24535 raft_consensus.cc:2932] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.413743 25373 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.460372 25712 ts_tablet_manager.cc:1905] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:48.461577 24164 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.469630 24168 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:48.473140 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet d273006c869f4749852291bcee467158 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:48.495776 25712 ts_tablet_manager.cc:1918] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:48.496568 25712 log.cc:1196] T 34970da619114abc942b839d0862a300 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/34970da619114abc942b839d0862a300
I20260809 23:17:48.498467 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet 34970da619114abc942b839d0862a300 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:48.505074 24543 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.508328 25713 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:48.509459 25713 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:48.510031 25713 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:48.512426 25373 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.512429 24429 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.514293 25697 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:48.522892 25051 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:48.521785 25662 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.527135 24425 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.528329 25373 raft_consensus.cc:2932] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.531621 25713 ts_tablet_manager.cc:1905] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:48.539775 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ca74ab4e31434c79a52a994e74129545 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:48.541607 24170 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.549917 25713 ts_tablet_manager.cc:1918] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:48.550418 25713 log.cc:1196] T d273006c869f4749852291bcee467158 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/d273006c869f4749852291bcee467158
I20260809 23:17:48.555250 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet d273006c869f4749852291bcee467158 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:48.563302 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet ca74ab4e31434c79a52a994e74129545 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:48.564215 25713 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:48.565194 25713 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:48.565848 25713 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:48.569006 25713 ts_tablet_manager.cc:1905] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:48.584060 25713 ts_tablet_manager.cc:1918] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:48.584460 25713 log.cc:1196] T ca74ab4e31434c79a52a994e74129545 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/ca74ab4e31434c79a52a994e74129545
I20260809 23:17:48.586025 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet ca74ab4e31434c79a52a994e74129545 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:48.807857 24321 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:48.812330 25373 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.813144 24425 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:48.813879 25694 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:48.815071 25693 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:48.821987 25694 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.823712 24429 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.826372 25373 raft_consensus.cc:2932] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.840972 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet deaa6cc935614b7bbb64f5ee8faf4c74 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:48.841943 24168 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:48.852923 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet deaa6cc935614b7bbb64f5ee8faf4c74 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:48.854718 25714 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:48.855953 25714 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:48.856660 25714 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:48.859771 25714 ts_tablet_manager.cc:1905] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:48.875479 25714 ts_tablet_manager.cc:1918] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:48.875821 25714 log.cc:1196] T deaa6cc935614b7bbb64f5ee8faf4c74 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/deaa6cc935614b7bbb64f5ee8faf4c74
I20260809 23:17:48.877359 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet deaa6cc935614b7bbb64f5ee8faf4c74 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:49.915824 24425 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:17:49.922554 24378 consensus_peers.cc:487] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dad562edaec4eebaa047293c085c0c4. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:49.923822 24543 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:49.923892 24321 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:49.925796 25716 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:49.926981 25544 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:49.937307 25544 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:49.943514 24321 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:49.945545 24543 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:49.961658 24314 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:49.969853 24168 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:17:49.978193 24269 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8a29ab049db4236a3198e8545cafb9c. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:49.981194 24540 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:49.983052 25693 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:49.984864 24429 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:49.991513 25693 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:50.001997 25693 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.003492 24540 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.004858 24429 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.020462 24429 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.032976 24314 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:50.034212 24540 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:50.035297 25716 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:50.038123 25716 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:17:50.039005 24378 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a682e0d089354a44a680bd8ad9b4b789. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:50.039775 24168 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.060426 25723 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.062147 24314 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.073801 24540 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.085523 24321 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.090793 24170 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:50.104146 24540 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:50.105068 24269 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af1ef8fe019645f3b99232df6916cb2c. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:50.105588 24429 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:50.106963 25693 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:50.108556 25620 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:50.118867 25693 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.120836 24540 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.127460 24429 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:50.143082 24168 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:50.421224 25750 ts_tablet_manager.cc:927] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:50.425671 25750 tablet_copy_client.cc:316] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:50.428396 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet 9dad562edaec4eebaa047293c085c0c4 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:50.428920 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet 9dad562edaec4eebaa047293c085c0c4 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-9dad562edaec4eebaa047293c085c0c4
I20260809 23:17:50.436677 24450 tablet_copy_source_session.cc:205] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:50.440366 25750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dad562edaec4eebaa047293c085c0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:50.442457 25754 ts_tablet_manager.cc:927] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:50.467157 25750 tablet_copy_client.cc:791] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:50.467739 25750 tablet_copy_client.cc:655] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:50.471347 25750 tablet_copy_client.cc:531] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:50.476604 25754 tablet_copy_client.cc:316] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:50.478235 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet b8a29ab049db4236a3198e8545cafb9c from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:50.478741 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet b8a29ab049db4236a3198e8545cafb9c from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-b8a29ab049db4236a3198e8545cafb9c
I20260809 23:17:50.481395 25750 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:50.486042 24334 tablet_copy_source_session.cc:205] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:50.490753 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a29ab049db4236a3198e8545cafb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:50.509820 25754 tablet_copy_client.cc:791] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:50.509974 25750 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:50.510360 25754 tablet_copy_client.cc:655] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:50.510910 25750 tablet_bootstrap.cc:492] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:50.511976 25750 ts_tablet_manager.cc:1397] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.031s	user 0.021s	sys 0.009s
I20260809 23:17:50.514783 25750 raft_consensus.cc:348] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.515604 25750 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:50.516552 25750 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.519555 25754 tablet_copy_client.cc:531] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:50.526036 25750 ts_tablet_manager.cc:1428] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.004s
I20260809 23:17:50.527495 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-9dad562edaec4eebaa047293c085c0c4 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:50.527896 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-9dad562edaec4eebaa047293c085c0c4 on tablet 9dad562edaec4eebaa047293c085c0c4 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:50.531095 25750 ts_tablet_manager.cc:927] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:50.534911 25754 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:50.546689 25750 tablet_copy_client.cc:316] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:50.548314 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet a682e0d089354a44a680bd8ad9b4b789 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:50.548840 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet a682e0d089354a44a680bd8ad9b4b789 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-a682e0d089354a44a680bd8ad9b4b789
I20260809 23:17:50.557227 24450 tablet_copy_source_session.cc:205] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:50.559226 25754 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:50.560097 25754 tablet_bootstrap.cc:492] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:50.560835 25750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a682e0d089354a44a680bd8ad9b4b789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:50.561105 25754 ts_tablet_manager.cc:1397] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.008s
I20260809 23:17:50.564023 25754 raft_consensus.cc:348] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.564836 25754 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:50.565442 25754 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.567807 25754 ts_tablet_manager.cc:1428] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:17:50.569849 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-b8a29ab049db4236a3198e8545cafb9c received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:50.570235 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-b8a29ab049db4236a3198e8545cafb9c on tablet b8a29ab049db4236a3198e8545cafb9c with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:50.577950 25750 tablet_copy_client.cc:791] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:50.578670 25750 tablet_copy_client.cc:655] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:50.584292 25750 tablet_copy_client.cc:531] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:50.592274 25750 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:50.614563 25750 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:50.615338 25750 tablet_bootstrap.cc:492] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:50.615989 25750 ts_tablet_manager.cc:1397] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.024s	user 0.017s	sys 0.006s
I20260809 23:17:50.618760 25750 raft_consensus.cc:348] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.619654 25750 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:50.620185 25750 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.622596 25750 ts_tablet_manager.cc:1428] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260809 23:17:50.633294 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-a682e0d089354a44a680bd8ad9b4b789 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:50.633718 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-a682e0d089354a44a680bd8ad9b4b789 on tablet a682e0d089354a44a680bd8ad9b4b789 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:50.651234 25750 ts_tablet_manager.cc:927] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:50.654102 25750 tablet_copy_client.cc:316] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:50.655726 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet af1ef8fe019645f3b99232df6916cb2c from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:50.656280 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet af1ef8fe019645f3b99232df6916cb2c from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-af1ef8fe019645f3b99232df6916cb2c
I20260809 23:17:50.663311 24334 tablet_copy_source_session.cc:205] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:50.666117 25750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1ef8fe019645f3b99232df6916cb2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:50.680600 25750 tablet_copy_client.cc:791] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:50.681145 25750 tablet_copy_client.cc:655] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:50.685575 25750 tablet_copy_client.cc:531] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:50.693662 25750 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:50.712401 25750 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:50.713438 25750 tablet_bootstrap.cc:492] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:50.714205 25750 ts_tablet_manager.cc:1397] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.021s	user 0.019s	sys 0.000s
I20260809 23:17:50.716416 25750 raft_consensus.cc:348] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.717212 25750 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:50.717765 25750 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:50.719894 25750 ts_tablet_manager.cc:1428] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260809 23:17:50.721838 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-af1ef8fe019645f3b99232df6916cb2c received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:50.722213 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-af1ef8fe019645f3b99232df6916cb2c on tablet af1ef8fe019645f3b99232df6916cb2c with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:50.870795 25373 raft_consensus.cc:1203] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:50.885146 25373 raft_consensus.cc:1203] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:51.147050 25370 raft_consensus.cc:1203] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:51.171437 25370 raft_consensus.cc:1203] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:51.329782 25718 raft_consensus.cc:1050] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:51.332870 25718 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:51.521519 25772 raft_consensus.cc:1050] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:51.524441 25772 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:51.532230 25370 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.533401 25693 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.617493 24539 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.619418 25747 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.624624 24429 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.639961 25747 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.644069 24535 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.646669 25620 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.671324 24539 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.679675 24314 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.687599 25779 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.692595 25718 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.696797 25370 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.697829 25724 raft_consensus.cc:1050] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:51.700946 25724 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:51.717777 25781 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:51.729970 25776 raft_consensus.cc:1050] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:51.732494 25776 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:51.736507 25372 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.742430 25372 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.744060 25772 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.744692 24320 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.756551 24540 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.758859 25620 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:51.767081 25718 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:51.783506 24425 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.786088 25718 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:51.798269 25693 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:51.818408 24540 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:51.847359 24435 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.844403 25776 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.838539 25769 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260809 23:17:51.854562 25371 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.849903 24170 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.882251 25781 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.886188 24320 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.898922 25769 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.907004 25372 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:51.903916 25370 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:17:52.022303 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 97321.5us
W20260809 23:17:52.054570 25800 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.103s	user 0.005s	sys 0.006s
I20260809 23:17:52.103080 24540 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.110078 24429 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.162997 24545 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:17:52.196265 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 112113us
I20260809 23:17:52.212838 25375 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.210126 24314 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:17:52.265007 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 90254.8us
I20260809 23:17:52.273854 24545 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.293320 24165 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.295467 24321 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:52.295187 24165 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.300233 24544 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.303500 25371 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.305861 25819 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:52.307590 25776 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:52.315275 25693 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.323865 25371 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.327162 24545 raft_consensus.cc:2932] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.329409 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b8a29ab049db4236a3198e8545cafb9c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:52.354956 24170 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.373453 24165 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.377755 24435 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:52.382407 24435 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:52.383587 25371 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.384934 25718 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:52.388836 25375 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.391327 25718 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:52.392031 24545 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.394117 25816 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:52.394657 25826 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.403813 24314 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.404810 25370 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.421406 24314 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:52.420240 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a29ab049db4236a3198e8545cafb9c with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:52.415053 25718 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260809 23:17:52.434474 24544 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.434402 25375 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:52.436321 25780 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.438776 25819 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:52.440284 25371 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.443125 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a682e0d089354a44a680bd8ad9b4b789 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:52.446413 25819 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:52.450145 24314 raft_consensus.cc:2932] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.448798 24544 raft_consensus.cc:2932] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.452636 24542 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.462379 24164 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.449347 25776 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.484223 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet a682e0d089354a44a680bd8ad9b4b789 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:52.492975 24164 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.497339 25832 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:52.500586 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet af1ef8fe019645f3b99232df6916cb2c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:52.505961 25832 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:52.503116 25370 raft_consensus.cc:2932] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.508033 25832 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:52.530230 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dad562edaec4eebaa047293c085c0c4 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:52.533345 25832 ts_tablet_manager.cc:1905] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:52.534209 25833 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:52.537616 25833 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:52.538731 25833 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:52.542142 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9dad562edaec4eebaa047293c085c0c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:52.544037 24165 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:52.545536 25835 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
W20260809 23:17:52.557335 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080005bb80 after lost signal to thread 22536
W20260809 23:17:52.558430 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800085360 after lost signal to thread 24220
W20260809 23:17:52.559685 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6f80 after lost signal to thread 24355
W20260809 23:17:52.560951 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800324120 after lost signal to thread 24469
I20260809 23:17:52.561350 25833 ts_tablet_manager.cc:1905] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260809 23:17:52.562011 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08002f8a20 after lost signal to thread 24579
I20260809 23:17:52.559621 25835 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:52.563702 25835 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:52.566026 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet af1ef8fe019645f3b99232df6916cb2c with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:52.568190 25835 ts_tablet_manager.cc:1905] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260809 23:17:52.583418 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800304bc0 after lost signal to thread 25410
I20260809 23:17:52.602489 25835 ts_tablet_manager.cc:1918] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:52.603277 25835 log.cc:1196] T 9dad562edaec4eebaa047293c085c0c4 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/9dad562edaec4eebaa047293c085c0c4
I20260809 23:17:52.605352 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 9dad562edaec4eebaa047293c085c0c4 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:52.614379 25832 ts_tablet_manager.cc:1918] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:52.614884 25832 log.cc:1196] T b8a29ab049db4236a3198e8545cafb9c P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/b8a29ab049db4236a3198e8545cafb9c
I20260809 23:17:52.616532 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet b8a29ab049db4236a3198e8545cafb9c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:52.617100 25832 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:52.619593 25832 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:52.620420 25832 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:52.621273 25833 ts_tablet_manager.cc:1918] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:52.623550 25833 log.cc:1196] T a682e0d089354a44a680bd8ad9b4b789 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/a682e0d089354a44a680bd8ad9b4b789
I20260809 23:17:52.624296 25832 ts_tablet_manager.cc:1905] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:52.625779 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet a682e0d089354a44a680bd8ad9b4b789 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:52.645392 25832 ts_tablet_manager.cc:1918] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:52.645898 25832 log.cc:1196] T af1ef8fe019645f3b99232df6916cb2c P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/af1ef8fe019645f3b99232df6916cb2c
I20260809 23:17:52.647578 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet af1ef8fe019645f3b99232df6916cb2c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:53.665099 24429 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:53.673040 24544 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.673907 24321 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.675457 25829 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:17:53.678236 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3063532c2cd49f8b7075e771a2e818a. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:53.679467 25781 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:53.711860 25829 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.713366 24544 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.728477 24314 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.731712 24429 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:53.753125 24165 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.770762 24544 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.780102 25781 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:17:53.791016 24378 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c37e848050ec44069438f4c88a76f9ef. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:53.793071 24313 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.796754 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:53.808431 25781 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.811765 24313 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.821339 24544 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.836303 24313 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:53.846498 24170 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 23:17:53.856745 24269 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c66997a772ef46a5ae07d530d26ab8a0. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:53.859376 24542 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.869159 25776 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:53.891888 24429 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.893536 25776 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:53.907267 25813 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.913578 24542 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.936048 24429 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:53.950568 24542 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:53.951859 24170 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:53.973134 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:53.974614 24498 consensus_peers.cc:487] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2c4cf897ac64cb8981b99dda67dbc92. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:53.974671 25662 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:53.977993 24314 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:53.986374 25853 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:54.000032 25774 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:54.006402 24313 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:54.001663 24427 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:54.020855 24164 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:54.063963 25867 ts_tablet_manager.cc:927] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:54.066469 25867 tablet_copy_client.cc:316] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:54.069335 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet b3063532c2cd49f8b7075e771a2e818a from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:54.069829 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet b3063532c2cd49f8b7075e771a2e818a from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-b3063532c2cd49f8b7075e771a2e818a
I20260809 23:17:54.077788 24450 tablet_copy_source_session.cc:205] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:54.084089 25867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3063532c2cd49f8b7075e771a2e818a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:54.109793 25867 tablet_copy_client.cc:791] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:54.110373 25867 tablet_copy_client.cc:655] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:54.115512 25867 tablet_copy_client.cc:531] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:54.126189 25867 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:54.161543 25867 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:54.162489 25867 tablet_bootstrap.cc:492] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:54.163265 25867 ts_tablet_manager.cc:1397] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.015s
I20260809 23:17:54.167083 25867 raft_consensus.cc:348] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.168629 25867 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:54.169546 25867 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.173485 25867 ts_tablet_manager.cc:1428] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260809 23:17:54.175383 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-b3063532c2cd49f8b7075e771a2e818a received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:54.175812 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-b3063532c2cd49f8b7075e771a2e818a on tablet b3063532c2cd49f8b7075e771a2e818a with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:54.315014 25867 ts_tablet_manager.cc:927] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:54.330516 25867 tablet_copy_client.cc:316] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:54.332782 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet c37e848050ec44069438f4c88a76f9ef from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:54.333381 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet c37e848050ec44069438f4c88a76f9ef from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-c37e848050ec44069438f4c88a76f9ef
I20260809 23:17:54.358227 24450 tablet_copy_source_session.cc:205] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:54.361699 25867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37e848050ec44069438f4c88a76f9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:54.386677 25867 tablet_copy_client.cc:791] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:54.387331 25867 tablet_copy_client.cc:655] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:54.392710 25867 tablet_copy_client.cc:531] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:54.403648 25867 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:54.420156 25873 ts_tablet_manager.cc:927] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:54.426810 25873 tablet_copy_client.cc:316] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:54.436254 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet c66997a772ef46a5ae07d530d26ab8a0 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:54.436831 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet c66997a772ef46a5ae07d530d26ab8a0 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-c66997a772ef46a5ae07d530d26ab8a0
I20260809 23:17:54.444213 25867 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:54.444796 24334 tablet_copy_source_session.cc:205] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:54.445245 25867 tablet_bootstrap.cc:492] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:54.446265 25867 ts_tablet_manager.cc:1397] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.043s	user 0.024s	sys 0.003s
I20260809 23:17:54.448668 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66997a772ef46a5ae07d530d26ab8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:54.469136 25867 raft_consensus.cc:348] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.470146 25867 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:54.470890 25867 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.480093 25867 ts_tablet_manager.cc:1428] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260809 23:17:54.488271 25873 tablet_copy_client.cc:791] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:54.489579 25873 tablet_copy_client.cc:655] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:54.493253 25873 tablet_copy_client.cc:531] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:54.494678 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-c37e848050ec44069438f4c88a76f9ef received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:54.495150 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-c37e848050ec44069438f4c88a76f9ef on tablet c37e848050ec44069438f4c88a76f9ef with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:54.505327 25873 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:54.507378 25370 raft_consensus.cc:1203] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:54.541100 25873 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:54.542024 25873 tablet_bootstrap.cc:492] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:54.544684 25873 ts_tablet_manager.cc:1397] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.040s	user 0.024s	sys 0.006s
I20260809 23:17:54.547971 25873 raft_consensus.cc:348] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.549224 25873 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:54.550010 25873 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.553364 25873 ts_tablet_manager.cc:1428] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260809 23:17:54.555117 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-c66997a772ef46a5ae07d530d26ab8a0 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:54.555539 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-c66997a772ef46a5ae07d530d26ab8a0 on tablet c66997a772ef46a5ae07d530d26ab8a0 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:54.580855 25873 ts_tablet_manager.cc:927] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:54.582845 25873 tablet_copy_client.cc:316] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:54.584475 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet b2c4cf897ac64cb8981b99dda67dbc92 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:54.585003 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet b2c4cf897ac64cb8981b99dda67dbc92 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-b2c4cf897ac64cb8981b99dda67dbc92
I20260809 23:17:54.592456 24557 tablet_copy_source_session.cc:205] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:54.595644 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c4cf897ac64cb8981b99dda67dbc92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:54.630290 25873 tablet_copy_client.cc:791] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:54.631009 25873 tablet_copy_client.cc:655] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:54.635128 25873 tablet_copy_client.cc:531] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:54.645138 25873 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:54.676537 25873 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:54.681970 25873 tablet_bootstrap.cc:492] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:54.682729 25873 ts_tablet_manager.cc:1397] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.038s	user 0.023s	sys 0.000s
I20260809 23:17:54.685447 25873 raft_consensus.cc:348] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.686630 25873 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:54.687357 25873 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:54.691069 25873 ts_tablet_manager.cc:1428] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260809 23:17:54.693125 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-b2c4cf897ac64cb8981b99dda67dbc92 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:54.693562 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-b2c4cf897ac64cb8981b99dda67dbc92 on tablet b2c4cf897ac64cb8981b99dda67dbc92 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:54.702888 25370 raft_consensus.cc:1203] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:54.965067 25829 raft_consensus.cc:1050] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:54.967378 25829 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:54.982647 24542 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:54.984138 25829 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:54.988022 25370 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:54.990213 24314 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:54.993325 25781 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:54.999909 25780 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:55.025451 25375 raft_consensus.cc:1203] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:55.038396 25829 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.053640 24313 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.055835 25370 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.081081 24535 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.095762 24164 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:55.141247 24427 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:55.163534 25375 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:55.164912 24313 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:55.165714 25781 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:55.167462 25829 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:55.176790 25891 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.179239 25375 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.195645 24313 raft_consensus.cc:2932] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.197908 24165 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.213202 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b3063532c2cd49f8b7075e771a2e818a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:55.215226 25370 raft_consensus.cc:1203] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:55.242535 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet b3063532c2cd49f8b7075e771a2e818a with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:55.244553 25898 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:55.245733 25898 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:55.246520 25898 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:55.259127 25898 ts_tablet_manager.cc:1905] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:55.287078 25898 ts_tablet_manager.cc:1918] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:55.287971 25898 log.cc:1196] T b3063532c2cd49f8b7075e771a2e818a P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/b3063532c2cd49f8b7075e771a2e818a
I20260809 23:17:55.290992 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet b3063532c2cd49f8b7075e771a2e818a (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:55.312028 25890 raft_consensus.cc:1050] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:55.317899 25890 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:55.351855 24542 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.352814 25375 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.353747 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:55.356321 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:55.380018 24313 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.381659 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:55.417587 25888 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.436959 25371 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.471338 24165 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.480388 24535 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.504565 24314 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.635717 25855 raft_consensus.cc:1050] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:55.643194 25855 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:55.650519 24435 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:17:55.652108 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66412.3us
I20260809 23:17:55.725013 24544 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:55.726662 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:55.728358 24435 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.730360 25905 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:55.732969 25373 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.740023 25909 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.745334 24535 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.752060 25858 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:55.757503 24544 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.766876 25858 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:55.782613 25375 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:55.787124 25863 raft_consensus.cc:1050] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:55.789084 25780 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:55.799898 25863 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:55.810098 24165 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.833377 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c37e848050ec44069438f4c88a76f9ef with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:55.845619 25371 raft_consensus.cc:2932] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.852604 25776 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.863679 24435 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.867280 24429 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.868788 25863 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:55.877226 25375 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.878338 24312 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:55.883921 25774 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260809 23:17:55.888995 25662 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:17:55.901648 25919 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260809 23:17:55.914508 25373 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.925737 24542 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:55.933612 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet c37e848050ec44069438f4c88a76f9ef with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:55.980568 24165 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:55.994408 25662 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.016520 24314 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.014256 24312 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:56.017058 25371 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.023368 25373 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:56.025193 25908 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:56.028375 24544 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:56.031605 25926 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:56.032701 25926 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:56.038195 24429 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.039984 25855 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.041635 25375 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.054483 25926 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:56.056128 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c66997a772ef46a5ae07d530d26ab8a0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:56.058707 25908 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:56.065763 24544 raft_consensus.cc:2932] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.069957 24168 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.071164 25926 ts_tablet_manager.cc:1905] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:56.084533 24165 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.097491 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet c66997a772ef46a5ae07d530d26ab8a0 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
I20260809 23:17:56.100987 25926 ts_tablet_manager.cc:1918] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:56.101686 25926 log.cc:1196] T c37e848050ec44069438f4c88a76f9ef P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/c37e848050ec44069438f4c88a76f9ef
I20260809 23:17:56.104094 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet c37e848050ec44069438f4c88a76f9ef (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:56.137596 24544 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:56.167784 25371 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:56.167805 24435 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:56.178499 25863 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:17:56.193588 25923 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.195086 25774 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:56.202629 24435 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.205272 25928 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:56.199836 25375 raft_consensus.cc:2932] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.223276 25928 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:56.224128 25928 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:56.227077 24168 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:56.232403 25928 ts_tablet_manager.cc:1905] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:56.239492 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c4cf897ac64cb8981b99dda67dbc92 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:56.240464 25926 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:56.241567 25926 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:56.242259 25926 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:56.242620 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b2c4cf897ac64cb8981b99dda67dbc92 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:56.252288 25926 ts_tablet_manager.cc:1905] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:56.257915 25928 ts_tablet_manager.cc:1918] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:56.258613 25928 log.cc:1196] T c66997a772ef46a5ae07d530d26ab8a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/c66997a772ef46a5ae07d530d26ab8a0
I20260809 23:17:56.260864 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet c66997a772ef46a5ae07d530d26ab8a0 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:56.270417 25926 ts_tablet_manager.cc:1918] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:56.271286 25926 log.cc:1196] T b2c4cf897ac64cb8981b99dda67dbc92 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/b2c4cf897ac64cb8981b99dda67dbc92
I20260809 23:17:56.274411 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet b2c4cf897ac64cb8981b99dda67dbc92 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:57.320470 24435 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.326913 24544 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:57.328387 24312 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:57.329159 24378 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3538d14b2274bf99cd72e3872fda1fa. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:57.329783 25891 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:57.330717 25916 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:57.339619 25891 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.341073 24544 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.343201 24312 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.360603 24165 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.366042 24545 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.414729 24435 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:57.417016 25923 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:17:57.418780 24498 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65d375db793a4542808e1cff72fb6617. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:57.420433 24312 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:57.421648 25929 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:17:57.452167 25923 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.458127 24312 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.464594 24435 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.473915 24312 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.481082 24168 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:17:57.484916 24545 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:57.485884 24269 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48af4d51aa8f41a486c4fe6e7541700e. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:57.487262 25950 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:57.486577 24435 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:57.489537 25776 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:57.503036 25931 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.505563 24545 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.510411 24429 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.519546 24435 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.526515 24545 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:17:57.530050 25891 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:57.533926 24164 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.544005 24312 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:17:57.544838 24378 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c831643e28e34208b6edf775607155e4. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:17:57.547623 25942 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:17:57.562013 25943 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.571769 24544 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.592864 24165 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.601892 24316 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:17:57.798986 25958 ts_tablet_manager.cc:927] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:17:57.801327 25958 tablet_copy_client.cc:316] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:17:57.803277 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 65d375db793a4542808e1cff72fb6617 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:17:57.803798 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 65d375db793a4542808e1cff72fb6617 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-65d375db793a4542808e1cff72fb6617
I20260809 23:17:57.812192 24557 tablet_copy_source_session.cc:205] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:57.815591 25958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d375db793a4542808e1cff72fb6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:57.833353 25958 tablet_copy_client.cc:791] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:57.834052 25958 tablet_copy_client.cc:655] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:57.839406 25958 tablet_copy_client.cc:531] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:57.853773 25958 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:57.889314 25958 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:57.890396 25958 tablet_bootstrap.cc:492] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:57.891382 25958 ts_tablet_manager.cc:1397] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.038s	user 0.029s	sys 0.009s
I20260809 23:17:57.894567 25958 raft_consensus.cc:348] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.895449 25958 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:57.896579 25958 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:57.903527 25958 ts_tablet_manager.cc:1428] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260809 23:17:57.906322 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-65d375db793a4542808e1cff72fb6617 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:17:57.906877 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-65d375db793a4542808e1cff72fb6617 on tablet 65d375db793a4542808e1cff72fb6617 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:57.916663 25958 ts_tablet_manager.cc:927] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:17:57.919951 25958 tablet_copy_client.cc:316] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:17:57.922842 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 48af4d51aa8f41a486c4fe6e7541700e from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:17:57.923565 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 48af4d51aa8f41a486c4fe6e7541700e from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-48af4d51aa8f41a486c4fe6e7541700e
I20260809 23:17:57.933717 24334 tablet_copy_source_session.cc:205] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:57.939590 25958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48af4d51aa8f41a486c4fe6e7541700e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:57.946918 25964 ts_tablet_manager.cc:927] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:57.961380 25964 tablet_copy_client.cc:316] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:57.968536 25958 tablet_copy_client.cc:791] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:57.969200 25958 tablet_copy_client.cc:655] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:57.970530 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet b3538d14b2274bf99cd72e3872fda1fa from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:57.972395 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet b3538d14b2274bf99cd72e3872fda1fa from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-b3538d14b2274bf99cd72e3872fda1fa
I20260809 23:17:57.977334 25958 tablet_copy_client.cc:531] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:57.985303 24450 tablet_copy_source_session.cc:205] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:57.990411 25958 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:57.992960 25964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3538d14b2274bf99cd72e3872fda1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:58.031315 25964 tablet_copy_client.cc:791] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:58.034889 25964 tablet_copy_client.cc:655] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:58.040648 25964 tablet_copy_client.cc:531] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:58.050251 25958 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:58.051429 25958 tablet_bootstrap.cc:492] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:58.052171 25958 ts_tablet_manager.cc:1397] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.062s	user 0.037s	sys 0.021s
I20260809 23:17:58.054512 25964 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:58.055295 25958 raft_consensus.cc:348] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.056252 25958 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:58.056974 25958 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.060482 25958 ts_tablet_manager.cc:1428] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:17:58.064785 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-48af4d51aa8f41a486c4fe6e7541700e received from {username='slave'} at 127.0.0.1:51376
I20260809 23:17:58.065377 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-48af4d51aa8f41a486c4fe6e7541700e on tablet 48af4d51aa8f41a486c4fe6e7541700e with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:58.109506 25958 ts_tablet_manager.cc:927] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:17:58.113972 25958 tablet_copy_client.cc:316] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:17:58.160137 25964 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:58.161391 25964 tablet_bootstrap.cc:492] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:58.162257 25964 ts_tablet_manager.cc:1397] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.108s	user 0.030s	sys 0.036s
I20260809 23:17:58.165302 25964 raft_consensus.cc:348] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.166435 25964 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:58.167152 25964 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.168290 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet c831643e28e34208b6edf775607155e4 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:17:58.168838 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet c831643e28e34208b6edf775607155e4 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-c831643e28e34208b6edf775607155e4
I20260809 23:17:58.177148 24450 tablet_copy_source_session.cc:205] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:17:58.182412 25964 ts_tablet_manager.cc:1428] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.020s	user 0.000s	sys 0.009s
I20260809 23:17:58.186035 25958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c831643e28e34208b6edf775607155e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:17:58.188225 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-b3538d14b2274bf99cd72e3872fda1fa received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:58.195117 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-b3538d14b2274bf99cd72e3872fda1fa on tablet b3538d14b2274bf99cd72e3872fda1fa with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:58.193311 25371 raft_consensus.cc:1203] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:58.209728 25958 tablet_copy_client.cc:791] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:17:58.210381 25958 tablet_copy_client.cc:655] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:17:58.221331 25958 tablet_copy_client.cc:531] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:17:58.230085 25958 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:17:58.246206 25958 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:17:58.247054 25958 tablet_bootstrap.cc:492] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:17:58.247668 25958 ts_tablet_manager.cc:1397] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.018s	user 0.019s	sys 0.001s
I20260809 23:17:58.250064 25958 raft_consensus.cc:348] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.250965 25958 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:17:58.251533 25958 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:17:58.253988 25958 ts_tablet_manager.cc:1428] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:17:58.256086 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-c831643e28e34208b6edf775607155e4 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:17:58.256469 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-c831643e28e34208b6edf775607155e4 on tablet c831643e28e34208b6edf775607155e4 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:17:58.417493 25375 raft_consensus.cc:1203] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:58.478152 25375 raft_consensus.cc:1203] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:58.508898 25375 raft_consensus.cc:1203] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:17:58.573580 25973 raft_consensus.cc:1050] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:58.575642 25973 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:58.588394 25375 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.588485 24427 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.589910 25973 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:58.591111 25923 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:58.590763 24314 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.592877 25929 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:58.614184 25923 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.616283 25375 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.617105 24316 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.638270 24165 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.677007 24427 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.754328 24535 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:58.764878 25371 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:58.765244 24427 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:58.766258 25973 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:58.767367 25863 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:58.774005 25929 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.778877 24427 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.779647 25371 raft_consensus.cc:2932] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.787962 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 65d375db793a4542808e1cff72fb6617 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:58.797449 24165 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.813845 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d375db793a4542808e1cff72fb6617 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:17:58.818168 25986 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:17:58.819694 25986 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:58.820948 25986 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:58.829908 25986 ts_tablet_manager.cc:1905] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:58.851702 25986 ts_tablet_manager.cc:1918] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:58.852293 25986 log.cc:1196] T 65d375db793a4542808e1cff72fb6617 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/65d375db793a4542808e1cff72fb6617
I20260809 23:17:58.854625 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 65d375db793a4542808e1cff72fb6617 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:58.894547 25931 raft_consensus.cc:1050] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:58.898236 25931 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:58.907907 24535 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.909035 25371 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.909404 25931 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:58.910831 25776 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:58.911229 24427 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:58.913076 25776 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:58.942015 25776 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.950023 24535 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.955777 24429 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.973376 24170 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:58.979715 25371 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.021000 25938 raft_consensus.cc:1050] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:59.023680 25938 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:59.026304 24312 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:59.031443 24535 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.032840 25891 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:59.039517 24535 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.039680 25371 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.041499 25997 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:59.044294 25998 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:59.051131 25375 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.052314 25943 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:59.058580 25997 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.062590 25375 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.059890 24545 raft_consensus.cc:2932] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.071226 25985 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.074602 24312 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.074330 25371 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.081645 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 48af4d51aa8f41a486c4fe6e7541700e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:59.086890 25891 raft_consensus.cc:1050] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:17:59.089217 25891 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:59.090531 24170 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.096140 24535 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.107105 25375 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.108566 24544 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.112686 25938 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:59.115479 25938 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:59.115262 24168 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:17:59.120030 25985 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:17:59.127027 24314 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:17:59.129522 25985 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:17:59.163331 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet 48af4d51aa8f41a486c4fe6e7541700e with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34762
W20260809 23:17:59.179191 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62418.4us
I20260809 23:17:59.184353 26020 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:17:59.185578 26020 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:59.186339 26020 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:59.190812 26020 ts_tablet_manager.cc:1905] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:59.194811 25943 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.196266 24316 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.210739 25373 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.217025 26020 ts_tablet_manager.cc:1918] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:59.217476 26020 log.cc:1196] T 48af4d51aa8f41a486c4fe6e7541700e P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/48af4d51aa8f41a486c4fe6e7541700e
I20260809 23:17:59.221740 24312 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.224370 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet 48af4d51aa8f41a486c4fe6e7541700e (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:59.218585 24544 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.233134 24168 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.264741 24435 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:59.269353 25373 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.270032 24312 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.270519 25891 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:59.272226 25891 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260809 23:17:59.282384 25943 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.287271 24316 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.289165 25373 raft_consensus.cc:2932] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.304971 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b3538d14b2274bf99cd72e3872fda1fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:59.305207 24164 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.325184 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet b3538d14b2274bf99cd72e3872fda1fa with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:59.328191 26024 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:59.329339 26024 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:59.330092 26024 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:59.336278 26024 ts_tablet_manager.cc:1905] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:59.350307 26024 ts_tablet_manager.cc:1918] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:59.350625 26024 log.cc:1196] T b3538d14b2274bf99cd72e3872fda1fa P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/b3538d14b2274bf99cd72e3872fda1fa
I20260809 23:17:59.353302 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet b3538d14b2274bf99cd72e3872fda1fa (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:59.516759 24427 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:17:59.521240 25373 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.522815 24316 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:17:59.523161 26007 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:59.528044 25938 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:17:59.531291 26007 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.536216 25373 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.543018 24316 raft_consensus.cc:2932] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.550717 24164 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:17:59.548842 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c831643e28e34208b6edf775607155e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:17:59.567257 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet c831643e28e34208b6edf775607155e4 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:17:59.568315 26024 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:17:59.582742 26024 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:17:59.583542 26024 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:17:59.587865 26024 ts_tablet_manager.cc:1905] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:17:59.609833 26024 ts_tablet_manager.cc:1918] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:17:59.610273 26024 log.cc:1196] T c831643e28e34208b6edf775607155e4 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/c831643e28e34208b6edf775607155e4
I20260809 23:17:59.612454 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet c831643e28e34208b6edf775607155e4 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:17:59.986023 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:00.041005 24429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:00.041800 24429 raft_consensus.cc:595] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:00.057623 24544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:00.058231 24544 raft_consensus.cc:595] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:00.060765 24544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:00.061230 24544 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:00.063341 24535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:00.063776 24535 raft_consensus.cc:595] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:00.078230 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:00.078771 24312 raft_consensus.cc:595] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:00.081755 24535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:00.082361 24535 raft_consensus.cc:595] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:00.083397 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:18:00.084764 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:00.085336 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:00.230962 25973 raft_consensus.cc:979] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.231484 25923 raft_consensus.cc:1067] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.232884 24316 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:00.233464 24316 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.233868 24316 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.240172 24316 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.242558 24316 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:00.242611 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:00.243358 24429 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.247040 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:00.247601 24535 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.247924 24535 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:00.248569 24535 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.249782 24535 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.251070 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:00.252439 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:00.253386 25997 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.253839 25997 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:00.254653 25997 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.256731 24535 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:00.267213 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:00.271474 25891 raft_consensus.cc:979] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:00.271962 25938 raft_consensus.cc:1067] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:00.273355 24535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:00.273773 24535 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.274338 24535 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.281219 24535 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.282662 24535 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:00.283550 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:00.283478 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:00.284138 24429 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.284194 24312 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.284456 24429 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:00.285110 24429 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.286046 24429 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.289746 24312 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:00.290849 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:00.291620 25923 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.292091 25923 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:00.292344 24429 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:00.292898 25923 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.321456 24164 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:00.336177 25923 raft_consensus.cc:979] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:00.337145 26041 raft_consensus.cc:1067] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:00.339262 25375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37942
I20260809 23:18:00.340126 25375 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.340484 25375 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.346228 25375 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.348160 25375 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:00.349043 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:00.349073 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:00.349858 24429 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.349934 24535 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.350373 24535 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:00.352061 24535 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.353228 24535 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.357245 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:00.358323 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:00.358793 24535 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:00.366369 26046 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.377562 26046 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:00.378624 26046 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.393880 26026 raft_consensus.cc:979] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:00.394369 26050 raft_consensus.cc:1067] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:00.396816 25375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:00.397416 25375 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.397776 25375 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.398023 24164 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:00.407831 25375 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.410074 25375 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:00.410897 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:00.410894 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:00.411736 24535 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.411764 24316 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.412205 24316 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:00.412906 24316 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.414036 24316 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.418324 24535 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:00.419534 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:00.420316 26046 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.420516 24316 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:00.420797 26046 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:00.421797 26046 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:00.431443 24164 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:00.460083 26041 raft_consensus.cc:979] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.460539 25923 raft_consensus.cc:1067] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.462167 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:00.462666 24312 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.463033 24312 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.469295 24312 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.470782 24312 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:00.471524 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:00.471690 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:00.472227 24535 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.472083 24429 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.472518 24535 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:00.473208 24535 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.474089 24535 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.477581 24429 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:00.478765 24266 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:00.479846 24535 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:00.479734 26050 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.480654 26050 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:00.481580 26050 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.498894 24164 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:00.609644 25923 raft_consensus.cc:979] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.610134 25923 raft_consensus.cc:1067] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:00.611507 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:00.612232 24312 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:00.612581 24312 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:00.622408 24244 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260809 23:18:00.623644 24312 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.625857 24312 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:00.625965 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:00.626549 24427 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.626627 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:00.629029 24544 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:00.629731 24544 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:00.630669 24544 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.632068 24544 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:00.635142 24427 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
W20260809 23:18:00.628273 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5fcb08685ecd486c8f1d951336f39982, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.638043 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:00.638340 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5fcb08685ecd486c8f1d951336f39982, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.639557 26050 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:00.641193 26050 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:00.643152 24544 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
W20260809 23:18:00.643457 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 137c4d66939a48e59f84422216ede16f, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:00.643760 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 137c4d66939a48e59f84422216ede16f, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.644709 26050 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:00.648062 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 54afb87e83f14564b7c4ec761e92304a, TS 60d5c69728134c8bb918a900eb6a8c21 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:00.648468 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 54afb87e83f14564b7c4ec761e92304a, TS 60d5c69728134c8bb918a900eb6a8c21 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:00.652966 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 07c57d179ede43aebb4ed26fdaf60001, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:00.653460 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 07c57d179ede43aebb4ed26fdaf60001, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:00.655555 24164 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:00.673749 24429 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.675011 25923 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.685736 24312 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.687472 25923 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.706459 24429 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.708333 26050 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.714063 24544 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.716316 26050 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.816035 24545 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:00.817534 26046 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:00.832099 24312 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:00.833500 26046 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:00.897748 24545 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:00.898558 24427 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.899312 26046 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:00.900404 26050 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.919803 24535 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:00.921701 25998 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:00.924610 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:00.932343 26048 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260809 23:18:01.092671 24535 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:01.094097 26050 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.104267 24429 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:01.105878 25997 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.131165 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:01.182083 24545 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:01.182545 24545 raft_consensus.cc:595] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:01.184978 24429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:01.185770 24429 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:01.188215 24545 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:01.188663 24545 raft_consensus.cc:595] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:01.191473 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:01.192109 24314 raft_consensus.cc:595] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:01.194654 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:01.195429 24314 raft_consensus.cc:595] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:01.199107 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:01.199653 24314 raft_consensus.cc:595] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:01.200793 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:18:01.202726 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:01.203574 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:01.387768 25997 raft_consensus.cc:979] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:01.388319 25997 raft_consensus.cc:1067] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:01.389802 25371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:01.390429 25371 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.390756 25371 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.397567 25371 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.399437 25371 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:01.400521 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:01.401108 24312 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.401404 24312 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:01.401463 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:01.401973 24545 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.402209 24312 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.405655 24312 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.412837 24545 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:01.413936 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:01.414043 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:01.414734 26084 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.415222 26084 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:01.416045 26084 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.428702 24164 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:01.440186 26050 raft_consensus.cc:979] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:01.440728 26050 raft_consensus.cc:1067] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:01.442895 25371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:01.443554 25371 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.443914 25371 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.454607 26041 raft_consensus.cc:979] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:01.453701 25371 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.455179 26044 raft_consensus.cc:1067] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:01.456319 25371 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:01.457026 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:01.457603 24312 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.457706 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:01.457993 24312 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:01.457755 24314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:01.458493 24427 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.458907 24312 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.458745 24314 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.460258 24314 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.460227 24312 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.465952 24427 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:01.467993 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:01.468109 24312 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:01.468849 26084 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.469332 26084 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:01.470189 26084 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:01.469412 24314 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.472697 24314 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:01.473254 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:01.473239 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:01.474112 24545 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.474465 24545 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:01.474817 24427 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.476063 24545 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.477280 24545 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.486553 24427 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:01.487980 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:01.489066 25997 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.489573 25997 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:01.490398 25997 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.499274 24545 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:01.512300 24164 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:01.515955 24170 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:01.521377 26041 raft_consensus.cc:979] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:01.522090 26041 raft_consensus.cc:1067] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:01.524652 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:01.525322 24312 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.525982 24312 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.540820 25998 raft_consensus.cc:979] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:01.541631 25998 raft_consensus.cc:1067] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:01.541411 24312 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.543478 24312 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:01.543730 24545 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:01.544229 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:01.544458 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:01.544809 24427 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.545015 24535 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.545365 24535 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:01.546030 24535 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.547319 24535 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.544247 24545 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.551367 24545 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.552810 25938 raft_consensus.cc:979] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:01.553282 25938 raft_consensus.cc:1067] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:01.554646 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:01.554986 24535 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:01.555197 24544 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:01.555532 24544 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.556226 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:01.557037 25998 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.557502 25998 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:01.557238 24545 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.558434 25998 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.559942 24427 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:01.559883 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:01.560689 24429 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.562498 24545 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:01.565151 24544 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.560001 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:01.567152 24314 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.567467 24314 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:01.567862 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:01.568447 24316 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.568141 24314 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.568831 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:01.569381 24314 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.569998 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:01.571049 26044 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.571872 26044 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:01.573065 26044 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.575896 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:01.577813 24544 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:01.578781 24316 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:01.579988 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:01.581099 25923 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:01.582011 25923 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:01.582734 25923 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.584420 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:01.585094 24429 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:01.585439 24429 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:01.586218 24429 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.589340 24429 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:01.594146 24164 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:01.596324 24170 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:01.604043 24429 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:01.617429 24170 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:01.726001 24544 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:01.733630 24427 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:18:01.733999 24498 consensus_peers.cc:487] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef1955f952024cb2815f7065f861cf25. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:01.735016 26044 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.737406 24316 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:18:01.738579 26044 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.742983 25923 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.744576 24427 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.750221 24316 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.759387 24314 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:01.791648 24545 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:18:01.791769 24269 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f20a6f5f014e45d3a917a197db7a0998. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:01.793362 26050 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.782382 24170 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.805260 24427 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:18:01.807087 26050 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:01.817592 25998 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.819098 24427 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:01.834578 24545 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:01.836833 24244 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260809 23:18:01.839983 24168 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 23:18:01.862076 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 827f95d8d9f946f6b673db8272386c5f, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.863217 24316 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260809 23:18:01.867725 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 827f95d8d9f946f6b673db8272386c5f, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.869074 26084 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:01.876178 24429 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:01.877648 26084 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260809 23:18:01.886564 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 137c4d66939a48e59f84422216ede16f, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:01.886976 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 137c4d66939a48e59f84422216ede16f, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:01.991863 24316 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:01.993934 26084 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:02.012198 24544 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:02.013309 26046 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:02.074359 24535 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.075603 25998 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:02.088387 24429 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.090663 25997 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:02.101876 24311 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.103379 26102 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:02.107583 24535 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.110262 25997 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260809 23:18:02.124331 24429 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.134109 25997 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:02.140024 24429 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.142205 26041 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:02.153326 24311 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.159605 26103 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:02.167805 24427 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:02.179208 26103 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:02.227234 26134 ts_tablet_manager.cc:927] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:02.261853 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:02.283905 26134 tablet_copy_client.cc:316] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:02.287467 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet f20a6f5f014e45d3a917a197db7a0998 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:02.288106 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet f20a6f5f014e45d3a917a197db7a0998 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-f20a6f5f014e45d3a917a197db7a0998
I20260809 23:18:02.303915 24334 tablet_copy_source_session.cc:205] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:02.307739 26134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f20a6f5f014e45d3a917a197db7a0998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:02.327819 26134 tablet_copy_client.cc:791] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:02.328485 26134 tablet_copy_client.cc:655] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:02.329957 26137 ts_tablet_manager.cc:927] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:02.339951 26134 tablet_copy_client.cc:531] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:02.345829 26137 tablet_copy_client.cc:316] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:02.347417 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet ef1955f952024cb2815f7065f861cf25 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:02.347918 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet ef1955f952024cb2815f7065f861cf25 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-ef1955f952024cb2815f7065f861cf25
I20260809 23:18:02.349041 24535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:02.349606 24535 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:02.351214 26134 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:02.356148 24535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:02.356689 24535 raft_consensus.cc:595] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:02.360569 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:02.361120 24316 raft_consensus.cc:595] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:02.365386 24435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:02.366096 24435 raft_consensus.cc:595] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:02.365365 24557 tablet_copy_source_session.cc:205] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:02.369702 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:02.370885 24311 raft_consensus.cc:595] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:02.372368 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:18:02.374351 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:02.374972 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:02.378213 26137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1955f952024cb2815f7065f861cf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:02.395100 26137 tablet_copy_client.cc:791] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:02.395628 26137 tablet_copy_client.cc:655] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:02.402205 26137 tablet_copy_client.cc:531] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:02.403614 26134 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:02.404736 26134 tablet_bootstrap.cc:492] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:02.405454 26134 ts_tablet_manager.cc:1397] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.054s	user 0.023s	sys 0.004s
I20260809 23:18:02.412317 26134 raft_consensus.cc:348] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:02.413684 26134 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:02.414726 26134 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:02.419063 26134 ts_tablet_manager.cc:1428] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.005s
I20260809 23:18:02.424134 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-f20a6f5f014e45d3a917a197db7a0998 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:02.424808 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-f20a6f5f014e45d3a917a197db7a0998 on tablet f20a6f5f014e45d3a917a197db7a0998 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:02.426846 26137 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:02.478078 26137 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:02.479403 26137 tablet_bootstrap.cc:492] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:02.480203 26137 ts_tablet_manager.cc:1397] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.054s	user 0.021s	sys 0.021s
I20260809 23:18:02.480693 25938 raft_consensus.cc:979] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:02.481961 26109 raft_consensus.cc:1067] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:02.483791 25375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37950
I20260809 23:18:02.484462 25375 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:02.484819 25375 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.484841 26137 raft_consensus.cc:348] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:02.486354 26137 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:02.487179 26137 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:02.493163 26137 ts_tablet_manager.cc:1428] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.005s
I20260809 23:18:02.495014 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-ef1955f952024cb2815f7065f861cf25 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:02.495558 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-ef1955f952024cb2815f7065f861cf25 on tablet ef1955f952024cb2815f7065f861cf25 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:02.504221 25375 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.506382 25375 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:02.506968 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:02.507133 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:02.507531 24427 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:02.507597 24312 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.507887 24427 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:02.508555 24427 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.509593 24427 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.518251 24427 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:02.519451 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:02.520398 26046 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:02.520849 26046 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:02.522421 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:02.521705 26046 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.536983 24164 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:02.560508 26041 raft_consensus.cc:979] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:02.561118 26041 raft_consensus.cc:1067] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:02.563256 24311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:02.563959 24311 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:02.564509 24311 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:02.580044 24311 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.583056 24311 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:02.583606 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:02.584187 24427 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:02.587280 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:02.587790 24544 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:02.588023 24544 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:02.588763 24544 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.590205 24544 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:02.592790 24427 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:02.593809 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:02.595768 26050 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:02.596603 26050 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:02.598248 24544 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:02.597730 26050 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.631105 24164 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:02.716598 25375 raft_consensus.cc:1203] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:18:02.760116 26102 raft_consensus.cc:979] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:02.760874 26102 raft_consensus.cc:1067] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:02.762794 24311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:02.763938 24311 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:02.764585 24311 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.780007 26136 raft_consensus.cc:979] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:02.780794 25997 raft_consensus.cc:1067] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:02.780143 24311 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.782874 24545 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:02.783484 24545 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:02.783546 24311 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:02.783628 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:02.783910 24545 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.784361 24435 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.784408 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:02.784978 24535 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:02.785166 24535 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:02.785710 24535 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.786401 24535 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.790581 24435 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:02.790612 24545 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.791900 24266 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:02.792711 26050 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:02.793444 26050 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:02.794792 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:02.793948 24545 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:02.804432 24435 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.794699 26050 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.810016 25375 raft_consensus.cc:1203] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:18:02.817179 24535 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:02.818260 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:02.821072 24435 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:02.823021 24311 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:02.823518 24311 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:02.824753 24311 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.826594 24311 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.834957 24496 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:02.836673 26103 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:02.837201 26103 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:02.838032 26103 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:02.855042 26136 raft_consensus.cc:979] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:02.855531 26136 raft_consensus.cc:1067] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:02.857048 25371 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:02.857571 25371 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:02.857908 25371 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.865767 25371 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.870802 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:02.872051 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:02.873059 24314 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:02.873339 24314 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:02.874035 24314 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.875113 24314 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.878424 24535 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:02.883204 25371 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:02.884044 24314 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:02.885990 24535 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:02.886794 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:02.890213 26046 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:02.892051 24311 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:02.904062 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:02.914217 24314 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:02.921603 26048 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:02.928793 24170 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:02.948128 26046 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:02.949458 26046 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:02.978322 24435 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:02.984160 26150 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:02.988843 24164 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:03.033423 24544 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:03.034868 26094 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:03.048367 24435 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:03.049654 26148 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:03.170316 26041 raft_consensus.cc:1050] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:03.175271 26041 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:03.198702 25371 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:03.200196 24316 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:03.202060 26102 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:03.205269 26041 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:03.242607 26103 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:03.243517 25371 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:03.252727 24428 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:03.292678 24430 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:03.449373 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:03.847383 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.015s	sys 0.019s
W20260809 23:18:03.847823 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.015s	sys 0.019s
W20260809 23:18:03.957775 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.020s	sys 0.030s
W20260809 23:18:03.958226 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.022s	sys 0.030s
W20260809 23:18:03.958515 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 713679us
I20260809 23:18:04.013953 26176 raft_consensus.cc:1050] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:04.019079 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:04.020434 24314 raft_consensus.cc:595] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:04.024757 26176 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.045773 26169 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260809 23:18:04.056571 24430 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:04.055516 26164 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260809 23:18:04.078656 26190 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260809 23:18:04.096521 25373 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:04.097920 26176 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:04.107563 24430 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:04.114735 24428 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260809 23:18:04.117300 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 836648us
I20260809 23:18:04.117262 26094 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:04.273236 25938 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.277758 24314 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:04.280668 26166 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:04.281316 26166 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.285687 25938 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.294292 25938 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.299316 26173 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.300091 26173 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.304134 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:04.305835 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.306638 24435 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:04.308801 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:04.288537 26176 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.288558 26094 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.302848 26173 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:04.359079 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 244724us
I20260809 23:18:04.315107 26169 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:04.311780 26166 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.339366 25997 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:04.374845 24316 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:04.339422 26176 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.339211 26094 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.367033 26169 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:04.377168 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08001676e0 after lost signal to thread 22536
I20260809 23:18:04.376049 26046 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260809 23:18:04.388139 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800167dc0 after lost signal to thread 24220
I20260809 23:18:04.388739 26046 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:04.389816 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08003375e0 after lost signal to thread 24355
I20260809 23:18:04.392876 26176 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.393565 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.389345 26046 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.399506 26218 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.404522 26203 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.405936 26046 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.414245 26190 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.416390 26094 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.385138 26169 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:04.373557 25997 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.427233 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.428037 26218 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.429538 26203 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.470110 26203 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.436169 26192 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:04.406828 26103 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.161s	user 0.016s	sys 0.000s
I20260809 23:18:04.444602 26046 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.449363 26217 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.467382 26176 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:04.429774 26190 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.487323 26174 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.482785 26046 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:04.472936 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08002024a0 after lost signal to thread 24469
I20260809 23:18:04.487866 26174 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.491753 26046 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.511839 24168 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 232) took 1401 ms. Trace:
W20260809 23:18:04.501870 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08003005c0 after lost signal to thread 24470
I20260809 23:18:04.482501 26176 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.529832 26176 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.470991 26192 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.482895 26050 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.496680 26217 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.483178 26218 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.530536 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.534576 26192 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.536262 25997 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.537494 26050 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.512353 24168 rpcz_store.cc:276] 0809 23:18:03.092628 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:03.220409 (+127781us) service_pool.cc:227] Handling call
0809 23:18:04.493714 (+1273305us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
W20260809 23:18:04.543656 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 247359us
I20260809 23:18:04.599287 26173 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.600005 26173 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.602243 26173 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.602444 26050 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.611248 26176 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.611790 26174 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:04.603956 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080019c860 after lost signal to thread 24579
I20260809 23:18:04.618907 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.618788 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.612564 25997 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:04.624320 25938 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.624769 25938 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.617904 26177 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.631043 26150 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.631850 26150 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.634536 26046 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.635805 26046 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.638582 26046 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.645655 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.647804 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.619292 26176 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:04.669685 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 293562us
I20260809 23:18:04.669901 24425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:04.670522 24425 raft_consensus.cc:595] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:04.671442 25938 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.664743 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.626572 25997 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.664738 26150 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:04.676188 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080031d8a0 after lost signal to thread 25410
I20260809 23:18:04.702906 26190 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.675859 26148 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:04.705188 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.705928 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.706559 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:04.708379 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:04.677131 26218 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.704551 24170 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:04.745537 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:04.746757 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59490.1us
I20260809 23:18:04.749404 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.703210 26169 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.752035 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.754814 24313 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:04.754639 26224 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:04.756052 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70364.9us
I20260809 23:18:04.686244 26217 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.761840 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:04.763037 26225 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.765830 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.670948 26177 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.763814 26225 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.790812 26225 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:04.793430 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109235us
I20260809 23:18:04.759627 26169 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.797497 26169 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.798454 26169 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.799682 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:04.801462 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:04.805351 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.803668 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.813776 26234 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.784802 26174 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.757297 26218 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.827293 26226 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.808425 26166 raft_consensus.cc:979] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:04.793991 26094 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.833992 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.829912 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:04.749608 26148 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.797592 26196 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.760798 26192 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260809 23:18:04.815531 26177 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.848701 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.870561 26222 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.814548 26234 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.810567 26169 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.832545 26190 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:04.865216 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:04.860440 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.867547 26218 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.786341 26224 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.869244 25997 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.898794 25997 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:04.863824 26094 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.890673 24321 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 2: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:04.835826 26041 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:04.859671 26196 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.857000 26174 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.895798 26227 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:04.879014 26173 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:04.865826 26166 raft_consensus.cc:1067] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:04.856113 26226 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.878548 26176 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.907644 26235 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:04.909132 26094 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:04.909971 24320 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 1.
I20260809 23:18:04.889565 26169 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:04.910861 26041 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:04.917943 26148 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:04.873489 26222 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.922761 26150 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:04.923218 26196 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.891880 26190 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:04.852934 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:04.945318 26235 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:04.954481 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 209516us
I20260809 23:18:04.957558 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:04.916523 26174 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:04.965255 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 180263us
I20260809 23:18:04.952513 26150 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.013499 26041 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.014959 26234 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:04.980203 26224 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.018144 26190 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.020537 26150 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:05.024251 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.025425 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.027040 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.010120 26196 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260809 23:18:05.008672 26226 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:05.028491 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:04.999747 24320 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:05.030175 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.042948 24422 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:05.052778 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.053653 26196 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:05.039443 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.055620 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.057840 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.059414 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.054453 26192 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:05.060463 26234 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.068581 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.078662 26212 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.080476 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.068352 26192 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:05.089731 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.089751 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.091075 26192 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.087878 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.054472 26196 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.073498 26164 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.082546 26234 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.094306 26164 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.100420 26222 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.102739 26235 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.103286 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.104558 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.080056 26212 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:05.125226 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 312045us
I20260809 23:18:05.107018 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.134702 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:05.139079 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 337733us
I20260809 23:18:05.139556 24545 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:05.140116 24545 raft_consensus.cc:595] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:05.119087 26164 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.143988 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.145829 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.147817 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.147781 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.148627 25375 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:05.154089 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.154325 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.154548 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.156698 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.157773 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.158427 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.133046 26234 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.144172 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.165233 25938 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.166648 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.138095 26196 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.167605 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.171977 25373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37950
I20260809 23:18:05.172472 25373 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:05.172777 25373 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.142561 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.181030 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.176741 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.174551 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.283777 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:05.284651 24316 raft_consensus.cc:595] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.286962 24545 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:05.288612 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.290170 26103 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.290653 26103 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.291893 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.292534 26103 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.185245 26244 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.218655 26185 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:05.216187 26217 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.297416 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:05.219508 26212 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.216266 26196 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:05.294541 25373 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.302352 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.303447 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.305716 25373 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.307086 26185 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.308116 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.308779 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.309140 26185 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:05.309923 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.280673 26224 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:05.313560 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.311132 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:05.329919 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.024s	sys 0.036s
I20260809 23:18:05.331341 26224 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.306605 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.310978 26196 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.338220 26224 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:05.341992 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.025s	sys 0.036s
W20260809 23:18:05.342798 26239 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260809 23:18:05.342916 26185 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.345456 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.342815 24543 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:05.340005 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.351927 24380 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:05.356072 25373 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:05.362596 25370 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:05.369127 24427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:05.369689 24427 raft_consensus.cc:595] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:05.373766 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.374694 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.374864 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.379343 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:05.379902 24321 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.381623 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.382608 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.369975 26109 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.387900 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.374555 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:05.371147 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.302290 26244 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.393889 24546 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:05.418853 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:18:05.420884 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:05.421732 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:05.432818 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.420123 26166 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260809 23:18:05.439610 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.423898 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.467346 24321 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:05.442456 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.471051 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:05.472677 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.473695 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.474562 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.479984 26254 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.486162 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.467089 26196 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:05.488978 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69842.8us
I20260809 23:18:05.475841 26222 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.476038 26041 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.508895 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.515893 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.516536 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.520376 24498 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.522480 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.524796 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.522719 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:05.526417 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.529220 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:05.529450 26178 raft_consensus.cc:979] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:05.530408 26178 raft_consensus.cc:1067] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:05.530633 26217 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.530337 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.531613 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.531904 26261 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.527315 26254 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.527557 26050 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:05.532752 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.533241 26050 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:05.533648 26050 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:05.526381 26211 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.520804 26041 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.540932 24321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:05.523483 26224 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.541867 24321 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:05.542951 24321 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.543579 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:05.544317 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.546157 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.547883 26041 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.548108 26178 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.547955 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.555419 26190 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.559448 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.561168 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.523700 26222 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.563273 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.565503 26185 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.568004 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.572741 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.572459 26263 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:05.574110 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.574384 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.575956 26227 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.578819 24309 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:05.582118 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.582473 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:05.578894 26178 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:05.588027 26178 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.588865 26223 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.587193 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.580186 26192 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.605775 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:05.607146 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61985.4us
I20260809 23:18:05.606225 24321 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.604992 26185 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260809 23:18:05.613408 24321 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:05.613351 26169 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.618564 24270 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.590816 26222 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:05.598891 26225 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.623948 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:05.632081 26273 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.620661 26263 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.663188 26267 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.650071 24431 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:05.683422 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 175753us
I20260809 23:18:05.684634 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:05.686007 26267 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.686141 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:05.687703 26225 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:05.688301 26225 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:05.689474 26225 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.706869 26178 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.712378 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.719071 26238 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.682291 26212 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:05.726812 26212 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.696094 24422 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:05.740694 24422 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:18:05.751524 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 127702us
W20260809 23:18:05.759865 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 117499us
I20260809 23:18:05.765630 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.772547 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:05.776202 26050 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.780831 26050 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:05.781023 26244 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:05.758569 26212 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:05.786361 26136 raft_consensus.cc:979] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:05.787199 26136 raft_consensus.cc:1067] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:05.760746 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:05.858700 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97938.5us
I20260809 23:18:05.742162 24422 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:05.870110 24544 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:05.873206 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.879801 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:05.881644 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:05.896852 26075 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:05.907433 24422 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.937695 26238 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:05.940776 25370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:05.942021 25370 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:05.942569 25370 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:05.942729 26050 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:05.943758 26050 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:05.950053 26050 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:05.995927 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57700.2us
I20260809 23:18:06.007135 26238 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.007866 24422 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
W20260809 23:18:06.011391 26248 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260809 23:18:06.010111 25370 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.031615 26266 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:06.038868 25370 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.040351 26238 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.042879 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.055924 24309 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:06.032492 26266 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.066165 26185 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:06.067124 26185 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.069788 26185 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:06.074812 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 371825us
I20260809 23:18:06.076969 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.079340 26261 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:06.086977 26220 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:06.087767 26220 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:06.093953 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:06.099586 24309 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:06.100410 24309 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:06.107653 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.102447 24309 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.112694 24309 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.116638 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.120158 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:06.121239 24543 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.120216 26255 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 23:18:06.130275 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87725.9us
I20260809 23:18:06.140740 26220 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.143013 26220 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.159466 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.161375 26266 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:06.173605 26238 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.174443 24543 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:06.176544 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:06.178395 26288 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:06.179054 26288 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:06.180059 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:06.183015 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.185163 26261 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:06.189435 26265 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.185884 26261 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.197422 26261 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.180331 26288 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.201086 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.201620 26266 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 23:18:06.211546 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 151686us
I20260809 23:18:06.225878 24309 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
W20260809 23:18:06.227269 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67999.1us
I20260809 23:18:06.238160 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.240600 26217 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.240794 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.242820 26103 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.245774 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.251524 26103 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.251752 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.252388 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.253398 26103 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.254463 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.255023 24170 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 143) took 1584 ms. Trace:
I20260809 23:18:06.254369 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:06.257411 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.084s	sys 0.093s
W20260809 23:18:06.257871 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.085s	sys 0.093s
W20260809 23:18:06.265152 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 646625us
I20260809 23:18:06.266683 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.269449 26285 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.255522 24170 rpcz_store.cc:276] 0809 23:18:04.670561 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:04.670708 (+   147us) service_pool.cc:227] Handling call
0809 23:18:06.254751 (+1584043us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:18:05.348959 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:18:05.464718 (+115759us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:18:05.464753 (+    35us) write_op.cc:435] Acquired schema lock
0809 23:18:05.464771 (+    18us) tablet.cc:660] Decoding operations
0809 23:18:05.465102 (+   331us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:18:05.465158 (+    56us) write_op.cc:641] Partition lock acquired for write op
0809 23:18:05.465190 (+    32us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:18:05.465399 (+   209us) tablet.cc:699] Row locks acquired
0809 23:18:05.465422 (+    23us) write_op.cc:260] PREPARE: finished
0809 23:18:05.465583 (+   161us) write_op.cc:270] Start()
0809 23:18:05.469722 (+  4139us) write_op.cc:276] Timestamp: P: 1786317485465537 usec, L: 0
0809 23:18:05.475640 (+  5918us) op_driver.cc:331] REPLICATION: starting
0809 23:18:05.528641 (+ 53001us) log.cc:842] Serialized 595 byte log entry
0809 23:18:05.818278 (+289637us) op_driver.cc:444] REPLICATION: finished
0809 23:18:06.217179 (+398901us) write_op.cc:301] APPLY: starting
0809 23:18:06.217495 (+   316us) tablet.cc:1365] starting BulkCheckPresence
0809 23:18:06.217715 (+   220us) tablet.cc:1368] finished BulkCheckPresence
0809 23:18:06.217916 (+   201us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:18:06.220138 (+  2222us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:18:06.220481 (+   343us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 23:18:06.220564 (+    83us) write_op.cc:312] APPLY: finished
0809 23:18:06.223115 (+  2551us) log.cc:842] Serialized 23 byte log entry
0809 23:18:06.233560 (+ 10445us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:06.233692 (+   132us) write_op.cc:454] Released schema lock
0809 23:18:06.233745 (+    53us) write_op.cc:341] FINISH: Updating metrics
0809 23:18:06.234132 (+   387us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:06.234163 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":398361,"apply.run_cpu_time_us":3640,"apply.run_wall_time_us":17303,"num_ops":1,"prepare.queue_time_us":576467,"prepare.run_cpu_time_us":18575,"prepare.run_wall_time_us":274923,"raft.queue_time_us":126397,"raft.run_cpu_time_us":162201,"raft.run_wall_time_us":426135,"replication_time_us":340763,"thread_start_us":992341,"threads_started":4,"wal-append.queue_time_us":100616}]]}
I20260809 23:18:06.258788 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:06.280629 24168 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 233) took 1265 ms. Trace:
I20260809 23:18:06.283962 24168 rpcz_store.cc:276] 0809 23:18:05.015368 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:05.015652 (+   284us) service_pool.cc:227] Handling call
0809 23:18:06.280419 (+1264767us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2470,"reported_tablets":5}
I20260809 23:18:06.288081 24380 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:06.279606 26261 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.291793 26220 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.293215 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:06.283138 26285 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260809 23:18:06.302245 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.310623 25997 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:06.305476 26238 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:06.316161 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:06.317313 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:06.319761 24313 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:06.311288 25997 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.310760 26245 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.324492 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:06.325683 26238 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.325791 26267 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:06.336846 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.338860 26238 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.339545 25997 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.319824 24427 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.350648 24316 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:06.351316 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62402us
I20260809 23:18:06.350898 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:06.365856 24321 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:06.364882 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.367074 26285 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.368811 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.379046 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.380056 26238 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.381268 25997 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.386080 24535 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.390863 26285 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:06.391290 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:06.391390 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.392622 26103 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:06.392539 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.394021 26285 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.398207 26217 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:06.399260 26217 raft_consensus.cc:979] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:06.399734 26217 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:06.400066 26217 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:06.400113 26244 raft_consensus.cc:1067] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:06.398488 24164 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.403398 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:06.394598 24270 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.403903 26217 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.411580 24164 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:06.412658 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:06.425925 25997 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.427335 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.413551 24431 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:06.428263 25997 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.415273 26285 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.419955 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:06.429282 26285 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:06.429756 26285 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:06.430804 26285 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.412827 26217 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:06.441198 26227 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:06.443902 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:06.444491 24544 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.449379 25372 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:06.426134 24164 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:06.451241 26223 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260809 23:18:06.452937 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:06.443887 24316 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:06.455212 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:06.462183 24316 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:06.462800 24316 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.470412 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.475521 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.478318 24427 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:06.481591 26102 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:06.477996 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.505954 24316 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.510185 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:06.510401 24316 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.510783 24423 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.526583 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.515857 24543 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.541571 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:06.555055 24534 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:06.559949 24534 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:06.547430 26220 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:06.561107 24534 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.577080 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.577741 24534 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:06.578022 26238 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.579298 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.581278 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.582798 26238 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.576293 26220 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.591853 25370 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.580247 26235 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:06.599732 26220 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.600786 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.602880 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.589864 26223 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.634684 24423 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:06.637027 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:06.650095 24534 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
W20260809 23:18:06.650473 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46153us
W20260809 23:18:06.637805 26298 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.003s
I20260809 23:18:06.637908 24535 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:06.596411 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:06.639292 26297 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260809 23:18:06.648342 26301 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.059s	user 0.002s	sys 0.004s
W20260809 23:18:06.656841 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54115us
I20260809 23:18:06.658746 24266 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:06.665228 26238 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:06.665719 26238 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:06.666622 26238 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.665746 24312 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:06.663480 24423 raft_consensus.cc:2932] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.651888 26266 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.673130 26263 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:06.655311 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66102.9us
I20260809 23:18:06.676430 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.687114 26286 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.688390 24309 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:06.690039 26288 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:06.693819 26261 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:06.694482 26261 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.701197 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:06.709007 26286 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:06.711617 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:06.729312 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:06.713755 26261 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:06.715493 26286 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:06.767949 26286 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:06.773231 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ef1955f952024cb2815f7065f861cf25 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:06.782128 24431 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:06.765357 24170 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 1 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:06.796718 26245 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:06.801939 25370 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:06.805883 26307 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:06.826169 25997 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.830078 24423 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.841653 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1955f952024cb2815f7065f861cf25 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40854
I20260809 23:18:06.883147 24428 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:06.885558 26245 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260809 23:18:06.895957 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:06.900066 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f20a6f5f014e45d3a917a197db7a0998 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:06.900671 25370 raft_consensus.cc:2932] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.907912 26312 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:06.936518 24168 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:06.944643 26315 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:18:06.945863 26315 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:06.946731 26315 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:18:06.973218 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69752.9us
I20260809 23:18:06.976380 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet f20a6f5f014e45d3a917a197db7a0998 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:18:06.998800 24539 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:07.000614 26286 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:07.005770 24170 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:18:07.010360 26315 ts_tablet_manager.cc:1905] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:07.039083 26323 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:18:07.041990 26323 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:07.043392 26323 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:07.047577 24168 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:07.054821 24168 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:07.063952 26323 ts_tablet_manager.cc:1905] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:07.112278 26323 ts_tablet_manager.cc:1918] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:18:07.112814 26323 log.cc:1196] T f20a6f5f014e45d3a917a197db7a0998 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/f20a6f5f014e45d3a917a197db7a0998
I20260809 23:18:07.115630 26315 ts_tablet_manager.cc:1918] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:18:07.115949 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet f20a6f5f014e45d3a917a197db7a0998 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:07.116230 26315 log.cc:1196] T ef1955f952024cb2815f7065f861cf25 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/ef1955f952024cb2815f7065f861cf25
I20260809 23:18:07.118808 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet ef1955f952024cb2815f7065f861cf25 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:07.164880 24423 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:07.166275 26307 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:07.169538 24541 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:07.171139 26286 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:07.301108 24535 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:07.305172 24423 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:07.305850 26307 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:07.436123 24423 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:07.497712 26319 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:07.498435 26319 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:07.500630 26319 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:07.609894 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:07.612960 26332 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:07.668442 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:07.669133 24316 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:07.682305 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:07.687813 26103 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:07.688350 26103 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:07.690320 26103 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:07.687878 26220 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:07.691733 26220 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:07.692346 26220 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:07.703889 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58455.7us
I20260809 23:18:07.802821 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:07.803444 24539 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:07.807066 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:07.862177 26220 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:07.874572 24427 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:07.875039 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:07.875669 24316 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:07.879776 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:07.880659 24321 raft_consensus.cc:595] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:07.884857 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:07.885537 24321 raft_consensus.cc:595] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:07.887413 26220 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:07.888589 24316 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:07.889675 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:07.889791 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:07.892059 26220 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:07.890648 24321 raft_consensus.cc:595] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:07.892832 26220 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:07.894784 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:07.894057 26220 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:07.898711 24539 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:07.896063 24427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d273006c869f4749852291bcee467158"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:07.900774 24427 raft_consensus.cc:595] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:07.904402 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:07.905217 24321 raft_consensus.cc:595] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:07.907219 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:18:07.927943 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:07.929174 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:07.939047 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 315543us
I20260809 23:18:07.952553 24539 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:07.965338 26245 raft_consensus.cc:979] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:07.965826 26245 raft_consensus.cc:1067] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:07.967782 24539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:07.968334 24539 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:07.968698 24539 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:07.979930 24539 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:07.982118 24539 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:07.983074 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:07.983707 25372 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:07.990798 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:07.991865 24321 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:08.000136 25372 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.001385 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:08.002216 26347 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:08.002741 26347 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:08.005087 26347 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.009042 26238 raft_consensus.cc:979] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.012166 26286 raft_consensus.cc:1067] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.014626 24541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:08.015266 24541 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:08.015699 24541 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.027509 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:08.028247 24316 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:08.028538 24316 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:08.029379 24316 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.031061 24316 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.026396 24541 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.037314 24541 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:08.041010 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:08.041548 24435 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.048733 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:08.050674 26345 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:08.032009 24164 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:08.058915 24316 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.084082 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:08.083580 24170 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:08.084759 24321 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:08.085225 24321 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:08.085952 24321 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.087378 24321 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.092774 24435 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.094543 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:08.095410 26343 raft_consensus.cc:979] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:08.095912 26343 raft_consensus.cc:1067] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:08.096256 26223 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:08.097324 26223 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:08.098225 25372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:08.099146 25372 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:08.098593 26223 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.099614 25372 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.118006 25372 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.125252 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:08.125936 24312 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:08.126284 24312 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:08.127074 24312 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.128856 24312 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.135358 24321 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.139463 24312 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:08.147188 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:08.148775 24428 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.148211 25372 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:08.165239 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:08.166213 26338 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:08.166711 26338 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:08.167764 26338 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.207192 24428 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:08.228643 24168 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:08.279830 26261 raft_consensus.cc:979] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.280596 26220 raft_consensus.cc:1067] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.282797 24539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d273006c869f4749852291bcee467158"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:08.282992 26238 raft_consensus.cc:979] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.283380 24539 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:08.283471 26238 raft_consensus.cc:1067] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:08.283761 24539 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.285712 24535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:08.286238 24535 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:08.286556 24535 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:08.301617 24539 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.304390 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:08.305312 24428 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:08.305819 24428 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:08.307325 24428 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.316416 24428 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.326100 24535 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.328778 24535 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:08.328967 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:08.329526 24316 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:08.329591 24539 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:08.329913 24316 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:08.330199 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:08.331044 24422 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:08.332839 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:08.333401 25372 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:08.339720 24316 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.343770 24422 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:08.345304 24496 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:08.346246 26347 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:08.347075 26347 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:08.348448 26347 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:08.342732 24316 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:08.361940 25372 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.363013 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:08.364012 26223 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:08.364885 26223 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:08.366744 26223 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:08.375780 24316 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:08.376380 24428 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:08.385078 24164 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:08.472972 24539 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:08.474701 26316 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:08.491318 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:08.496279 24164 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:08.518972 25370 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:08.529332 26345 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:08.530174 24316 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:08.532440 26261 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:08.536217 24312 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:08.540534 26223 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:08.549240 24321 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:08.551445 26347 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:08.565160 24313 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:08.574728 26227 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:08.587477 24428 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:08.622452 26340 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:08.644071 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08002b0120 after lost signal to thread 22536
W20260809 23:18:08.712984 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800107620 after lost signal to thread 24220
W20260809 23:18:08.765049 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 84809.7us
W20260809 23:18:08.791124 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800345100 after lost signal to thread 24355
W20260809 23:18:08.815930 26370 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
W20260809 23:18:08.818681 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800353240 after lost signal to thread 24469
W20260809 23:18:08.819072 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84866us
W20260809 23:18:08.820504 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08002828c0 after lost signal to thread 24579
W20260809 23:18:08.820917 26376 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260809 23:18:08.829640 25405 debug-util.cc:398] Leaking SignalData structure 0x7b080030a380 after lost signal to thread 25410
I20260809 23:18:08.834551 24312 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:18:08.877007 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 214135us
I20260809 23:18:08.881367 24435 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:18:08.902573 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101745us
W20260809 23:18:08.914464 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 93145.1us
W20260809 23:18:08.917138 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 109638us
I20260809 23:18:08.973023 24539 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:08.975575 25370 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:08.995247 26383 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:09.020429 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 43712.6us
I20260809 23:18:09.020819 26238 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:09.064474 24435 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:09.081058 24312 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:09.087445 26397 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:09.091490 26411 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:09.100317 24435 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:09.131001 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:09.136112 24269 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cadfc1bde6cb4d6997f3fb215dc184ec. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:09.140735 26421 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:09.155606 24422 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:09.166651 26410 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:09.186061 26380 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:09.238550 24539 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:09.223652 26404 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:09.291085 24541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:09.307489 24541 raft_consensus.cc:595] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:09.324357 24428 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:09.337913 24427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:09.338603 24427 raft_consensus.cc:595] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:09.347249 24427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:09.347805 24427 raft_consensus.cc:595] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:09.358306 26429 raft_consensus.cc:979] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:09.358831 26429 raft_consensus.cc:1067] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:09.358331 24168 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:09.367223 24541 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:09.368108 24543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:09.368757 24543 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:09.370289 24543 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:09.370803 24244 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20260809 23:18:09.368073 24541 raft_consensus.cc:595] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:09.385479 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:18:09.387163 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:09.387715 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:09.391762 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e, TS 60d5c69728134c8bb918a900eb6a8c21 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:09.392096 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e, TS 60d5c69728134c8bb918a900eb6a8c21 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.396431 24543 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.408236 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:09.408809 24429 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:09.409116 24429 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:09.409789 24429 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.410833 24429 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.422259 24429 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:09.424376 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:09.424937 25373 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.423231 24543 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:09.424818 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:09.427597 26410 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:09.428231 26410 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:09.429340 26410 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.455262 26448 ts_tablet_manager.cc:927] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:09.475001 26448 tablet_copy_client.cc:316] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:09.486150 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet cadfc1bde6cb4d6997f3fb215dc184ec from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:09.486909 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet cadfc1bde6cb4d6997f3fb215dc184ec from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-cadfc1bde6cb4d6997f3fb215dc184ec
I20260809 23:18:09.520478 25373 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:09.542599 24334 tablet_copy_source_session.cc:205] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
W20260809 23:18:09.546396 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 827f95d8d9f946f6b673db8272386c5f, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:09.546756 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 827f95d8d9f946f6b673db8272386c5f, TS ab5aefaeb3d14775b30a74a2a6d3357f -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.550467 26429 raft_consensus.cc:979] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:09.550958 26429 raft_consensus.cc:1067] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:09.552229 26422 raft_consensus.cc:979] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:09.552675 26410 raft_consensus.cc:1067] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:09.554450 24311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:09.554991 24311 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:09.555341 24541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:09.555915 24541 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:09.556270 24541 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:09.562551 24244 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 974355bb93274291bb6be768b656cc67, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:09.562938 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 974355bb93274291bb6be768b656cc67, TS 808fed3ed3f04882ad757a3db9abfdd9 -> TS b9b54436d0e94a5595cdded73fe33da9 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.555359 24311 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.616472 24165 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:09.631975 24311 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.634141 24311 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:09.635004 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:09.635856 24428 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.640789 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:09.641608 24543 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:09.642019 24543 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:09.642979 24543 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.645401 26448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cadfc1bde6cb4d6997f3fb215dc184ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:09.649694 24428 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:09.650995 24266 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:09.652690 26397 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:09.653674 26397 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:09.654696 26397 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.655462 24543 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.629700 24541 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.677224 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:09.677865 24428 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:09.678161 24428 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:09.680341 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:09.680934 25372 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.683032 24164 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:09.684949 24541 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:09.689491 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:09.691893 24428 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.695192 24428 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.692412 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:09.698869 26411 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:09.702060 24543 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:09.699436 26411 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:09.705780 26411 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:09.710508 24428 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
W20260809 23:18:09.714205 26452 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260809 23:18:09.726183 26448 tablet_copy_client.cc:791] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:09.726773 26448 tablet_copy_client.cc:655] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:09.745632 24164 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:09.751718 26448 tablet_copy_client.cc:531] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:09.764770 26448 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:09.813195 26448 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:09.816545 26448 tablet_bootstrap.cc:492] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:09.817790 26448 ts_tablet_manager.cc:1397] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.053s	user 0.030s	sys 0.017s
I20260809 23:18:09.826610 26448 raft_consensus.cc:348] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:09.827613 25372 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:09.830560 26223 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:09.827666 26448 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:09.832362 26448 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:09.836251 26448 ts_tablet_manager.cc:1428] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.003s
I20260809 23:18:09.838833 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-cadfc1bde6cb4d6997f3fb215dc184ec received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:09.839473 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-cadfc1bde6cb4d6997f3fb215dc184ec on tablet cadfc1bde6cb4d6997f3fb215dc184ec with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:09.884040 24435 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:09.885771 26345 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:09.889703 26411 raft_consensus.cc:979] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:09.890242 26411 raft_consensus.cc:1067] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:09.891937 24309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:09.892642 24309 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:09.893100 24309 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.929608 24309 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.934180 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:09.934831 24543 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:09.935204 24543 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:09.938024 24309 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:09.938489 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:09.939653 24428 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.942317 24543 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.945317 24543 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:09.961354 24428 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:09.976804 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:09.978434 26343 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:09.978889 26343 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:09.979842 26343 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:09.995949 24543 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:10.034260 25372 raft_consensus.cc:1203] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260809 23:18:10.050652 24429 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:10.052737 26386 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:10.087163 24164 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:10.201025 25372 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:10.202615 26410 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:10.217080 24535 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:10.219128 26343 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:10.220994 24429 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:10.222749 26411 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:10.394358 24543 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:10.395941 26386 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:10.447330 24428 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:10.449678 26386 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:10.525466 26343 raft_consensus.cc:1050] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:10.528034 26343 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:10.543604 24543 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:10.543643 25373 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:10.545388 26404 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:10.550276 24429 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:10.552368 26386 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:10.569041 26404 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:10.602613 26397 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.604075 24535 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.641556 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:10.642802 24165 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:10.653745 25373 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.664113 24428 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.746138 24309 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:10.754657 25373 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:10.757507 26397 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:10.766180 24535 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:10.770431 26343 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:10.773118 26343 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.777513 25372 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.797035 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet cadfc1bde6cb4d6997f3fb215dc184ec with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:10.800005 24543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:10.800660 24543 raft_consensus.cc:595] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:10.803427 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:10.804226 24309 raft_consensus.cc:595] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:10.840881 24170 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:10.847625 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:10.848289 24309 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:10.852728 24543 raft_consensus.cc:2932] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:10.855645 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet cadfc1bde6cb4d6997f3fb215dc184ec with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34762
I20260809 23:18:10.857787 26488 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:18:10.863189 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55440us
I20260809 23:18:10.877322 24539 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:10.877987 24539 raft_consensus.cc:595] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:10.887596 24435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:10.888216 24435 raft_consensus.cc:595] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:10.889582 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:18:10.891237 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:10.892009 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:10.971372 26488 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:10.972332 26488 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:10.976089 26488 ts_tablet_manager.cc:1905] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:11.001348 26343 raft_consensus.cc:979] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:11.002010 26343 raft_consensus.cc:1067] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:11.012478 24435 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:11.013144 24435 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:11.013515 24435 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:11.024647 24435 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.027254 26411 raft_consensus.cc:979] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:11.027756 26411 raft_consensus.cc:1067] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:11.029150 26471 raft_consensus.cc:979] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:11.029378 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34778
I20260809 23:18:11.029603 26471 raft_consensus.cc:1067] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:11.029759 26488 ts_tablet_manager.cc:1918] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260809 23:18:11.029978 24428 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:11.030153 26488 log.cc:1196] T cadfc1bde6cb4d6997f3fb215dc184ec P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/cadfc1bde6cb4d6997f3fb215dc184ec
I20260809 23:18:11.030428 24428 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.031528 24543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:11.032066 24543 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:11.032416 24543 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.035563 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:11.035969 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet cadfc1bde6cb4d6997f3fb215dc184ec (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:11.036152 24312 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:11.036523 24312 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:11.037200 24312 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.038280 24312 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:11.038923 24428 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.042101 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:11.042601 24535 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:11.042889 24535 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:11.043710 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:11.043601 24535 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.044178 25373 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.044662 24535 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.047096 24428 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:11.051132 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:11.051669 24539 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:11.053661 24535 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:11.054766 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:11.055423 24435 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:11.055652 26382 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:11.056170 26382 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:11.057029 26382 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.059278 24539 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:11.060528 25373 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:11.072268 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:11.073365 26466 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:11.073832 26466 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:11.074688 26466 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.070801 24543 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.097668 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:11.099030 24309 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.100356 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:11.101011 24435 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:11.101293 24435 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:11.102020 24435 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.103401 24435 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.113471 24435 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:11.117789 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:11.118736 26223 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:11.119253 26223 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:11.084527 24312 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:11.097790 24543 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:11.120281 26223 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.123054 26386 raft_consensus.cc:979] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:11.123522 26386 raft_consensus.cc:1067] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:11.125415 25373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:11.125962 25373 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:11.126308 25373 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.134052 25373 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.141788 25373 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:11.143966 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:11.146019 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:11.146333 24312 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:11.146698 24312 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:11.146714 24535 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.148277 24309 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:11.149720 24312 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.166356 24170 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:11.174979 24312 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.182000 24535 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:11.183771 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:11.185101 26150 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:11.185755 26150 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:11.186728 26150 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:11.200526 24312 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:11.219808 24165 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:11.226558 24170 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:11.262629 24170 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:11.333645 26223 raft_consensus.cc:979] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:11.334151 26223 raft_consensus.cc:1067] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:11.335609 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:11.336342 24312 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:11.336755 24312 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.345223 24312 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.348364 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:11.348963 24535 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:11.349311 24535 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:11.350190 24535 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.351361 24535 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.353612 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:11.354185 24435 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:11.356726 24312 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:11.362053 24535 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:11.362825 24435 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:11.363163 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:11.364014 26386 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:11.364504 26386 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:11.365394 26386 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:11.376201 24170 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:11.587273 24544 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:11.590756 26497 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:11.598095 24312 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:11.601469 26471 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:11.606645 24544 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:11.610482 26471 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:11.624629 25373 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:11.626296 26382 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:11.716876 24435 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:11.731177 26223 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:11.741226 24312 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:11.750537 24535 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:11.763128 24312 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:11.777112 26150 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:11.796080 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44101.8us
I20260809 23:18:11.831274 24535 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:18:11.826205 26505 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.122s	user 0.003s	sys 0.000s
W20260809 23:18:11.880604 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 113156us
I20260809 23:18:11.797746 26510 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260809 23:18:11.887748 24428 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:11.863227 26508 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
W20260809 23:18:11.954211 26489 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:18:12.012357 26397 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:18:11.952106 26500 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260809 23:18:12.085747 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 205504us
W20260809 23:18:12.089951 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 200583us
W20260809 23:18:12.177027 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69364us
I20260809 23:18:12.184150 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:12.123010 26520 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.112s
W20260809 23:18:12.284629 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64749us
I20260809 23:18:12.293484 24311 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:18:12.316557 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 425283us
W20260809 23:18:12.360543 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 409215us
W20260809 23:18:12.363227 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 167237us
I20260809 23:18:12.402154 24435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:12.402861 24435 raft_consensus.cc:595] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:12.434281 24435 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:12.437295 26550 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:12.454248 24422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7be87796c0684430b07c73073572ee50"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:12.455444 24422 raft_consensus.cc:595] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:12.472945 24544 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:12.483884 26551 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:12.485697 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
W20260809 23:18:12.494428 24269 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e43c87eb09044548379ab2c35cce3c1. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:12.494716 24311 raft_consensus.cc:595] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:12.527980 24435 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:12.520993 26535 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:12.542501 24535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
W20260809 23:18:12.559158 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61301.6us
I20260809 23:18:12.597239 24535 raft_consensus.cc:595] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:12.603577 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:18:12.606189 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:12.533264 26525 raft_consensus.cc:979] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:12.615299 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:12.613315 24546 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:12.627163 26471 raft_consensus.cc:1067] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:12.635316 24541 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:12.640218 24534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:12.640829 24534 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:12.662026 24534 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:12.662312 24170 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:12.691694 24311 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260809 23:18:12.694523 24498 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 974355bb93274291bb6be768b656cc67. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:12.695343 26471 raft_consensus.cc:979] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:12.695950 26471 raft_consensus.cc:1067] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:12.696884 26411 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:12.697805 24428 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:12.698669 24323 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7be87796c0684430b07c73073572ee50"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:12.699294 24323 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:12.699666 24323 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:12.704282 26548 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:12.721745 26577 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260809 23:18:12.707568 24534 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:12.725692 24534 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:12.732189 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:12.733004 24428 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:12.733424 24428 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:12.734103 24428 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:12.736403 24428 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:12.736672 24323 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:12.741292 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:12.742044 24429 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:12.742408 24429 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:12.745638 24429 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:12.746771 24429 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:12.739122 24323 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:12.755999 24429 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:12.760867 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:12.762069 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:12.762986 24534 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:12.763916 26535 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:12.764422 26535 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:12.771178 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:12.771950 25373 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:12.773950 24428 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:12.774974 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:12.826550 26576 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260809 23:18:12.797394 26537 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:12.935045 24534 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:12.768316 26535 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:12.938330 25373 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:18:13.014099 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49243.5us
I20260809 23:18:13.006007 26537 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:13.080219 26537 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:13.101770 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48971.5us
W20260809 23:18:13.106395 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67584.7us
I20260809 23:18:13.090533 26410 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:13.151428 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 137558us
I20260809 23:18:13.149399 26592 ts_tablet_manager.cc:927] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:18:13.201191 26582 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260809 23:18:13.230731 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44210.4us
W20260809 23:18:13.231792 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79520.6us
I20260809 23:18:13.241742 24428 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.287468 24312 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.278610 26592 tablet_copy_client.cc:316] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:13.312270 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 8e43c87eb09044548379ab2c35cce3c1 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:13.312777 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 8e43c87eb09044548379ab2c35cce3c1 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-8e43c87eb09044548379ab2c35cce3c1
W20260809 23:18:13.356945 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54978.8us
I20260809 23:18:13.355448 24334 tablet_copy_source_session.cc:205] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:13.363561 26592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e43c87eb09044548379ab2c35cce3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:13.394629 24313 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:13.390621 24168 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:13.408044 26614 raft_consensus.cc:979] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:13.408576 26614 raft_consensus.cc:1067] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:13.424418 24422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34778
I20260809 23:18:13.425133 24422 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:13.425595 24422 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.393616 26622 ts_tablet_manager.cc:927] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:13.445597 24422 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:13.450629 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:13.451321 24535 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:13.451673 24535 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:13.452512 24535 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:13.455648 24535 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.464497 26622 tablet_copy_client.cc:316] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:13.466532 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 974355bb93274291bb6be768b656cc67 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:13.467080 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 974355bb93274291bb6be768b656cc67 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-974355bb93274291bb6be768b656cc67
I20260809 23:18:13.464234 24535 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:13.468834 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:13.469641 26591 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:13.470163 26591 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:13.471119 26591 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:13.479287 24429 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260809 23:18:13.480302 24269 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8039b4f7b5b04cd0a5fb1f9d6c999b5e. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:13.480798 26604 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:13.488919 24557 tablet_copy_source_session.cc:205] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:13.491616 24422 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:13.462998 24165 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.504894 24535 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:13.503309 26592 tablet_copy_client.cc:791] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:13.506223 26592 tablet_copy_client.cc:655] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:13.508051 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:13.508658 25370 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.511400 26585 raft_consensus.cc:979] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:13.514513 26622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 974355bb93274291bb6be768b656cc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:13.519485 26592 tablet_copy_client.cc:531] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:13.521720 26551 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260809 23:18:13.524405 26535 raft_consensus.cc:1067] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:13.533159 24165 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:13.535548 24422 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:13.536736 24435 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:13.539096 26560 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:13.549032 26584 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:13.556120 25370 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:13.563750 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:13.564538 24544 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:13.564999 24544 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.578398 24544 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:13.564092 24170 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:13.609448 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:13.610070 24312 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:13.610347 24312 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:13.615801 24544 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:13.620174 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:13.621050 24422 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.626302 26592 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:13.628276 24312 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:13.640810 24312 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:13.652988 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:13.654479 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:13.655447 26612 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:13.655930 26612 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:13.656905 26612 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:13.710453 26604 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.713526 24544 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:13.719198 26545 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:13.721657 24429 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.724107 25370 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:13.813760 24546 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.826908 24165 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:13.739147 26596 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260809 23:18:13.882956 24168 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:18:13.962525 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87206.9us
W20260809 23:18:13.962689 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 103389us
I20260809 23:18:13.962589 24422 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:13.986522 24535 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:13.995048 24435 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:13.997264 26601 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260809 23:18:14.008738 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 97036.5us
I20260809 23:18:14.009009 26622 tablet_copy_client.cc:791] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:14.025040 26622 tablet_copy_client.cc:655] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:14.056749 26622 tablet_copy_client.cc:531] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260809 23:18:14.081784 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58876.1us
W20260809 23:18:14.104619 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82515.9us
W20260809 23:18:14.195807 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59628.8us
W20260809 23:18:14.204144 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92754us
W20260809 23:18:14.222458 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 186547us
I20260809 23:18:14.285897 26622 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:14.053936 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:14.343593 26635 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.405s	user 0.007s	sys 0.009s
W20260809 23:18:14.415205 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 398249us
I20260809 23:18:14.465083 24312 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:18:14.475005 26590 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260809 23:18:14.525305 24313 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:14.556537 24544 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:18:14.697032 26471 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:18:14.700415 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91107us
W20260809 23:18:14.702416 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105852us
I20260809 23:18:14.663143 26410 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260809 23:18:14.754251 25372 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260809 23:18:14.767030 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 162121us
W20260809 23:18:14.768035 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113904us
I20260809 23:18:14.836519 24422 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260809 23:18:14.894567 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 284416us
W20260809 23:18:14.895569 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 287837us
W20260809 23:18:14.896215 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 201751us
W20260809 23:18:14.895910 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 299995us
I20260809 23:18:14.890028 26680 ts_tablet_manager.cc:927] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:18:14.901672 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 133920us
I20260809 23:18:14.910086 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:14.921914 24423 raft_consensus.cc:595] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:14.950726 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a4c69723b934fd5869e2f079439b2f9. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:18:14.966292 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69475us
I20260809 23:18:14.943363 26682 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260809 23:18:14.968712 26538 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260809 23:18:15.036419 24542 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:15.042547 24542 raft_consensus.cc:595] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:15.023901 26680 tablet_copy_client.cc:316] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:15.063503 24546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65d375db793a4542808e1cff72fb6617"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:15.064091 24546 raft_consensus.cc:595] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:15.055745 26650 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:15.190853 26650 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:15.193598 26650 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:15.148206 26635 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:18:15.186683 26592 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260809 23:18:15.226508 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 258788us
W20260809 23:18:15.299127 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 251747us
I20260809 23:18:15.235255 26604 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:15.313246 26592 tablet_bootstrap.cc:492] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:15.324955 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:15.326709 26510 raft_consensus.cc:979] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:15.329121 24311 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:15.329244 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:15.330745 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:15.332443 26538 raft_consensus.cc:1067] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:15.335968 26592 ts_tablet_manager.cc:1397] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 1.710s	user 0.030s	sys 0.043s
I20260809 23:18:15.323163 26604 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:15.459944 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 161716us
I20260809 23:18:15.438253 26592 raft_consensus.cc:348] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:18:15.528512 26622 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
W20260809 23:18:15.581156 26706 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260809 23:18:15.533802 26604 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:15.628480 26640 raft_consensus.cc:979] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:15.629372 26640 raft_consensus.cc:1067] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:15.632054 24534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:15.633005 24534 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:15.633569 24534 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:15.711561 24428 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65d375db793a4542808e1cff72fb6617"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34778
I20260809 23:18:15.712556 24428 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:15.713019 24428 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:15.740041 24535 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:15.486670 26592 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:15.759912 24496 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:15.760143 26592 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:15.765348 26556 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:15.768167 26651 raft_consensus.cc:979] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:15.765990 26556 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:15.777561 26556 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:15.785836 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:15.798231 26651 raft_consensus.cc:1067] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:15.827634 26622 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:15.828667 26622 tablet_bootstrap.cc:492] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
W20260809 23:18:15.891131 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.022s	sys 0.058s
W20260809 23:18:15.891644 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.022s	sys 0.058s
I20260809 23:18:15.897631 26624 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:15.898272 26624 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:15.945477 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.027s	sys 0.104s
W20260809 23:18:15.946003 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.027s	sys 0.104s
I20260809 23:18:15.949215 26596 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:15.949481 26624 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:15.950381 26592 ts_tablet_manager.cc:1428] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.614s	user 0.011s	sys 0.002s
W20260809 23:18:15.950754 26592 ts_tablet_manager.cc:1460] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Tablet startup took 614ms
W20260809 23:18:15.951081 26592 ts_tablet_manager.cc:1462] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Trace:
0809 23:18:12.999408 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:12.999646 (+   238us) service_pool.cc:227] Handling call
0809 23:18:13.186607 (+186961us) ts_tablet_manager.cc:928] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
0809 23:18:13.362344 (+175737us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:13.428781 (+ 66437us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:13.429114 (+   333us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:13.494198 (+ 65084us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:13.494487 (+   289us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:13.494794 (+   307us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:13.494869 (+    75us) inbound_call.cc:173] Queueing success response
0809 23:18:13.625424 (+130555us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0809 23:18:15.336320 (+1710896us) ts_tablet_manager.cc:1429] Starting tablet replica
0809 23:18:15.435606 (+ 99286us) tablet_replica.cc:259] Starting instrumentation
0809 23:18:15.436373 (+   767us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":33737,"raft.queue_time_us":215,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":291,"tablet-copy.queue_time_us":120078,"tablet-download-pool.queue_time_us":120,"tablet-download-pool.run_cpu_time_us":1104,"tablet-download-pool.run_wall_time_us":12647,"thread_start_us":53173,"threads_started":2}
W20260809 23:18:15.955372 26690 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.005s	sys 0.014s
I20260809 23:18:15.812338 26590 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:15.959013 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.004s	sys 0.015s
W20260809 23:18:15.959414 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.004s	sys 0.015s
W20260809 23:18:15.960798 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 595152us
W20260809 23:18:15.955807 26690 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.005s	sys 0.014s
I20260809 23:18:15.966326 24534 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:15.972215 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
W20260809 23:18:16.066224 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 250325us
W20260809 23:18:16.067333 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105808us
I20260809 23:18:16.037031 26622 ts_tablet_manager.cc:1397] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 1.752s	user 0.056s	sys 0.033s
I20260809 23:18:16.077152 24428 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.094480 24428 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.096062 24312 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:16.098122 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:18:16.100466 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:16.136518 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:16.158875 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63829.8us
I20260809 23:18:16.167304 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.168792 26719 ts_tablet_manager.cc:927] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:16.169643 24333 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-8e43c87eb09044548379ab2c35cce3c1 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:16.170142 24333 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-8e43c87eb09044548379ab2c35cce3c1 on tablet 8e43c87eb09044548379ab2c35cce3c1 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:16.186107 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:16.187016 25372 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:16.086809 26669 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.197561 26669 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.200371 26669 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.204047 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:16.204741 24534 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:16.413165 26613 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:16.459762 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 387488us
I20260809 23:18:16.473649 26624 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.503624 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:16.504310 24544 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:16.504616 24544 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:16.510679 25373 raft_consensus.cc:1203] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260809 23:18:16.470404 26613 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.555814 24544 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.563436 24544 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:16.567410 26590 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.609s	user 0.006s	sys 0.006s
I20260809 23:18:16.568602 26622 raft_consensus.cc:348] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:16.569746 26622 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:16.509481 26624 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.573443 26624 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.577557 26613 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.570598 26622 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:16.566275 26719 tablet_copy_client.cc:316] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:18:16.591635 26622 ts_tablet_manager.cc:1428] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.506s	user 0.012s	sys 0.024s
W20260809 23:18:16.595137 26622 ts_tablet_manager.cc:1460] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Tablet startup took 523ms
I20260809 23:18:16.600410 26590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:47446 (request call id 10) took 2627 ms. Trace:
I20260809 23:18:16.600955 26590 rpcz_store.cc:276] 0809 23:18:13.981524 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:13.982589 (+  1065us) service_pool.cc:227] Handling call
0809 23:18:13.992720 (+ 10131us) spinlock_profiling.cc:243] Waited 7.6 ms on lock 0x7b540035fd20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5dc0 00007fac23866bd2 00007fac23865890 00007fac287a2f39 00007fac21e8e704 00007fac21e8e692 00007fac21e8e5c1 00007fac21e8e53c 00007fac21e8d7f2 00007fac21c8a94c
0809 23:18:14.015047 (+ 22327us) log.cc:842] Serialized 499 byte log entry
0809 23:18:16.600320 (+2585273us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24710,"raft.queue_time_us":50247,"raft.run_cpu_time_us":2264,"raft.run_wall_time_us":110308,"spinlock_wait_cycles":17489152,"thread_start_us":468821,"threads_started":2,"wal-append.queue_time_us":177290,"wal-append.run_cpu_time_us":3282,"wal-append.run_wall_time_us":1034409}
W20260809 23:18:16.596386 26622 ts_tablet_manager.cc:1462] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Trace:
0809 23:18:13.288367 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:13.288780 (+   413us) service_pool.cc:227] Handling call
0809 23:18:13.438142 (+149362us) ts_tablet_manager.cc:928] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
0809 23:18:13.513378 (+ 75236us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:13.581019 (+ 67641us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:13.581422 (+   403us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:13.891002 (+309580us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:13.919762 (+ 28760us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:14.000433 (+ 80671us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900f98 00007fac20900ebb 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ed919 00007fac2881ea8e 00007fac288045ad 00007fac28804559
0809 23:18:14.000663 (+   230us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:14.000851 (+   188us) inbound_call.cc:173] Queueing success response
0809 23:18:14.285147 (+284296us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0809 23:18:16.085598 (+1800451us) ts_tablet_manager.cc:1429] Starting tablet replica
0809 23:18:16.170591 (+ 84993us) tablet_replica.cc:259] Starting instrumentation
0809 23:18:16.520494 (+349903us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":248,"raft.queue_time_us":3088,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"spinlock_wait_cycles":9472384,"tablet-copy.queue_time_us":57457,"tablet-download-pool.queue_time_us":440,"tablet-download-pool.run_cpu_time_us":1391,"tablet-download-pool.run_wall_time_us":29237,"thread_start_us":71501,"threads_started":2}
W20260809 23:18:16.646531 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.005s	sys 0.006s
W20260809 23:18:16.647744 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.006s	sys 0.006s
W20260809 23:18:16.648242 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 529584us
W20260809 23:18:16.653854 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 514156us
I20260809 23:18:16.494825 26711 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:16.656875 26711 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.658401 25372 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:16.484357 26712 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.659957 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:16.662642 26711 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.673367 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:16.677778 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 215522us
W20260809 23:18:16.679076 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 170673us
I20260809 23:18:16.680732 26667 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:16.682515 26712 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.683260 26666 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.684592 26667 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.685863 26712 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.687728 26667 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.691380 26722 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:16.685873 26666 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.695551 26613 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.696588 26613 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:16.706393 24535 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:16.709100 24557 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-974355bb93274291bb6be768b656cc67 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:16.709384 26613 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:16.709877 24557 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-974355bb93274291bb6be768b656cc67 on tablet 974355bb93274291bb6be768b656cc67 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:16.715780 26656 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:16.716813 26656 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.718451 26612 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.719146 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:16.719429 26612 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.720818 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-8039b4f7b5b04cd0a5fb1f9d6c999b5e
I20260809 23:18:16.721084 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.692476 26722 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.738090 26665 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:16.744240 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:16.746115 26665 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:16.747957 26665 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.726069 26612 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:16.727813 24312 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.005s
I20260809 23:18:16.764775 26590 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:16.767851 25371 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:16.770339 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97370.1us
I20260809 23:18:16.775236 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.776894 26683 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:16.779196 26689 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:16.779685 26689 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:16.781045 26689 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.783833 26666 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.784963 26656 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.786713 26722 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.753688 26711 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.787760 26712 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.787716 24334 tablet_copy_source_session.cc:205] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:16.798435 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:16.788499 26711 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.800055 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.807168 25372 raft_consensus.cc:1203] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260809 23:18:16.807372 26624 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:16.810904 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.813927 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.827978 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:16.828176 26556 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:16.829008 26556 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.831171 24544 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
W20260809 23:18:16.834792 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94472us
I20260809 23:18:16.789021 26712 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.835041 26556 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:16.836927 26711 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:16.840713 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57202.9us
I20260809 23:18:16.853991 26666 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.854656 26666 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.856917 26666 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.856698 26711 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:16.861934 26556 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.861114 26711 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:16.864252 26556 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.865944 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.868917 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.852046 26590 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:16.842685 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:16.880637 26590 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:16.883728 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49737.6us
I20260809 23:18:16.884349 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:16.885581 26666 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:16.889030 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:16.891808 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:16.892041 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.892184 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:16.895366 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.856382 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:16.906230 24424 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:16.910497 26689 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:16.882824 26712 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:16.916646 24314 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:16.919814 26711 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:16.929435 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:16.945353 26680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:18:16.948505 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51113.4us
I20260809 23:18:16.951061 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:16.951629 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:18:16.955615 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83238.4us
I20260809 23:18:16.948858 24314 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:16.962828 24314 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:16.963990 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:16.965808 26560 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:16.966185 26560 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:16.966552 26560 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:16.952600 26556 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:16.987550 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:16.999742 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46972.5us
W20260809 23:18:16.997802 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99956.2us
I20260809 23:18:17.006326 24170 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:17.036489 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.036631 26712 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.037297 26712 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.039238 26624 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.039721 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.041302 26712 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.044169 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.047710 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.049574 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.053674 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.054239 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.056054 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.060420 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.059517 26712 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.071137 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.088339 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.095031 26712 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.127158 26546 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.140457 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.123214 26624 raft_consensus.cc:1050] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:17.140668 26546 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.121102 24314 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.121598 26712 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.143756 26712 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:17.145270 26624 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:17.147756 26546 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.148399 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.145885 26712 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:17.152786 26650 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.154336 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.155076 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.156199 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.157624 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.158215 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.158339 24422 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:17.153134 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.161283 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:17.167178 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113524us
I20260809 23:18:17.167516 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:17.168108 24312 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:17.172047 24314 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.173121 26722 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.173730 26722 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.175786 26722 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.172574 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:40856 (request call id 1400) took 1098 ms. Trace:
I20260809 23:18:17.177688 24314 rpcz_store.cc:276] 0809 23:18:16.074567 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:16.916289 (+841722us) service_pool.cc:227] Handling call
0809 23:18:17.172493 (+256204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:17.184842 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.186013 26560 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.187638 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.189386 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.192247 25371 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:17.196877 26683 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:17.203630 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet 4a4c69723b934fd5869e2f079439b2f9 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:18:17.204267 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet 4a4c69723b934fd5869e2f079439b2f9 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-4a4c69723b934fd5869e2f079439b2f9
I20260809 23:18:17.211197 24320 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:17.211277 24425 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:17.219295 24543 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:17.219473 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:17.222229 26560 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:17.222738 26556 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:17.223655 26556 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:17.229111 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.230674 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.231649 24546 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:17.234578 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:17.235139 24450 tablet_copy_source_session.cc:205] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:17.249643 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:17.249676 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.250514 24543 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:17.250813 24543 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:17.251624 24543 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.252067 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.252526 24543 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:17.256171 25375 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:17.257647 26546 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:17.265626 24312 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:17.225178 26556 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:17.311646 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46768.3us
I20260809 23:18:17.314530 24543 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:17.312665 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:17.316962 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.319773 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.328114 24546 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:17.328783 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:17.331265 26667 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.335177 26639 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:17.338209 26665 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:17.343948 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.345628 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:17.351703 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:17.335950 26639 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.356384 26604 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:17.348548 26740 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:17.362224 26690 raft_consensus.cc:1050] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:17.348831 26712 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.381875 24270 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:17.382982 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.383677 26719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a4c69723b934fd5869e2f079439b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:17.384157 26725 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:17.384912 26725 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.387585 26725 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:17.392787 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.393647 24423 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:17.395812 24496 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:17.403563 26601 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.404201 26601 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:17.406170 26601 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:17.407606 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:17.413292 26624 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:17.413666 26624 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:17.414458 26639 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.423234 26651 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:17.411240 24168 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:17.424093 26651 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:17.424853 26651 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:17.414530 26624 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.427934 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:17.428525 24425 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:17.428766 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.429459 26604 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:17.429976 26604 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:17.428088 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46302.1us
I20260809 23:18:17.440421 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.440335 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.442117 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.444118 26596 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.444751 26596 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.445835 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.436028 26604 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.447782 26560 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.452083 26680 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37956 (request call id 369) took 2922 ms. Trace:
I20260809 23:18:17.426966 26624 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:17.463616 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.421052 26690 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:17.461905 26740 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:17.473855 24423 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:17.465241 26642 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260809 23:18:17.465548 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.465365 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:17.453136 26680 rpcz_store.cc:276] 0809 23:18:14.529874 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:14.529987 (+   113us) service_pool.cc:227] Handling call
0809 23:18:14.902479 (+372492us) ts_tablet_manager.cc:928] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
0809 23:18:16.943094 (+2040615us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:17.135340 (+192246us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:17.135626 (+   286us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:17.401882 (+266256us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:17.450090 (+ 48208us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:17.451346 (+  1256us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:17.451793 (+   447us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":29128,"tablet-copy.queue_time_us":340900,"thread_start_us":316508,"threads_started":2}
I20260809 23:18:17.494148 26680 tablet_copy_client.cc:791] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:17.495582 26680 tablet_copy_client.cc:655] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:17.500010 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.509106 24425 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:17.505895 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.520299 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.507721 26650 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.509927 26560 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.529678 24543 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:17.533752 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.536006 26712 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.523995 26665 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.557044 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.558168 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.559037 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:17.553699 26667 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260809 23:18:17.561107 26732 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.563170 26711 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.551752 26560 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:17.581576 26711 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.589578 26680 tablet_copy_client.cc:531] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:17.613063 24266 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:17.617561 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.629163 26604 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.629628 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.630527 26604 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:17.618634 26711 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:17.631794 26711 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:17.632369 26711 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:17.621630 26651 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.636886 24546 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:17.639827 24539 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:17.645330 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:17.646224 24429 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:17.647763 24266 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.644158 26680 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:17.680868 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:17.661518 26651 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:17.684391 24380 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:17.686538 24170 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 283) took 1569 ms. Trace:
I20260809 23:18:17.677655 26560 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:17.686656 26722 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.723325 26543 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.741600 26719 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37950 (request call id 468) took 2184 ms. Trace:
I20260809 23:18:17.693346 24170 rpcz_store.cc:276] 0809 23:18:16.114354 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:16.114916 (+   562us) service_pool.cc:227] Handling call
0809 23:18:17.683468 (+1568552us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:18:17.104493 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:18:17.200472 (+ 95979us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:18:17.200528 (+    56us) write_op.cc:435] Acquired schema lock
0809 23:18:17.200566 (+    38us) tablet.cc:660] Decoding operations
0809 23:18:17.201033 (+   467us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:18:17.201163 (+   130us) write_op.cc:641] Partition lock acquired for write op
0809 23:18:17.201214 (+    51us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:18:17.201535 (+   321us) tablet.cc:699] Row locks acquired
0809 23:18:17.201571 (+    36us) write_op.cc:260] PREPARE: finished
0809 23:18:17.201737 (+   166us) write_op.cc:270] Start()
0809 23:18:17.201895 (+   158us) write_op.cc:276] Timestamp: P: 1786317497201675 usec, L: 0
0809 23:18:17.201942 (+    47us) op_driver.cc:331] REPLICATION: starting
0809 23:18:17.202662 (+   720us) log.cc:842] Serialized 595 byte log entry
0809 23:18:17.627972 (+425310us) op_driver.cc:444] REPLICATION: finished
0809 23:18:17.665243 (+ 37271us) write_op.cc:301] APPLY: starting
0809 23:18:17.665343 (+   100us) tablet.cc:1365] starting BulkCheckPresence
0809 23:18:17.665446 (+   103us) tablet.cc:1368] finished BulkCheckPresence
0809 23:18:17.665608 (+   162us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:18:17.666015 (+   407us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:18:17.666353 (+   338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 23:18:17.666421 (+    68us) write_op.cc:312] APPLY: finished
0809 23:18:17.666819 (+   398us) log.cc:842] Serialized 23 byte log entry
0809 23:18:17.667062 (+   243us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:17.667131 (+    69us) write_op.cc:454] Released schema lock
0809 23:18:17.667185 (+    54us) write_op.cc:341] FINISH: Updating metrics
0809 23:18:17.667566 (+   381us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:17.667607 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":89598,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":36934,"apply.run_cpu_time_us":2499,"apply.run_wall_time_us":2638,"num_ops":1,"prepare.queue_time_us":75766,"prepare.run_cpu_time_us":96715,"prepare.run_wall_time_us":272176,"raft.queue_time_us":145724,"raft.run_cpu_time_us":46459,"raft.run_wall_time_us":81898,"replication_time_us":425793,"thread_start_us":274725,"threads_started":4,"wal-append.queue_time_us":234475}]]}
I20260809 23:18:17.747964 26719 rpcz_store.cc:276] 0809 23:18:15.556848 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:15.557020 (+   172us) service_pool.cc:227] Handling call
0809 23:18:16.169101 (+612081us) ts_tablet_manager.cc:928] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
0809 23:18:17.371073 (+1201972us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:17.542705 (+171632us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:17.582516 (+ 39811us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:17.737920 (+155404us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:17.738213 (+   293us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:17.741446 (+  3233us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:17.741547 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":59636,"tablet-copy.queue_time_us":611240,"thread_start_us":737589,"threads_started":2}
I20260809 23:18:17.755278 26719 tablet_copy_client.cc:791] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:17.755986 26719 tablet_copy_client.cc:655] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
W20260809 23:18:17.776948 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94404.4us
I20260809 23:18:17.714133 26731 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260809 23:18:17.789305 24429 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:17.734090 26560 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.797880 26731 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:17.800380 24425 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:17.813274 26639 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:17.808423 26543 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260809 23:18:17.820116 26560 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:17.820712 24423 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:17.803242 26731 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:17.823197 26731 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.813004 24496 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:17.829115 26763 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260809 23:18:17.833597 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:17.834744 26719 tablet_copy_client.cc:531] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:17.840924 26783 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:17.824488 26731 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:17.824539 26711 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.850291 26711 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:17.846031 26596 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:17.857821 26596 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:17.865365 26732 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:17.870431 26596 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:17.871989 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:17.874760 24543 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:17.896248 24424 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:17.892176 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:17.914582 26719 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:17.903716 26756 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260809 23:18:17.927757 25372 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:17.954753 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:17.941969 24165 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:17.964331 25369 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:17.966195 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:17.966483 24312 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:17.979480 26787 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260809 23:18:17.981382 26556 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:17.992377 24321 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:18.019891 24544 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:18.023629 26732 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:18.020268 24164 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:18.031495 24164 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.035250 24313 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:18.042223 24320 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:18.045732 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:18.048506 26596 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:18.049582 26783 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:18.049932 26783 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:18.052289 26783 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:18.056586 26783 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:18.062325 26560 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:18.056735 26777 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:18.065161 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:18.069761 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:18.068886 26775 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:18.074025 24424 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:18.096814 24312 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:18.105140 24266 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:18.107296 26711 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:18.111927 25372 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:18.086400 24423 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.154345 24544 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260809 23:18:18.155087 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47838.4us
I20260809 23:18:18.156332 26546 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:18.160341 26731 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:18.144708 26777 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.161535 26546 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:18.195015 26792 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.108s	user 0.001s	sys 0.001s
I20260809 23:18:18.207588 25371 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.204782 26642 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:18.287149 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57047.5us
I20260809 23:18:18.196367 25369 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.353739 24321 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:18.353965 24430 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.371906 24544 raft_consensus.cc:2932] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.376710 24320 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.361100 26556 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260809 23:18:18.419435 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8e43c87eb09044548379ab2c35cce3c1 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260809 23:18:18.422879 26797 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260809 23:18:18.437054 26680 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:18.438061 26680 tablet_bootstrap.cc:492] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:18.453464 26680 ts_tablet_manager.cc:1397] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.810s	user 0.035s	sys 0.014s
I20260809 23:18:18.469165 24423 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:18.474612 26719 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:18.479851 26560 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:18.491082 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 164567us
I20260809 23:18:18.511229 26680 raft_consensus.cc:348] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:18.512183 26680 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:18.520504 24170 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.531530 26719 tablet_bootstrap.cc:492] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:18.532403 26719 ts_tablet_manager.cc:1397] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.618s	user 0.027s	sys 0.013s
I20260809 23:18:18.512828 26680 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:18.523947 24170 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:18.541590 26680 ts_tablet_manager.cc:1428] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.088s	user 0.008s	sys 0.000s
I20260809 23:18:18.550797 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-8039b4f7b5b04cd0a5fb1f9d6c999b5e received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:18.553500 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-8039b4f7b5b04cd0a5fb1f9d6c999b5e on tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:18.562556 26719 raft_consensus.cc:348] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:18.563632 26719 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:18.564365 26719 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:18.571290 26719 ts_tablet_manager.cc:1428] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.039s	user 0.004s	sys 0.004s
I20260809 23:18:18.573395 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-4a4c69723b934fd5869e2f079439b2f9 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:18:18.573848 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-4a4c69723b934fd5869e2f079439b2f9 on tablet 4a4c69723b934fd5869e2f079439b2f9 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:18.598867 25369 raft_consensus.cc:1203] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:18:18.622238 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e43c87eb09044548379ab2c35cce3c1 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34762
I20260809 23:18:18.626204 24544 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:18.635664 25369 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:18.638084 26556 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:18.644248 24423 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:18.646919 26560 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:18.652638 26651 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.659484 25372 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.672753 26813 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:18:18.679497 26813 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:18.681229 26813 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:18.681488 24425 raft_consensus.cc:2932] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.708338 26813 ts_tablet_manager.cc:1905] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:18.711397 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 974355bb93274291bb6be768b656cc67 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:18.720331 24170 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:18.746749 26813 ts_tablet_manager.cc:1918] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260809 23:18:18.747452 26813 log.cc:1196] T 8e43c87eb09044548379ab2c35cce3c1 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/8e43c87eb09044548379ab2c35cce3c1
I20260809 23:18:18.765210 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 974355bb93274291bb6be768b656cc67 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40854
I20260809 23:18:18.765829 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet 8e43c87eb09044548379ab2c35cce3c1 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
W20260809 23:18:18.832887 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61948.3us
I20260809 23:18:18.852972 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:18.897919 26814 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:18:18.905455 26814 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:18.907670 26731 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:18.908241 26731 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:18.911201 26731 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:18.912271 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:18.912902 24430 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:18.913957 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:18.914578 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:18.916939 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:18.916427 24534 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34970da619114abc942b839d0862a300"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:18.918000 24534 raft_consensus.cc:595] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:18.918182 26731 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:18.918565 26731 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:18.918918 26731 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:18.923557 26814 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:18.928534 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:18.940814 25371 raft_consensus.cc:1203] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260809 23:18:18.974054 26814 ts_tablet_manager.cc:1905] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:18.978350 26731 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:18.985325 24321 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:18.985711 24321 raft_consensus.cc:603] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:18:18.990662 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:18.991415 24430 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:19.006369 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:19.007344 24544 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:19.014779 24544 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:19.020058 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c831643e28e34208b6edf775607155e4"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:19.017417 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:19.019456 24430 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:19.020817 24423 raft_consensus.cc:595] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:19.023644 26604 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:19.024149 26604 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:19.024963 26604 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:19.019212 26731 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:19.032532 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:19.034310 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:19.034996 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:19.067314 24164 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:19.092166 26814 ts_tablet_manager.cc:1918] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260809 23:18:19.092764 26814 log.cc:1196] T 974355bb93274291bb6be768b656cc67 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/974355bb93274291bb6be768b656cc67
I20260809 23:18:19.098497 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 974355bb93274291bb6be768b656cc67 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:19.107429 26556 raft_consensus.cc:979] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:19.107903 26556 raft_consensus.cc:1067] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:19.109617 24320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34970da619114abc942b839d0862a300"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:19.110127 24320 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:19.110455 24320 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.118273 24320 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.124339 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:19.124884 24544 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:19.125149 24544 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:19.125835 24544 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.126907 24544 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.133774 24544 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:19.143738 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:19.144584 26731 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:19.144970 24320 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:19.145081 26731 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:19.146782 26731 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.150537 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:19.151139 25372 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.158047 25372 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:19.203480 26785 raft_consensus.cc:1050] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:19.206167 26785 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.216311 24164 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:19.247337 24544 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:19.250495 25371 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:19.248955 26815 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260809 23:18:19.251827 24320 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:19.256130 26779 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:19.260216 26601 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:19.301098 26770 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:19.301609 26770 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:19.303856 26770 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:19.306661 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:19.316200 24321 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:19.317404 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:19.320114 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:19.317734 26785 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:19.326628 24534 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:19.332252 26770 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:19.332667 26770 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:19.332984 26770 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:19.371057 26770 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:19.376322 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:19.377056 24321 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:19.390762 24320 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:19.410532 26779 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:19.415318 26601 raft_consensus.cc:979] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:19.416045 26601 raft_consensus.cc:1067] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:19.421437 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:19.422677 26770 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:19.455194 24321 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:19.454813 25372 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:19.411232 26779 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:19.425354 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c831643e28e34208b6edf775607155e4"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:19.462287 24425 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 3
I20260809 23:18:19.476156 24312 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:19.476817 24312 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.479871 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:19.480746 26775 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:19.481413 26775 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:19.482544 26775 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:19.488333 26779 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:19.493409 26543 raft_consensus.cc:1050] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:19.496515 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:19.500919 24321 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:19.496953 26543 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.502251 24378 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:19.502586 24378 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:18:19.501519 24544 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:19.524081 24164 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:19.527397 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:19.542042 24425 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:19.542606 26779 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:19.543540 26604 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:19.554013 24430 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:19.555650 26711 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260809 23:18:19.629464 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53372.5us
I20260809 23:18:19.658015 24312 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.660085 24312 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:19.682220 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:19.682883 24429 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:19.683240 24429 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:19.690091 24429 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.691519 24429 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.690528 25369 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:19.693344 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:19.695501 25372 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:19.708922 24429 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:19.725845 25372 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:19.757752 24534 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:19.762554 24543 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:19.774461 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:19.776217 26844 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:19.776691 26844 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:19.777513 26844 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:19.851460 26711 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:19.852582 25372 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:19.766039 26835 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260809 23:18:19.914657 26844 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:19.836483 24422 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:19.933830 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132658us
W20260809 23:18:19.922138 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 105139us
W20260809 23:18:19.949717 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 137100us
I20260809 23:18:19.933236 26860 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260809 23:18:19.993582 24429 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260809 23:18:20.067154 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66780.7us
I20260809 23:18:20.068640 26883 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:20.075083 24168 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:18:20.104260 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 142466us
I20260809 23:18:20.104533 25369 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:20.111534 24423 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:20.117210 24312 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:20.129150 26832 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:20.130489 24313 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260809 23:18:20.131428 26864 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:20.195403 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:20.190522 24168 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:20.224557 26842 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:20.208688 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46641.7us
I20260809 23:18:20.220804 26877 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:20.244369 26848 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.246312 26852 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:20.243664 25369 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.278822 24423 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.279183 25373 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.295599 24164 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 1 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.309552 24320 raft_consensus.cc:2932] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.310443 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a4c69723b934fd5869e2f079439b2f9 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38080
I20260809 23:18:20.322490 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4a4c69723b934fd5869e2f079439b2f9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260809 23:18:20.389000 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52179.4us
I20260809 23:18:20.420293 24164 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.428249 24534 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:20.429034 24534 raft_consensus.cc:595] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:18:20.456235 24270 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.21.255.62:37921, user_credentials={real_user=slave}} blocked reactor thread for 155675us
I20260809 23:18:20.460210 26895 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:18:20.290885 24542 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:20.472523 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 174617us
I20260809 23:18:20.492399 25371 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:20.506453 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7be87796c0684430b07c73073572ee50"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:20.507318 24313 raft_consensus.cc:595] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:20.510205 26895 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:20.510898 26895 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:20.520284 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:20.520867 24313 raft_consensus.cc:595] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:20.500931 26842 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260809 23:18:20.527720 24314 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:20.532452 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:20.533077 24312 raft_consensus.cc:595] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:20.534152 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:18:20.536000 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:20.536644 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:20.539667 25369 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:20.541754 26884 raft_consensus.cc:979] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:20.542312 26884 raft_consensus.cc:1067] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:20.516309 26895 ts_tablet_manager.cc:1905] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:20.545445 24425 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:20.548058 26842 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:20.558044 24425 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:20.592795 26543 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260809 23:18:20.605898 26543 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260809 23:18:20.614300 26873 raft_consensus.cc:979] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:20.622356 26873 raft_consensus.cc:1067] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:20.612064 26874 raft_consensus.cc:979] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:20.624111 26874 raft_consensus.cc:1067] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:20.640945 26895 ts_tablet_manager.cc:1918] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260809 23:18:20.641528 26895 log.cc:1196] T 4a4c69723b934fd5869e2f079439b2f9 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/4a4c69723b934fd5869e2f079439b2f9
I20260809 23:18:20.643832 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet 4a4c69723b934fd5869e2f079439b2f9 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:20.638473 26883 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:20.668604 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63374.1us
I20260809 23:18:20.670487 25371 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:20.673887 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79035.7us
I20260809 23:18:20.684440 24316 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:20.684980 24316 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:20.685304 24316 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.693248 24429 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:20.693811 24429 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:20.694139 24429 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:20.694494 24425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7be87796c0684430b07c73073572ee50"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:20.696249 24425 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:20.695533 24431 raft_consensus.cc:2932] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:20.696599 24425 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.706368 24429 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.710839 24316 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:20.712638 24316 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:20.714306 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:20.709787 24425 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.721315 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:20.720114 24170 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 4 to 5, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:20.726949 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:20.727504 25373 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.729580 24542 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:20.729918 24542 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:20.730604 24542 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:20.733901 24542 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.736740 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:20.737317 24314 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:20.737617 24314 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:20.738677 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:20.740960 24425 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:20.742389 24542 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:20.743755 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:20.743921 24429 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:20.739444 24314 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.745574 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:20.746033 24542 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.748142 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:20.748667 24543 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:20.749969 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:20.750499 24316 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:20.750751 24316 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:20.752360 24542 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:20.761518 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:20.766616 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:20.770135 24314 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:20.770859 24316 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.772094 24316 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.775156 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:20.779232 26900 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:20.779750 26877 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:20.786401 24316 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:20.791005 26601 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:20.791543 26601 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:20.792483 26601 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.798249 26900 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:20.807717 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38080
I20260809 23:18:20.808663 26895 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:18:20.787937 26877 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:20.809815 26895 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:20.810644 26895 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:20.812181 26900 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:20.810318 26877 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.822381 26895 ts_tablet_manager.cc:1905] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:20.829052 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:20.850390 24168 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:20.877774 26895 ts_tablet_manager.cc:1918] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260809 23:18:20.882087 26895 log.cc:1196] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/8039b4f7b5b04cd0a5fb1f9d6c999b5e
I20260809 23:18:20.886071 26842 raft_consensus.cc:979] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:20.886591 26842 raft_consensus.cc:1067] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:20.890455 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet 8039b4f7b5b04cd0a5fb1f9d6c999b5e (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:20.891666 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:20.892238 24544 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:20.892604 24544 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.894605 24168 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:20.902482 24168 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 4 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:20.907827 24544 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.918288 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:20.919011 24316 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:20.918816 24544 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:20.920954 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:20.919391 24316 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:20.921535 24425 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.924168 24316 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.928674 24425 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:20.929942 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:20.930730 26852 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:20.931437 26852 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:20.932231 26852 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:20.925530 24316 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:20.974895 24316 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:20.985833 24168 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:21.271384 24316 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:21.276041 26877 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:21.299787 24542 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:21.305210 24544 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:21.310768 26877 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:21.314632 26601 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:21.326460 24314 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:21.328035 26848 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:21.343209 25373 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:21.344668 26900 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:21.346539 24312 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:21.353629 26852 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:21.483771 24423 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:21.485611 26832 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:21.501432 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106124us
I20260809 23:18:21.516036 24542 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:21.519953 26842 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:21.628818 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:21.690404 24543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:21.691084 24543 raft_consensus.cc:595] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:21.693934 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:21.694540 24312 raft_consensus.cc:595] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:21.698375 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "904621b566f64064bc1a7411ab27c677"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:21.699302 24423 raft_consensus.cc:595] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:21.701968 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:21.702419 24423 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:21.703428 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:18:21.705022 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:21.706082 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:21.745313 26932 raft_consensus.cc:979] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:21.745872 26932 raft_consensus.cc:1067] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:21.757256 24316 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:21.758486 24316 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:21.759753 24316 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.803515 24316 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.805402 24316 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:21.806025 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:21.806607 24423 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.812688 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:21.813301 24542 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:21.813589 24542 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:21.814342 24542 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.817961 24542 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.848006 24423 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:21.852787 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:21.853686 26900 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:21.854128 26900 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:21.854887 26900 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.859234 24542 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:21.873754 26842 raft_consensus.cc:979] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:21.874217 26842 raft_consensus.cc:1067] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:21.877275 25369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:21.877811 25369 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:21.878187 25369 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.892092 26929 raft_consensus.cc:979] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:21.892587 26929 raft_consensus.cc:1067] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:21.893548 25369 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:21.896780 25369 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:21.901219 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:21.901813 24423 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.910645 24423 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:21.911767 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:21.912575 26878 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:21.913044 26878 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:21.913893 26878 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:21.915474 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:21.916147 24316 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:21.916424 24316 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:21.917250 24316 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:21.918286 24316 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:21.929287 24170 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:21.931473 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:21.932018 24312 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:21.932374 24312 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:21.934353 24316 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:21.944831 24164 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 3 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:21.956889 24312 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.959368 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:21.959920 24312 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:21.960026 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:21.961062 24543 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:21.960592 24423 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:21.963358 24423 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:21.964329 24423 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.965729 24423 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:21.970976 24543 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:21.972026 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:21.972813 26849 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:21.973278 26849 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:21.974061 26849 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:21.979406 24423 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:21.986505 24543 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:22.003518 24423 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:22.005015 26832 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260809 23:18:22.015818 24498 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3a8a92891fb40e8911ac15230653aab. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:22.017117 24312 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:22.018615 26832 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:22.023113 26936 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.028728 24425 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.049283 24543 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:22.069159 24422 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:22.071148 26852 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:22.077899 24312 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.080808 24316 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:22.087713 24168 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260809 23:18:22.094026 24498 consensus_peers.cc:487] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9429f7311dd744bea4301956e143e9a7. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:22.102237 26770 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:22.104135 26601 raft_consensus.cc:979] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:22.104710 26601 raft_consensus.cc:1067] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:22.109958 24543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "904621b566f64064bc1a7411ab27c677"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38092
I20260809 23:18:22.110708 24543 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:22.111260 24543 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:22.147113 24543 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:22.148391 26832 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.150188 24543 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:22.163372 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:22.164350 24423 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:22.164647 24423 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:22.165606 24423 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:22.166814 24423 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:22.174539 24314 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.183362 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:22.184068 24316 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:22.224803 24423 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:22.226861 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:22.228318 26770 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:22.234879 26770 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:22.236109 24422 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.238485 26770 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:22.261641 24542 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:22.267235 24170 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 4, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:22.278533 24165 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index -1 to 4, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.303038 24422 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:22.304551 26936 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:22.306140 24316 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:18:22.306788 24498 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac3c46a49aea44a2b2721eb2a750f055. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:22.318732 24316 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:22.320194 26936 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:22.350875 26936 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.353574 24165 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 4, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:22.363585 24423 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:22.365239 26878 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:22.376715 24316 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.415248 24165 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.421182 24422 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:22.437011 24542 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:22.437578 24425 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:22.439391 26849 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:22.442193 26849 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:22.445974 24543 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:22.449208 26543 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:22.486694 24423 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:22.488149 26849 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:22.490379 24316 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:22.511332 26943 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260809 23:18:22.543889 24534 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:18:22.602571 26960 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260809 23:18:22.615804 26975 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260809 23:18:22.648397 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46073.3us
W20260809 23:18:22.761845 26940 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260809 23:18:22.787914 24431 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:22.772401 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:22.809969 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 116822us
W20260809 23:18:22.825043 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 131218us
I20260809 23:18:22.831054 24312 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:22.813488 26770 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260809 23:18:22.849710 27001 ts_tablet_manager.cc:927] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:22.856334 24312 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:18:22.948968 26966 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:22.983192 27007 ts_tablet_manager.cc:927] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:22.984081 27001 tablet_copy_client.cc:316] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:22.990561 24544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:22.991163 24544 raft_consensus.cc:595] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:23.019122 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet d3a8a92891fb40e8911ac15230653aab from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:23.019760 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet d3a8a92891fb40e8911ac15230653aab from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-d3a8a92891fb40e8911ac15230653aab
I20260809 23:18:22.976958 26989 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260809 23:18:23.038700 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:23.039801 24314 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:23.056213 24498 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e2662351f0e4b14bede1be0deaba6c8. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:23.133023 24557 tablet_copy_source_session.cc:205] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:23.134878 27007 tablet_copy_client.cc:316] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:23.138628 24423 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:18:23.139138 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:23.148684 24313 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:23.162432 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 237969us
W20260809 23:18:23.166460 27016 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260809 23:18:23.176205 24312 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:23.178915 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:23.180680 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:23.181319 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:23.164825 27022 ts_tablet_manager.cc:927] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:23.197000 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 9429f7311dd744bea4301956e143e9a7 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:23.197811 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 9429f7311dd744bea4301956e143e9a7 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-9429f7311dd744bea4301956e143e9a7
I20260809 23:18:23.199150 26900 raft_consensus.cc:979] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:23.208232 27001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a92891fb40e8911ac15230653aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:23.215191 26997 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:23.285013 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67530.5us
I20260809 23:18:23.310770 27022 tablet_copy_client.cc:316] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:23.311199 26900 raft_consensus.cc:1067] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:23.311806 24557 tablet_copy_source_session.cc:205] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
W20260809 23:18:23.312563 27024 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260809 23:18:23.323629 24425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:23.324168 24425 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:23.324563 24425 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:23.331544 24425 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:23.338603 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46012.1us
I20260809 23:18:23.339164 24425 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:23.364962 24557 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet ac3c46a49aea44a2b2721eb2a750f055 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:23.365703 24557 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet ac3c46a49aea44a2b2721eb2a750f055 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-ac3c46a49aea44a2b2721eb2a750f055
I20260809 23:18:23.367538 27007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9429f7311dd744bea4301956e143e9a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:23.392961 26985 raft_consensus.cc:979] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:23.393497 26985 raft_consensus.cc:1067] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:23.397282 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:23.397853 24312 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:23.398196 24312 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:23.422886 24312 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.426671 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:23.427330 24309 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:23.427917 24309 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:23.428797 24309 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.430280 24309 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:23.427220 27001 tablet_copy_client.cc:791] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:23.463738 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:23.466444 24544 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:23.463944 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:23.468863 24425 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:23.469444 24312 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:23.470338 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 127177us
I20260809 23:18:23.472975 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:23.485498 24425 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:23.487089 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:23.488322 26998 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:23.488508 24557 tablet_copy_source_session.cc:205] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:23.488852 26998 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:23.489732 26998 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.492872 27022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac3c46a49aea44a2b2721eb2a750f055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:23.478000 27001 tablet_copy_client.cc:655] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:23.500684 24425 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:23.506534 24544 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:23.507792 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:23.519972 26931 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:23.520771 24309 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:23.520872 27032 ts_tablet_manager.cc:927] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:23.510456 26543 raft_consensus.cc:979] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:23.537153 26931 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:23.553409 27032 tablet_copy_client.cc:316] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.3:37765
I20260809 23:18:23.556243 24555 tablet_copy_service.cc:140] P 808fed3ed3f04882ad757a3db9abfdd9: Received BeginTabletCopySession request for tablet 4e2662351f0e4b14bede1be0deaba6c8 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:46304)
I20260809 23:18:23.556875 27001 tablet_copy_client.cc:531] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:23.556895 24555 tablet_copy_service.cc:161] P 808fed3ed3f04882ad757a3db9abfdd9: Beginning new tablet copy session on tablet 4e2662351f0e4b14bede1be0deaba6c8 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:46304: session id = b9b54436d0e94a5595cdded73fe33da9-4e2662351f0e4b14bede1be0deaba6c8
I20260809 23:18:23.560923 26931 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.553558 27005 raft_consensus.cc:1067] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:23.568809 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:23.569380 24430 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:23.569728 24430 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:23.583611 26966 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:23.589529 27031 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.108s	user 0.000s	sys 0.003s
I20260809 23:18:23.593685 24430 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.595316 27007 tablet_copy_client.cc:791] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:23.595728 24430 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:23.597421 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:23.597982 24312 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:23.598253 24312 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:23.598918 24312 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.602051 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:23.602617 24534 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:23.602717 24312 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:23.595890 27007 tablet_copy_client.cc:655] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:23.595327 24542 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:23.615017 24542 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:23.615754 24542 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.616935 24542 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:23.617991 27001 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:23.644716 27007 tablet_copy_client.cc:531] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:23.648998 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:23.655684 27022 tablet_copy_client.cc:791] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:23.642544 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:18:23.640188 24168 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:23.657612 27007 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:23.667531 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:23.680970 27011 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:23.681995 27011 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:23.682835 27011 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:23.668610 24312 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:18:23.659771 27022 tablet_copy_client.cc:655] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:23.675616 24168 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:23.703338 27022 tablet_copy_client.cc:531] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:23.667905 26966 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:46300 (request call id 13) took 1193 ms. Trace:
I20260809 23:18:23.711333 26966 rpcz_store.cc:276] 0809 23:18:22.474929 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:22.475096 (+   167us) service_pool.cc:227] Handling call
0809 23:18:22.542182 (+ 67086us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22252ada 00007fac22252763 00007fac22252701 00007fac2389bb12 00007fac2389b8ed 00007fac23866f84 00007fac23866743 00007fac287a2f39 00007fac21e8e704 00007fac21e8e692
0809 23:18:22.573766 (+ 31584us) log.cc:842] Serialized 499 byte log entry
0809 23:18:22.766070 (+192304us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b54001e1020. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f9416 00007fac237f2daa 00007fac237f2014 00007fac237da545 00007fac237df771 00007fac237df6f9 00007fac237df689 00007fac237df651 00007fac237de81d 00007fac29b9c204
0809 23:18:22.806685 (+ 40615us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b54001e1020. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f1a4f 00007fac237da545 00007fac237df771 00007fac237df6f9 00007fac237df689 00007fac237df651 00007fac237de81d 00007fac29b9c204 00007fac29b9c039 00007fac209386b6
0809 23:18:22.842713 (+ 36028us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b54001e1020. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237ee771 00007fac237f9220 00007fac237f91c8 00007fac237f9148 00007fac237f9103 00007fac237f838c 00007fac29869206 00007fac29861808 00007fac2383538b 00007fac23837e80
0809 23:18:22.999336 (+156623us) spinlock_profiling.cc:243] Waited 96.8 ms on lock 0x7b54001e1020. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f9416 00007fac237f2daa 00007fac237f2014 00007fac237da545 00007fac237df771 00007fac237df6f9 00007fac237df689 00007fac237df651 00007fac237de81d 00007fac29b9c204
0809 23:18:23.667809 (+668473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37670,"raft.queue_time_us":28977,"raft.run_cpu_time_us":10693,"raft.run_wall_time_us":495369,"spinlock_wait_cycles":319939072,"thread_start_us":68669,"threads_started":1,"wal-append.queue_time_us":69030}
I20260809 23:18:23.720927 24555 tablet_copy_source_session.cc:205] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:23.736963 27032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2662351f0e4b14bede1be0deaba6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:23.760032 27022 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:23.800125 27001 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:23.808648 27001 tablet_bootstrap.cc:492] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:23.829464 27032 tablet_copy_client.cc:791] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:23.830231 27032 tablet_copy_client.cc:655] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:23.809684 27001 ts_tablet_manager.cc:1397] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.192s	user 0.030s	sys 0.010s
I20260809 23:18:23.868968 27032 tablet_copy_client.cc:531] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:23.881987 27001 raft_consensus.cc:348] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:23.883723 27001 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:23.913730 27032 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:23.933112 27007 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:23.886405 27001 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:23.934805 27007 tablet_bootstrap.cc:492] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:23.940167 27007 ts_tablet_manager.cc:1397] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.283s	user 0.027s	sys 0.018s
I20260809 23:18:23.959803 24534 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:23.962364 26849 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:23.954571 27001 ts_tablet_manager.cc:1428] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.007s
I20260809 23:18:23.965734 27007 raft_consensus.cc:348] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:23.982071 24555 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-d3a8a92891fb40e8911ac15230653aab received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:23.982544 24555 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-d3a8a92891fb40e8911ac15230653aab on tablet d3a8a92891fb40e8911ac15230653aab with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:24.000554 24542 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:18:24.002473 24312 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:18:24.006102 25372 raft_consensus.cc:1203] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260809 23:18:23.976266 27007 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:24.004237 24170 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 6, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 6 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:24.008235 24170 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 5, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 5 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:24.005806 27011 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260809 23:18:24.011845 26929 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.014837 27007 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:24.042284 27007 ts_tablet_manager.cc:1428] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.098s	user 0.006s	sys 0.001s
I20260809 23:18:24.087306 24555 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-9429f7311dd744bea4301956e143e9a7 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:24.087836 24555 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-9429f7311dd744bea4301956e143e9a7 on tablet 9429f7311dd744bea4301956e143e9a7 with peer b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:24.118036 27050 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.068s	user 0.000s	sys 0.004s
I20260809 23:18:24.142710 24431 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:24.145457 26849 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:24.173944 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81556.2us
I20260809 23:18:24.239048 27022 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:24.240109 27022 tablet_bootstrap.cc:492] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:24.261441 27022 ts_tablet_manager.cc:1397] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.502s	user 0.020s	sys 0.032s
I20260809 23:18:24.296940 25375 raft_consensus.cc:1203] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4]   Dedup: 3.4->[]
I20260809 23:18:24.335716 27022 raft_consensus.cc:348] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:24.336673 27022 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:24.337321 27022 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:24.341997 27032 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:24.360296 27032 tablet_bootstrap.cc:492] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:24.361564 27032 ts_tablet_manager.cc:1397] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.448s	user 0.033s	sys 0.037s
I20260809 23:18:24.366487 27022 ts_tablet_manager.cc:1428] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.084s	user 0.004s	sys 0.003s
I20260809 23:18:24.392314 24555 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-ac3c46a49aea44a2b2721eb2a750f055 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:24.393435 24555 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-ac3c46a49aea44a2b2721eb2a750f055 on tablet ac3c46a49aea44a2b2721eb2a750f055 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:24.404043 27032 raft_consensus.cc:348] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:24.405432 27032 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:24.413785 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:24.419741 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111527us
W20260809 23:18:24.421275 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 117329us
I20260809 23:18:24.428810 24542 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:18:24.430204 24312 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:18:24.406464 27032 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:24.454175 26967 raft_consensus.cc:1050] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:24.456714 26967 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.461867 27032 ts_tablet_manager.cc:1428] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.100s	user 0.005s	sys 0.004s
I20260809 23:18:24.462473 26848 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.465078 24555 tablet_copy_service.cc:342] P 808fed3ed3f04882ad757a3db9abfdd9: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-4e2662351f0e4b14bede1be0deaba6c8 received from {username='slave'} at 127.0.0.1:46304
I20260809 23:18:24.465508 24555 tablet_copy_service.cc:433] P 808fed3ed3f04882ad757a3db9abfdd9: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-4e2662351f0e4b14bede1be0deaba6c8 on tablet 4e2662351f0e4b14bede1be0deaba6c8 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:24.485124 25372 raft_consensus.cc:1203] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
W20260809 23:18:24.489042 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47936.2us
I20260809 23:18:24.493577 27069 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.564241 24431 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "137c4d66939a48e59f84422216ede16f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:24.564864 24431 raft_consensus.cc:595] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:24.568650 24431 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:24.569185 24431 raft_consensus.cc:595] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:24.574369 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:24.574982 24312 raft_consensus.cc:595] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:24.578171 24431 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:24.579483 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:24.580013 26852 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.581168 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:24.581800 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:24.592324 25372 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:24.594257 24312 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:24.596161 26852 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.599741 26852 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:24.683704 25375 raft_consensus.cc:1203] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260809 23:18:24.689083 26985 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:24.703591 24312 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:24.706087 24431 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:24.725870 24164 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:24.748026 25375 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:24.786252 26849 raft_consensus.cc:979] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:24.786772 26849 raft_consensus.cc:1067] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:24.788443 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:24.788996 24544 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:24.789342 24544 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:24.812072 24544 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.835537 26848 raft_consensus.cc:979] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:24.836107 27069 raft_consensus.cc:1067] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:24.837934 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:24.838634 24312 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:24.839017 24312 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.849009 24312 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.851684 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:24.852269 24422 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:24.852603 24422 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:24.853338 24422 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.854524 24422 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.862022 24422 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:24.863175 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:24.864042 27052 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:24.864682 27052 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:24.865509 27052 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.868368 26967 raft_consensus.cc:1050] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:24.875428 24544 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:24.876598 24312 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:24.876716 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:24.877305 24321 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:24.877753 24321 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:24.877629 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:24.878379 25369 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.885951 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:24.886555 25372 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:24.878603 24321 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.896881 24321 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:24.897300 26967 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.902905 25369 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:24.903930 25372 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:24.904918 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:24.905732 26852 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:24.906231 26852 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:24.907109 26852 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.908361 24321 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:24.911017 26848 raft_consensus.cc:979] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:24.911473 26848 raft_consensus.cc:1067] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:24.913156 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "137c4d66939a48e59f84422216ede16f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:24.913705 24312 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:24.914021 24312 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.948638 24312 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:24.951519 24312 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:24.951689 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:24.952669 24422 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:24.952916 24422 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:24.953490 24422 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:24.954543 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:24.955035 24544 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.958731 24422 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:24.964102 24544 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:24.968499 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:24.961457 24168 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:24.970604 24422 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:24.967760 24542 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:24.976081 27065 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
W20260809 23:18:25.008317 27095 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260809 23:18:25.004207 27065 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:25.059557 27065 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:25.076496 24165 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:18:25.081651 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87144.9us
W20260809 23:18:25.085342 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 108964us
I20260809 23:18:25.096470 25372 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260809 23:18:25.087234 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86159.3us
I20260809 23:18:25.107281 25369 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.107967 27102 raft_consensus.cc:1050] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:25.112082 27008 raft_consensus.cc:1050] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:25.110900 27102 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:25.128288 24321 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.131253 24321 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:25.133621 26932 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:25.137495 24431 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:25.157569 24165 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:25.169237 27008 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:25.176638 27104 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:25.115818 26967 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260809 23:18:25.189107 26932 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:25.192374 26989 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:25.213133 24422 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:25.216049 27103 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260809 23:18:25.229873 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81357.7us
I20260809 23:18:25.236105 25375 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:25.237830 24316 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:25.247308 26985 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260809 23:18:25.252508 26852 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:25.262683 24422 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:25.266875 27099 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:25.269520 26989 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.295598 25369 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.303017 25373 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEARNER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:25.309963 25373 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:25.313835 26932 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:25.314752 24425 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:25.315192 27097 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:25.316479 26932 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:25.334230 24321 raft_consensus.cc:2932] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.354360 24316 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.356554 24316 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:25.357239 24165 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 5 to 6, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.372841 26932 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:25.374432 26852 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.374943 27100 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.395623 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d3a8a92891fb40e8911ac15230653aab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:25.392014 25372 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.400808 26985 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:25.399884 26967 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.418107 24425 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.407975 25375 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.439955 24309 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.449118 25373 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.464993 24431 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.490737 24170 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:25.493631 24422 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.530745 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a8a92891fb40e8911ac15230653aab with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34762
I20260809 23:18:25.542285 25375 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.543716 26985 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:25.546649 24165 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.554590 24534 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:18:25.556420 27133 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260809 23:18:25.632702 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59954.3us
I20260809 23:18:25.643543 27140 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:18:25.685218 24544 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:25.610874 27098 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260809 23:18:25.718267 27140 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:25.723362 27140 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:25.740214 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:25.744295 24431 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:25.788269 27140 ts_tablet_manager.cc:1905] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:25.824735 24431 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.826505 27102 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:25.827291 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:25.828233 24316 raft_consensus.cc:595] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:25.831543 24539 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:25.832374 24539 raft_consensus.cc:595] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:25.833851 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260809 23:18:25.837345 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87028.6us
I20260809 23:18:25.840744 24431 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.859226 25375 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:25.860450 24312 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.844323 24316 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.841594 27140 ts_tablet_manager.cc:1918] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260809 23:18:25.887746 27140 log.cc:1196] T d3a8a92891fb40e8911ac15230653aab P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/d3a8a92891fb40e8911ac15230653aab
I20260809 23:18:25.848744 27145 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260809 23:18:25.910809 27154 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260809 23:18:25.837637 27102 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.917884 27097 raft_consensus.cc:979] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:25.918332 27097 raft_consensus.cc:1067] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:25.919979 24431 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:25.920495 24431 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:25.920809 24431 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:25.960728 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:25.961673 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:25.962620 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9429f7311dd744bea4301956e143e9a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:25.934017 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet d3a8a92891fb40e8911ac15230653aab (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:25.967625 24431 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:25.971601 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95522.7us
I20260809 23:18:25.971621 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:25.972358 24321 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:25.972671 24321 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:25.973371 24321 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:25.973901 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:25.974423 24321 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:25.974915 24431 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:25.974423 24542 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:25.979712 25373 raft_consensus.cc:2932] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:25.975605 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.008749 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:26.012935 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:26.018502 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:26.021662 27142 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:26.022192 27142 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:26.023042 27142 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:26.133893 27164 raft_consensus.cc:979] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:26.134400 27164 raft_consensus.cc:1067] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:26.141249 24309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:26.141784 24309 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:26.142163 24309 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:26.159406 24168 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index 5 to 6, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:26.170962 24309 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:26.172923 24309 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:26.173246 24534 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:26.181010 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:26.181416 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:26.181787 24431 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:26.182310 24542 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:26.182729 24542 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:26.183652 24542 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:26.185120 24542 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:26.192946 24431 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:26.195917 25372 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:26.196635 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:26.198395 27052 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:26.198518 27164 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:26.199141 27052 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:26.203151 24542 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:26.200695 27052 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:26.215430 24170 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 4 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:26.223297 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 9429f7311dd744bea4301956e143e9a7 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40854
I20260809 23:18:26.232869 24429 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:26.238511 24534 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:26.244186 27157 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260809 23:18:26.242373 26967 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.269414 24429 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.288581 24309 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:26.290894 26985 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:26.293197 25372 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:26.279574 27172 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:18:26.296329 27157 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:26.313715 24165 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:26.314139 27104 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.316403 25372 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.322414 24170 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.328984 25373 raft_consensus.cc:2932] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.345714 24309 raft_consensus.cc:2932] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.354156 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4e2662351f0e4b14bede1be0deaba6c8 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:26.369782 24303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2662351f0e4b14bede1be0deaba6c8 with delete_type TABLET_DATA_TOMBSTONED (TS 60d5c69728134c8bb918a900eb6a8c21 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40854
I20260809 23:18:26.375548 27172 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:26.376760 27172 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:26.372901 24165 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 4 to 5, VOTER ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) evicted. New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:26.380033 24152 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ac3c46a49aea44a2b2721eb2a750f055 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:26.397091 27172 ts_tablet_manager.cc:1905] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:26.404829 24404 tablet_service.cc:1514] Processing DeleteTablet for tablet ac3c46a49aea44a2b2721eb2a750f055 with delete_type TABLET_DATA_TOMBSTONED (TS ab5aefaeb3d14775b30a74a2a6d3357f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34762
I20260809 23:18:26.406373 27140 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:18:26.407550 27140 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:26.408278 27140 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:26.412388 27140 ts_tablet_manager.cc:1905] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:26.424026 27172 ts_tablet_manager.cc:1918] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20260809 23:18:26.424707 27172 log.cc:1196] T 9429f7311dd744bea4301956e143e9a7 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/9429f7311dd744bea4301956e143e9a7
I20260809 23:18:26.426836 27172 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:18:26.427873 27172 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:26.428566 27172 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:26.435747 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 9429f7311dd744bea4301956e143e9a7 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:26.445200 27140 ts_tablet_manager.cc:1918] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260809 23:18:26.445786 27140 log.cc:1196] T ac3c46a49aea44a2b2721eb2a750f055 P ab5aefaeb3d14775b30a74a2a6d3357f: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/wals/ac3c46a49aea44a2b2721eb2a750f055
I20260809 23:18:26.448340 24150 catalog_manager.cc:4841] TS ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): tablet ac3c46a49aea44a2b2721eb2a750f055 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:26.456811 27172 ts_tablet_manager.cc:1905] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:26.480527 27172 ts_tablet_manager.cc:1918] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260809 23:18:26.480986 27172 log.cc:1196] T 4e2662351f0e4b14bede1be0deaba6c8 P 60d5c69728134c8bb918a900eb6a8c21: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/wals/4e2662351f0e4b14bede1be0deaba6c8
I20260809 23:18:26.482448 24151 catalog_manager.cc:4841] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): tablet 4e2662351f0e4b14bede1be0deaba6c8 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:26.585193 24309 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:26.587423 26848 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:26.597536 24542 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:26.605485 27160 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:26.669008 24429 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:26.683558 27099 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:26.744046 24534 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:26.748538 27000 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260809 23:18:27.020639 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 90794.9us
W20260809 23:18:27.084319 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61122us
I20260809 23:18:27.314865 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:27.520295 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:27.521807 24309 raft_consensus.cc:595] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:27.525722 24429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:27.526297 24429 raft_consensus.cc:595] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:27.538679 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:27.539554 24309 raft_consensus.cc:595] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:27.541081 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:27.543322 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:27.544183 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:27.646832 24312 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:27.651674 27160 raft_consensus.cc:979] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:27.652465 26848 raft_consensus.cc:1067] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:27.657334 24321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:27.657953 24321 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:27.658398 24321 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.830708 27000 raft_consensus.cc:979] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:27.831439 27000 raft_consensus.cc:1067] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:27.833218 24542 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:27.833740 24542 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:27.834188 24542 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.855022 24542 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.859141 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:27.859874 24320 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:27.860193 24320 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:27.860937 24320 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.861423 24542 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:27.862099 24320 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.870544 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:27.878129 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:27.883909 24431 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.884542 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:27.888104 27164 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:27.888885 27164 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:27.889837 27164 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.891574 24431 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:27.880868 24321 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.907213 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:27.907835 27000 raft_consensus.cc:979] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:27.907845 24431 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:27.908433 24431 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:27.908512 27000 raft_consensus.cc:1067] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:27.909310 24431 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.910082 24542 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:27.910585 24542 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:27.911019 24542 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:27.910563 24431 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.928594 24542 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.931912 24542 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:27.933746 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:27.934260 24320 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:27.934520 24320 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:27.935163 24320 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.936373 24320 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:27.937599 24431 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:27.944907 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:27.945847 27000 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:27.946376 27000 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:27.947412 27000 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:27.946494 24320 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:27.955611 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:27.956987 24321 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:27.958882 26967 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:27.959863 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:27.961880 26967 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:27.962641 24431 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:27.965204 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:27.966032 24542 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:27.990454 24431 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:28.007632 24542 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:28.008675 24534 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260809 23:18:28.009742 24269 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fcb08685ecd486c8f1d951336f39982. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:28.010257 27000 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:28.038293 24431 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 23:18:28.047619 27099 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:28.047644 26967 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:28.137064 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 138106us
I20260809 23:18:28.156819 24165 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:18:28.268055 27206 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.121s	user 0.003s	sys 0.001s
W20260809 23:18:28.359112 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52354.2us
I20260809 23:18:28.363565 24170 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:18:28.460268 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68440.8us
W20260809 23:18:28.510514 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58826.5us
I20260809 23:18:28.519605 27204 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:28.534063 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 395913us
I20260809 23:18:28.586158 24425 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:18:28.599066 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 220288us
I20260809 23:18:28.624393 27212 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
W20260809 23:18:28.682410 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55233.6us
W20260809 23:18:28.683787 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60027.5us
I20260809 23:18:28.684636 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:28.701093 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 195669us
W20260809 23:18:28.702136 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 142501us
I20260809 23:18:28.722196 24431 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260809 23:18:28.755208 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44801us
W20260809 23:18:28.759805 27226 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260809 23:18:28.764289 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50615.7us
I20260809 23:18:28.769094 24429 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:28.769698 27204 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:51366 (request call id 13) took 1129 ms. Trace:
W20260809 23:18:28.780133 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 74685.6us
I20260809 23:18:28.794777 24312 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:18:28.795809 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72700.6us
I20260809 23:18:28.799081 24546 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:28.800719 27154 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260809 23:18:28.784411 24534 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:28.770143 27204 rpcz_store.cc:276] 0809 23:18:27.639927 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:27.640648 (+   721us) service_pool.cc:227] Handling call
0809 23:18:27.874512 (+233864us) log.cc:842] Serialized 499 byte log entry
0809 23:18:28.769634 (+895122us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":3070,"mutex_wait_us":2206,"raft.queue_time_us":7203,"raft.run_cpu_time_us":1982,"raft.run_wall_time_us":15618,"spinlock_wait_cycles":1079424,"thread_start_us":136666,"threads_started":2,"wal-append.queue_time_us":135485}
I20260809 23:18:28.831035 27225 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:28.830374 24424 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:28.840000 27216 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:28.848059 27217 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:28.892035 24312 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:28.916999 24316 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:28.914830 24170 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:28.921929 27208 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:28.952256 24165 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:28.998041 27250 ts_tablet_manager.cc:927] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:29.027097 24422 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 23:18:29.032387 24542 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260809 23:18:29.043180 24269 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4ed6a400f2643da95b8f744ebab6f4c. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:18:29.045590 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 143983us
I20260809 23:18:29.090996 27239 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260809 23:18:29.107518 24534 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:29.108392 24534 raft_consensus.cc:595] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:29.182749 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 83092.9us
W20260809 23:18:29.199731 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94739.6us
W20260809 23:18:29.202306 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60277.6us
I20260809 23:18:29.100039 27217 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.107s
W20260809 23:18:29.286634 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93572.6us
W20260809 23:18:29.286334 27246 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.116s	user 0.005s	sys 0.001s
I20260809 23:18:29.287577 27250 tablet_copy_client.cc:316] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
W20260809 23:18:29.374042 27266 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260809 23:18:29.382287 27257 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260809 23:18:29.457752 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 207228us
W20260809 23:18:29.469841 27259 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.256s	user 0.003s	sys 0.000s
I20260809 23:18:29.535774 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet 5fcb08685ecd486c8f1d951336f39982 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:29.536742 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet 5fcb08685ecd486c8f1d951336f39982 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-5fcb08685ecd486c8f1d951336f39982
W20260809 23:18:29.598506 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66344.8us
I20260809 23:18:29.588665 27219 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:29.652048 27248 raft_consensus.cc:979] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:29.652770 27248 raft_consensus.cc:1067] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:29.675078 24424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:29.680877 24424 raft_consensus.cc:595] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:29.745918 24334 tablet_copy_source_session.cc:205] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:29.731820 24422 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
W20260809 23:18:29.794484 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102903us
I20260809 23:18:29.845798 27250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcb08685ecd486c8f1d951336f39982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:29.850255 25373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37942
W20260809 23:18:29.927613 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58195.1us
W20260809 23:18:29.929167 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113529us
I20260809 23:18:29.932317 25373 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:29.936655 25373 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:29.943885 27289 ts_tablet_manager.cc:927] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:18:29.958240 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 113746us
W20260809 23:18:29.985507 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 139686us
I20260809 23:18:29.999526 24534 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.117707 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:30.118407 24321 raft_consensus.cc:595] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:30.125967 24165 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 2, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 1 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:30.131064 25373 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.140137 25368 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:30.153810 26848 raft_consensus.cc:979] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:30.166344 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:30.168306 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:30.169008 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:30.155251 24422 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:30.165933 27193 raft_consensus.cc:1067] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:30.163986 27289 tablet_copy_client.cc:316] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:30.205747 25373 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:30.222769 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70038.4us
W20260809 23:18:30.220889 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75047us
W20260809 23:18:30.230441 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71558.2us
W20260809 23:18:30.235669 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64670.1us
I20260809 23:18:30.241243 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet e4ed6a400f2643da95b8f744ebab6f4c from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:30.247956 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet e4ed6a400f2643da95b8f744ebab6f4c from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-e4ed6a400f2643da95b8f744ebab6f4c
I20260809 23:18:30.248638 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:30.249248 24425 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:30.258387 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:30.259176 24534 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:30.259531 24534 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:30.262038 24534 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.266136 24534 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:30.268621 24321 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:30.269177 24321 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:30.269455 24321 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:30.282732 24321 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.301528 24334 tablet_copy_source_session.cc:205] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:30.302560 24425 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:30.305706 27289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ed6a400f2643da95b8f744ebab6f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:30.305372 24534 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:30.342730 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:30.318003 27250 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37956 (request call id 522) took 1497 ms. Trace:
I20260809 23:18:30.336540 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:30.347877 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:30.348526 25369 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:30.355100 27245 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:30.355940 27245 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:30.357054 27245 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.346659 24424 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:30.363180 24424 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:30.364094 24424 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.383816 24321 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:30.366321 24539 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 23:18:30.388513 24321 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 23:18:30.390269 25369 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:30.394559 27193 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260809 23:18:30.399773 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3876fa8158d49ec9c1128fbebe6f994. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:30.401046 27230 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:30.401707 24424 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:30.404763 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:30.405655 27304 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:30.407092 27304 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:30.412381 27304 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:30.420920 27278 raft_consensus.cc:979] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:30.421638 27278 raft_consensus.cc:1067] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:18:30.350741 27250 rpcz_store.cc:276] 0809 23:18:28.820702 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:28.820851 (+   149us) service_pool.cc:227] Handling call
0809 23:18:28.998455 (+177604us) ts_tablet_manager.cc:928] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
0809 23:18:29.824518 (+826063us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:30.230339 (+405821us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:30.244547 (+ 14208us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:30.317037 (+ 72490us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:30.317453 (+   416us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:30.317821 (+   368us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:30.317894 (+    73us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1032,"tablet-copy.queue_time_us":176094,"thread_start_us":358885,"threads_started":2}
I20260809 23:18:30.423579 24539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:18:30.423848 27250 tablet_copy_client.cc:791] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:30.424152 24539 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:30.424536 24539 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:30.433743 24539 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:30.440307 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:30.440882 24316 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:30.441169 24316 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:30.424533 27250 tablet_copy_client.cc:655] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:30.442147 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:30.442907 24430 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:30.441893 24316 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:30.442408 24539 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:30.441447 24424 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:30.466162 27250 tablet_copy_client.cc:531] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:30.474761 24316 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:30.485464 24430 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:30.491672 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:30.477751 27230 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.504846 27211 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:30.505535 27211 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:30.512626 24321 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.506665 27211 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:30.567072 24546 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.583472 24316 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:30.582326 24168 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 3 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:30.592774 24321 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:30.627089 27250 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:30.648372 24168 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:30.650974 27289 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:37956 (request call id 532) took 1074 ms. Trace:
I20260809 23:18:30.650247 24168 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: config changed from index -1 to 5, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 4 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.651393 27289 rpcz_store.cc:276] 0809 23:18:29.576153 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:29.579576 (+  3423us) service_pool.cc:227] Handling call
0809 23:18:29.944286 (+364710us) ts_tablet_manager.cc:928] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
0809 23:18:30.304519 (+360233us) tablet_copy_client.cc:361] Tablet copy session begun
0809 23:18:30.567466 (+262947us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:30.583946 (+ 16480us) tablet_copy_client.cc:490] Wrote new tablet metadata
0809 23:18:30.650223 (+ 66277us) tablet_copy_client.cc:411] Wrote new consensus metadata
0809 23:18:30.650543 (+   320us) tablet_replica.cc:205] Creating consensus instance
0809 23:18:30.650823 (+   280us) ts_tablet_manager.cc:983] Replica registered.
0809 23:18:30.650888 (+    65us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":13493,"tablet-copy.queue_time_us":363575,"thread_start_us":306483,"threads_started":2}
I20260809 23:18:30.657122 27289 tablet_copy_client.cc:791] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:30.660122 27289 tablet_copy_client.cc:655] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:30.660168 24430 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:30.663523 27219 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260809 23:18:30.664742 24269 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 -> Peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Couldn't send request to peer b9b54436d0e94a5595cdded73fe33da9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd2b577e2c3e4021bc8ebdfa3462437d. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:18:30.665238 24546 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260809 23:18:30.667199 27304 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:30.670576 27289 tablet_copy_client.cc:531] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:30.704419 24170 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:30.781421 24544 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:30.782328 24424 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:30.784525 27293 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:30.789841 27320 ts_tablet_manager.cc:927] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:30.791311 27320 tablet_copy_client.cc:316] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.2:42099
I20260809 23:18:30.792981 24450 tablet_copy_service.cc:140] P ab5aefaeb3d14775b30a74a2a6d3357f: Received BeginTabletCopySession request for tablet c3876fa8158d49ec9c1128fbebe6f994 from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:47454)
I20260809 23:18:30.793548 24450 tablet_copy_service.cc:161] P ab5aefaeb3d14775b30a74a2a6d3357f: Beginning new tablet copy session on tablet c3876fa8158d49ec9c1128fbebe6f994 from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:47454: session id = b9b54436d0e94a5595cdded73fe33da9-c3876fa8158d49ec9c1128fbebe6f994
I20260809 23:18:30.811882 27219 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.817842 24546 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.802186 27293 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260809 23:18:30.836319 24424 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } } }
I20260809 23:18:30.854923 24450 tablet_copy_source_session.cc:205] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:30.891058 27320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3876fa8158d49ec9c1128fbebe6f994. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:30.896814 27289 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:30.922137 24164 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: config changed from index -1 to 3, NON_VOTER b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) added. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:30.925232 25369 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:30.928378 27278 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:30.938848 24430 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:30.940825 27219 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:31.005540 27320 tablet_copy_client.cc:791] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:31.011957 27320 tablet_copy_client.cc:655] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:31.109418 27320 tablet_copy_client.cc:531] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:31.117488 24316 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:31.119601 27300 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:31.129032 24424 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:31.151873 27300 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260809 23:18:31.182068 27334 ts_tablet_manager.cc:927] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Initiating tablet copy from peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:31.217399 27320 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:31.243650 27334 tablet_copy_client.cc:316] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Beginning tablet copy session from remote peer at address 127.21.255.1:36681
I20260809 23:18:31.250321 24334 tablet_copy_service.cc:140] P 60d5c69728134c8bb918a900eb6a8c21: Received BeginTabletCopySession request for tablet cd2b577e2c3e4021bc8ebdfa3462437d from peer b9b54436d0e94a5595cdded73fe33da9 ({username='slave'} at 127.0.0.1:51376)
I20260809 23:18:31.250896 24334 tablet_copy_service.cc:161] P 60d5c69728134c8bb918a900eb6a8c21: Beginning new tablet copy session on tablet cd2b577e2c3e4021bc8ebdfa3462437d from peer b9b54436d0e94a5595cdded73fe33da9 at {username='slave'} at 127.0.0.1:51376: session id = b9b54436d0e94a5595cdded73fe33da9-cd2b577e2c3e4021bc8ebdfa3462437d
I20260809 23:18:31.265671 24334 tablet_copy_source_session.cc:205] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 23:18:31.269249 27334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2b577e2c3e4021bc8ebdfa3462437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:18:31.273586 27250 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:31.274657 27250 tablet_bootstrap.cc:492] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:31.248334 27289 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:31.282557 27289 tablet_bootstrap.cc:492] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:31.283663 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:31.285637 27250 ts_tablet_manager.cc:1397] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.659s	user 0.048s	sys 0.020s
I20260809 23:18:31.291463 27289 ts_tablet_manager.cc:1397] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.395s	user 0.028s	sys 0.000s
I20260809 23:18:31.317477 27289 raft_consensus.cc:348] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.318514 27289 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:31.319286 27289 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.345111 27289 ts_tablet_manager.cc:1428] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260809 23:18:31.338234 27250 raft_consensus.cc:348] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.351581 27250 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:31.352365 27250 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.356774 27250 ts_tablet_manager.cc:1428] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.071s	user 0.007s	sys 0.001s
I20260809 23:18:31.357384 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-e4ed6a400f2643da95b8f744ebab6f4c received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:31.357869 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-e4ed6a400f2643da95b8f744ebab6f4c on tablet e4ed6a400f2643da95b8f744ebab6f4c with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:31.359838 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-5fcb08685ecd486c8f1d951336f39982 received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:31.360209 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-5fcb08685ecd486c8f1d951336f39982 on tablet 5fcb08685ecd486c8f1d951336f39982 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:31.365787 27334 tablet_copy_client.cc:791] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 0 data blocks...
I20260809 23:18:31.366478 27334 tablet_copy_client.cc:655] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Starting download of 1 WAL segments...
I20260809 23:18:31.384219 25368 raft_consensus.cc:1203] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5]   Dedup: 4.5->[]
I20260809 23:18:31.389977 27334 tablet_copy_client.cc:531] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 23:18:31.412667 27334 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap starting.
I20260809 23:18:31.426062 27320 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:31.447218 27320 tablet_bootstrap.cc:492] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:31.447988 27320 ts_tablet_manager.cc:1397] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.231s	user 0.029s	sys 0.002s
I20260809 23:18:31.450862 27320 raft_consensus.cc:348] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.467875 27320 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:31.483266 27320 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.490623 27320 ts_tablet_manager.cc:1428] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260809 23:18:31.492214 24450 tablet_copy_service.cc:342] P ab5aefaeb3d14775b30a74a2a6d3357f: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-c3876fa8158d49ec9c1128fbebe6f994 received from {username='slave'} at 127.0.0.1:47454
I20260809 23:18:31.492653 24450 tablet_copy_service.cc:433] P ab5aefaeb3d14775b30a74a2a6d3357f: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-c3876fa8158d49ec9c1128fbebe6f994 on tablet c3876fa8158d49ec9c1128fbebe6f994 with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:31.496516 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:31.497257 24312 raft_consensus.cc:595] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:31.500432 24546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cec68a6991264e72a23a8750a420795d"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:31.500998 24546 raft_consensus.cc:595] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:31.503556 24546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:18:31.504051 24546 raft_consensus.cc:595] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:31.505308 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260809 23:18:31.507041 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:31.509382 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:31.573964 27334 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:18:31.592759 27334 tablet_bootstrap.cc:492] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Bootstrap complete.
I20260809 23:18:31.594420 27334 ts_tablet_manager.cc:1397] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent bootstrapping tablet: real 0.182s	user 0.025s	sys 0.003s
I20260809 23:18:31.609810 27334 raft_consensus.cc:348] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.611004 27334 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Initialized, Role: LEARNER
I20260809 23:18:31.611785 27334 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:31.621419 27334 ts_tablet_manager.cc:1428] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260809 23:18:31.623520 24334 tablet_copy_service.cc:342] P 60d5c69728134c8bb918a900eb6a8c21: Request end of tablet copy session b9b54436d0e94a5595cdded73fe33da9-cd2b577e2c3e4021bc8ebdfa3462437d received from {username='slave'} at 127.0.0.1:51376
I20260809 23:18:31.623898 24334 tablet_copy_service.cc:433] P 60d5c69728134c8bb918a900eb6a8c21: ending tablet copy session b9b54436d0e94a5595cdded73fe33da9-cd2b577e2c3e4021bc8ebdfa3462437d on tablet cd2b577e2c3e4021bc8ebdfa3462437d with peer b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:31.628931 27301 raft_consensus.cc:979] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:31.629443 27300 raft_consensus.cc:1067] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:31.634125 24424 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cec68a6991264e72a23a8750a420795d"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34778
I20260809 23:18:31.634850 24424 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:31.635363 24424 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:31.651291 24424 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:31.655011 27300 raft_consensus.cc:979] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:31.655560 27300 raft_consensus.cc:1067] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:31.657128 24312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40856
I20260809 23:18:31.657773 24312 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:31.658165 24312 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:31.668630 24424 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:31.669250 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:31.669507 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:31.671504 24546 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:31.671531 25368 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:31.671983 24546 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:31.673452 24546 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:31.674646 24546 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:31.678884 25368 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:31.682415 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:31.682888 24312 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:31.684639 27256 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:31.693405 27256 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:31.694731 27256 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:31.699517 24312 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:31.692245 25373 raft_consensus.cc:1203] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260809 23:18:31.688907 24546 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:31.703274 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:31.706956 24542 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:31.707239 24542 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:31.707921 24542 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:31.709069 24542 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:31.717267 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:31.717900 24430 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:31.718583 24542 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:31.731307 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:31.732154 27278 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:31.733263 27278 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:31.734200 27278 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:31.739943 24164 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:31.744107 25368 raft_consensus.cc:1203] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
I20260809 23:18:31.752750 24430 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:31.782092 24170 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:31.913389 25373 raft_consensus.cc:1203] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5]   Dedup: 4.5->[]
I20260809 23:18:31.948225 27351 raft_consensus.cc:1050] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:31.955022 27351 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:31.984892 24431 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:31.985533 24546 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:31.987710 27350 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:31.988960 25373 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:31.998432 27351 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:32.001369 27219 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:32.081604 27350 raft_consensus.cc:979] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:32.082158 27285 raft_consensus.cc:1067] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:18:32.081120 27278 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.086081 24430 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:18:32.086545 25369 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.086606 24430 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:32.096251 24431 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.096182 24430 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:32.126247 24430 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:32.128445 24170 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.132378 24430 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:32.133630 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:32.204005 24546 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:32.148926 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:32.208822 24316 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:32.209162 24316 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:32.209908 24316 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:32.155129 25369 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:32.163982 24544 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.180217 27304 raft_consensus.cc:1050] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:32.216740 24316 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:32.219005 24546 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:32.219529 27256 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:32.228466 27304 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:32.245954 24316 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:32.251813 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:32.261627 27256 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:32.262463 27256 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:32.263803 27256 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:32.265753 24316 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:32.286609 27351 raft_consensus.cc:1050] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
I20260809 23:18:32.293790 24430 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:32.297686 27219 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:32.290493 27351 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:32.304881 25369 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:32.306766 27232 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:32.308284 24430 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:32.311357 27219 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260809 23:18:32.320155 24544 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:32.320956 24546 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:32.338124 27256 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:32.343250 27350 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260809 23:18:32.381222 25373 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:32.423265 27285 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260809 23:18:32.527105 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131203us
W20260809 23:18:32.529757 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119356us
W20260809 23:18:32.575654 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 241820us
W20260809 23:18:32.647096 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 202452us
W20260809 23:18:32.651672 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79950.5us
I20260809 23:18:32.651810 24544 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260809 23:18:32.663079 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86432.3us
W20260809 23:18:32.675719 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50834us
I20260809 23:18:32.676517 24164 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:18:32.701912 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49546.5us
I20260809 23:18:32.715830 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:32.739123 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 88854.5us
I20260809 23:18:32.707211 27350 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.824855 24542 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260809 23:18:32.833022 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 127183us
I20260809 23:18:32.848582 25371 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 LEARNER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:32.850384 25369 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:32.863417 27278 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
W20260809 23:18:33.037127 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 172836us
I20260809 23:18:33.072885 27406 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260809 23:18:33.089645 27382 raft_consensus.cc:1050] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: attempting to promote NON_VOTER b9b54436d0e94a5595cdded73fe33da9 to VOTER
W20260809 23:18:33.103847 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 43489.3us
I20260809 23:18:33.115615 27382 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:33.144500 24424 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260809 23:18:33.147022 24431 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260809 23:18:33.144048 24422 raft_consensus.cc:2932] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Committing config change with OpId 4.7: config changed from index 6 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:33.186131 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46382.5us
W20260809 23:18:33.187681 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48657.8us
I20260809 23:18:33.189687 27232 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:33.221058 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 6 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:33.289598 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81003us
W20260809 23:18:33.335536 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107468us
I20260809 23:18:33.348615 24316 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:33.349879 27405 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260809 23:18:33.355119 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5fcb08685ecd486c8f1d951336f39982 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260809 23:18:33.350442 27285 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260809 23:18:33.345446 27382 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:33.400381 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 263735us
I20260809 23:18:33.476711 25369 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEARNER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260809 23:18:33.479545 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132433us
I20260809 23:18:33.480684 24539 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:33.492534 24543 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:33.493428 24314 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260809 23:18:33.497783 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 224767us
I20260809 23:18:33.529686 27365 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260809 23:18:33.540851 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65d375db793a4542808e1cff72fb6617"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:33.541560 24423 raft_consensus.cc:595] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:33.551978 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 112979us
W20260809 23:18:33.610383 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56069.2us
I20260809 23:18:33.450449 27382 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
W20260809 23:18:33.648433 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 154541us
W20260809 23:18:33.692723 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 145659us
W20260809 23:18:33.735638 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 211313us
W20260809 23:18:33.737995 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 366775us
I20260809 23:18:33.829149 27395 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:33.829802 27395 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:33.736078 27389 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.190s
W20260809 23:18:33.906385 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 288952us
I20260809 23:18:33.790242 27411 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
W20260809 23:18:33.908370 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.004s	sys 0.004s
W20260809 23:18:33.909664 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.004s	sys 0.004s
W20260809 23:18:33.969185 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 123296us
W20260809 23:18:33.999078 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 143316us
W20260809 23:18:34.004885 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 293439us
W20260809 23:18:34.021065 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 168002us
I20260809 23:18:33.998325 27435 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:34.091063 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 213134us
W20260809 23:18:34.101621 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 194033us
I20260809 23:18:34.093341 27435 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:34.418211 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 298363us
I20260809 23:18:34.176113 27442 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:34.438155 27442 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:34.213104 27457 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260809 23:18:34.225196 27447 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260809 23:18:34.225819 27292 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:34.216609 27437 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.345s	user 0.000s	sys 0.000s
I20260809 23:18:34.217969 27445 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.217074 27448 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.530157 24544 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:34.211776 27380 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:34.231138 27382 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:34.217481 27443 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:34.166476 27395 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:34.131186 27411 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.324s
I20260809 23:18:34.471946 27372 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:34.473199 27455 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:34.462651 27451 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:34.467891 27431 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:34.221539 27369 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:34.481801 27396 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:34.523164 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 414822us
I20260809 23:18:34.579502 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1657) took 1087 ms. Trace:
I20260809 23:18:34.561529 27395 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:34.556490 27372 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:34.584167 27395 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.586140 27372 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:34.587093 27384 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:34.580071 24430 rpcz_store.cc:276] 0809 23:18:33.480550 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.481583 (+  1033us) service_pool.cc:227] Handling call
0809 23:18:33.482035 (+   452us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:33.749530 (+267495us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:33.749559 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:33.750772 (+  1213us) log.cc:842] Serialized 556 byte log entry
0809 23:18:33.860404 (+109632us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:33.860438 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:34.541254 (+680816us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:34.541330 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:34.567509 (+ 26179us) raft_consensus.cc:1671] finished
0809 23:18:34.567528 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:34.567733 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":614207232,"thread_start_us":109117,"threads_started":1,"wal-append.queue_time_us":248995}
W20260809 23:18:34.596546 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.010s	sys 0.061s
W20260809 23:18:34.598193 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.010s	sys 0.061s
I20260809 23:18:34.602670 27463 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.603574 27463 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:34.588764 27449 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260809 23:18:34.606487 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51134.3us
W20260809 23:18:34.622133 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.012s	sys 0.026s
W20260809 23:18:34.625864 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.012s	sys 0.026s
I20260809 23:18:34.587953 27384 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:34.629715 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.021s	sys 0.004s
W20260809 23:18:34.669925 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.022s	sys 0.005s
I20260809 23:18:34.534073 27445 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.680860 27445 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:34.660771 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.012s	sys 0.017s
W20260809 23:18:34.701699 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.013s	sys 0.018s
W20260809 23:18:34.704015 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 697442us
I20260809 23:18:34.626802 27382 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:34.708570 27400 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:34.542922 27380 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:34.712805 27444 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.713492 27444 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:34.531788 27292 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.536957 27448 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.751328 27470 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:34.750799 27486 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.709492 27400 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.563997 27431 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.547714 27443 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.592867 27396 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.708660 27382 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.558872 27451 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:34.558655 27455 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:34.786247 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.044s	sys 0.138s
I20260809 23:18:34.872257 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 599) took 1055 ms. Trace:
I20260809 23:18:34.866364 27463 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:34.873931 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1906) took 1523 ms. Trace:
W20260809 23:18:34.899425 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.081s	sys 0.204s
I20260809 23:18:34.899828 25369 rpcz_store.cc:276] 0809 23:18:33.816677 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.816885 (+   208us) service_pool.cc:227] Handling call
0809 23:18:33.817548 (+   663us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:33.820810 (+  3262us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:33.820847 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:33.822831 (+  1984us) log.cc:842] Serialized 556 byte log entry
0809 23:18:33.907261 (+ 84430us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:33.907334 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:33.907478 (+   144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:34.563470 (+655992us) raft_consensus.cc:1671] finished
0809 23:18:34.563502 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:34.872171 (+308669us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":71776,"threads_started":1,"wal-append.queue_time_us":200843}
I20260809 23:18:34.874965 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:34.900553 24546 rpcz_store.cc:276] 0809 23:18:33.350577 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.372026 (+ 21449us) service_pool.cc:227] Handling call
0809 23:18:33.372707 (+   681us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:33.384932 (+ 12225us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:33.384970 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:33.388877 (+  3907us) log.cc:842] Serialized 556 byte log entry
0809 23:18:33.910047 (+521170us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:33.910089 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:33.910178 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:34.727803 (+817625us) raft_consensus.cc:1671] finished
0809 23:18:34.727827 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:34.873872 (+146045us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":520847,"threads_started":1,"wal-append.queue_time_us":1073403}
I20260809 23:18:34.872084 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1658) took 1367 ms. Trace:
I20260809 23:18:34.939631 24422 rpcz_store.cc:276] 0809 23:18:33.504619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.505539 (+   920us) service_pool.cc:227] Handling call
0809 23:18:33.506201 (+   662us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:33.506621 (+   420us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:33.506661 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:33.510053 (+  3392us) log.cc:842] Serialized 556 byte log entry
0809 23:18:33.649433 (+139380us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:33.649481 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:33.649566 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:34.558776 (+909210us) raft_consensus.cc:1671] finished
0809 23:18:34.558804 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:34.872017 (+313213us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139074,"threads_started":1,"wal-append.queue_time_us":312478}
W20260809 23:18:34.941542 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 259730us
I20260809 23:18:34.724509 27442 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:34.980270 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:34.945360 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:35.077039 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:34500 (request call id 19) took 1600 ms. Trace:
W20260809 23:18:34.786433 27430 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260809 23:18:34.955611 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:35.309566 24543 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:35.312601 24423 rpcz_store.cc:276] 0809 23:18:33.476234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.477833 (+  1599us) service_pool.cc:227] Handling call
0809 23:18:35.076429 (+1598596us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b540035d520. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5dc0 00007fac2385a9c2 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86
0809 23:18:35.076913 (+   484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179840768}
W20260809 23:18:35.302115 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.011s	sys 0.013s
W20260809 23:18:35.345444 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.012s	sys 0.013s
I20260809 23:18:34.902695 27451 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:34.785728 27395 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.343410 27382 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.009907 27365 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.344602 27211 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.348302 27365 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.348403 27211 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.263142 27474 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:34.946514 27380 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.264371 27426 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.263687 27444 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:35.376452 27211 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:35.158682 27489 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.379797 27211 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.376143 27426 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.157814 27492 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.381767 27211 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:35.390619 27485 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.594s	user 0.002s	sys 0.000s
I20260809 23:18:35.006963 27480 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.409135 27211 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.411069 27400 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.362429 27482 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:35.374213 27443 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:34.953971 27470 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.457674 27370 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:34.995514 27486 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.392233 27462 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.472020 27486 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:35.451496 27431 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.483239 27232 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.487078 27232 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.458540 27370 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.388842 27474 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.552994 27470 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.429075 27489 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.543960 27350 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.487418 27458 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.560657 27350 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.580129 27426 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.608283 27398 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:35.555394 27395 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.603116 25373 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:35.557231 27442 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.561820 27431 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.647758 27476 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.670817 27475 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.671515 27474 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.526117 27435 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.464263 27482 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.611908 27426 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.446947 27492 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.609155 27398 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.515398 27473 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.494346 27462 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.671648 27475 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.715128 27408 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:35.652885 27470 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.454613 27480 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:35.733242 27369 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.045s	user 0.014s	sys 0.000s
W20260809 23:18:35.780015 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 177490us
I20260809 23:18:35.782603 27350 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.787921 27232 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.730643 27193 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.642237 27442 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.793159 27473 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.653188 27443 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.613951 27458 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:35.804535 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.017s	sys 0.037s
I20260809 23:18:35.639602 27395 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.488951 27365 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.825510 27475 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:35.826691 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.033s	sys 0.023s
I20260809 23:18:35.789366 27498 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:35.879439 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.925s	user 0.033s	sys 0.023s
I20260809 23:18:35.650563 27431 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.880405 27193 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.882339 27400 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:35.817497 24544 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.105s	user 0.071s	sys 0.130s
W20260809 23:18:35.860787 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.018s	sys 0.037s
W20260809 23:18:35.891336 24544 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.179s	user 0.072s	sys 0.130s
W20260809 23:18:35.891481 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.0189e+06us
I20260809 23:18:35.892021 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1919) took 1362 ms. Trace:
I20260809 23:18:35.892364 24544 rpcz_store.cc:276] 0809 23:18:34.529440 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:34.529585 (+   145us) service_pool.cc:227] Handling call
0809 23:18:34.529874 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:34.530057 (+   183us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:18:34.692968 (+162911us) log.cc:842] Serialized 14 byte log entry
0809 23:18:35.891683 (+1198715us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:35.891732 (+    49us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:35.891775 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:35.891841 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:35.891970 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18978,"raft.queue_time_us":91495,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":59701,"thread_start_us":1179260,"threads_started":1}
I20260809 23:18:35.897166 27473 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.648813 27476 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.796319 27426 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.915083 27372 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:35.827589 27462 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.826385 27470 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.929618 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:36.076424 27372 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.059070 27431 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:36.010459 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.914s	user 0.024s	sys 0.053s
W20260809 23:18:36.079931 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.164s	user 0.024s	sys 0.061s
I20260809 23:18:36.017484 27370 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.930675 27498 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:35.978564 27193 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.086006 27193 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:35.982190 27400 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:36.087734 25323 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.933s (SENT) [suppressed 2 similar messages]
I20260809 23:18:36.086843 27193 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.089545 27400 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.090289 27400 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.063159 27480 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.980667 27489 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.058357 27395 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:36.034225 27398 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.096596 27470 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.097374 27470 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.098019 27470 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.032380 27458 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:35.888311 27408 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:35.887981 27443 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.119145 27443 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.995664 27428 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:36.121320 27428 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.063911 27482 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:35.932932 27442 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:36.045261 27476 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.065992 27365 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260809 23:18:36.126593 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 1.
I20260809 23:18:36.080750 27426 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.137580 27428 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.139708 27372 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.139561 27498 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.143879 27193 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:36.151015 25373 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.530s	user 0.005s	sys 0.006s
I20260809 23:18:36.154346 27193 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:36.142357 27400 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.134526 27408 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:36.167452 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.000s	sys 0.019s
W20260809 23:18:36.169250 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.169s	user 0.000s	sys 0.019s
W20260809 23:18:36.195604 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.489s	user 0.002s	sys 0.034s
W20260809 23:18:36.196197 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.490s	user 0.002s	sys 0.034s
W20260809 23:18:36.202117 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.004s	sys 0.015s
W20260809 23:18:36.202466 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.004s	sys 0.015s
W20260809 23:18:36.202917 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.167s	user 0.005s	sys 0.033s
W20260809 23:18:36.208927 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.215s	user 0.005s	sys 0.033s
W20260809 23:18:36.209748 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.777s	user 0.022s	sys 0.066s
W20260809 23:18:36.210074 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.777s	user 0.022s	sys 0.066s
I20260809 23:18:36.166181 27193 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:36.334571 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.018s	sys 0.023s
W20260809 23:18:36.521530 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.021s	sys 0.032s
I20260809 23:18:36.224354 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.270167 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:36.344165 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:36.338068 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:36.222154 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:36.531282 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.535105 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.536453 25366 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:36.546677 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.575148 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.512107 27371 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:36.582232 27371 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.227003 27428 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:36.583824 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.584880 27371 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:36.583765 27428 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:36.364894 27365 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:36.586526 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.586436 27365 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.219151 27499 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:36.596663 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.015s	sys 0.049s
W20260809 23:18:36.606127 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.015s	sys 0.050s
I20260809 23:18:36.588124 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:36.595826 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.576352 24316 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 1.
I20260809 23:18:36.516975 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.615103 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:36.611783 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:36.219933 27408 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:36.620086 24314 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:36.227465 27372 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:36.631590 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.011s	sys 0.014s
W20260809 23:18:36.641882 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.011s	sys 0.014s
W20260809 23:18:36.642280 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 554545us
I20260809 23:18:36.279464 27500 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:36.650270 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.595381 27435 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:36.619501 27369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.004s	sys 0.020s
I20260809 23:18:36.658347 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.612895 27371 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:36.656338 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.685523 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:36.660009 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:36.712206 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.002s	sys 0.011s
W20260809 23:18:36.712666 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.002s	sys 0.011s
I20260809 23:18:36.641556 27230 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:36.783201 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140135us
I20260809 23:18:36.629244 27501 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:36.787588 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 138013us
W20260809 23:18:36.658449 27369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.005s	sys 0.020s
W20260809 23:18:36.677057 27485 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s	user 0.018s	sys 0.040s
I20260809 23:18:36.691553 27464 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:36.797374 27464 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.800374 27292 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.801117 27464 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.802394 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.658026 27406 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:36.807991 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.809515 27439 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.810189 27439 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.812863 27439 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.814990 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 606) took 1212 ms. Trace:
I20260809 23:18:36.743096 27304 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.691524 27441 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.815981 27400 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:36.755774 27371 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.651983 27372 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.927047 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:36.943048 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.376s	user 0.043s	sys 0.109s
W20260809 23:18:36.806332 27485 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.267s	user 0.018s	sys 0.040s
I20260809 23:18:36.806150 27428 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.784171 27508 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:36.675881 27435 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.085058 27504 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:36.648478 27408 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.815478 25373 rpcz_store.cc:276] 0809 23:18:35.602281 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:35.602402 (+   121us) service_pool.cc:227] Handling call
0809 23:18:35.602695 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:35.602967 (+   272us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:18:36.155602 (+552635us) log.cc:842] Serialized 14 byte log entry
0809 23:18:36.814536 (+658934us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:36.814607 (+    71us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:36.814664 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:36.814749 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:36.814905 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101350,"raft.queue_time_us":103083,"thread_start_us":557137,"threads_started":1}
I20260809 23:18:36.782657 27442 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:36.629783 27499 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.783128 27470 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.819389 27464 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:36.815043 27512 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:36.998216 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.046s	sys 0.050s
W20260809 23:18:37.123025 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.047s	sys 0.050s
I20260809 23:18:36.792264 27230 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:36.692140 27500 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.909386 27304 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:37.026873 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.460s	user 0.044s	sys 0.109s
W20260809 23:18:37.027166 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.375s	user 0.049s	sys 0.054s
W20260809 23:18:37.151096 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.509s	user 0.050s	sys 0.054s
I20260809 23:18:37.026909 25370 raft_consensus.cc:1799] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:37.008702 27193 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:36.993975 27441 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.083886 27372 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.084165 27508 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:36.795661 27501 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.105686 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 347) took 2410 ms. Trace:
I20260809 23:18:37.109687 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.112502 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:36.898407 27406 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.006827 27400 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:37.134939 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 310230us
I20260809 23:18:36.991402 27507 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:36.955864 27522 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.144484 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:37.148164 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:37.083923 27371 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.155085 25370 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:18:37.157470 27530 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.169620 27464 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:37.171157 27519 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260809 23:18:37.027926 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.171793 27408 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.124248 27442 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.105406 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.085861 27504 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.487447 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:37.430420 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 562380us
I20260809 23:18:37.473191 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:37.223397 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54220.6us
I20260809 23:18:37.131111 27396 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.474527 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 742) took 2804 ms. Trace:
I20260809 23:18:37.174171 27499 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.488536 27529 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.489144 27230 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.489401 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.489773 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.492023 25363 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:37.150625 27365 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.530825 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.567257 25373 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:37.565197 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.568128 25372 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
W20260809 23:18:37.476747 27527 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.302s	user 0.000s	sys 0.000s
I20260809 23:18:37.174341 27435 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.555502 27455 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.137116 27512 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.572705 27529 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.585178 27500 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.585451 27504 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.585520 27464 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.531951 27530 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.590399 27512 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:37.624385 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.001s	sys 0.002s
W20260809 23:18:37.625567 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.001s	sys 0.002s
W20260809 23:18:37.628739 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.011s	sys 0.004s
W20260809 23:18:37.629191 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.011s	sys 0.004s
W20260809 23:18:37.157305 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.025s	user 0.005s	sys 0.009s
I20260809 23:18:37.491309 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:37.642458 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.008s	sys 0.059s
W20260809 23:18:37.643023 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.008s	sys 0.060s
I20260809 23:18:37.643034 24435 raft_consensus.cc:1799] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:37.643743 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 838797us
I20260809 23:18:37.644501 24435 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:18:37.512198 24430 rpcz_store.cc:276] 0809 23:18:34.695068 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:34.696527 (+  1459us) service_pool.cc:227] Handling call
0809 23:18:34.696902 (+   375us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:35.518630 (+821728us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:35.518670 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:35.518724 (+    54us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:35.518775 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:37.105428 (+1586653us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:37.105472 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:37.105609 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1889236864}
W20260809 23:18:37.641896 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.509s	user 0.008s	sys 0.014s
I20260809 23:18:37.657176 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:37.657301 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:37.657259 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:37.681062 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.887s	user 0.012s	sys 0.090s
W20260809 23:18:37.681784 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.013s	sys 0.090s
W20260809 23:18:37.684197 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123477us
I20260809 23:18:37.536924 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:37.529896 25368 rpcz_store.cc:276] 0809 23:18:34.669832 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:34.670831 (+   999us) service_pool.cc:227] Handling call
0809 23:18:34.727386 (+ 56555us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:34.850672 (+123286us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:34.873822 (+ 23150us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:34.903036 (+ 29214us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:34.915617 (+ 12581us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:37.474179 (+2558562us) spinlock_profiling.cc:243] Waited 50.6 ms on lock 0x7b540055f854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:37.474235 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:37.474434 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116451712}
I20260809 23:18:37.537549 27400 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.711177 27369 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:37.723603 27230 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.727540 27376 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.728706 27304 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.728425 27230 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.728763 27365 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.723470 27370 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:37.537386 27507 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.808239 27406 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.712261 27441 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.810285 27411 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.808135 27504 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:37.775676 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 352) took 1864 ms (client timeout 1621 ms). Trace:
I20260809 23:18:37.778265 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 699) took 1541 ms. Trace:
W20260809 23:18:37.761276 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 224371us
W20260809 23:18:37.857671 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 224278us
I20260809 23:18:37.861054 27372 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.861330 27499 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:37.863642 27532 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.863463 27499 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: NON_VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: true } }
I20260809 23:18:37.864130 27365 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.864691 27532 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.766251 27439 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:37.851936 25375 rpcz_store.cc:276] 0809 23:18:36.236824 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:36.237722 (+   898us) service_pool.cc:227] Handling call
0809 23:18:36.519212 (+281490us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:37.175449 (+656237us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:37.175493 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:37.175547 (+    54us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:37.175599 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:37.777742 (+602143us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:37.777800 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:37.778039 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1508570368}
I20260809 23:18:37.860347 27230 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.537607 27522 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.888526 27544 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.795560 27369 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.889885 27504 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.891422 27492 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.892172 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2198) took 1237 ms. Trace:
I20260809 23:18:37.753065 27409 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:37.830878 24423 rpcz_store.cc:269] 0809 23:18:35.911647 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:35.912404 (+   757us) service_pool.cc:227] Handling call
0809 23:18:37.775202 (+1862798us) spinlock_profiling.cc:243] Waited 49.4 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:37.775436 (+   234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113506688}
I20260809 23:18:37.898236 27530 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.899354 27369 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.900094 27544 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.742642 27495 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:37.869526 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:37.866849 27365 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.909621 27448 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.901388 24165 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 350) took 4372 ms. Trace:
I20260809 23:18:37.756814 27408 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:37.917411 27301 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:37.866837 27532 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:37.893409 27439 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.916124 27553 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.911082 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.916996 27463 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:37.914397 24165 rpcz_store.cc:276] 0809 23:18:33.518405 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.518584 (+   179us) service_pool.cc:227] Handling call
0809 23:18:37.890601 (+4372017us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":682,"reported_tablets":2}
I20260809 23:18:37.728724 27376 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.917938 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:37.918228 27301 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.867388 27529 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.863483 27411 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:37.862746 27372 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.867398 27370 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:37.911303 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:37.867903 27538 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:37.925453 27439 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:37.915287 27499 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.950220 27448 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:37.936041 27530 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.954281 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.917884 27507 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.150609 24168 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 338) took 4714 ms. Trace:
W20260809 23:18:38.044962 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57747.8us
I20260809 23:18:38.202890 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:37.901077 24309 rpcz_store.cc:276] 0809 23:18:36.654538 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:36.659098 (+  4560us) service_pool.cc:227] Handling call
0809 23:18:36.659446 (+   348us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:37.628062 (+968616us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:37.628154 (+    92us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:37.628213 (+    59us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:37.628271 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:37.891596 (+263325us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:37.891661 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:37.891915 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2226081152}
I20260809 23:18:38.158125 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:37.982700 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.015126 24425 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:38.061733 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1201) took 1272 ms. Trace:
I20260809 23:18:38.215065 24168 rpcz_store.cc:276] 0809 23:18:33.436027 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.436171 (+   144us) service_pool.cc:227] Handling call
0809 23:18:38.150401 (+4714230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":323378,"reported_tablets":3}
I20260809 23:18:38.040314 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.015647 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.195688 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.196427 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:37.995078 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.169224 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.199137 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.037704 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.012040 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.041090 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.149504 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:38.015345 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122533us
I20260809 23:18:38.043977 24170 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 368) took 4743 ms. Trace:
I20260809 23:18:38.054983 24164 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 236) took 4874 ms. Trace:
W20260809 23:18:38.359925 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 508234us
W20260809 23:18:38.132978 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107946us
I20260809 23:18:38.411463 24320 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:18:38.572075 24170 rpcz_store.cc:276] 0809 23:18:33.300663 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.300866 (+   203us) service_pool.cc:227] Handling call
0809 23:18:33.410952 (+110086us) spinlock_profiling.cc:243] Waited 9.49 ms on lock 0x7b5400000910. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900d92 00007fac298517c9 00007fac297e9c36 00007fac297bac28 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec
0809 23:18:38.043712 (+4632760us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":150668,"reported_tablets":1,"spinlock_wait_cycles":21832320}
I20260809 23:18:38.585593 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.586560 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.203410 27501 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.573588 24321 rpcz_store.cc:276] 0809 23:18:36.789317 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:36.792540 (+  3223us) service_pool.cc:227] Handling call
0809 23:18:38.059750 (+1267210us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:38.061295 (+  1545us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":825665408}
W20260809 23:18:38.048552 27531 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.541s	user 0.005s	sys 0.001s
I20260809 23:18:37.949182 27392 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.036216 27510 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:38.227458 27448 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:38.537921 27461 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.977152 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.157367 27301 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:37.981845 27411 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.040350 27508 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.168653 27504 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.601490 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:37.997903 27376 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.363412 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.426710 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:37.979656 27482 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:37.998015 27442 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.133221 27397 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:37.955289 27369 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.086220 27370 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.984910 27384 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.157714 27428 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.254463 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.061098 27372 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:37.992092 27300 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:38.253518 27400 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.173907 27522 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.039412 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.610553 24317 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:38.608865 25375 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.254742 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.203014 27439 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.012323 27472 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.598488 27448 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:38.617971 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.630566 27439 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.631083 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.634699 27428 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.636780 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.637617 24541 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:38.637933 27428 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.638583 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.014839 27463 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:38.617918 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.615317 24429 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:18:38.043591 27409 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:38.623497 24430 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:38.604360 27544 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.622697 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.334378 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:38.603401 27510 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:38.200946 27553 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.622424 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:38.466918 27530 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:38.627681 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.629113 27472 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.255968 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcb08685ecd486c8f1d951336f39982 with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:38080
I20260809 23:18:38.622227 24424 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:37.975998 27495 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:38.237323 27538 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.697378 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.752694 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 697) took 2529 ms. Trace:
I20260809 23:18:38.753839 27538 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.753126 25371 rpcz_store.cc:276] 0809 23:18:36.223607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:36.223726 (+   119us) service_pool.cc:227] Handling call
0809 23:18:36.320279 (+ 96553us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:36.320775 (+   496us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:36.320810 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:36.582180 (+261370us) log.cc:842] Serialized 556 byte log entry
0809 23:18:36.997636 (+415456us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:36.997686 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:36.997771 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:37.983469 (+985698us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:18:38.752258 (+768789us) raft_consensus.cc:1671] finished
0809 23:18:38.752275 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:38.752513 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31991040,"thread_start_us":414891,"threads_started":1,"wal-append.queue_time_us":572321}
I20260809 23:18:38.620082 27549 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:38.648675 24425 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:38.754987 27549 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.689515 27553 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.685709 27439 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:38.641669 27369 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.674444 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:38.653257 27408 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.733s	user 0.005s	sys 0.009s
I20260809 23:18:38.625818 27461 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.703559 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.730209 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:38.679340 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76544.3us
I20260809 23:18:38.702546 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.696590 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.663501 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.659255 24541 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.659364 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.756848 27439 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:38.746353 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:18:38.675340 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.677093 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.650141 24309 raft_consensus.cc:595] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:38.743806 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.690037 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.716423 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.663288 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.693565 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:38.762212 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.666237 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.679476 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.650465 27510 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:38.765149 24431 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:38.696769 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:38.766916 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.767375 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.767598 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.767743 24435 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:38.767958 27461 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.768142 24425 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:38.768339 24545 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:38.768770 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.769049 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.757797 27549 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.760564 24542 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:38.573268 24164 rpcz_store.cc:276] 0809 23:18:33.134229 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:33.139070 (+  4841us) service_pool.cc:227] Handling call
0809 23:18:33.152387 (+ 13317us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5400002dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac297bad6c 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec 00007fac27bc60a2 00007fac21c8a94c 00007fac21c89d86
0809 23:18:37.985089 (+4832702us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:18:33.909128 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:18:34.225538 (+316410us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:18:34.225607 (+    69us) write_op.cc:435] Acquired schema lock
0809 23:18:34.225667 (+    60us) tablet.cc:660] Decoding operations
0809 23:18:34.226389 (+   722us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:18:34.226476 (+    87us) write_op.cc:641] Partition lock acquired for write op
0809 23:18:34.226534 (+    58us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:18:34.226856 (+   322us) tablet.cc:699] Row locks acquired
0809 23:18:34.226910 (+    54us) write_op.cc:260] PREPARE: finished
0809 23:18:34.227138 (+   228us) write_op.cc:270] Start()
0809 23:18:34.227297 (+   159us) write_op.cc:276] Timestamp: P: 1786317514227088 usec, L: 0
0809 23:18:34.227336 (+    39us) op_driver.cc:331] REPLICATION: starting
0809 23:18:34.228424 (+  1088us) log.cc:842] Serialized 533 byte log entry
0809 23:18:37.053617 (+2825193us) op_driver.cc:444] REPLICATION: finished
0809 23:18:37.587267 (+533650us) write_op.cc:301] APPLY: starting
0809 23:18:37.587406 (+   139us) tablet.cc:1365] starting BulkCheckPresence
0809 23:18:37.587745 (+   339us) tablet.cc:1368] finished BulkCheckPresence
0809 23:18:37.587992 (+   247us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:18:37.588319 (+   327us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:18:37.588919 (+   600us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0809 23:18:37.588975 (+    56us) write_op.cc:312] APPLY: finished
0809 23:18:37.589367 (+   392us) log.cc:842] Serialized 23 byte log entry
0809 23:18:37.589783 (+   416us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:37.590115 (+   332us) write_op.cc:454] Released schema lock
0809 23:18:37.590144 (+    29us) write_op.cc:341] FINISH: Updating metrics
0809 23:18:37.590354 (+   210us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:37.590373 (+    19us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":269512,"reported_tablets":1,"spinlock_wait_cycles":26805120,"child_traces":[["op",{"apply.queue_time_us":461436,"apply.run_cpu_time_us":2242,"apply.run_wall_time_us":3437,"num_ops":1,"prepare.queue_time_us":536323,"prepare.run_cpu_time_us":15580,"prepare.run_wall_time_us":659264,"raft.queue_time_us":1429787,"raft.run_cpu_time_us":39254,"raft.run_wall_time_us":600717,"replication_time_us":2800768,"thread_start_us":2438215,"threads_started":4,"wal-append.queue_time_us":406471}]]}
I20260809 23:18:38.771775 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:38.772825 24311 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 23:18:38.769301 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.649138 25365 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:18:38.773838 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.691627 27409 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:38.776197 24317 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:38.776471 27369 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.783296 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.783517 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.784268 24317 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:18:38.781157 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.776890 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.777925 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.778541 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.721462 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.660009 27472 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:38.762557 27439 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:38.789997 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.677659 27550 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.776842 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.751658 25366 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.793985 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:38.786166 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92741.3us
I20260809 23:18:38.785485 24422 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:38.795545 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.795884 24424 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:38.783360 25372 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:38.798877 27472 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:38.799363 27472 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:38.799430 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.799844 27472 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:38.799916 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.800660 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.801120 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.801753 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.802150 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.803354 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:18:38.793670 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.792384 25368 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:38.792315 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.792755 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.806573 25375 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:38.765571 24323 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.804776 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.788687 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:38.793392 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96877.3us
I20260809 23:18:38.783407 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.799904 27448 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.780786 24435 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:38.651844 27300 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:38.805351 24320 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:38.811004 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:38.811970 24309 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:38.812686 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:38.813035 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.813571 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:38.815232 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.815490 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.815810 27530 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:38.816748 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.809489 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.815846 24535 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:38.818246 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.819279 27369 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:38.819875 27369 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:38.820062 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.820278 27369 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:38.821238 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.822355 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.819167 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:38.823458 24541 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:38.819164 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.824738 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.827898 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.829155 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.830148 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.671160 27397 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.807112 27409 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:38.833254 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.833468 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.834146 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.834766 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:38.835556 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.817567 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.838202 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.833292 24539 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:38.841528 25371 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 2: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:38.842168 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.769838 27392 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.846634 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.847645 24309 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 23:18:38.840471 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.832064 27428 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:38.849050 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.850770 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.853066 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.676290 24428 raft_consensus.cc:2932] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:38.824486 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:38.856094 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.858958 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.836969 27409 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:38.859015 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:38.859061 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.025s	sys 0.039s
W20260809 23:18:38.864730 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.025s	sys 0.039s
I20260809 23:18:38.848531 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:38.788923 27482 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.866349 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.857445 24313 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:38.862591 24423 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:38.871091 24425 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:38.872151 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.874523 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.874672 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:38.874564 24535 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:38.877241 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.878468 24541 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:38.855171 27428 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:38.881986 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.883121 24321 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:38.884809 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:38.867415 27409 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:38.887591 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.855357 27530 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:38.885975 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.892649 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.891659 24542 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:38.894474 24534 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260809 23:18:38.897090 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.836027 27550 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:38.877084 27549 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:38.895112 24545 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:38.899075 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:38.899240 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:38.874166 27463 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:38.886065 27428 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:38.904842 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.906729 24543 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:18:38.906795 27549 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.894459 27301 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:38.876339 25363 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:38.913424 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:38.916635 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.048s	sys 0.088s
W20260809 23:18:38.919998 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.048s	sys 0.088s
I20260809 23:18:38.921835 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.922449 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:38.914819 27549 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:38.925930 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.933043 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.915644 27301 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:38.919526 27538 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:38.942760 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:38.943944 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.951627 24320 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 1.
I20260809 23:18:38.955608 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.957192 27551 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:38.957772 27551 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:38.958189 27551 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:38.958557 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.944613 27301 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:38.944365 27448 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:38.967242 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.963449 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.965297 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.944855 27538 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:38.946798 27370 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:38.966581 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:38.969334 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:39.172963 27566 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:39.040643 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 443887us
I20260809 23:18:38.956465 24539 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:38.950480 27472 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:39.026304 24425 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:39.055940 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:38.973691 27538 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.135531 27482 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.041215 27300 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:39.022627 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47420.5us
I20260809 23:18:39.041033 27392 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.104249 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.106742 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.023756 24422 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:39.039820 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:39.020581 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57401.8us
I20260809 23:18:39.067699 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.057870 27442 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.051054 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.188853 24545 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:38.937232 27530 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:39.011940 27397 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.021206 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:38.977222 27370 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.071817 27439 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.035542 27515 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 23:18:39.180536 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 253270us
I20260809 23:18:39.198907 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.176188 27566 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.097386 27550 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.190013 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:39.195955 24540 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:39.195999 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:39.192353 27558 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.002s	sys 0.000s
I20260809 23:18:39.197464 27461 raft_consensus.cc:979] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:39.200374 27370 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.203310 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.199306 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.182397 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:39.210335 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.210953 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:39.207369 27566 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.214345 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.210062 27550 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:39.228143 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.234369 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.202057 27515 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.235960 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.237272 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.228964 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.233858 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:39.241792 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.247648 27577 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:39.248073 27577 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:39.248474 27577 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.239104 27515 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:39.250727 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 146567us
I20260809 23:18:39.244603 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:39.246171 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:39.254629 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44357.8us
I20260809 23:18:39.261668 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:39.262333 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.263751 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.264451 24546 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:39.262367 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:39.267143 24312 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.270226 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.271965 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.276373 27408 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.276897 27408 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.279326 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:39.252115 24498 connection.cc:232] RPC call timeout handler was delayed by -1.2930046028923243s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
I20260809 23:18:39.281486 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.277436 27408 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.213415 27442 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.283891 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:39.291781 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 410746us
I20260809 23:18:39.292600 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:39.281731 27510 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.500s	user 0.010s	sys 0.002s
I20260809 23:18:39.285274 27544 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.307796 27571 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.308398 27571 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.308871 27571 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.318099 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.320966 27580 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.321580 27580 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.322101 27580 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.323191 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.323412 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.324792 27553 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.325320 27553 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.309494 27369 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.323683 27536 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:39.333801 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:39.333797 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 112867us
I20260809 23:18:39.297065 27562 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:18:39.347298 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.293538 27300 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.353011 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.297732 27568 raft_consensus.cc:1067] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Signalling peer b9b54436d0e94a5595cdded73fe33da9 to start an election
I20260809 23:18:39.312101 27397 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:39.348090 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 1975) took 1332 ms. Trace:
I20260809 23:18:39.355965 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:39.357519 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1333998641930521s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2211s for a response
W20260809 23:18:39.369316 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1323571661487222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2211s for a response
W20260809 23:18:39.369758 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0548163671046495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260809 23:18:39.280100 27550 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.325810 27553 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:39.381686 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 117778us
I20260809 23:18:39.354394 27522 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.366802 24544 rpcz_store.cc:276] 0809 23:18:37.994281 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:37.996773 (+  2492us) service_pool.cc:227] Handling call
0809 23:18:37.997611 (+   838us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:37.998194 (+   583us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:37.998226 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:38.086631 (+ 88405us) log.cc:842] Serialized 556 byte log entry
0809 23:18:38.865094 (+778463us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:38.865130 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:38.865193 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:39.326311 (+461118us) raft_consensus.cc:1671] finished
0809 23:18:39.326331 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:39.326464 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":777989,"threads_started":1,"wal-append.queue_time_us":853953}
W20260809 23:18:39.385149 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 116482us
W20260809 23:18:39.387147 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 176265us
I20260809 23:18:39.382253 24312 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:39.353766 27508 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:39.378191 27559 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.269s	user 0.001s	sys 0.001s
I20260809 23:18:39.355260 27300 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.371052 27397 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.393898 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.390836 27522 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.310695 27510 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.398568 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:39.387662 27461 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:39.404886 27495 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.396528 27300 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:39.411144 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55271us
I20260809 23:18:39.391947 27439 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.425421 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2206) took 1233 ms. Trace:
I20260809 23:18:39.429152 27573 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:39.397819 27508 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.399235 27397 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.508855 27586 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.514845 27586 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:39.446899 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51266.9us
I20260809 23:18:39.515249 27586 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:39.398523 27536 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.509158 24314 rpcz_store.cc:276] 0809 23:18:38.191977 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.192136 (+   159us) service_pool.cc:227] Handling call
0809 23:18:38.193062 (+   926us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:38.193739 (+   677us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:38.193776 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:38.333991 (+140215us) log.cc:842] Serialized 556 byte log entry
0809 23:18:38.920535 (+586544us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:38.920578 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:38.920631 (+    53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:39.425209 (+504578us) raft_consensus.cc:1671] finished
0809 23:18:39.425229 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:39.425358 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":586093,"threads_started":1,"wal-append.queue_time_us":711202}
I20260809 23:18:39.396687 27472 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:39.400815 27442 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.411831 27522 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.522485 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:39.513671 27574 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260809 23:18:39.514233 27384 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:39.521566 27536 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:39.549559 27536 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:39.407428 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:39.560711 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 268208us
I20260809 23:18:39.548820 27384 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.562222 27442 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.567710 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:39.574188 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 180364us
W20260809 23:18:39.574767 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 267557us
W20260809 23:18:39.575340 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 173656us
I20260809 23:18:39.574180 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:39.576628 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 186136us
I20260809 23:18:39.575084 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.579429 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.579830 27550 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:39.585279 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69347.7us
W20260809 23:18:39.590550 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 197360us
I20260809 23:18:39.593674 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:39.594696 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.562227 27384 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.603995 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.602010 27593 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:39.581014 27530 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:39.618489 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:39.617862 27442 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.634804 27585 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.635565 27594 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:39.615103 27510 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.617722 27544 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.629019 27442 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:39.655236 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 1973) took 1650 ms (client timeout 1918 ms). Trace:
I20260809 23:18:39.659502 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.614568 27593 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:39.704530 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 123102us
I20260809 23:18:39.627034 27550 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:39.735101 24534 rpcz_store.cc:269] 0809 23:18:37.993335 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:37.994595 (+  1260us) service_pool.cc:227] Handling call
0809 23:18:39.643778 (+1649183us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54000b4d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:39.643975 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":709229312}
I20260809 23:18:39.699342 27562 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260809 23:18:39.749208 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1202) took 1065 ms. Trace:
I20260809 23:18:39.754276 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:39.755411 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 187758us
I20260809 23:18:39.757874 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:39.760726 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 198889us
W20260809 23:18:39.765355 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 186026us
W20260809 23:18:39.766263 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106885us
I20260809 23:18:39.759615 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1942) took 1153 ms. Trace:
I20260809 23:18:39.663992 27369 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.766018 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:39.689020 27544 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:39.700894 27589 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:39.743793 27442 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:39.773373 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 186524us
I20260809 23:18:39.777760 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:39.745226 27593 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:39.766956 24539 rpcz_store.cc:276] 0809 23:18:38.606232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.913310 (+307078us) service_pool.cc:227] Handling call
0809 23:18:38.923788 (+ 10478us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:38.944347 (+ 20559us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:18:39.353545 (+409198us) log.cc:842] Serialized 14 byte log entry
0809 23:18:39.759179 (+405634us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:39.759249 (+    70us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:39.759321 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:39.759407 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:39.759548 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31824,"raft.queue_time_us":108700,"thread_start_us":255280,"threads_started":1}
I20260809 23:18:39.751127 27510 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.745060 27550 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.777307 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:39.781869 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 206853us
I20260809 23:18:39.787041 27369 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:39.787920 27550 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.788245 27369 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:39.787247 27508 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:39.788679 27369 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:39.791689 27567 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:39.775811 27573 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:39.792740 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.793485 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:39.795164 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.799875 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:39.792668 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.776109 27544 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:39.757391 24323 rpcz_store.cc:276] 0809 23:18:38.684070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.832233 (+148163us) service_pool.cc:227] Handling call
0809 23:18:39.748264 (+916031us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:39.749117 (+   853us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":735113600}
I20260809 23:18:39.793941 27585 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:39.810920 27508 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.872035 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:39.873587 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 104024us
W20260809 23:18:39.875080 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70777us
I20260809 23:18:39.875761 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:39.876498 27596 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.876653 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.876998 27596 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.877460 27596 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:39.811652 27409 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.906s	user 0.004s	sys 0.041s
I20260809 23:18:39.811966 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:39.879657 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87072.3us
I20260809 23:18:39.881561 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:39.882403 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74996.2us
I20260809 23:18:39.883477 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.880872 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:39.877702 27480 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:39.806494 27567 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:39.883232 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99876.3us
W20260809 23:18:39.886835 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91756.4us
I20260809 23:18:39.887704 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.888548 27575 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.888993 27575 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.889334 27575 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:39.891942 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 102540us
I20260809 23:18:39.853397 27585 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.858770 27448 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.893939 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:39.894299 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87977.8us
I20260809 23:18:39.893864 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:39.892566 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.896036 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.883410 27562 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260809 23:18:39.901078 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.890626 27567 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:39.895654 27585 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:39.891212 27597 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:39.918169 24422 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.715s	user 0.017s	sys 0.042s
I20260809 23:18:39.918040 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:39.930020 27573 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:39.931478 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.902184 27499 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:39.924223 27515 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.674s	user 0.020s	sys 0.007s
I20260809 23:18:39.912041 27400 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:39.926944 27512 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:39.886178 27409 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:39.922379 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:39.902172 27589 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:39.936614 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:39.938767 27583 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:39.950047 24317 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:39.946583 24316 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:39.948060 27588 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:39.949471 27409 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.932415 27597 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:39.933288 27448 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:39.954572 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1717) took 1242 ms. Trace:
I20260809 23:18:39.957031 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.957556 27595 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:39.957432 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:39.958662 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.936801 27515 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:39.966015 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.946172 27590 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:39.933272 27499 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.961527 27589 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:39.967156 27499 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:39.967648 27515 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:39.966734 27590 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:39.961550 24317 rpcz_store.cc:276] 0809 23:18:38.712637 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.806777 (+ 94140us) service_pool.cc:227] Handling call
0809 23:18:39.954367 (+1147590us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:39.954494 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2586022784}
I20260809 23:18:39.962785 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:39.961309 27597 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:39.976257 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:39.974529 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:39.977139 25373 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:39.950304 27583 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:39.981491 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:39.967932 27589 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.988298 27515 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:39.976524 27590 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:39.993328 27582 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.964174 27595 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:39.931762 27369 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.000828 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:40.009537 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:39.999944 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:39.993696 27550 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:39.984799 27588 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:40.023916 24309 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:39.990657 27583 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.004045 27595 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.032651 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.035866 27563 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:40.034910 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.037333 25371 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:39.991074 27300 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.020655 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.044523 27563 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.026063 27372 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:40.029285 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:40.056473 27428 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.141s	user 0.019s	sys 0.003s
I20260809 23:18:40.030156 27588 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.056924 27428 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.057379 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.048631 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:40.009390 27536 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:40.040035 27571 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.731s	user 0.006s	sys 0.020s
I20260809 23:18:40.041996 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:39.993925 27582 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.067818 24545 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:40.067883 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 158129us
W20260809 23:18:40.048496 27301 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.036s	user 0.000s	sys 0.023s
I20260809 23:18:40.076166 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.079151 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:40.067771 27553 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.685s	user 0.018s	sys 0.018s
I20260809 23:18:40.082671 25373 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.069742 27530 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:40.078280 27301 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.058601 27588 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.050239 27372 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:40.089499 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 193518us
I20260809 23:18:40.081821 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.072525 27428 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.101611 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.082913 27584 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:40.106210 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.108691 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.111958 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.115236 24535 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.118541 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.118634 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.091487 27554 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.077039 27571 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.093927 27530 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:40.131191 27530 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.131616 27369 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.094785 27550 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.136976 24546 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.138427 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1949) took 1447 ms. Trace:
W20260809 23:18:40.139571 27607 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260809 23:18:40.140488 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.137002 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.141182 25371 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:40.105286 27372 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.127619 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:40.111897 27397 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.596s	user 0.029s	sys 0.028s
I20260809 23:18:40.129452 27510 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:40.129984 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.130290 27301 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.130527 27576 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:40.124375 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.132243 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.132862 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:40.132988 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.091987 27448 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260809 23:18:40.114571 27569 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:40.135280 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2007) took 1116 ms. Trace:
I20260809 23:18:40.136147 27562 ts_tablet_manager.cc:1905] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260809 23:18:40.101153 27384 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.111177 27584 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.143909 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:40.138736 27577 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.890s	user 0.014s	sys 0.035s
I20260809 23:18:40.110180 27588 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.133096 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.138767 24546 rpcz_store.cc:276] 0809 23:18:38.690782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.284609 (+593827us) service_pool.cc:227] Handling call
0809 23:18:40.136801 (+852192us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.137436 (+   635us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.138244 (+   808us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54000eecd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:40.138367 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1959031168}
I20260809 23:18:40.121732 27594 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.149842 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:40.150457 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71906.9us
I20260809 23:18:40.088441 27553 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.152128 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:40.140523 27408 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.839s	user 0.019s	sys 0.015s
I20260809 23:18:40.162467 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.160006 24545 rpcz_store.cc:276] 0809 23:18:39.019465 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.214032 (+194567us) service_pool.cc:227] Handling call
0809 23:18:40.135086 (+921054us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:40.135222 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1892712832}
I20260809 23:18:40.163143 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.154039 27606 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:40.166275 27582 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.166502 27448 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:40.168543 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1948) took 1478 ms. Trace:
I20260809 23:18:40.169399 24316 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:40.197137 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.175467 27536 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:40.154697 27566 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.939s	user 0.027s	sys 0.004s
I20260809 23:18:40.164078 27554 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:40.181135 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.175974 27584 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.188289 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.199312 24312 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.200003 27300 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.159909 27569 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:40.185195 27428 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:40.219583 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.147735 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.195562 24535 rpcz_store.cc:276] 0809 23:18:38.690376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.276573 (+586197us) service_pool.cc:227] Handling call
0809 23:18:39.276925 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.115729 (+838804us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.168329 (+ 52600us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b5400080a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:40.168489 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1927363328}
I20260809 23:18:40.202267 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.182739 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.204919 24539 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.206902 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:40.208458 27370 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.982s	user 0.005s	sys 0.021s
I20260809 23:18:40.209007 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.147639 27397 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.177392 27594 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.215342 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.230734 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1692) took 1276 ms. Trace:
I20260809 23:18:40.194175 27448 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:40.232724 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 99696.6us
I20260809 23:18:40.233861 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.235945 25371 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:40.237910 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.228237 27512 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:40.240137 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.244273 24320 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 2: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:40.232527 24422 rpcz_store.cc:276] 0809 23:18:38.941738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.941882 (+   144us) service_pool.cc:227] Handling call
0809 23:18:38.955778 (+ 13896us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:39.019307 (+ 63529us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:18:39.932854 (+913547us) log.cc:842] Serialized 14 byte log entry
0809 23:18:40.217978 (+285124us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:40.218028 (+    50us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:40.218073 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.218141 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:40.218241 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5988,"raft.queue_time_us":109216,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":22632,"thread_start_us":244661,"threads_started":2,"wal-append.queue_time_us":129206}
I20260809 23:18:40.245390 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.249606 24321 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:40.249959 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.222843 27569 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.187835 27408 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:40.255754 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71417.6us
I20260809 23:18:40.256342 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.250511 25375 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:18:40.217020 27563 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.255897 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:40.181989 27577 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.262012 27550 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:18:40.263000 27397 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.263131 24423 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.263722 27577 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.266000 27439 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:18:40.267024 27439 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:40.267381 27439 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:40.267611 27605 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260809 23:18:40.267784 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.268208 27439 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.238430 27594 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.269510 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.270116 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:40.259116 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.279206 27439 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.280231 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.281653 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.237043 27400 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.244230 27536 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:40.244000 27588 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:40.293649 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:40.256361 27549 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.311s	user 0.026s	sys 0.015s
I20260809 23:18:40.295274 24317 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.286659 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:40.259424 27580 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.937s	user 0.015s	sys 0.025s
I20260809 23:18:40.258823 27301 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:40.210952 27566 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.255343 27582 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.320084 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.294665 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.214453 27554 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.324046 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.236761 27370 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.338383 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:40.324150 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.325415 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.337018 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:40.343544 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57067.5us
I20260809 23:18:40.297075 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.344144 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.344280 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.345376 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.346477 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.353524 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.324275 27549 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.357599 27549 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.359987 24309 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 2: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:40.360674 24320 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:40.324823 27536 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:40.365228 24320 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.361922 24311 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.363052 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.351097 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.270830 27428 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20260809 23:18:40.343137 27300 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.369549 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.337648 27582 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.380874 27582 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.381909 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.382468 24316 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:40.383057 27613 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.139s	user 0.002s	sys 0.000s
I20260809 23:18:40.384457 27596 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.386626 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:40.370437 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.372515 24323 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.378330 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:40.378623 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 81603.9us
I20260809 23:18:40.379551 27428 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:40.380010 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 1.
I20260809 23:18:40.352501 27571 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.391543 27553 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.395138 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.396550 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.397957 27428 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:40.399024 27384 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.399322 27428 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.400967 27428 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.373293 27472 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.373942 27463 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260809 23:18:40.403873 27397 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:40.404017 27463 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:40.404309 27397 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:40.405512 27408 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.405416 27397 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.405686 24316 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:40.407660 24422 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:40.395202 27596 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.404589 27463 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:18:40.386552 27544 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.508s	user 0.006s	sys 0.003s
I20260809 23:18:40.326233 27580 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.411178 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.340664 27567 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.323726 27448 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.373068 27608 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:40.375572 27512 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260809 23:18:40.415366 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45037.2us
I20260809 23:18:40.379963 27554 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.426759 27544 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.443217 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.444177 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.444571 27544 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.445654 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.440806 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.420033 27508 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.433290 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:40.431504 27463 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.442185 27580 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:40.429092 27551 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.469s	user 0.017s	sys 0.026s
I20260809 23:18:40.447132 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:40.411468 27586 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.896s	user 0.021s	sys 0.006s
I20260809 23:18:40.450958 24544 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
W20260809 23:18:40.453042 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 159462us
I20260809 23:18:40.455426 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.455971 24424 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.456476 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.376016 27530 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.457692 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.480609 24313 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:40.459573 27563 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:40.461730 27612 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.462066 27397 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.462312 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.463737 27621 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.469197 27586 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.470009 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 566) took 1240 ms. Trace:
I20260809 23:18:40.472950 27571 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.438591 27608 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:40.458024 27538 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.251s	user 0.012s	sys 0.028s
I20260809 23:18:40.445777 27544 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:40.478868 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:40.483287 27585 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.002s
I20260809 23:18:40.557579 24313 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:40.456350 27554 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.532912 24539 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.535003 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:40.536671 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52005.9us
W20260809 23:18:40.439704 27510 consensus_queue.cc:1159] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
W20260809 23:18:40.537186 27590 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.000s
I20260809 23:18:40.465466 27508 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:40.543282 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85693us
I20260809 23:18:40.542694 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.537472 27612 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.545959 27463 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.467533 27596 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:40.545378 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.548933 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.553228 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.581207 24540 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:40.547452 24311 rpcz_store.cc:276] 0809 23:18:39.229727 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.229874 (+   147us) service_pool.cc:227] Handling call
0809 23:18:39.268180 (+ 38306us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.368459 (+1100279us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.469796 (+101337us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:40.469948 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2509139200}
I20260809 23:18:40.555279 27608 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.447702 27597 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:40.559733 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 100813us
I20260809 23:18:40.562608 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.563392 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:40.564170 27595 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.000s
I20260809 23:18:40.542578 27616 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.574476 24428 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.573665 27612 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.572038 27585 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.564693 27552 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.559108 27538 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.466413 27551 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.533264 27300 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260809 23:18:40.576766 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:40.587877 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.612708 24535 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.554606 27571 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.592754 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2129) took 1009 ms. Trace:
I20260809 23:18:40.590296 27621 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.591764 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.565335 27370 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.573194 27563 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:40.603426 27510 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:40.537808 27536 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.580600 27554 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.575703 27566 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.593679 24544 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:40.615417 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 146416us
I20260809 23:18:40.613322 24424 rpcz_store.cc:276] 0809 23:18:39.583626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.589291 (+  5665us) service_pool.cc:227] Handling call
0809 23:18:39.617942 (+ 28651us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.477814 (+859872us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.592521 (+114707us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:40.592673 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904888064}
I20260809 23:18:40.589397 27472 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.580689 27590 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.616910 27571 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.609541 24546 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:40.617808 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.625242 24542 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:40.626448 24543 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.598397 27597 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.568512 27594 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.661950 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:40.594583 27580 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260809 23:18:40.666729 27585 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.667624 27510 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:40.618332 27392 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:18:40.625146 24380 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 1.627s (SENT)
I20260809 23:18:40.575457 27622 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260809 23:18:40.665508 27530 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.617853 27621 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.623706 27552 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260809 23:18:40.684677 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66989.5us
I20260809 23:18:40.686115 27597 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:40.686317 27551 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.687119 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:40.682729 27472 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.683043 27510 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:40.672881 27553 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260809 23:18:40.688839 27538 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.697752 24312 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:40.598726 27595 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.697086 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:40.700261 24544 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
W20260809 23:18:40.680985 27583 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.646s	user 0.009s	sys 0.002s
I20260809 23:18:40.623656 27573 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.154s
I20260809 23:18:40.627038 27515 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:40.717844 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 108445us
W20260809 23:18:40.837360 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 245698us
I20260809 23:18:40.840982 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:40.746711 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84876.6us
I20260809 23:18:40.836187 24535 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:40.852015 27562 ts_tablet_manager.cc:1918] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.6
I20260809 23:18:40.852502 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.852567 27562 log.cc:1196] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/5fcb08685ecd486c8f1d951336f39982
I20260809 23:18:40.853732 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.854183 27562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38080 (request call id 754) took 2912 ms. Trace:
I20260809 23:18:40.855225 27566 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:40.854542 27562 rpcz_store.cc:276] 0809 23:18:37.941743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:37.949956 (+  8213us) service_pool.cc:227] Handling call
0809 23:18:39.177138 (+1227182us) ts_tablet_manager.cc:1145] Deleting tablet 5fcb08685ecd486c8f1d951336f39982
0809 23:18:39.216016 (+ 38878us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0809 23:18:40.357700 (+1141684us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:40.843956 (+486256us) tablet_metadata.cc:641] Metadata flushed
0809 23:18:40.854110 (+ 10154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":315950,"tablet-delete.queue_time_us":418181,"thread_start_us":395183,"threads_started":1}
I20260809 23:18:40.855650 27566 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:40.693639 27567 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:40.856079 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.844715 27621 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.857003 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:40.856547 27566 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.845641 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.860145 24425 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
W20260809 23:18:40.784377 27522 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.232s	user 0.022s	sys 0.005s
I20260809 23:18:40.866415 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.867112 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.867750 24320 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
W20260809 23:18:40.869021 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 243881us
I20260809 23:18:40.830168 27508 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:40.870122 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.831274 27594 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.869145 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.874626 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:40.737993 27576 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:40.872246 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.875265 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.875963 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1944) took 2264 ms. Trace:
I20260809 23:18:40.877620 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.845073 27597 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:40.880182 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.880405 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.881525 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.844812 27552 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:40.819943 27618 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.245s
I20260809 23:18:40.865535 27515 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.884115 27626 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:40.912142 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.884326 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet 5fcb08685ecd486c8f1d951336f39982 (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:18:40.871034 27522 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:40.858745 27593 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.056s	user 0.012s	sys 0.001s
I20260809 23:18:40.887605 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.879549 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:40.872457 27567 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.903812 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.904261 24541 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.930584 25375 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:18:40.899974 24430 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:40.895296 25370 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.902575 27409 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:40.873167 27369 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20260809 23:18:40.959193 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:40.959604 27409 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:40.960228 27369 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:40.960670 27369 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:40.960489 27409 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.961567 27515 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.912591 27552 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:40.961512 27369 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.962402 24431 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.703336 27582 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.907537 27551 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:40.896764 25371 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:40.889308 24543 rpcz_store.cc:276] 0809 23:18:38.611134 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:38.932483 (+321349us) service_pool.cc:227] Handling call
0809 23:18:38.949872 (+ 17389us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.674125 (+1724253us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:40.875568 (+201443us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001d5e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:40.875883 (+   315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3855194368}
I20260809 23:18:40.900100 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.657992 27625 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.277s
W20260809 23:18:40.903192 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2004) took 1885 ms (client timeout 1627 ms). Trace:
I20260809 23:18:40.921088 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:40.975952 24542 rpcz_store.cc:269] 0809 23:18:39.018140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.045547 (+ 27407us) service_pool.cc:227] Handling call
0809 23:18:40.902859 (+1857312us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:40.903118 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3303563520}
W20260809 23:18:40.922149 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46935.5us
I20260809 23:18:40.979516 24498 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.922257 24320 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.874246 27586 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:40.872721 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.952153 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:40.949254 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:40.895773 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:40.905119 27597 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:40.967512 24429 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.968293 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:40.886969 27575 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.991s	user 0.001s	sys 0.021s
I20260809 23:18:40.991119 24535 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:40.991206 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.862313 27583 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:40.826865 27563 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:40.997159 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.960233 27522 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:40.972375 24311 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:40.986438 24309 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:40.993250 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:40.995244 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:40.995877 24430 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.966482 27369 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:40.888700 27576 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:41.023105 27576 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:41.024459 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:41.001861 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:40.990311 27442 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.186s	user 0.006s	sys 0.007s
W20260809 23:18:41.007509 24498 raft_consensus.cc:2670] T 5fcb08685ecd486c8f1d951336f39982 P 808fed3ed3f04882ad757a3db9abfdd9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260809 23:18:40.909169 25365 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:18:41.028414 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49000.1us
I20260809 23:18:41.029253 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:40.965112 27608 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.032428 27608 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.033232 24423 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:40.996814 27573 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:41.033485 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.034180 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:40.948083 27593 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:41.014254 27499 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.047s	user 0.007s	sys 0.007s
I20260809 23:18:41.045850 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.046583 27568 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:41.001689 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:41.056736 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.071823 24323 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:41.074177 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.043898 27551 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:41.073503 24544 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:41.091281 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.048977 27384 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
W20260809 23:18:41.058717 27606 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.737s	user 0.009s	sys 0.002s
I20260809 23:18:41.048561 27590 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.094059 27628 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.095010 27512 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:41.095945 27472 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.097533 24317 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:41.042219 27408 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260809 23:18:40.997057 27575 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.097640 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.108147 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 394) took 2348 ms. Trace:
I20260809 23:18:41.109066 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:18:41.105872 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60357.4us
I20260809 23:18:41.125046 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.072778 27595 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:41.104720 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.504s (SENT) [suppressed 26 similar messages]
I20260809 23:18:41.123483 24541 rpcz_store.cc:276] 0809 23:18:38.765822 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.294407 (+528585us) service_pool.cc:227] Handling call
0809 23:18:40.904094 (+1609687us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.955043 (+ 50949us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:41.107917 (+152874us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400271fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:41.108071 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3677703424}
I20260809 23:18:41.037990 27442 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.132136 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.083214 27582 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.042083 27370 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260809 23:18:41.082468 27634 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:41.137758 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.102948 27384 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:41.050467 27499 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:41.147962 24314 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.154343 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.156374 27637 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:41.163868 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.164057 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.164558 24312 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:41.166605 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.167292 24317 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:41.165210 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.175540 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.176940 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.179672 27554 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
W20260809 23:18:41.180049 27632 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260809 23:18:41.180271 27554 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:41.181372 27554 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.182683 25372 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:41.097115 27573 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260809 23:18:41.142448 27301 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:41.181964 27499 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:41.163671 24425 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:41.211640 24431 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:41.187832 24534 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.186782 24312 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:41.195641 27638 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.211927 24425 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:41.197389 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:41.238867 24312 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
W20260809 23:18:41.239215 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 100669us
I20260809 23:18:41.237262 27619 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:41.245499 27619 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:41.246768 27619 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:41.249214 24317 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:41.188864 27584 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.961s	user 0.007s	sys 0.004s
W20260809 23:18:41.183729 27571 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.000s
W20260809 23:18:41.147423 27372 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.993s	user 0.012s	sys 0.001s
I20260809 23:18:41.251387 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.252485 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.258267 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.259356 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:41.264299 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190216us
I20260809 23:18:41.269392 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.201472 24309 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:41.274740 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85094.4us
I20260809 23:18:41.306636 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.261735 24425 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.237807 27553 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260809 23:18:41.159219 27633 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:41.311978 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.312031 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.159003 27627 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:41.161422 27628 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
I20260809 23:18:41.321882 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.349449 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.350209 25375 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:41.350581 25373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:37956
I20260809 23:18:41.351104 25373 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:41.351421 25373 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:41.344247 24425 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:41.188206 27551 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:41.159396 27634 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.269865 25368 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:41.357148 27634 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:41.360829 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84239.9us
I20260809 23:18:41.361836 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.278062 27583 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.260006 27584 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:41.439230 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:41.262140 27585 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260809 23:18:41.264168 27372 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.263561 27571 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.470867 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.432430 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.485699 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.488011 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.489033 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.489717 27409 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.490358 24425 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:41.361079 27633 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:41.493383 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.502821 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.503612 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.504446 25370 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:41.506119 25369 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:41.513103 24309 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.516309 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.517390 24316 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:41.517374 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.519163 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.519382 27442 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.519011 27597 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.521399 24311 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 2: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:41.527275 24431 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.545369 24431 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.536947 24539 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:41.528013 25373 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.541296 24435 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.437646 27593 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:41.541965 25371 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:41.306610 27552 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.266384 27384 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:41.548988 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 237055us
I20260809 23:18:41.551326 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.554952 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.509910 27630 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:41.554294 25373 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:41.562435 24541 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:41.563133 24430 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.563825 25375 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 4. Ops: []
I20260809 23:18:41.569809 24428 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:41.571862 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.555565 27575 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:41.567991 27621 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.000s
I20260809 23:18:41.577057 24535 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.575838 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.580001 24313 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:41.580386 24313 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:41.577422 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:41.581470 24317 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:41.584527 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:41.584784 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.587869 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.569346 27630 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:41.588136 24313 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.612166 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:41.616417 27648 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260809 23:18:41.566470 27553 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260809 23:18:41.573984 27647 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:41.624316 24313 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 4
I20260809 23:18:41.587020 27442 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:18:41.633950 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 130787us
I20260809 23:18:41.638541 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.612429 27630 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:41.598095 27408 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:41.654714 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:41.653250 24320 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.524s	user 0.009s	sys 0.000s
I20260809 23:18:41.655709 24320 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:41.653293 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.654357 27408 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:41.629297 27463 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.044s	user 0.011s	sys 0.001s
I20260809 23:18:41.611886 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.574036 27641 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260809 23:18:41.666244 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.671447 24425 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 2: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:41.585340 27621 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:41.677215 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 104900us
I20260809 23:18:41.658456 27538 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:41.657610 27480 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:41.714841 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.721694 24542 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:41.611824 27651 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20260809 23:18:41.654390 27660 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:41.733911 24313 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:41.734725 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80079.4us
W20260809 23:18:41.721053 27569 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.437s	user 0.010s	sys 0.003s
I20260809 23:18:41.714823 27553 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:41.774449 24546 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:41.775300 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.777529 24313 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:41.779680 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.719858 27538 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:41.778637 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:41.790139 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:41.777292 27652 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260809 23:18:41.729588 27480 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:41.799471 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.778223 27553 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:41.800412 27553 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:41.800630 25368 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:41.800944 27480 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.801640 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.801754 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.731881 27372 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.732671 27584 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:41.820530 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46848.3us
I20260809 23:18:41.822355 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:41.736042 27571 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.581436 27633 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.829972 27575 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:41.788285 27666 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:41.651012 27647 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.791877 27597 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.775936 27641 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260809 23:18:41.854107 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:41.853295 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 200088us
I20260809 23:18:41.857156 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:41.803007 27646 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:41.721218 27510 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.866621 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:41.856113 27369 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.822s	user 0.007s	sys 0.003s
I20260809 23:18:41.873711 25370 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:41.781356 27569 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:41.877712 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 265658us
I20260809 23:18:41.877942 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:41.877063 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:41.880187 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44609.6us
I20260809 23:18:41.858539 27666 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:41.881646 25370 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.882802 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:41.883644 24542 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 2: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:41.864171 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:41.889410 24424 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:41.889650 24321 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:41.872309 27530 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.873507 27660 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:41.890398 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.875489 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:41.902784 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:41.903470 24313 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 2: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:41.883553 27552 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.911161 24317 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.874914 27646 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:41.902289 25368 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:41.919569 27638 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:41.916440 27640 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260809 23:18:41.920270 27638 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:41.921083 27663 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:41.921855 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1974) took 1373 ms. Trace:
I20260809 23:18:41.922204 24539 rpcz_store.cc:276] 0809 23:18:40.547992 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.609322 (+ 61330us) service_pool.cc:227] Handling call
0809 23:18:40.624947 (+ 15625us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.549389 (+924442us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:41.921638 (+372249us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:41.921798 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2096937856}
W20260809 23:18:41.923637 24269 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.868s (SENT)
I20260809 23:18:41.929296 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:41.921257 27638 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:41.893746 27660 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:41.940167 24269 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.580s (SENT)
I20260809 23:18:41.940598 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:41.943260 27653 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:41.945205 24424 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:41.898016 27678 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.947266 27569 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.937760 27408 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:41.934916 27584 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:41.957180 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:41.948426 27678 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:41.934338 27621 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:41.966019 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 576) took 1954 ms. Trace:
I20260809 23:18:41.966451 24314 rpcz_store.cc:276] 0809 23:18:40.015750 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.019142 (+  3392us) service_pool.cc:227] Handling call
0809 23:18:40.031000 (+ 11858us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.148325 (+1117325us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:41.965784 (+817459us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:41.965956 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2568361216}
I20260809 23:18:41.941803 27660 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:18:42.029306 27612 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.421s	user 0.007s	sys 0.007s
I20260809 23:18:42.032258 27647 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:42.032780 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 174855us
I20260809 23:18:42.033352 27553 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:42.035948 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 170238us
I20260809 23:18:41.966192 25368 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:42.037197 24431 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:41.909880 27670 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
I20260809 23:18:42.038556 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.047920 24317 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:42.048434 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:42.052186 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.053202 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:42.029875 27612 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.048902 24313 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:42.038972 27589 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:42.065874 24542 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:42.064505 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:42.074791 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:42.063215 27584 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260809 23:18:42.046792 27679 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:41.966987 27651 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
W20260809 23:18:42.059203 27567 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.108s	user 0.008s	sys 0.004s
I20260809 23:18:42.078116 24311 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:42.084064 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:42.089298 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:42.091326 24422 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:42.093366 24431 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:34500
I20260809 23:18:42.068547 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.094251 24431 raft_consensus.cc:595] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:42.098680 27683 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260809 23:18:42.100368 27683 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:42.105628 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.107611 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:42.072335 27589 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:42.085199 27653 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:42.125020 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:42.126461 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:42.077764 27672 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:42.128393 27672 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:42.083925 27685 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:42.093974 27567 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:42.131975 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:42.103091 25363 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:42.096868 27552 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:42.087193 27647 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 23:18:42.104609 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.077978 27655 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.248s
I20260809 23:18:42.162011 24545 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:42.188880 24540 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:42.189294 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83498.7us
I20260809 23:18:42.111989 27569 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:18:42.157155 25365 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.085s	user 0.006s	sys 0.005s
W20260809 23:18:42.161512 27665 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260809 23:18:42.212189 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.213438 25363 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:42.191255 27594 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:42.214831 27594 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.215417 27594 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.218144 27594 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.219834 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.220616 25368 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:42.203568 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71664.4us
I20260809 23:18:42.161856 27668 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:42.129876 27672 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.101164 27683 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.124123 27653 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:42.228477 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:42.205603 27301 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.965s	user 0.007s	sys 0.005s
I20260809 23:18:42.230700 27609 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:42.041083 27678 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.317178 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:42.134191 27575 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:42.214608 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.233209 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:42.326694 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93583.9us
I20260809 23:18:42.234404 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:42.329309 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94979.8us
I20260809 23:18:42.218698 27594 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:42.239838 27552 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:42.240753 27566 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:42.136698 27583 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20260809 23:18:42.343798 27678 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.351006 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:42.318902 27301 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:42.366098 25363 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:42.313939 27666 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:42.352933 27630 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.687s	user 0.004s	sys 0.006s
I20260809 23:18:42.389460 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:42.333938 27612 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.244132 27668 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:42.333640 27397 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:42.366458 27594 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:42.395558 24543 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:42.132457 27679 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.359359 27575 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.415294 27552 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:42.231957 27596 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.129s
I20260809 23:18:42.155958 27685 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.422230 27397 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:42.428315 27679 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.431912 27584 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:42.365996 27629 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:42.384924 27686 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260809 23:18:42.223577 27589 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.432507 27666 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.494534 27621 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:42.440986 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.386436 27680 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260809 23:18:42.243712 27653 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.416651 27609 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:42.431025 27630 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.422071 27668 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:42.439028 27576 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.415s	user 0.008s	sys 0.007s
I20260809 23:18:42.424319 27553 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.366973 27544 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.068s
I20260809 23:18:42.460700 25368 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:42.462159 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2024) took 1836 ms. Trace:
I20260809 23:18:42.510632 27575 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.466538 27566 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:42.453270 27606 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
W20260809 23:18:42.490427 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.000s	sys 0.005s
W20260809 23:18:42.644083 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.001s	sys 0.005s
W20260809 23:18:42.644520 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 754276us
I20260809 23:18:42.645356 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.493633 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 725) took 1627 ms. Trace:
W20260809 23:18:42.495244 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.005s	sys 0.008s
I20260809 23:18:42.624507 27596 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:42.646559 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.005s	sys 0.008s
I20260809 23:18:42.630674 25363 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 2: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:42.540079 24539 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:42.619661 27589 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:42.539922 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:42.476498 27409 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.552640 24312 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:42.441134 27636 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:42.615873 27629 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.651018 27566 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.648538 27653 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:42.608659 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.017s	sys 0.043s
W20260809 23:18:42.663662 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.017s	sys 0.044s
W20260809 23:18:42.449388 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59979.4us
I20260809 23:18:42.665024 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:42.584262 27301 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:42.556728 27584 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:42.583333 27679 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.682543 27679 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.646481 25365 rpcz_store.cc:276] 0809 23:18:40.865933 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.870647 (+  4714us) service_pool.cc:227] Handling call
0809 23:18:40.908702 (+ 38055us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:40.908986 (+   284us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:18:42.231736 (+1322750us) log.cc:842] Serialized 14 byte log entry
0809 23:18:42.493239 (+261503us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:42.493311 (+    72us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:42.493357 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:42.493433 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:42.493569 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":129936,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":3969920,"thread_start_us":125621,"threads_started":1,"wal-append.queue_time_us":178272}
I20260809 23:18:42.684651 27679 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.626845 27397 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.453873 27681 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.513s
I20260809 23:18:42.627203 24544 rpcz_store.cc:276] 0809 23:18:40.625984 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.882101 (+256117us) service_pool.cc:227] Handling call
0809 23:18:40.930416 (+ 48315us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.092382 (+161966us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:42.461855 (+1369473us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:42.462080 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":328614144}
I20260809 23:18:42.434561 27594 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:42.631259 24541 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:42.694746 27681 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:42.650192 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.652242 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.695216 27681 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:42.695775 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 43648us
I20260809 23:18:42.695473 27594 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.582643 27687 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.384s
I20260809 23:18:42.659564 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:42.663972 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 618864us
I20260809 23:18:42.653779 27690 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:42.679011 27566 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.681505 27630 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:42.697889 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47757.9us
I20260809 23:18:42.626966 27576 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.419123 27678 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20260809 23:18:42.731676 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.003s	sys 0.034s
W20260809 23:18:42.821600 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.003s	sys 0.034s
I20260809 23:18:42.646749 27596 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.662691 27636 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:42.817366 24544 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:42.817730 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.661926 27683 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.823261 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:42.833871 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:42.835475 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:42.836918 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:42.742980 27691 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:18:42.696094 27681 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.828557 27685 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.697213 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.656289 27409 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.823843 27593 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.684679 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.745048 27300 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.840355 27636 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:42.668383 27703 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:42.827028 27575 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.986239 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.986653 27575 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:42.859072 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:42.782254 27567 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:42.796269 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:42.890164 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47779.4us
I20260809 23:18:42.894585 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.905565 24539 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:42.993647 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 645) took 1507 ms (client timeout 1720 ms). Trace:
I20260809 23:18:42.992290 27567 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:42.994087 25373 rpcz_store.cc:269] 0809 23:18:41.486359 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.894312 (+1407953us) service_pool.cc:227] Handling call
0809 23:18:42.993587 (+ 99275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:42.995234 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:42.997079 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:42.960517 27687 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:42.686333 27301 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:42.739619 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 228322us
I20260809 23:18:42.908263 27409 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.908607 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:42.841101 27690 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:18:42.893486 27660 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.855s	user 0.014s	sys 0.001s
I20260809 23:18:42.838308 27679 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260809 23:18:43.021884 27301 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.948932 24425 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:42.951645 27554 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:42.923981 27681 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:42.959353 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 965) took 2537 ms. Trace:
I20260809 23:18:42.898140 27683 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260809 23:18:42.905544 27629 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.891094 27584 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.961419 27538 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.967721 27596 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:42.972247 27576 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:42.987279 27575 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.998819 27687 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.987784 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:42.986284 27703 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:42.992993 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:43.018050 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:43.018846 27660 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:42.947687 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2154) took 2084 ms (client timeout 1879 ms). Trace:
I20260809 23:18:43.023393 24427 rpcz_store.cc:276] 0809 23:18:40.417121 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.417292 (+   171us) service_pool.cc:227] Handling call
0809 23:18:42.953905 (+2536613us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:42.954501 (+   596us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2885542272}
I20260809 23:18:43.032503 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:42.890717 27627 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:42.975594 27593 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:42.991097 27508 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:42.990573 27300 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.031291 27567 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:43.119596 24429 rpcz_store.cc:269] 0809 23:18:40.863307 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.863429 (+   122us) service_pool.cc:227] Handling call
0809 23:18:42.947422 (+2083993us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:42.947576 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161578368}
W20260809 23:18:43.146139 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113732us
I20260809 23:18:43.150837 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:43.162748 27683 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:43.156332 27653 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.196369 27630 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:43.197418 27575 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.196345 27653 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.110376 25368 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:43.200012 27301 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:43.079419 27596 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:43.122162 27584 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:43.122582 27627 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:43.253706 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2147) took 2766 ms. Trace:
I20260809 23:18:43.056732 27704 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.259250 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1713) took 2578 ms. Trace:
I20260809 23:18:43.119406 27508 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:43.263314 24423 rpcz_store.cc:276] 0809 23:18:40.437718 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.437864 (+   146us) service_pool.cc:227] Handling call
0809 23:18:40.442619 (+  4755us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.033917 (+591298us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.203794 (+2169877us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.203990 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1357825920}
I20260809 23:18:43.278190 24544 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:43.282964 24428 rpcz_store.cc:276] 0809 23:18:40.624266 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.624448 (+   182us) service_pool.cc:227] Handling call
0809 23:18:40.668264 (+ 43816us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.572841 (+904577us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.201878 (+1629037us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54003eb2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.202093 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2072913664}
I20260809 23:18:43.308825 27703 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:43.261942 27668 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.630s	user 0.012s	sys 0.000s
I20260809 23:18:43.280632 27704 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.206471 27596 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:43.303581 27616 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:43.136757 25363 raft_consensus.cc:2362] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4. Re-sending same reply.
I20260809 23:18:43.328548 27687 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:43.311169 27690 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.362502 27596 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.331038 27653 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.380648 27619 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:43.382048 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2044) took 1346 ms. Trace:
I20260809 23:18:43.384315 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2151) took 2523 ms. Trace:
I20260809 23:18:43.399935 24422 rpcz_store.cc:276] 0809 23:18:40.861024 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.861183 (+   159us) service_pool.cc:227] Handling call
0809 23:18:40.861459 (+   276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.091910 (+1230451us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.384013 (+1292103us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.384228 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2819507072}
I20260809 23:18:43.385499 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2159) took 2281 ms. Trace:
I20260809 23:18:43.342379 27687 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.387753 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 634) took 2065 ms. Trace:
I20260809 23:18:43.421639 25375 rpcz_store.cc:276] 0809 23:18:41.322826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.351921 (+ 29095us) service_pool.cc:227] Handling call
0809 23:18:41.514493 (+162572us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.568329 (+ 53836us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.387356 (+1819027us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.387668 (+   312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110200192}
I20260809 23:18:43.389609 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 387) took 1678 ms. Trace:
I20260809 23:18:43.394331 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2216) took 2555 ms. Trace:
I20260809 23:18:43.394291 27508 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.401266 24435 rpcz_store.cc:276] 0809 23:18:41.104174 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.104523 (+   349us) service_pool.cc:227] Handling call
0809 23:18:41.106305 (+  1782us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.547103 (+440798us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.385208 (+1838105us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.385426 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":998092032}
I20260809 23:18:43.386574 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 640) took 1901 ms. Trace:
I20260809 23:18:43.405989 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 587) took 2948 ms. Trace:
I20260809 23:18:43.426422 24321 rpcz_store.cc:276] 0809 23:18:40.457520 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.457654 (+   134us) service_pool.cc:227] Handling call
0809 23:18:40.481032 (+ 23378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.890732 (+1409700us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.405693 (+1514961us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54002530d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.405924 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3239096704}
I20260809 23:18:43.424000 24316 rpcz_store.cc:276] 0809 23:18:40.839000 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.839209 (+   209us) service_pool.cc:227] Handling call
0809 23:18:41.517229 (+678020us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.517779 (+   550us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.394083 (+1876304us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b54003c37d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:43.394238 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1529821440}
I20260809 23:18:43.425377 25370 rpcz_store.cc:276] 0809 23:18:41.484955 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.891000 (+406045us) service_pool.cc:227] Handling call
0809 23:18:41.932873 (+ 41873us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.956275 (+ 23402us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:41.956306 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:41.956366 (+    60us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:41.956406 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.386293 (+1429887us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.386357 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:43.386505 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37691136}
I20260809 23:18:43.399814 24543 rpcz_store.cc:276] 0809 23:18:42.036018 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.036157 (+   139us) service_pool.cc:227] Handling call
0809 23:18:42.036506 (+   349us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.418252 (+381746us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.381733 (+963481us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.381961 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":825170176}
I20260809 23:18:43.423012 24425 rpcz_store.cc:276] 0809 23:18:41.711251 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.714402 (+  3151us) service_pool.cc:227] Handling call
0809 23:18:42.948735 (+1234333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:43.022695 (+ 73960us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.389383 (+366688us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54003bb350. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:43.389541 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2695268608}
I20260809 23:18:43.408408 27553 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:43.432426 27553 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.345156 27616 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:43.416361 27538 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:43.326711 27668 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.420305 27619 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:43.461444 27553 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:43.438783 27596 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:43.469178 25366 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:43.341815 27627 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:43.475615 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 765) took 3441 ms (client timeout 2009 ms). Trace:
W20260809 23:18:43.476066 25372 rpcz_store.cc:269] 0809 23:18:40.032191 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.032433 (+   242us) service_pool.cc:227] Handling call
0809 23:18:43.473095 (+3440662us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540073f5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:43.473256 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2605599104}
I20260809 23:18:43.478747 27627 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:43.369215 27690 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.431254 27687 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.483740 27619 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.436005 27616 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.482390 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 723) took 2620 ms. Trace:
I20260809 23:18:43.490602 25366 rpcz_store.cc:276] 0809 23:18:40.861489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.865202 (+  3713us) service_pool.cc:227] Handling call
0809 23:18:43.468080 (+2602878us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900bce 000000000034aea9 00007fac2872ca30 00007fac287f27b4 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:18:43.468579 (+   499us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:43.480787 (+ 12208us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.482258 (+  1471us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21307136}
I20260809 23:18:43.505986 27448 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:43.512385 27619 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:43.516589 27590 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:43.521106 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2042) took 1634 ms. Trace:
I20260809 23:18:43.521647 24542 rpcz_store.cc:276] 0809 23:18:41.886300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.887603 (+  1303us) service_pool.cc:227] Handling call
0809 23:18:41.897276 (+  9673us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.076204 (+178928us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.520775 (+1444571us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.521024 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":387123584}
I20260809 23:18:43.480958 27668 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.517596 27577 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260809 23:18:43.525298 27704 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:43.516124 27690 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:43.524441 27590 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:43.529310 27590 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:43.531133 27589 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.876s	user 0.012s	sys 0.002s
I20260809 23:18:43.531000 27590 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:43.510557 27300 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:43.542692 27397 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:43.532119 27633 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:43.592722 27702 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.096s	user 0.005s	sys 0.012s
I20260809 23:18:43.487358 27646 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.611806 25371 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:43.612498 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.732s	user 0.042s	sys 0.094s
W20260809 23:18:43.618507 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.738s	user 0.042s	sys 0.095s
I20260809 23:18:43.620340 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:43.623798 27554 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.007s
I20260809 23:18:43.630017 27687 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.653494 27668 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.640950 27638 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:43.646337 27552 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
W20260809 23:18:43.683856 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63631.1us
I20260809 23:18:43.692438 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2035) took 1976 ms. Trace:
I20260809 23:18:43.697543 24541 rpcz_store.cc:276] 0809 23:18:41.716022 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.716139 (+   117us) service_pool.cc:227] Handling call
0809 23:18:41.730012 (+ 13873us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.695065 (+965053us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.692157 (+997092us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.692370 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2072158080}
I20260809 23:18:43.705147 24535 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:43.749178 27448 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:43.750125 27448 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.753429 27448 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:43.754206 27448 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:43.755453 27448 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.761332 25372 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:43.761468 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:43.660171 27370 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.769840 25370 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:43.770066 25372 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:43.759445 27651 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.772329 27651 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.774191 25375 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:43.694960 27637 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:43.777622 27595 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:43.778083 27595 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:18:43.747607 27636 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.699s	user 0.007s	sys 0.012s
I20260809 23:18:43.697009 27552 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:43.799124 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2161) took 2642 ms. Trace:
I20260809 23:18:43.798137 25370 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:43.803514 24430 rpcz_store.cc:276] 0809 23:18:41.156510 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.157314 (+   804us) service_pool.cc:227] Handling call
0809 23:18:41.157652 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.569463 (+411811us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.798819 (+2229356us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.799048 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":932151680}
I20260809 23:18:43.768525 27588 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.779923 27658 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:43.813786 25370 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:43.816919 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:43.796787 27370 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:43.705883 27668 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:43.794691 27595 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:44.210242 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.502s	user 0.011s	sys 0.026s
I20260809 23:18:43.921830 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2012) took 3951 ms. Trace:
W20260809 23:18:43.900625 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2017) took 3422 ms (client timeout 1593 ms). Trace:
W20260809 23:18:44.031266 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.930s	user 0.007s	sys 0.005s
W20260809 23:18:43.839241 27544 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.184s	user 0.004s	sys 0.046s
I20260809 23:18:43.911711 27370 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:44.378829 27672 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.458s	user 0.013s	sys 0.040s
I20260809 23:18:44.374897 27593 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:44.384424 27672 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.464s	user 0.013s	sys 0.040s
W20260809 23:18:43.840113 27686 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.030s	sys 0.076s
W20260809 23:18:44.383770 27544 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.734s	user 0.007s	sys 0.077s
I20260809 23:18:43.818514 27448 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:44.125999 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 643) took 2640 ms (client timeout 1795 ms). Trace:
W20260809 23:18:44.385221 27619 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.865s	user 0.005s	sys 0.005s
I20260809 23:18:43.919775 27384 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:43.876714 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1759) took 2389 ms (client timeout 1596 ms). Trace:
W20260809 23:18:44.122740 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.283s	user 0.001s	sys 0.030s
I20260809 23:18:44.379329 27651 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:44.012890 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.210s	user 0.012s	sys 0.013s
W20260809 23:18:44.423806 24312 rpcz_store.cc:269] 0809 23:18:41.487645 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.487758 (+   113us) service_pool.cc:227] Handling call
0809 23:18:43.876364 (+2388606us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:43.876587 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2206440704}
I20260809 23:18:43.968604 27660 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:44.393028 24545 rpcz_store.cc:269] 0809 23:18:40.456035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:40.456196 (+   161us) service_pool.cc:227] Handling call
0809 23:18:43.878593 (+3422397us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:43.878712 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2752489728}
I20260809 23:18:44.382422 24534 rpcz_store.cc:276] 0809 23:18:39.970151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:39.970264 (+   113us) service_pool.cc:227] Handling call
0809 23:18:39.981678 (+ 11414us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:41.241515 (+1259837us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:43.920784 (+2679269us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:43.921025 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2761970432}
W20260809 23:18:44.383347 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.494s	user 0.012s	sys 0.018s
I20260809 23:18:43.925043 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:43.818796 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.597s	user 0.022s	sys 0.073s
W20260809 23:18:43.978531 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.467s	user 0.013s	sys 0.056s
W20260809 23:18:44.222210 27573 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.968s	user 0.029s	sys 0.068s
W20260809 23:18:44.448736 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.227s	user 0.036s	sys 0.096s
W20260809 23:18:44.449573 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.223s	user 0.026s	sys 0.071s
I20260809 23:18:43.795022 27636 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:44.000377 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.455s	user 0.029s	sys 0.050s
I20260809 23:18:43.874153 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:44.451261 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.906s	user 0.038s	sys 0.053s
W20260809 23:18:43.994400 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.872s	user 0.008s	sys 0.042s
W20260809 23:18:44.452001 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.333s	user 0.020s	sys 0.051s
I20260809 23:18:43.938743 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:44.413197 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.573s	user 0.023s	sys 0.030s
W20260809 23:18:44.452793 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.613s	user 0.024s	sys 0.032s
I20260809 23:18:44.453483 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.454257 24422 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:44.456614 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:44.397317 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.500s	user 0.016s	sys 0.033s
W20260809 23:18:44.004916 27666 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.313s	user 0.045s	sys 0.103s
I20260809 23:18:44.413404 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2030) took 2876 ms. Trace:
W20260809 23:18:44.161285 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.416s	user 0.015s	sys 0.024s
I20260809 23:18:43.936295 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:44.459714 24540 rpcz_store.cc:276] 0809 23:18:41.536685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.547219 (+ 10534us) service_pool.cc:227] Handling call
0809 23:18:41.566421 (+ 19202us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.206754 (+640333us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:44.413056 (+2206302us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400147b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:44.413259 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1375371648}
W20260809 23:18:44.424824 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.585s	user 0.008s	sys 0.040s
W20260809 23:18:43.914304 27667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.464s	user 0.020s	sys 0.044s
I20260809 23:18:43.942353 27655 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
W20260809 23:18:44.463987 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.567s	user 0.017s	sys 0.034s
W20260809 23:18:44.427942 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.625s	user 0.017s	sys 0.031s
W20260809 23:18:44.381640 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.674s	user 0.011s	sys 0.036s
W20260809 23:18:44.405131 27686 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.780s	user 0.045s	sys 0.089s
W20260809 23:18:44.470088 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.79375e+06us
I20260809 23:18:44.376287 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:43.819190 27588 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:44.172617 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.525s	user 0.031s	sys 0.053s
I20260809 23:18:44.451495 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:18:44.454789 27538 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.012s	user 0.011s	sys 0.005s
W20260809 23:18:44.495327 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.848s	user 0.032s	sys 0.058s
I20260809 23:18:44.422716 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:44.460083 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.715s	user 0.021s	sys 0.032s
W20260809 23:18:44.460775 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 522092us
I20260809 23:18:43.845198 27552 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:44.461300 25321 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.016s	sys 0.042s
W20260809 23:18:44.501950 25321 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.016s	sys 0.042s
W20260809 23:18:44.502316 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 877424us
I20260809 23:18:44.455181 27370 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:44.418895 25371 rpcz_store.cc:269] 0809 23:18:41.485779 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.596541 (+110762us) service_pool.cc:227] Handling call
0809 23:18:44.125680 (+2529139us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:44.125907 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4354221312}
W20260809 23:18:43.922644 27606 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.262s	user 0.039s	sys 0.023s
I20260809 23:18:44.506244 25366 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:18:44.455581 27573 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.206s	user 0.033s	sys 0.082s
W20260809 23:18:44.433677 27590 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.896s	user 0.014s	sys 0.000s
W20260809 23:18:44.468374 27667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.021s	user 0.024s	sys 0.059s
I20260809 23:18:44.461755 27695 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:44.517398 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:44.517882 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:44.520285 24422 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:44.474392 27655 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:44.506314 27606 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.847s	user 0.042s	sys 0.040s
I20260809 23:18:44.472656 24425 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:44.458724 27568 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:44.467171 27666 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.773s	user 0.051s	sys 0.122s
I20260809 23:18:44.462376 27679 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.462638 27583 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:44.492741 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:44.503870 27596 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.029s	user 0.011s	sys 0.003s
I20260809 23:18:44.467566 27660 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 23:18:44.518435 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.235s	user 0.013s	sys 0.049s
I20260809 23:18:43.832629 27658 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.517969 24435 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 4
I20260809 23:18:44.537416 27685 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:44.464540 27677 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.684s	user 0.011s	sys 0.031s
I20260809 23:18:44.508481 27629 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.525251 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.516767 27552 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:44.515053 27538 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:44.519279 27593 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:44.545445 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73923.5us
I20260809 23:18:44.538511 27568 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.543982 27588 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.542412 27679 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.526880 27606 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.604698 27728 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:44.524912 27695 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.605962 27636 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.609675 27573 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:44.526675 27712 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:44.513010 27370 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.611531 27655 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.541586 27711 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260809 23:18:44.520828 27590 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:44.614476 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.331s	user 0.013s	sys 0.049s
I20260809 23:18:44.629096 27668 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.643440 27686 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:44.612980 27714 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:44.627593 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:44.658118 27588 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.627859 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.625360 24422 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:44.646201 24425 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:44.648140 27593 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.706092 27593 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.628563 27685 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.706385 27695 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.707800 27685 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.708361 27568 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.682387 27573 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.709064 27568 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.709630 27568 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.675519 27728 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.713167 27728 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.703783 27448 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.714437 27448 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.623082 27715 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:44.715446 27606 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.715972 27715 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.720208 27370 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:44.624223 27716 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260809 23:18:44.708403 27588 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.724170 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:44.621352 27722 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260809 23:18:44.726070 27588 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.726817 27686 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.729334 27686 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.710709 27573 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.631397 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:44.651381 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 1995) took 2156 ms. Trace:
I20260809 23:18:44.735824 24546 rpcz_store.cc:276] 0809 23:18:42.494913 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.677255 (+182342us) service_pool.cc:227] Handling call
0809 23:18:42.797868 (+120613us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:42.798263 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:42.798300 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:42.822414 (+ 24114us) log.cc:842] Serialized 26 byte log entry
0809 23:18:44.499189 (+1676775us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:44.499227 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:44.499291 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:44.651150 (+151859us) raft_consensus.cc:1671] finished
0809 23:18:44.651173 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:44.651318 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34557,"raft.queue_time_us":22610,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":18239,"thread_start_us":1572609,"threads_started":1,"wal-append.queue_time_us":1786947}
W20260809 23:18:44.628630 27698 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260809 23:18:44.644019 27538 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.656926 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:44.740288 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 109002us
I20260809 23:18:44.740550 27538 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.630548 27629 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.742189 27626 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:44.742581 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2174) took 2582 ms. Trace:
W20260809 23:18:44.626390 27677 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.846s	user 0.011s	sys 0.031s
I20260809 23:18:44.623929 27689 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 23:18:44.659852 27638 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.951s	user 0.014s	sys 0.000s
I20260809 23:18:44.693200 27655 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:44.693647 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.850s	user 0.031s	sys 0.052s
W20260809 23:18:44.748934 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.978s	user 0.032s	sys 0.052s
I20260809 23:18:44.694453 24540 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:44.257750 27633 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.709s	user 0.003s	sys 0.015s
I20260809 23:18:44.694600 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:44.761950 24545 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:44.696144 27590 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.613848 27583 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.700347 24429 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:44.763964 27590 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.764082 27583 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.764729 27568 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.765071 27718 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.765930 27718 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.628741 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.701910 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:44.768497 27718 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.769130 27718 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.769694 27718 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.613341 27697 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.742671 27626 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.771257 27697 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.771703 27626 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.773020 27626 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:44.773720 27697 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.743850 27680 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:44.629855 27666 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.745536 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:44.718917 27715 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.743330 24431 rpcz_store.cc:276] 0809 23:18:42.160000 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.160166 (+   166us) service_pool.cc:227] Handling call
0809 23:18:42.160712 (+   546us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:42.190731 (+ 30019us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:42.190762 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:42.201541 (+ 10779us) log.cc:842] Serialized 26 byte log entry
0809 23:18:44.449234 (+2247693us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:44.449282 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:44.449371 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:44.742340 (+292969us) raft_consensus.cc:1671] finished
0809 23:18:44.742368 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:44.742513 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29512,"raft.queue_time_us":30063,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":10432,"thread_start_us":2227667,"threads_started":1,"wal-append.queue_time_us":2304628}
I20260809 23:18:44.777065 24435 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:44.778271 24429 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:44.780680 25369 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:44.786012 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:44.809661 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:44.775375 27680 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.752014 27714 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.783743 27709 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.776492 27636 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:44.814215 27510 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:44.804247 27711 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:44.809597 27715 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:18:44.688282 27712 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.785364 27686 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260809 23:18:44.810984 27596 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260809 23:18:44.816144 27552 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260809 23:18:44.820720 27679 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.826349 27633 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.813721 27722 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260809 23:18:44.803959 27666 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:44.836562 24270 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.047s	sys 0.088s
I20260809 23:18:44.827517 27711 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:18:44.837024 24270 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.047s	sys 0.088s
W20260809 23:18:44.837739 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 638514us
I20260809 23:18:44.835824 27686 consensus_queue.cc:1225] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 23:18:44.834969 27633 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.830336 27714 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.848664 27712 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.846990 27725 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:44.853860 27721 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:44.819476 27709 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:44.863754 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.661s	user 0.021s	sys 0.060s
W20260809 23:18:44.864063 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.662s	user 0.021s	sys 0.060s
I20260809 23:18:44.852483 27711 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:44.853382 27725 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:18:44.866501 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.66862e+06us
I20260809 23:18:44.876927 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.877876 24435 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:44.881618 24427 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:44.884152 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.886997 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.887764 24427 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:44.888439 24431 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:44.888659 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.884814 24429 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:44.856835 27718 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:44.863550 27594 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.876s	user 0.025s	sys 0.059s
I20260809 23:18:44.915045 27649 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:44.915473 27649 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:44.915966 27730 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:44.916286 27649 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.917357 27593 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:44.918565 27628 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:44.919051 27628 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:44.919405 27667 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:44.919903 27667 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.919986 27628 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.921011 27667 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.923187 27667 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:44.923449 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 423) took 3337 ms. Trace:
I20260809 23:18:44.917860 27730 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:44.924685 27730 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:44.926036 27629 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:44.925685 27730 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.923924 24535 rpcz_store.cc:276] 0809 23:18:41.586342 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.586500 (+   158us) service_pool.cc:227] Handling call
0809 23:18:41.624521 (+ 38021us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:43.748621 (+2124100us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:44.923159 (+1174538us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b5400148554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:44.923376 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4764574080}
I20260809 23:18:44.927009 27595 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:44.917872 27649 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:44.933476 27730 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:44.940747 27448 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.948962 27628 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:44.842111 27680 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.866284 27711 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.883001 27638 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:44.958158 27638 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:44.960016 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.005s	sys 0.009s
I20260809 23:18:44.873885 27666 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:44.882653 27679 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:44.882987 27538 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.854429 27721 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:44.967804 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51794.5us
I20260809 23:18:44.935523 27612 raft_consensus.cc:979] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: : Instructing follower 60d5c69728134c8bb918a900eb6a8c21 to start an election
W20260809 23:18:44.963442 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 647) took 3476 ms (client timeout 1565 ms). Trace:
W20260809 23:18:44.981523 25369 rpcz_store.cc:269] 0809 23:18:41.486840 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:41.518912 (+ 32072us) service_pool.cc:227] Handling call
0809 23:18:44.963215 (+3444303us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:44.963347 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7354223488}
I20260809 23:18:44.970280 27638 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:44.955093 27594 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.922s	user 0.025s	sys 0.059s
I20260809 23:18:44.970978 27648 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:44.972788 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.974439 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.975751 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.991520 24431 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:44.976370 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.992157 24422 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:44.978192 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.978870 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:44.964128 27639 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:44.990964 27648 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:44.989941 27567 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:45.004622 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.007170 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.017347 24427 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:45.008963 24422 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:45.018220 24422 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:44.985359 27712 raft_consensus.cc:1067] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Signalling peer 60d5c69728134c8bb918a900eb6a8c21 to start an election
I20260809 23:18:45.036317 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.035318 27651 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260809 23:18:45.043171 24422 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.044297 24422 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:45.048768 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.049947 24428 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:45.032928 27568 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:45.051719 27568 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.038779 27590 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.054157 27590 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.056977 27590 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.057562 27706 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.058184 27706 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.016155 27639 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.060634 27706 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.061444 27639 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.063004 27721 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.064740 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.065526 24425 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.067291 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.068239 24427 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:45.068933 27633 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.069134 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.036768 27508 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:44.998467 27680 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:44.990828 27638 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260809 23:18:45.072662 27612 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:18:45.073107 27612 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:45.087047 27648 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.087721 27711 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.089372 27711 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:44.992372 27725 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.092284 27725 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.066839 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:45.101935 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65702.6us
I20260809 23:18:45.103194 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.054311 27568 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:44.982586 27538 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:44.980726 27679 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.145063 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.074147 27612 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:45.146273 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.018s	sys 0.013s
W20260809 23:18:45.146991 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.018s	sys 0.014s
W20260809 23:18:45.147392 24269 connection.cc:232] RPC call timeout handler was delayed by -1.35911883902736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260809 23:18:45.120538 24320 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
W20260809 23:18:45.148757 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.009s	sys 0.021s
I20260809 23:18:45.145260 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:45.123440 24313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.009s	sys 0.018s
I20260809 23:18:45.148168 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.149927 24316 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:45.098311 27569 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.151490 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:45.149163 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.010s	sys 0.021s
I20260809 23:18:45.152083 24321 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:45.152916 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:45.150676 24313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.011s	sys 0.019s
I20260809 23:18:45.153741 24311 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:45.153932 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2243) took 3105 ms. Trace:
I20260809 23:18:45.142894 24422 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:18:45.104089 27628 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:45.016662 27619 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260809 23:18:45.151631 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.154237 24313 rpcz_store.cc:276] 0809 23:18:42.048041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.048230 (+   189us) service_pool.cc:227] Handling call
0809 23:18:42.048568 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:42.048771 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:18:42.118663 (+ 69892us) log.cc:842] Serialized 14 byte log entry
0809 23:18:45.153588 (+3034925us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:45.153642 (+    54us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:45.153691 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:45.153762 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:45.153869 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8672,"raft.queue_time_us":156911,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"spinlock_wait_cycles":1280,"thread_start_us":3025827,"threads_started":2,"wal-append.queue_time_us":2382521}
W20260809 23:18:45.164834 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61742us
I20260809 23:18:45.160214 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:45.142438 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.010s	sys 0.015s
W20260809 23:18:45.168078 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.010s	sys 0.015s
W20260809 23:18:45.142503 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.001s	sys 0.025s
W20260809 23:18:45.169219 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.001s	sys 0.025s
W20260809 23:18:45.169564 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 672523us
I20260809 23:18:45.169916 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:45.142323 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.021s	sys 0.035s
I20260809 23:18:45.170574 24422 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:45.140095 27734 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:45.172644 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.021s	sys 0.035s
I20260809 23:18:45.180294 24435 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:45.155479 27566 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:18:45.172489 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 362899us
I20260809 23:18:45.140327 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:45.172461 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:45.172897 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.015s	sys 0.020s
I20260809 23:18:45.162133 27568 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.184834 24424 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:45.184190 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.162336 27573 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:45.262423 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.015s	sys 0.039s
I20260809 23:18:45.184346 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.262632 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.156536 27633 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:18:45.185693 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.000s	sys 0.040s
I20260809 23:18:45.266482 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
W20260809 23:18:45.282490 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.008s	sys 0.047s
W20260809 23:18:45.262202 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 449610us
W20260809 23:18:45.267777 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89166.2us
W20260809 23:18:45.268409 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 128177us
I20260809 23:18:45.285400 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:45.160116 27628 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:45.173381 27567 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:45.289685 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.291426 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.292126 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:45.292937 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:45.293638 24427 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:45.116575 27583 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:45.274616 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90373.6us
I20260809 23:18:45.296792 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.176928 27734 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:45.376616 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83170.9us
I20260809 23:18:45.300513 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:45.301280 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.378661 24546 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.305369 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.380275 24422 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:45.180485 27666 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.381381 27666 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.306541 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:45.383716 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77254.8us
I20260809 23:18:45.384608 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.293069 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.298475 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:45.387333 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94354.6us
I20260809 23:18:45.387385 24427 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.177456 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:45.388862 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.390295 27706 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.391228 27590 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.401090 27666 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:45.410032 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110838us
I20260809 23:18:45.189152 27612 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:45.415522 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.264041 27638 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:45.421940 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.423461 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:45.416606 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.173849 27593 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.202s
I20260809 23:18:45.418658 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.266040 27742 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:45.433511 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.401741 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:45.279593 27723 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.179045 27689 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.231s
I20260809 23:18:45.446384 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:45.434912 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 150659us
I20260809 23:18:45.297750 27567 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.298051 27648 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.304030 27619 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.161846 27569 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.283305 27749 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:45.431385 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.374069 27639 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.449708 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.409662 27734 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.451364 27567 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.499606 27538 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.499998 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.500411 27538 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:18:45.501546 27709 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.455946 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:45.457930 27619 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.418059 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.505545 27619 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.506068 27619 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 23:18:45.506654 27510 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.001s
I20260809 23:18:45.429970 27590 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:45.281883 27568 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.303107 27633 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.500672 24546 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:45.302949 27677 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
I20260809 23:18:45.521786 24312 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
W20260809 23:18:45.461923 27730 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.003s
I20260809 23:18:45.294909 27573 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.491338 27723 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:45.499702 27672 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:18:45.520766 24269 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.720s (SENT)
I20260809 23:18:45.546303 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.415745 27583 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:45.506721 27595 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.578s	user 0.006s	sys 0.001s
I20260809 23:18:45.451562 27606 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:45.501495 27538 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:45.499464 27593 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.432365 27638 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.562076 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.564630 27651 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:45.565167 27651 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:45.567065 27709 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.567727 27709 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.568171 27709 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.571303 27510 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:45.572014 27660 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:45.572448 27660 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:45.573443 27583 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.573292 27660 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.574476 24314 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:45.574623 27660 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.507344 27590 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.600220 27590 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:45.602763 24314 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:45.604713 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.605314 24312 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.605486 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.606169 24320 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:45.609014 24314 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:45.537606 27723 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.612046 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:45.515317 27626 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:45.613142 27626 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:45.613627 27626 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:45.510133 27742 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:45.619319 27742 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:45.620633 27742 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:45.567160 27715 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260809 23:18:45.552871 27730 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.578635 27569 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.622848 24316 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:45.623270 27679 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.579969 27593 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:45.532246 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 359833us
I20260809 23:18:45.625310 27593 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:45.594506 27711 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.000s
I20260809 23:18:45.580670 27573 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.626058 24425 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:45.544744 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.626430 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:45.627280 24540 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 2: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.567373 24430 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:45.595208 27658 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.572059 27510 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.456087 27748 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
I20260809 23:18:45.628053 27658 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:45.587496 27649 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.658s	user 0.006s	sys 0.004s
I20260809 23:18:45.584378 27616 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:45.566243 27651 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.612289 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.631881 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.555950 27538 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:45.627183 24424 raft_consensus.cc:2362] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3. Re-sending same reply.
I20260809 23:18:45.583393 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.597587 27568 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:45.636721 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 247931us
I20260809 23:18:45.547015 27666 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.550277 27672 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.628597 27658 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.638633 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.651655 24424 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.651754 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:45.652009 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.652487 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:45.652834 24541 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:45.653111 24424 raft_consensus.cc:2362] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4. Re-sending same reply.
I20260809 23:18:45.643448 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.645912 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.648043 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.491438 27714 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.656035 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.652292 24435 raft_consensus.cc:2362] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3. Re-sending same reply.
I20260809 23:18:45.657397 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.657966 24435 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.659149 24435 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:45.655480 24422 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.662385 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.662902 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.663455 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.664090 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.663522 25372 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:45.664888 24540 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.642289 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.668592 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:45.668736 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.669014 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.669389 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.663872 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.670040 24535 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:45.612848 27723 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.670341 25370 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.670799 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.718962 25375 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:45.715749 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.714841 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:45.724349 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 43746.6us
I20260809 23:18:45.724792 24541 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:45.714326 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.725623 24424 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:45.719342 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.728746 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.638213 27750 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:45.640156 27666 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:45.736351 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:45.641610 27655 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.880s	user 0.006s	sys 0.006s
I20260809 23:18:45.718416 27629 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.714341 27510 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.654738 27697 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.668763 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.735615 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 402) took 1105 ms. Trace:
I20260809 23:18:45.640717 27567 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.742242 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.719240 27538 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.742987 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:45.716966 27508 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.647s	user 0.005s	sys 0.004s
I20260809 23:18:45.718225 27626 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.773169 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.774037 27626 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.775223 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.775805 27567 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.775878 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.776988 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.777283 27655 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:45.777730 27655 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.778080 27655 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:45.777582 27628 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.652840 27672 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.721030 27633 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.756297 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.666489 27745 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:45.715361 27651 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.757532 25373 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.761788 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.762586 24430 rpcz_store.cc:276] 0809 23:18:44.630697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:44.650112 (+ 19415us) service_pool.cc:227] Handling call
0809 23:18:44.650399 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:45.627827 (+977428us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:45.735324 (+107497us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:45.735539 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2108127488}
I20260809 23:18:45.753635 27748 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:45.803583 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:45.668093 27714 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:45.804313 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.668916 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:45.765604 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 315973us
W20260809 23:18:45.766296 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 319999us
I20260809 23:18:45.764449 24534 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:45.722064 25365 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.653429 27590 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.720274 27658 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:45.754922 27725 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.662s	user 0.005s	sys 0.006s
I20260809 23:18:45.754094 27666 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.771093 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.771690 27538 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.782148 24535 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:45.799283 27745 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:45.775940 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.641480 27759 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:45.807294 27633 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.825683 27633 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.826205 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:45.826690 27590 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.827375 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.827764 27633 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.827971 24543 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:45.829916 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.806712 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:45.831089 27638 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:45.831840 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.832482 24321 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:18:45.832640 27680 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.762s	user 0.011s	sys 0.000s
I20260809 23:18:45.808643 27647 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.833443 27647 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.833765 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.834446 24314 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.836230 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.836418 27709 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.836992 24429 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:45.806075 27714 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.837555 25366 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:45.838421 27709 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.838956 27689 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:45.839354 27689 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.839684 27689 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.840566 27590 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.841387 24431 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.833454 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.813568 27718 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:45.843771 27718 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.844117 27718 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.757786 27629 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.837086 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:45.852023 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45356.5us
I20260809 23:18:45.852495 27629 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.758725 27750 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.854203 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.854839 24430 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:45.855249 27750 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.856145 24309 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:45.856318 27750 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:45.809687 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.856220 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.857223 24312 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.778920 27628 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.812872 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:45.857770 27746 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:45.858263 27746 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:45.807963 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 358914us
I20260809 23:18:45.858563 27746 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.818053 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.815259 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.859762 24544 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.859845 24534 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.743462 27764 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.815689 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:45.860503 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 658) took 3367 ms (client timeout 1720 ms). Trace:
I20260809 23:18:45.860705 27764 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:45.861444 27764 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.831035 27748 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:45.811908 25363 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
W20260809 23:18:45.860966 25365 rpcz_store.cc:269] 0809 23:18:42.492717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:42.684396 (+191679us) service_pool.cc:227] Handling call
0809 23:18:45.860286 (+3175890us) spinlock_profiling.cc:243] Waited 2.6 s on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:45.860429 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5980682496}
I20260809 23:18:45.863170 27633 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.863138 27748 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.863802 27633 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.866008 27633 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.866199 27677 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:45.815259 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.866639 27677 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:45.867522 27677 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.868816 27677 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.870255 27689 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.871263 27745 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.872598 27745 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.874418 27647 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.822045 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.804286 27626 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.853044 27658 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.876292 27658 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.876751 27658 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:45.878273 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.879230 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.879477 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.879981 24535 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 2: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.880661 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:45.880774 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.860951 24320 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.866658 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.883466 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.807695 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.885807 24431 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:45.887820 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.891528 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.892647 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:45.894054 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 222463us
I20260809 23:18:45.894534 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.895155 24309 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.892638 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.774838 27734 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.764561 27697 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.895164 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:45.910481 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 102857us
I20260809 23:18:45.911228 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.897389 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.897662 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:45.896605 24544 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:45.915964 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.916555 24544 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:45.916482 24309 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:45.876196 27626 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:45.923388 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.926885 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.930154 25372 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:45.930796 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.933307 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.934553 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.936112 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.933352 25373 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:45.938210 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:45.933540 24534 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:45.943111 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.943183 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.943750 27689 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.943859 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:45.943137 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.944830 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.945312 25365 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:45.945739 27697 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.946112 27730 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:45.946247 27568 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:45.946554 27730 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.946703 27568 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.947067 27730 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.947314 27766 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:45.947784 27766 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:45.948102 27766 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.948318 27569 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:45.948683 27569 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:45.949067 27569 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.949792 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.950516 25369 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:45.909463 27672 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:45.923238 27638 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:45.889853 27660 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:45.898306 27647 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.951167 27775 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.950887 27784 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260809 23:18:45.953512 27672 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:45.933344 27762 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:45.947158 27568 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:45.943228 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:45.969826 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.956310 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:45.971645 24545 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.971755 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:45.972265 24541 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:45.956804 24535 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:45.957244 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.954250 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.962199 27658 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:45.935189 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.976805 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:45.945683 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:45.957046 24423 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:45.980283 25370 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:45.982039 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.982659 24431 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:45.896611 27590 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.996186 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:45.897120 27721 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:45.979444 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.006916 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.009989 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.012236 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.013055 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.014314 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.016068 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.927274 27567 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:45.955020 27626 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.018649 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.016219 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.027375 24424 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.020874 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.979004 27577 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:45.979147 27400 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.021618 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.979233 27712 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.001102 27780 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.022343 27697 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.022269 27595 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:45.955448 27711 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260809 23:18:46.024471 27568 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.096686 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.955720 27647 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:46.100648 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:46.030704 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0641596899367869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260809 23:18:46.031540 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:45.951380 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.032505 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.104326 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.029162 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:45.996889 27709 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.111579 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:45.998788 27778 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.022251 27633 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:45.995929 27590 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.037286 24317 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:45.947160 27573 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260809 23:18:46.010286 27680 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:46.036862 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.152652 24428 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:46.005467 27689 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:46.041764 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.046605 25370 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:46.049026 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.052618 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.001282 27638 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.004176 27721 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.047835 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.023012 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.057407 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.161808 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.162060 25372 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:46.163035 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.063375 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.063445 27595 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:46.052551 27400 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.165162 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.109858 27709 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.064690 27734 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.052740 27712 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.177078 27712 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.066254 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.032449 27448 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.043512 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.179142 27448 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:46.179494 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 136083us
I20260809 23:18:46.067144 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.066958 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.181046 25365 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:46.063419 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.068979 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.068377 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.019943 27784 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260809 23:18:46.182251 24314 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.182782 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.183094 27784 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.183509 27784 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.035530 27672 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.184031 27784 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.077970 27762 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:46.184259 27672 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.086007 27714 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.186302 27672 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.017685 27750 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:46.035655 27569 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.186846 27750 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.187263 27750 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.188385 27569 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:45.954555 27660 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.190233 27660 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.027408 27775 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:46.096561 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.089628 24320 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260809 23:18:46.191243 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94765.5us
I20260809 23:18:46.191883 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.017109 27636 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.192970 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.092475 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.193660 24320 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.193825 24425 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:46.194423 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.093272 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:46.195264 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 102055us
I20260809 23:18:46.195919 27649 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:46.196317 27649 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.196434 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.196693 27649 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:46.196943 27568 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.197297 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.197486 27568 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.197906 27568 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.198192 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.099572 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.199018 27648 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.199399 27648 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.199489 27749 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.112416 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.118284 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.104341 27647 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.200433 24535 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.199838 27638 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.201222 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.125204 27658 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.120235 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.136935 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.036252 27751 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.100328 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.144788 24424 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:46.033200 27567 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.154582 25373 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:46.040685 27510 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260809 23:18:46.163913 24430 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.063934 27780 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.202777 24546 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:46.209064 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.209656 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:46.210084 24425 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:46.165205 24323 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:46.206221 24545 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:46.211949 27567 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.212193 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.210960 27751 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.213434 27751 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.179626 27448 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.208103 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.214671 27746 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.215046 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.199770 27648 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.216179 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.216676 27746 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.217401 24544 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5.
I20260809 23:18:46.217756 27759 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.218379 27759 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.149822 27590 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.205775 27628 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:46.175137 27709 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.219550 27628 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.220065 27628 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:46.166126 27721 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.202824 24542 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.221513 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.222095 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.222349 24546 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:46.222975 24542 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 2: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.223526 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.171657 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.224431 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.099992 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.102540 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:46.108042 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:46.113655 25325 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 2.173s (SENT) [suppressed 23 similar messages]
I20260809 23:18:46.137418 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.173018 27711 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260809 23:18:46.211220 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.236701 24423 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:46.225653 24422 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.137156 27633 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.248191 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.132697 27778 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.217217 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.222294 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.206023 24435 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:46.203217 24431 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:46.160398 27766 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.226330 27730 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.276199 24316 raft_consensus.cc:2354] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:18:46.276624 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.276909 27766 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.277094 27730 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.277261 24545 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.277776 27647 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.199898 27749 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.183604 24313 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.203146 24321 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:46.280239 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.280508 27647 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.280916 24544 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:46.281141 27778 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.281787 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.281896 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.173792 27689 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:46.198223 27568 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.283041 27749 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.164947 27666 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.205314 27595 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.226269 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.287563 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:46.288556 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.230458 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.227164 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.235562 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.232946 24535 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:46.224574 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.291250 24546 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:46.292225 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.292901 24546 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.293771 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.293923 25373 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
W20260809 23:18:46.226650 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 113037us
W20260809 23:18:46.226269 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 118310us
I20260809 23:18:46.296298 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.297873 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.289506 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:46.236066 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:46.299860 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63870.8us
I20260809 23:18:46.300666 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.304715 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.240486 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.306056 24542 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:46.225113 24311 raft_consensus.cc:2362] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2. Re-sending same reply.
I20260809 23:18:46.227463 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.307642 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.308415 24311 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:46.309365 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.311395 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.245129 24540 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.253043 24539 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:46.253585 24314 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:46.257887 24422 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:46.263510 27718 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.265380 27648 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.314386 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.315722 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.271486 27780 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.273257 27400 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.227854 27660 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.186375 27714 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.323515 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:46.285646 27711 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:46.280534 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.285558 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.329118 24321 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:46.285979 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.286046 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.229892 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.331140 24427 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.310045 24311 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:46.317984 25370 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:46.318748 27510 raft_consensus.cc:2741] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260809 23:18:46.332883 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:46.318236 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90865us
I20260809 23:18:46.334744 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.335928 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.315441 27636 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260809 23:18:46.319468 27590 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.336017 27569 raft_consensus.cc:2741] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 23:18:46.258754 27633 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.337893 27590 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.339264 24323 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.319782 27706 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.339711 27706 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:46.340088 27706 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.342018 27648 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.322438 27679 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260809 23:18:46.343250 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:46.343820 27679 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:46.344025 24422 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.344180 27679 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:46.321758 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.229463 27626 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.344663 24313 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:46.344898 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:46.345009 27679 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.345551 24313 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40860
I20260809 23:18:46.283392 27689 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.345901 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.346210 27679 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.296456 27742 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:46.347122 27742 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.347455 27742 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:18:46.284874 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 67725.2us
I20260809 23:18:46.322458 27746 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.319262 27647 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260809 23:18:46.351257 27746 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.336009 27748 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.351809 27746 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.319142 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.325032 27762 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:46.352504 24314 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:46.352846 27666 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.317018 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:46.353611 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.346009 27626 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.351867 27748 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.355250 27762 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.328548 27711 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:46.355571 27748 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.356154 27762 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.356387 27711 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.357813 27711 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.317489 27734 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.328008 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.332631 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:18:46.359859 24311 raft_consensus.cc:595] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.360267 24311 raft_consensus.cc:603] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:18:46.319875 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.361128 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.351387 24378 leader_election.cc:398] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:18:46.361835 24378 leader_election.cc:401] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:18:46.346037 24313 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:18:46.364059 24313 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.364176 24427 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:46.364450 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.296974 27759 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.359472 24320 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:46.367331 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:46.368592 24316 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.206804 27723 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.318665 27730 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.388885 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.389665 24321 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.369405 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:46.390723 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 331014us
I20260809 23:18:46.371797 24545 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:46.373415 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.376400 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.331676 24164 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 4, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:46.393412 24164 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 2 to 3, term changed from 1 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to <none>, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:46.332182 27567 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:46.396281 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.394730 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.401780 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.402194 24545 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.403615 24314 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:46.352746 27672 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.227762 27655 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.399156 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.420193 24542 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.366377 27714 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:18:46.364341 27780 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:46.397157 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.369141 27538 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:46.368872 27759 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:46.432617 24309 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.397758 24164 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:46.435000 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.388533 27400 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.398723 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:46.384995 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.410178 27775 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.396134 27666 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:46.389092 27730 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.451947 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.447724 27647 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 23:18:46.476997 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 191095us
W20260809 23:18:46.472608 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73965.7us
I20260809 23:18:46.420095 27763 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.479594 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.458138 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.480430 24535 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.456758 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.482623 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.389477 27751 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.421058 27712 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:18:46.446259 24313 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.464910 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:46.491302 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 395582us
I20260809 23:18:46.491456 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.493317 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.494002 24427 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5.
I20260809 23:18:46.495038 24422 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:46.495335 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.455965 24164 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 3 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:46.397614 27649 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.470692 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.419643 27672 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:46.472294 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87387.1us
I20260809 23:18:46.504884 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.506023 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.475384 24545 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:46.432610 27680 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:46.416702 27725 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.436160 27567 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.451483 27759 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.484675 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.483361 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.487097 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:46.498673 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.501432 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.454445 27780 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.423390 27593 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260809 23:18:46.488317 27590 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:46.509084 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.464742 27666 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.511605 24313 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.607465 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.608011 24423 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.514432 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.450265 27750 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:46.610458 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122113us
I20260809 23:18:46.476398 27730 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.485636 27763 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.476406 27796 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.585214 27648 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:46.508370 24164 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:18:46.584081 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 403549us
I20260809 23:18:46.530865 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.585229 24539 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:46.585536 24422 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:46.586916 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 99908.6us
I20260809 23:18:46.640452 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.641963 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.488834 27400 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.590728 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.643841 24424 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.479732 27619 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:46.644428 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.645058 24430 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.646427 24270 leader_election.cc:398] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5.
I20260809 23:18:46.647822 25371 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
W20260809 23:18:46.594050 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 85027.3us
I20260809 23:18:46.590194 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.614707 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.650123 24435 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:46.651057 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.507963 27510 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:46.510092 27677 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:46.621536 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 107189us
I20260809 23:18:46.632719 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.479043 27766 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.516638 27672 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.520498 27689 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:46.514732 27781 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:46.503397 27633 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.515489 27712 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:46.649088 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:46.488092 27720 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:46.611096 27666 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.654263 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.656049 24314 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260809 23:18:46.657150 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.663854 24164 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 2 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:18:46.672566 27780 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:46.692147 27800 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:18:46.631770 27763 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:46.675108 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 178377us
I20260809 23:18:46.628060 27804 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.657733 27769 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.517472 27568 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:46.654992 27764 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.793s	user 0.006s	sys 0.008s
I20260809 23:18:46.695477 24164 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 5 to 6, b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:18:46.663647 27714 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:46.705534 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48830.8us
I20260809 23:18:46.667768 27538 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.731768 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.671828 24422 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:46.655548 27510 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:46.665302 27677 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:46.720167 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.723618 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.663928 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.647804 27718 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.673961 27612 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:46.733470 24539 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:46.714557 27569 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.516937 27680 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:46.746140 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113493us
I20260809 23:18:46.705106 27573 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:46.739843 24311 raft_consensus.cc:2354] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4.
I20260809 23:18:46.726271 27593 raft_consensus.cc:2741] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 23:18:46.745857 24317 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.740927 24424 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:46.743578 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.769797 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.770493 24311 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.741974 27714 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.749529 24321 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:46.750157 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.781980 24320 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.750667 25370 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:46.751595 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.743813 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.783972 24314 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.718343 27804 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.782828 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.739720 27677 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.718837 27689 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
W20260809 23:18:46.755999 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99873.5us
I20260809 23:18:46.727483 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.744724 27769 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:46.789839 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.790349 24316 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.738848 24425 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:46.757609 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:46.667187 27784 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.705711 27781 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:46.725488 27805 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.718194 27721 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.775588 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.739682 27725 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:46.779536 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:46.729983 27633 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.780289 24313 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.824470 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.825821 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.734709 27510 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.730508 27590 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:46.822564 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.831431 24321 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.786276 24311 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:46.826713 24313 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.833452 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:46.834244 24311 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:46.730377 27723 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.752291 25366 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.730136 27655 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:46.793136 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.848219 24431 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 6
I20260809 23:18:46.793838 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.794387 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.803139 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.780341 27778 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.788215 27804 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.728790 27764 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.786697 27569 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260809 23:18:46.817917 27649 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:46.819886 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.810710 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.815200 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:46.861905 24424 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.826164 25323 leader_election.cc:398] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4.
I20260809 23:18:46.863332 25323 leader_election.cc:401] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:18:46.655041 27400 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.829751 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.831284 27590 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.877847 24323 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 1. Current term is 3. Ops: []
W20260809 23:18:46.810451 27448 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.596s	user 0.007s	sys 0.001s
I20260809 23:18:46.841809 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.795272 27636 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.849647 24316 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:46.806684 27805 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:46.850085 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55777.8us
I20260809 23:18:46.857028 27593 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:46.897091 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.890164 24534 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:46.898543 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.900646 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.902134 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.903631 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.823501 27751 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.810781 27648 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:46.853626 24320 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.789220 27619 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:46.808756 24542 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:46.821439 27720 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:46.818760 27689 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:46.844463 27709 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.620s	user 0.010s	sys 0.000s
I20260809 23:18:46.810319 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:46.858286 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:46.835273 27725 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.804652 27796 raft_consensus.cc:465] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Not starting pre-election -- already a leader
I20260809 23:18:46.739176 27567 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.835740 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:46.865965 25372 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:46.866873 27568 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:46.867326 27806 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260809 23:18:46.885968 27775 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:46.869343 27748 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.513s	user 0.005s	sys 0.002s
I20260809 23:18:46.870077 27633 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:46.806878 27816 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
W20260809 23:18:46.871227 27706 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.531s	user 0.006s	sys 0.004s
I20260809 23:18:46.841090 27538 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:46.874101 27628 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.654s	user 0.007s	sys 0.000s
I20260809 23:18:46.831008 27638 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260809 23:18:46.875753 27660 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.544s	user 0.006s	sys 0.002s
I20260809 23:18:46.876579 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:46.837729 24314 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:46.818027 27718 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260809 23:18:46.878609 24313 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:46.912578 27805 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.850443 24321 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:46.908253 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.912912 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:46.913071 25325 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.913563 24435 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:46.914618 24316 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:46.919102 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60910.4us
I20260809 23:18:46.807240 27750 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:46.923178 24311 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:46.819015 27712 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:46.925776 27810 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:46.853350 27811 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260809 23:18:46.879163 27762 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.011s	sys 0.038s
I20260809 23:18:46.916559 27651 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:46.920146 27749 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.637s	user 0.005s	sys 0.005s
I20260809 23:18:46.922821 27711 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:18:46.887387 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69235.4us
W20260809 23:18:46.954169 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 184094us
I20260809 23:18:46.897235 27448 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.031605 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:46.885800 27590 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:46.927091 27593 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:47.039273 24423 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:18:46.954365 27769 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:47.007692 25372 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260809 23:18:47.009047 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96098.7us
I20260809 23:18:47.009678 24542 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:47.010639 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:47.010047 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:47.012902 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 237648us
I20260809 23:18:47.014400 24431 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
W20260809 23:18:47.009626 27808 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260809 23:18:46.954062 27826 raft_consensus.cc:2741] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260809 23:18:47.037009 24424 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:18:47.007400 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99245.5us
I20260809 23:18:46.957341 27660 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:47.054943 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:47.029148 27762 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.011s	sys 0.038s
W20260809 23:18:47.035293 27746 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.683s	user 0.011s	sys 0.000s
I20260809 23:18:47.015718 27638 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:47.066825 24542 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:47.028277 27633 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:47.029390 27811 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:46.922439 27709 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.013775 27833 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.079200 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:47.079747 25369 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:47.031080 27651 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:47.080036 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:47.041950 27781 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:46.954594 27628 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.036546 27680 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.181102 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:47.182248 24321 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:47.149962 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91801.7us
I20260809 23:18:47.183790 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:46.954128 27748 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:46.955457 27706 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.184866 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:47.077504 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:47.185899 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 108504us
I20260809 23:18:47.040172 27714 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.041977 27593 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:47.058749 27745 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:47.078121 27638 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:47.193464 24266 leader_election.cc:398] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5.
I20260809 23:18:47.199900 24266 leader_election.cc:401] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260809 23:18:47.193943 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114816us
I20260809 23:18:47.036412 27749 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:47.202309 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 147486us
I20260809 23:18:47.013890 27619 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.167663 27816 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.078640 27721 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.076119 27784 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.166605 27826 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:47.075062 27834 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:46.958742 27775 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.218396 24320 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:47.228994 24431 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:47.180194 27651 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:47.219476 24544 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:47.235831 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52309.3us
I20260809 23:18:47.157249 27811 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:47.181470 27778 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.201380 27593 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:47.187979 27633 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:47.191720 27649 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:47.086110 27746 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.232247 25363 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:47.244618 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:47.216331 27647 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:47.217694 27648 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.033881 27720 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.196462 27842 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.126s
I20260809 23:18:47.248590 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:47.256937 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.212540 27764 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:47.308008 27764 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:47.309475 27837 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:47.310066 27837 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:47.311147 27448 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.311226 27709 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.311031 27837 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.312386 27749 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.315620 27706 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.283567 27638 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.290637 27725 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:47.238078 27730 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.603s	user 0.004s	sys 0.005s
W20260809 23:18:47.268579 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75287.6us
I20260809 23:18:47.236884 27826 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:47.239836 27745 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:47.051810 27689 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.240209 27568 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5.
I20260809 23:18:47.216883 27723 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260809 23:18:47.293696 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.330485 24424 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:47.256254 27400 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.239951 27834 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:47.296643 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52059.4us
I20260809 23:18:47.297642 24539 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:47.178781 27833 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.246085 27827 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:18:47.298668 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.021s	sys 0.076s
W20260809 23:18:47.294121 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.004s	sys 0.027s
W20260809 23:18:47.343941 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.021s	sys 0.076s
W20260809 23:18:47.344097 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.005s	sys 0.027s
W20260809 23:18:47.345106 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 696044us
I20260809 23:18:47.345943 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:47.188138 27840 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.143s
I20260809 23:18:47.228734 27750 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:47.285701 27649 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:47.279999 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:47.217399 27714 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:47.266307 27721 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260809 23:18:47.305932 27648 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.335590 27709 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:47.336105 27831 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.300s	user 0.001s	sys 0.000s
I20260809 23:18:47.336170 27723 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:47.347028 27745 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:47.334177 24430 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:47.257901 27816 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.331380 27826 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:47.341279 27835 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260809 23:18:47.305945 27647 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:47.327291 27672 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.634s	user 0.005s	sys 0.006s
I20260809 23:18:47.353320 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:47.324380 25366 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:47.356590 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.356206 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:47.359578 27750 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:47.360561 27839 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260809 23:18:47.238312 27769 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.328823 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:47.369954 27833 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:47.338816 27834 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:47.334061 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.413408 24321 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:47.364287 27748 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:47.364189 27714 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.365826 27709 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:47.349103 27784 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.370095 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.439990 24431 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:47.352684 27568 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:47.297713 27781 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.327380 27730 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.376353 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:47.447690 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71466.2us
I20260809 23:18:47.447048 27745 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.357635 27821 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.448875 27730 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.386518 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:47.365020 27628 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:47.391552 27751 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4.
I20260809 23:18:47.391976 27706 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:18:47.380267 27679 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.034s	user 0.010s	sys 0.005s
I20260809 23:18:47.388649 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.467288 24424 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:47.344303 27725 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:47.371330 27723 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:47.411942 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83164.1us
I20260809 23:18:47.446012 27568 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:47.415691 27749 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:47.348764 27847 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260809 23:18:47.410737 27649 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.417323 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:47.370980 27590 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.375623 27856 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
W20260809 23:18:47.411048 27759 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.789s	user 0.006s	sys 0.009s
I20260809 23:18:47.393322 27672 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.431608 27750 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.433400 27816 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.435525 27746 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.436335 27709 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.438330 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:47.423664 27666 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.707s	user 0.010s	sys 0.002s
W20260809 23:18:47.450311 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63878.4us
I20260809 23:18:47.456831 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:47.355759 27857 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.092s
I20260809 23:18:47.438604 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:47.358233 27633 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:47.487414 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 217795us
I20260809 23:18:47.414244 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:47.492214 24424 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
W20260809 23:18:47.466817 27510 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.616s	user 0.009s	sys 0.001s
I20260809 23:18:47.460665 27721 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:47.458588 27619 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.508575 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.511888 24320 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:47.505435 24321 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:47.445739 27784 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.552846 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.743953 24429 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:47.449167 27821 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.504405 27568 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:47.567943 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.746833 24423 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:47.747843 24431 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:47.747923 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.568125 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:18:47.750903 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:47.751789 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:47.607702 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:47.754029 24424 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:47.607894 24545 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:47.495318 27725 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.542903 27856 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:47.604015 24309 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:47.449172 27647 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:47.563091 27742 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.215s	user 0.003s	sys 0.010s
I20260809 23:18:47.694515 27626 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.621351 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:47.616608 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:47.557338 27833 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:47.728098 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.724690 27706 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:47.742238 24422 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:47.606269 27835 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:18:47.745379 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:47.563722 27400 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:47.769750 27709 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:47.605842 27666 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.609630 27679 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.780858 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.837754 24320 raft_consensus.cc:2354] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
W20260809 23:18:47.741644 27849 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.281s	user 0.000s	sys 0.002s
W20260809 23:18:47.613631 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 199533us
W20260809 23:18:47.790565 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 174062us
I20260809 23:18:47.722699 27842 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:47.694375 27648 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.794037 24321 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:47.513059 27749 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:47.567963 27857 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:47.605619 27836 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.558533 27759 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.816789 27626 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.767869 27721 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:47.775535 27778 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.781999 27823 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.942114 25370 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:47.832775 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87485.8us
I20260809 23:18:47.848644 24429 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:47.849359 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:47.948169 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98945.1us
I20260809 23:18:47.783885 27577 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:47.853880 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.953374 24423 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:47.856343 27636 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:47.846886 27400 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.933375 27648 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.828868 27628 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:18:47.568951 27723 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.817127 27538 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:47.769378 27510 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.836555 27865 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.937426 25373 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:47.951924 27869 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.860800 27857 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:47.954378 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2034) took 1696 ms. Trace:
W20260809 23:18:47.946291 25368 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.580s	user 0.002s	sys 0.020s
W20260809 23:18:47.967252 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 481640us
W20260809 23:18:47.941534 27677 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.136s	user 0.011s	sys 0.020s
W20260809 23:18:47.835410 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 214142us
W20260809 23:18:47.981770 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 736) took 1910 ms (client timeout 2032 ms). Trace:
I20260809 23:18:47.937132 27714 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:47.828105 27833 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:47.941534 27804 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.079s	user 0.014s	sys 0.007s
I20260809 23:18:47.820477 27742 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:47.747481 27619 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:47.959518 27636 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:47.963807 27616 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:47.863509 27679 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.024219 25371 rpcz_store.cc:269] 0809 23:18:46.047210 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.047338 (+   128us) service_pool.cc:227] Handling call
0809 23:18:47.957916 (+1910578us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:47.958044 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1031283584}
I20260809 23:18:48.000833 27697 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:47.956375 27778 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.008994 27595 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.009222 27628 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:47.976441 27577 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.030807 27590 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.027971 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:48.003556 27648 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.013921 27706 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:47.867197 27836 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.060882 27864 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.475s	user 0.002s	sys 0.000s
W20260809 23:18:48.061230 27854 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.588s	user 0.002s	sys 0.000s
I20260809 23:18:48.043792 27636 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:47.947355 27842 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.039470 27714 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:48.061600 27721 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.039516 27672 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:48.078214 27651 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.810s	user 0.012s	sys 0.002s
I20260809 23:18:48.015062 27865 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.030638 27677 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.075529 27784 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:47.962764 27823 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.111467 27863 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.548s	user 0.000s	sys 0.003s
I20260809 23:18:48.095129 27725 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.021790 27749 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.127521 27861 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.374s	user 0.002s	sys 0.000s
I20260809 23:18:48.022534 27869 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:48.122437 27868 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260809 23:18:48.040355 27804 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.131896 27763 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.406s	user 0.010s	sys 0.005s
I20260809 23:18:48.081019 27857 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.107924 27714 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.005205 27856 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.171361 27869 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.180739 27833 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.180626 27856 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:48.173056 27823 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:48.181187 27826 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.794s	user 0.009s	sys 0.017s
I20260809 23:18:48.196246 27842 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:48.209884 27865 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.189924 27836 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:48.225520 27834 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.786s	user 0.007s	sys 0.011s
I20260809 23:18:47.935482 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:47.977872 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:48.266690 24311 raft_consensus.cc:2354] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6.
W20260809 23:18:48.266512 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 305139us
I20260809 23:18:48.272055 24321 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:48.268260 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2268) took 1466 ms. Trace:
I20260809 23:18:48.275125 24320 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:48.274818 24314 rpcz_store.cc:276] 0809 23:18:46.801472 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.956377 (+154905us) service_pool.cc:227] Handling call
0809 23:18:47.008712 (+ 52335us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:47.008918 (+   206us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:47.008950 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:47.028586 (+ 19636us) log.cc:842] Serialized 26 byte log entry
0809 23:18:47.217763 (+189177us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:47.217812 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:47.217896 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:48.267426 (+1049530us) raft_consensus.cc:1671] finished
0809 23:18:48.267494 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:48.267990 (+   496us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139492,"threads_started":1,"wal-append.queue_time_us":256282}
I20260809 23:18:48.025177 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:48.020166 24546 rpcz_store.cc:276] 0809 23:18:46.258097 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.346743 (+ 88646us) service_pool.cc:227] Handling call
0809 23:18:47.954185 (+1607442us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:47.954320 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2528536832}
I20260809 23:18:48.072324 27595 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:48.385342 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 359969us
I20260809 23:18:48.386188 27595 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.327930 27577 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.329751 27619 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.329946 27714 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.118289 27651 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.078131 27590 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.077658 27510 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:48.118443 27666 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.387308 27826 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.154235 27763 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.377907 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 811) took 1136 ms. Trace:
I20260809 23:18:48.379961 27677 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.131254 27706 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.379530 27628 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.325388 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1774) took 1827 ms. Trace:
W20260809 23:18:48.208524 27811 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.910s	user 0.018s	sys 0.009s
W20260809 23:18:48.187247 27805 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.178s	user 0.010s	sys 0.008s
I20260809 23:18:48.062307 27616 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.404392 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.001s	sys 0.010s
I20260809 23:18:48.404528 27869 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:48.405660 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.000s	sys 0.024s
W20260809 23:18:48.406078 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.008s	sys 0.005s
W20260809 23:18:48.403703 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.002s	sys 0.007s
W20260809 23:18:48.198485 27810 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.171s	user 0.007s	sys 0.009s
I20260809 23:18:48.406363 27823 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.181872 27778 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:48.535987 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.002s	sys 0.030s
I20260809 23:18:48.429574 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 678) took 1831 ms. Trace:
W20260809 23:18:48.535682 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.008s	sys 0.014s
W20260809 23:18:48.539198 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.001s	sys 0.019s
W20260809 23:18:48.549542 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s	user 0.004s	sys 0.013s
W20260809 23:18:48.480134 25372 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.008s	sys 0.017s
W20260809 23:18:48.480471 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 120534us
W20260809 23:18:48.480345 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.002s	sys 0.015s
I20260809 23:18:48.417507 27742 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.417408 25368 rpcz_store.cc:276] 0809 23:18:47.241395 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.242032 (+   637us) service_pool.cc:227] Handling call
0809 23:18:48.377594 (+1135562us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:48.563331 25372 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.008s	sys 0.029s
I20260809 23:18:48.435081 27648 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.568354 27626 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:48.570885 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 356260us
I20260809 23:18:48.581282 24320 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:18:48.581470 27651 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:48.581712 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.002s	sys 0.025s
I20260809 23:18:48.581943 27836 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.592422 27836 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.583146 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:48.595283 27679 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:48.595611 24270 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 2.309s (SENT)
I20260809 23:18:48.596474 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.597584 25368 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 1.
I20260809 23:18:48.431185 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.430488 27759 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.590638 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.603524 24320 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:48.480068 27895 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:48.599419 27400 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.599814 27590 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.601531 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 751) took 1106 ms. Trace:
I20260809 23:18:48.556264 27826 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:48.594844 27836 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.613452 24321 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:48.615182 27778 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:48.611819 27895 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.618317 27590 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.618517 27804 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.602706 27510 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:48.612847 25371 rpcz_store.cc:276] 0809 23:18:47.494840 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.600078 (+1105238us) service_pool.cc:227] Handling call
0809 23:18:48.600639 (+   561us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:48.601163 (+   524us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:48.601189 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:48.601250 (+    61us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:48.601287 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.601352 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:48.601457 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.226387 27834 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.562048 27538 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:18:48.640085 24423 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.665s	user 0.000s	sys 0.016s
I20260809 23:18:48.598910 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:48.647171 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1809) took 1037 ms. Trace:
I20260809 23:18:48.648401 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:48.647614 24321 rpcz_store.cc:276] 0809 23:18:47.609547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.977497 (+367950us) service_pool.cc:227] Handling call
0809 23:18:48.642801 (+665304us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.651185 24546 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:48.530478 24427 rpcz_store.cc:276] 0809 23:18:46.497605 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.497757 (+   152us) service_pool.cc:227] Handling call
0809 23:18:46.498119 (+   362us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:46.515781 (+ 17662us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:46.515823 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:46.582942 (+ 67119us) log.cc:842] Serialized 26 byte log entry
0809 23:18:47.344438 (+761496us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:47.344481 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:47.344553 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:48.324529 (+979976us) raft_consensus.cc:1671] finished
0809 23:18:48.324584 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:48.325035 (+   451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35361,"raft.queue_time_us":20138,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":7951,"thread_start_us":716290,"threads_started":1,"wal-append.queue_time_us":772023}
I20260809 23:18:48.612721 27714 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.651867 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2074) took 1472 ms. Trace:
I20260809 23:18:48.657714 24546 rpcz_store.cc:276] 0809 23:18:47.179429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.388012 (+1208583us) service_pool.cc:227] Handling call
0809 23:18:48.651755 (+263743us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.666352 27697 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:48.666759 24541 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:48.651261 27510 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.647066 27538 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:48.672880 24314 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:48.659998 24423 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:48.668185 27697 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.679081 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 292278us
I20260809 23:18:48.681202 25368 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:48.676476 27568 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.852s	user 0.007s	sys 0.012s
I20260809 23:18:48.699592 27895 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.673918 27538 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:48.637272 27869 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.707671 27510 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.676651 27778 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.711256 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:48.676467 27896 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:48.652328 27823 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.727594 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.001s	sys 0.010s
I20260809 23:18:48.672740 27810 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:48.543177 25365 rpcz_store.cc:276] 0809 23:18:46.530061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.582018 (+ 51957us) service_pool.cc:227] Handling call
0809 23:18:47.958063 (+1376045us) spinlock_profiling.cc:243] Waited 5.66 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:18:48.005772 (+ 47709us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:48.021760 (+ 15988us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:48.029579 (+  7819us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:48.037224 (+  7645us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:48.039854 (+  2630us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.062721 (+ 22867us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:48.335040 (+272319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13015424}
W20260809 23:18:48.733229 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.002s	sys 0.011s
I20260809 23:18:48.736362 27616 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.716773 27805 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.747617 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.751741 25371 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:48.752142 27778 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.752782 24546 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:48.755043 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2010) took 2838 ms (client timeout 2150 ms). Trace:
W20260809 23:18:48.755561 24543 rpcz_store.cc:269] 0809 23:18:45.878922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:45.889958 (+ 11036us) service_pool.cc:227] Handling call
0809 23:18:48.717428 (+2827470us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.717721 (+   293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1555710720}
I20260809 23:18:48.756932 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:48.748940 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 462) took 1975 ms. Trace:
I20260809 23:18:48.712878 27568 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.760782 27626 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.761329 27568 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.761528 27626 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.763741 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52230.6us
W20260809 23:18:48.763870 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 798) took 2863 ms (client timeout 2447 ms). Trace:
I20260809 23:18:48.764226 27834 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:48.764691 25325 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 2.173s (SENT)
I20260809 23:18:48.765367 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:48.765559 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 740) took 2713 ms (client timeout 2004 ms). Trace:
I20260809 23:18:48.757843 24539 rpcz_store.cc:276] 0809 23:18:46.736670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.792439 (+ 55769us) service_pool.cc:227] Handling call
0809 23:18:46.792675 (+   236us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:47.333031 (+540356us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.711920 (+1378889us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:48.712237 (+   317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1160908544}
W20260809 23:18:48.768421 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 684) took 1449 ms (client timeout 1660 ms). Trace:
W20260809 23:18:48.769047 25370 rpcz_store.cc:269] 0809 23:18:47.318448 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.337389 (+ 18941us) service_pool.cc:227] Handling call
0809 23:18:48.767913 (+1430524us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.768195 (+   282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":915046272}
I20260809 23:18:48.771229 24542 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:48.771530 25375 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:48.773145 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 809) took 2037 ms. Trace:
I20260809 23:18:48.773453 25375 rpcz_store.cc:276] 0809 23:18:46.735607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.750572 (+ 14965us) service_pool.cc:227] Handling call
0809 23:18:48.771088 (+2020516us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:48.771989 (+   901us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.772928 (+   939us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54004a5bd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.773097 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4566346496}
I20260809 23:18:48.769248 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:48.775772 24539 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:48.776099 27836 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:48.776321 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2077) took 1591 ms. Trace:
I20260809 23:18:48.776823 24539 rpcz_store.cc:276] 0809 23:18:47.185403 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.768817 (+1583414us) service_pool.cc:227] Handling call
0809 23:18:48.776240 (+  7423us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.777004 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 680) took 2246 ms. Trace:
I20260809 23:18:48.777631 25373 rpcz_store.cc:276] 0809 23:18:46.530774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.530918 (+   144us) service_pool.cc:227] Handling call
0809 23:18:46.531273 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:48.009070 (+1477797us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.776497 (+767427us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:48.776774 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3224961280}
I20260809 23:18:48.778985 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.779680 27914 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:48.779774 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:48.770602 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:48.774586 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.771411 27626 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.784823 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.785660 25373 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:48.785852 27869 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.786895 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.788158 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:48.766129 25369 rpcz_store.cc:269] 0809 23:18:46.051675 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.051777 (+   102us) service_pool.cc:227] Handling call
0809 23:18:48.765118 (+2713341us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54004a5bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.765375 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2064230272}
W20260809 23:18:48.792855 27593 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.510s	user 0.010s	sys 0.014s
W20260809 23:18:48.793322 27636 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.685s	user 0.006s	sys 0.006s
I20260809 23:18:48.737723 27896 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.716120 27714 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.794576 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.014s	sys 0.043s
I20260809 23:18:48.833412 24313 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:48.844857 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:48.833609 24321 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:48.768409 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:48.845911 25373 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:48.833586 24539 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:48.799160 24498 leader_election.cc:398] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
I20260809 23:18:48.717806 27811 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.850984 24498 leader_election.cc:401] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:18:48.793465 27593 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.806834 24546 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:48.853404 25370 raft_consensus.cc:2354] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:18:48.856406 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:18:48.861426 27811 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.794143 27616 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:18:48.815076 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.015s	sys 0.043s
I20260809 23:18:48.795863 27778 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:48.867429 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2075) took 1683 ms (client timeout 2230 ms). Trace:
W20260809 23:18:48.871282 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2037) took 1299 ms (client timeout 1586 ms). Trace:
W20260809 23:18:48.872259 24539 rpcz_store.cc:269] 0809 23:18:47.183690 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.787592 (+1603902us) service_pool.cc:227] Handling call
0809 23:18:48.867348 (+ 79756us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.871203 24542 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:48.838581 27577 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.838785 27595 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:48.873792 24546 rpcz_store.cc:269] 0809 23:18:47.566743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.674202 (+1107459us) service_pool.cc:227] Handling call
0809 23:18:48.865582 (+191380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.838320 27720 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.846422 27895 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.793812 27636 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.879531 27706 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.885291 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:48.877277 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:48.890223 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:48.891214 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:48.886318 27706 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.892496 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:48.893450 27706 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.893784 27593 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.895066 27577 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.896108 27636 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.897535 27577 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.898325 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:48.899102 24422 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:48.874225 27778 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:48.901194 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1781) took 2837 ms (client timeout 2447 ms). Trace:
W20260809 23:18:48.901624 24317 rpcz_store.cc:269] 0809 23:18:46.064031 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.297589 (+233558us) service_pool.cc:227] Handling call
0809 23:18:48.900702 (+2603113us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.900929 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1028046208}
I20260809 23:18:48.903339 27763 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.903725 27595 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.904016 27917 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260809 23:18:48.905081 27714 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:48.905620 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2270) took 1896 ms (client timeout 1970 ms). Trace:
W20260809 23:18:48.906131 24311 rpcz_store.cc:269] 0809 23:18:47.009089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.009806 (+   717us) service_pool.cc:227] Handling call
0809 23:18:48.905363 (+1895557us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540032fad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.905543 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2801763072}
I20260809 23:18:48.907370 24427 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260809 23:18:48.907701 27709 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.950s	user 0.010s	sys 0.007s
W20260809 23:18:48.764313 25363 rpcz_store.cc:269] 0809 23:18:45.900853 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:45.908950 (+  8097us) service_pool.cc:227] Handling call
0809 23:18:48.763332 (+2854382us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400631354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.763631 (+   299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2435194112}
I20260809 23:18:48.909843 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:48.911880 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 424) took 2314 ms. Trace:
I20260809 23:18:48.912223 24425 rpcz_store.cc:276] 0809 23:18:46.596970 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.608447 (+ 11477us) service_pool.cc:227] Handling call
0809 23:18:46.732666 (+124219us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:46.790637 (+ 57971us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.911483 (+2120846us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540018c750. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:48.911699 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":285095296}
I20260809 23:18:48.913882 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 431) took 2055 ms. Trace:
I20260809 23:18:48.914214 24435 rpcz_store.cc:276] 0809 23:18:46.858187 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.858332 (+   145us) service_pool.cc:227] Handling call
0809 23:18:46.867593 (+  9261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:47.009394 (+141801us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.913463 (+1904069us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b54003eb7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:48.913724 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104897408}
I20260809 23:18:48.816651 27914 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:48.923983 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 199252us
I20260809 23:18:48.915849 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 631) took 2197 ms. Trace:
I20260809 23:18:48.923427 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.907388 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.928864 24309 rpcz_store.cc:276] 0809 23:18:46.718692 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.719102 (+   410us) service_pool.cc:227] Handling call
0809 23:18:47.295466 (+576364us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:18:47.295728 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:47.780334 (+484606us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.915339 (+1135005us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b540019dcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:48.915647 (+   308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":704494080}
I20260809 23:18:48.950908 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.953459 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:48.953990 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:48.960282 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2080) took 1670 ms (client timeout 2172 ms). Trace:
W20260809 23:18:48.960798 24539 rpcz_store.cc:269] 0809 23:18:47.289547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.885872 (+1596325us) service_pool.cc:227] Handling call
0809 23:18:48.960218 (+ 74346us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:48.961933 24309 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:18:48.962118 27878 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:48.963313 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.964573 27896 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:48.965009 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.965394 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:48.966080 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:48.966286 24429 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:48.967124 27718 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:48.967365 27895 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.968041 27718 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.962901 27878 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.975291 24317 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:48.973234 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:48.942759 27810 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:48.975438 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:48.985555 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:48.991389 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:48.898911 27538 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.993187 24431 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:48.957501 27778 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:48.962122 27856 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.746s	user 0.010s	sys 0.008s
I20260809 23:18:48.995802 27823 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.996615 27896 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:18:48.961385 27616 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.971650 27852 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:49.000582 27852 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.002386 27616 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:48.976961 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.971395 27834 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.003458 25363 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.004207 27834 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:48.972641 27593 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.007679 27718 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.007513 27593 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.007946 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:49.009747 27593 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.009907 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2033) took 2752 ms. Trace:
I20260809 23:18:49.010262 24545 rpcz_store.cc:276] 0809 23:18:46.257751 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.507613 (+249862us) service_pool.cc:227] Handling call
0809 23:18:46.511325 (+  3712us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:47.754859 (+1243534us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.009442 (+1254583us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:49.009730 (+   288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2521367936}
I20260809 23:18:49.010520 27766 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:49.011672 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.012531 27766 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:49.012966 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:48.971221 27763 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:49.013324 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.013926 24435 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:18:49.014195 25365 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:49.013825 27763 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.015416 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.015777 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.016593 25371 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:49.016785 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.017374 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.958269 27720 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:49.018225 25371 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:49.012620 25363 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.019272 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.019939 27720 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.020083 25369 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
W20260809 23:18:48.962906 27795 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.159s	user 0.005s	sys 0.011s
I20260809 23:18:49.011729 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.023644 24545 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:49.012636 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.018103 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.024224 27616 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.024485 24539 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:48.971163 27916 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:49.024983 24309 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:49.007539 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:48.971554 27595 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.024703 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 625) took 3132 ms. Trace:
I20260809 23:18:49.025770 25375 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:48.970901 27895 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.014781 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:48.998799 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2041) took 1252 ms. Trace:
I20260809 23:18:49.038554 27810 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:48.973544 27805 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.058543 24546 rpcz_store.cc:276] 0809 23:18:47.746247 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.883731 (+1137484us) service_pool.cc:227] Handling call
0809 23:18:48.998733 (+115002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.062371 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 810) took 1829 ms. Trace:
I20260809 23:18:49.069367 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.069707 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.076040 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:49.039762 27916 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.040926 24312 rpcz_store.cc:276] 0809 23:18:45.891960 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:45.892102 (+   142us) service_pool.cc:227] Handling call
0809 23:18:45.934029 (+ 41927us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:45.935183 (+  1154us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:45.935211 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:45.935687 (+   476us) log.cc:842] Serialized 26 byte log entry
0809 23:18:46.377183 (+441496us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:46.377237 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:46.377320 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:48.859949 (+2482629us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b240019ba58. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:18:49.024228 (+164279us) raft_consensus.cc:1671] finished
0809 23:18:49.024249 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.024520 (+   271us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":716,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":5730176,"thread_start_us":348424,"threads_started":1,"wal-append.queue_time_us":629306}
W20260809 23:18:49.077499 27912 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.279s	user 0.003s	sys 0.000s
I20260809 23:18:49.078279 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.078867 27916 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:49.079203 27918 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260809 23:18:49.079556 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2271) took 2070 ms (client timeout 2230 ms). Trace:
I20260809 23:18:49.080133 27896 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.080715 24429 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:49.080873 27896 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:49.081830 27573 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:49.082471 27573 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:49.083685 27573 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:49.085338 27827 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:49.085853 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2274) took 1719 ms. Trace:
I20260809 23:18:49.012959 27766 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:49.016786 27763 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.026160 27616 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:49.025760 27775 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:49.231925 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2069) took 2947 ms (client timeout 2106 ms). Trace:
I20260809 23:18:49.086505 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:49.069658 25372 rpcz_store.cc:276] 0809 23:18:47.233280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.771505 (+538225us) service_pool.cc:227] Handling call
0809 23:18:47.771839 (+   334us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:48.676682 (+904843us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:48.678479 (+  1797us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:48.679644 (+  1165us) log.cc:842] Serialized 26 byte log entry
0809 23:18:48.789957 (+110313us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:18:48.987637 (+197680us) log.cc:842] Serialized 14 byte log entry
0809 23:18:48.992666 (+  5029us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.992736 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:48.998108 (+  5372us) raft_consensus.cc:1671] finished
0809 23:18:48.998144 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.062301 (+ 64157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":198998,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":32277,"thread_start_us":1009687,"threads_started":3,"wal-append.queue_time_us":207739}
I20260809 23:18:49.079140 24312 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:49.233595 24544 rpcz_store.cc:269] 0809 23:18:46.284586 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.319373 (+ 34787us) service_pool.cc:227] Handling call
0809 23:18:49.231502 (+2912129us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:49.231723 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2037030784}
I20260809 23:18:49.213238 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:49.237591 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2043) took 1471 ms (client timeout 1880 ms). Trace:
W20260809 23:18:49.238085 24539 rpcz_store.cc:269] 0809 23:18:47.766218 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.978328 (+1212110us) service_pool.cc:227] Handling call
0809 23:18:49.237524 (+259196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.196543 24378 leader_election.cc:398] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6.
I20260809 23:18:49.240517 24378 leader_election.cc:401] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:18:49.190232 27896 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:49.201400 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81856.6us
I20260809 23:18:49.254895 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
W20260809 23:18:49.303570 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2042) took 1538 ms (client timeout 1957 ms). Trace:
W20260809 23:18:49.304307 24545 rpcz_store.cc:269] 0809 23:18:47.765867 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.011389 (+1245522us) service_pool.cc:227] Handling call
0809 23:18:49.303492 (+292103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.192482 24428 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:49.079947 24313 rpcz_store.cc:269] 0809 23:18:47.009402 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.010424 (+  1022us) service_pool.cc:227] Handling call
0809 23:18:49.079352 (+2068928us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:49.079498 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4030770432}
I20260809 23:18:49.061173 27852 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:49.201905 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 465) took 2380 ms (client timeout 1537 ms). Trace:
I20260809 23:18:49.249567 27730 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:49.254128 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.254280 24422 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:49.086402 24317 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.086227 24320 rpcz_store.cc:276] 0809 23:18:47.366684 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.619058 (+1252374us) service_pool.cc:227] Handling call
0809 23:18:48.619724 (+   666us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:48.852723 (+232999us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:48.852760 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:48.868270 (+ 15510us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:48.868322 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.085601 (+217279us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:49.085643 (+    42us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.085761 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":507001088}
I20260809 23:18:49.262810 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2254) took 3212 ms. Trace:
I20260809 23:18:49.317852 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.140573 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.080767 27812 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:49.163981 25365 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260809 23:18:49.770330 24309 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.664s	user 0.003s	sys 0.025s
W20260809 23:18:49.763470 24541 rpcz_store.cc:269] 0809 23:18:46.821917 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.829189 (+  7272us) service_pool.cc:227] Handling call
0809 23:18:49.201647 (+2372458us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:49.201827 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4009633152}
W20260809 23:18:49.662533 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 514028us
I20260809 23:18:49.773475 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:49.437213 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:49.333088 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 375814us
I20260809 23:18:49.318298 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:49.147210 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.659818 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:49.768787 24424 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.875s	user 0.014s	sys 0.015s
I20260809 23:18:49.318805 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2272) took 2186 ms. Trace:
I20260809 23:18:49.775290 24424 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:49.777796 24545 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:49.778371 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 469) took 2466 ms (client timeout 2706 ms). Trace:
W20260809 23:18:49.778805 24545 rpcz_store.cc:269] 0809 23:18:47.312447 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.560129 (+2247682us) service_pool.cc:227] Handling call
0809 23:18:49.778289 (+218160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.756357 24323 rpcz_store.cc:276] 0809 23:18:46.050448 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.340710 (+290262us) service_pool.cc:227] Handling call
0809 23:18:46.341039 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:46.878455 (+537416us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.234193 (+2355738us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54002535d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:49.262624 (+ 28431us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1233930624}
I20260809 23:18:49.163816 27834 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.785926 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.193269 27818 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:49.503466 27510 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.798146 27689 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.798404 27510 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.798861 27689 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.799372 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:49.438187 27821 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:49.800815 27810 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:49.319571 27697 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.482s
I20260809 23:18:49.650964 27766 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.769651 27812 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:49.440261 27805 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.802954 27766 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:49.803498 27805 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.522011 27781 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:49.806177 27805 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.805588 27781 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.806535 27797 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:49.807054 27797 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:49.807416 27797 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:49.809297 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2047) took 2022 ms (client timeout 2220 ms). Trace:
W20260809 23:18:49.809731 24539 rpcz_store.cc:269] 0809 23:18:47.767237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.263979 (+1496742us) service_pool.cc:227] Handling call
0809 23:18:49.789802 (+525823us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.810078 27914 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.810695 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.775302 24316 rpcz_store.cc:276] 0809 23:18:47.086337 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.103537 (+ 17200us) service_pool.cc:227] Handling call
0809 23:18:47.103977 (+   440us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:47.154646 (+ 50669us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:47.154677 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:47.155253 (+   576us) log.cc:842] Serialized 26 byte log entry
0809 23:18:48.904541 (+1749288us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b18002b0c00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:18:48.999344 (+ 94803us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:49.159358 (+160014us) log.cc:842] Serialized 14 byte log entry
0809 23:18:49.196026 (+ 36668us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.227799 (+ 31773us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:49.228017 (+   218us) raft_consensus.cc:1671] finished
0809 23:18:49.234404 (+  6387us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.272614 (+ 38210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21690,"raft.queue_time_us":74119,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":22672,"spinlock_wait_cycles":14827904,"thread_start_us":177601,"threads_started":2,"wal-append.queue_time_us":1897719}
I20260809 23:18:49.811600 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2084) took 1041 ms. Trace:
I20260809 23:18:49.811915 27763 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.811991 24539 rpcz_store.cc:276] 0809 23:18:48.770133 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.810396 (+1040263us) service_pool.cc:227] Handling call
0809 23:18:49.811526 (+  1130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.812167 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:49.812717 27810 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.813149 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:49.316740 27927 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260809 23:18:49.813295 27810 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.788275 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.797806 27818 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:49.812876 27763 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.800668 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:49.817054 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2045) took 2050 ms (client timeout 2169 ms). Trace:
W20260809 23:18:49.821820 24546 rpcz_store.cc:269] 0809 23:18:47.766722 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.142219 (+1375497us) service_pool.cc:227] Handling call
0809 23:18:49.816975 (+674756us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.442764 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.820206 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.823444 24317 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:49.648824 27827 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.563s	user 0.007s	sys 0.004s
I20260809 23:18:49.824137 24544 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:49.824642 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:49.770753 24309 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:49.825639 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 649) took 1075 ms. Trace:
I20260809 23:18:49.825914 24309 rpcz_store.cc:276] 0809 23:18:48.750165 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.966915 (+216750us) service_pool.cc:227] Handling call
0809 23:18:49.825592 (+858677us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.835467 27616 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.835526 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.836259 24321 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:49.836686 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.836992 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.837162 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.837271 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2281) took 1058 ms. Trace:
I20260809 23:18:49.837815 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 647) took 1166 ms. Trace:
I20260809 23:18:49.837661 24320 rpcz_store.cc:276] 0809 23:18:48.778531 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.819669 (+1041138us) service_pool.cc:227] Handling call
0809 23:18:49.837196 (+ 17527us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.839178 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.839874 24323 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:49.840341 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.841178 24323 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260809 23:18:49.823204 24270 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.018s	sys 0.008s
I20260809 23:18:49.841331 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.841856 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1808) took 2232 ms. Trace:
I20260809 23:18:49.842320 24309 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:49.842264 24323 rpcz_store.cc:276] 0809 23:18:47.609102 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.840474 (+2231372us) service_pool.cc:227] Handling call
0809 23:18:49.840860 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.841617 (+   757us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.841777 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.843053 27822 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:49.843852 27914 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.841063 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.866897 27711 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.867755 24424 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:49.860029 27822 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:49.867611 27914 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.868080 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.868355 27822 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:18:49.844767 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1780) took 2330 ms (client timeout 1880 ms). Trace:
W20260809 23:18:49.868975 24423 rpcz_store.cc:269] 0809 23:18:47.514390 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.747613 (+233223us) service_pool.cc:227] Handling call
0809 23:18:49.844537 (+2096924us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:49.844680 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470584704}
I20260809 23:18:49.869800 27914 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.870636 24423 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:18:49.841159 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2088) took 1069 ms. Trace:
I20260809 23:18:49.867798 27711 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.872910 27763 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.874306 27711 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.871577 24541 rpcz_store.cc:276] 0809 23:18:48.771469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.840098 (+1068629us) service_pool.cc:227] Handling call
0809 23:18:49.841102 (+  1004us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:49.866986 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46496us
I20260809 23:18:49.875633 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.876384 24541 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.845880 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:49.878845 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2090) took 1106 ms. Trace:
I20260809 23:18:49.879189 24539 rpcz_store.cc:276] 0809 23:18:48.772140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.812881 (+1040741us) service_pool.cc:227] Handling call
0809 23:18:49.878783 (+ 65902us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.838704 24321 rpcz_store.cc:276] 0809 23:18:48.674580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.836930 (+1162350us) service_pool.cc:227] Handling call
0809 23:18:49.837754 (+   824us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.848794 24425 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:18:49.848718 27810 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:49.653146 27878 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:49.809388 27781 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:49.850028 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2013) took 3918 ms (client timeout 2297 ms). Trace:
I20260809 23:18:49.850587 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.203011 27906 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.633s
I20260809 23:18:49.890652 24321 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:49.891188 24422 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:49.835726 24312 raft_consensus.cc:2396] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260809 23:18:49.841984 24270 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.018s	sys 0.008s
W20260809 23:18:49.892650 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 704887us
I20260809 23:18:49.841461 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.893472 24545 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:49.893966 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2085) took 1123 ms. Trace:
I20260809 23:18:49.894281 24545 rpcz_store.cc:276] 0809 23:18:48.770460 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.818020 (+1047560us) service_pool.cc:227] Handling call
0809 23:18:49.893911 (+ 75891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.852739 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:49.717672 27775 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 6
I20260809 23:18:49.835587 24427 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:49.896520 24309 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:18:49.703190 27538 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:49.845296 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 644) took 1894 ms. Trace:
I20260809 23:18:49.816412 27723 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:49.880376 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2081) took 2509 ms (client timeout 3287 ms). Trace:
W20260809 23:18:49.928089 24541 rpcz_store.cc:269] 0809 23:18:47.371418 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.875256 (+2503838us) service_pool.cc:227] Handling call
0809 23:18:49.880313 (+  5057us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:49.928345 24498 connection.cc:232] RPC call timeout handler was delayed by -1.8051130848471075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24470000000000003s for a response
I20260809 23:18:49.929123 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.883298 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.929912 24541 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:49.930244 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1992204918060452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
I20260809 23:18:49.930498 27764 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:49.800741 27510 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:49.820597 27818 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:49.887423 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.891348 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 645) took 1353 ms. Trace:
I20260809 23:18:49.940248 24321 rpcz_store.cc:276] 0809 23:18:48.541145 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.880021 (+1338876us) service_pool.cc:227] Handling call
0809 23:18:49.890290 (+ 10269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.891042 (+   752us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.891258 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.859028 24313 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:18:49.897368 24317 rpcz_store.cc:276] 0809 23:18:47.950501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.844376 (+1893875us) service_pool.cc:227] Handling call
0809 23:18:49.844711 (+   335us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.844935 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:49.844958 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:49.845013 (+    55us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:49.845067 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.845119 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.845216 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.871569 27812 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.897328 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2285) took 1117 ms. Trace:
I20260809 23:18:49.946866 24309 rpcz_store.cc:276] 0809 23:18:48.780199 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.895368 (+1115169us) service_pool.cc:227] Handling call
0809 23:18:49.895639 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.897007 (+  1368us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.897251 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:49.889835 24534 rpcz_store.cc:269] 0809 23:18:45.931791 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:45.942857 (+ 11066us) service_pool.cc:227] Handling call
0809 23:18:49.849434 (+3906577us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b5400270bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:49.849708 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2176509696}
I20260809 23:18:49.907289 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2280) took 1129 ms. Trace:
I20260809 23:18:49.903354 27538 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:49.948242 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1804) took 2452 ms. Trace:
I20260809 23:18:49.948642 24313 rpcz_store.cc:276] 0809 23:18:47.491365 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.838455 (+2347090us) service_pool.cc:227] Handling call
0809 23:18:49.838800 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.943420 (+104620us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.943615 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.856451 27827 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
I20260809 23:18:49.914417 27896 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:49.912110 25372 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:18:49.912384 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2212) took 3401 ms. Trace:
I20260809 23:18:49.912495 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.916450 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:49.917405 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.921341 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 643) took 2557 ms. Trace:
I20260809 23:18:49.921569 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.950958 24314 rpcz_store.cc:276] 0809 23:18:47.363758 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.282281 (+918523us) service_pool.cc:227] Handling call
0809 23:18:48.282981 (+   700us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:48.687418 (+404437us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.921069 (+1233651us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b540019cdd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:49.921281 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":818862976}
I20260809 23:18:49.922544 27775 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6.
I20260809 23:18:49.921902 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:49.923322 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:49.924130 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.929589 27763 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:49.898267 24546 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:49.930898 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.963114 24541 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:49.942441 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.943099 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:49.947589 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2277) took 1463 ms. Trace:
I20260809 23:18:49.964154 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.964234 24323 rpcz_store.cc:276] 0809 23:18:48.480524 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.878400 (+1397876us) service_pool.cc:227] Handling call
0809 23:18:49.939546 (+ 61146us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.941907 (+  2361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:49.943128 (+  1221us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:49.943197 (+    69us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:49.943241 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.943297 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.943409 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.947918 24312 rpcz_store.cc:276] 0809 23:18:48.778035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.244970 (+466935us) service_pool.cc:227] Handling call
0809 23:18:49.907228 (+662258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.910902 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.965543 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.965536 25370 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:49.943202 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:49.949805 24428 rpcz_store.cc:276] 0809 23:18:46.511143 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.511274 (+   131us) service_pool.cc:227] Handling call
0809 23:18:46.516547 (+  5273us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.821366 (+3304819us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.912074 (+ 90708us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:49.912319 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6049245440}
I20260809 23:18:49.945453 27834 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:49.967446 24317 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:18:49.968936 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.969210 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2089) took 1193 ms. Trace:
I20260809 23:18:49.969560 24541 rpcz_store.cc:276] 0809 23:18:48.771767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.930650 (+1158883us) service_pool.cc:227] Handling call
0809 23:18:49.965289 (+ 34639us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.970350 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.971139 24309 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:49.983203 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.984030 24312 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:49.984830 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.985600 24312 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:49.985294 27763 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:49.987146 27896 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:49.983213 25371 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:49.987670 27763 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:49.987177 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.989681 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.975597 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:49.988787 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 652) took 1099 ms. Trace:
I20260809 23:18:49.962460 25365 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:49.991712 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.991855 24312 rpcz_store.cc:276] 0809 23:18:48.916997 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.986138 (+1069141us) service_pool.cc:227] Handling call
0809 23:18:49.986425 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.986774 (+   349us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:49.986806 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:49.986846 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:49.986882 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.988156 (+  1274us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:49.988269 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:49.990825 24314 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:49.995683 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:49.995800 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.975144 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.996662 24545 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:49.996845 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:50.001487 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:49.997099 24431 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:50.004081 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.006860 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:49.996943 24534 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:50.007735 27852 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:50.008173 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.008996 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2087) took 1238 ms. Trace:
I20260809 23:18:50.009307 24534 rpcz_store.cc:276] 0809 23:18:48.771091 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.007895 (+1236804us) service_pool.cc:227] Handling call
0809 23:18:50.008935 (+  1040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.010453 27822 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:49.991585 27834 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:18:50.012575 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 457) took 3632 ms (client timeout 2173 ms). Trace:
I20260809 23:18:50.013051 27822 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.012710 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:49.967104 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:50.008869 24312 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:50.005366 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.014792 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2036) took 2448 ms. Trace:
W20260809 23:18:50.013211 24540 rpcz_store.cc:269] 0809 23:18:46.380370 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.456519 (+ 76149us) service_pool.cc:227] Handling call
0809 23:18:50.012021 (+3555502us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54000b5c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:50.012346 (+   325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647934848}
I20260809 23:18:50.020619 24546 rpcz_store.cc:276] 0809 23:18:47.566400 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.967497 (+2401097us) service_pool.cc:227] Handling call
0809 23:18:50.014727 (+ 47230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.006349 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:50.021415 24317 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:50.022423 24313 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:49.931351 27764 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.020462 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:50.027635 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2287) took 1111 ms. Trace:
I20260809 23:18:50.028185 24321 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:18:50.028101 24312 rpcz_store.cc:276] 0809 23:18:48.911715 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.992795 (+1081080us) service_pool.cc:227] Handling call
0809 23:18:50.008664 (+ 15869us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.022909 (+ 14245us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.023145 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.024526 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.010548 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.030876 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2050) took 1007 ms. Trace:
I20260809 23:18:50.031242 24545 rpcz_store.cc:276] 0809 23:18:49.026982 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.007226 (+980244us) service_pool.cc:227] Handling call
0809 23:18:50.030821 (+ 23595us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.032830 24541 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:50.034293 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.034737 24534 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:50.036006 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92865.7us
I20260809 23:18:50.038690 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.038832 24321 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:50.039279 24312 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:50.039499 24545 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.047355 24309 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:50.045214 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.050422 24317 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:50.054742 24540 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:50.056975 27903 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:18:50.059720 24545 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:50.060397 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2048) took 1038 ms. Trace:
I20260809 23:18:50.060752 24545 rpcz_store.cc:276] 0809 23:18:49.022370 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.031718 (+1009348us) service_pool.cc:227] Handling call
0809 23:18:50.060355 (+ 28637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.060876 24546 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:50.062559 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.059764 27903 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:50.067639 27903 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:50.068642 27903 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.070140 27903 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.012615 27834 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:50.071445 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.075743 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.078038 24428 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.052116 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.060220 27721 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:50.064113 24545 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:50.072636 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.083896 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:50.086225 24435 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:18:50.084923 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:50.091007 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:50.090682 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.100677 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2049) took 1078 ms. Trace:
I20260809 23:18:50.105326 24540 rpcz_store.cc:276] 0809 23:18:49.022694 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.055538 (+1032844us) service_pool.cc:227] Handling call
0809 23:18:50.100587 (+ 45049us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:50.103433 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2086) took 1332 ms (client timeout 1530 ms). Trace:
I20260809 23:18:50.080693 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 750) took 2710 ms. Trace:
I20260809 23:18:50.106755 24539 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:50.111589 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.124064 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.124982 24546 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 3: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:50.119551 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.215552 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.121196 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:50.106774 24544 rpcz_store.cc:269] 0809 23:18:48.770744 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.422377 (+651633us) service_pool.cc:227] Handling call
0809 23:18:50.103360 (+680983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.217478 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.217813 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.218676 24544 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.116623 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.215317 24540 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:50.220664 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2093) took 1197 ms. Trace:
I20260809 23:18:50.222388 24540 rpcz_store.cc:276] 0809 23:18:49.022834 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.106388 (+1083554us) service_pool.cc:227] Handling call
0809 23:18:50.220598 (+114210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.221544 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2095) took 1012 ms. Trace:
I20260809 23:18:50.223567 24545 rpcz_store.cc:276] 0809 23:18:49.209450 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.084382 (+874932us) service_pool.cc:227] Handling call
0809 23:18:50.221217 (+136835us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.080791 27721 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:18:50.230396 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 174516us
I20260809 23:18:50.238539 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.239288 24546 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:50.124111 27748 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:50.088616 27751 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:50.242766 24427 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:50.088474 27805 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:50.245306 24311 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:50.221015 27938 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:50.252425 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.230042 27510 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:50.112223 27764 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.258289 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:50.261152 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.142810 27797 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:50.267278 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 493909us
I20260809 23:18:50.268973 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.269819 24545 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:50.250509 27805 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:50.273675 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 653) took 1320 ms. Trace:
I20260809 23:18:50.274055 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:50.277429 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 959146us
I20260809 23:18:50.279897 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 466) took 3225 ms. Trace:
I20260809 23:18:50.280392 24545 rpcz_store.cc:276] 0809 23:18:47.047516 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.232070 (+3184554us) service_pool.cc:227] Handling call
0809 23:18:50.269403 (+ 37333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.272405 (+  3002us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.272634 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.274008 24314 rpcz_store.cc:276] 0809 23:18:48.955616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.993277 (+1037661us) service_pool.cc:227] Handling call
0809 23:18:50.020307 (+ 27030us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.273294 (+252987us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.273324 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.273362 (+    38us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:50.273396 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.273519 (+   123us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.273623 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":226216,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":12129,"thread_start_us":247910,"threads_started":1}
I20260809 23:18:50.281548 24543 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
W20260809 23:18:50.281582 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 256627us
I20260809 23:18:50.286163 24540 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.282053 24543 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:50.287221 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.210969 25366 rpcz_store.cc:276] 0809 23:18:47.368777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.567996 (+199219us) service_pool.cc:227] Handling call
0809 23:18:47.568475 (+   479us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:47.721231 (+152756us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:47.721279 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:47.722078 (+   799us) log.cc:842] Serialized 26 byte log entry
0809 23:18:49.996603 (+2274525us) spinlock_profiling.cc:243] Waited 7.41 ms on lock 0x7b540064c420. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac237f0028 00007fac23860a7b 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c
0809 23:18:50.023589 (+ 26986us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:50.033118 (+  9529us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.036961 (+  3843us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:50.038549 (+  1588us) raft_consensus.cc:1671] finished
0809 23:18:50.044529 (+  5980us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.079611 (+ 35082us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21057,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":26718,"spinlock_wait_cycles":17042304,"thread_start_us":869120,"threads_started":1,"wal-append.queue_time_us":860495,"wal-append.run_cpu_time_us":8519,"wal-append.run_wall_time_us":1012348}
W20260809 23:18:50.282624 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 192120us
I20260809 23:18:50.289785 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:50.290880 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 167353us
I20260809 23:18:50.293805 24545 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:50.275767 27764 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:50.302703 27649 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:50.303249 27649 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.304379 27721 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:50.305195 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 467) took 3244 ms (client timeout 1668 ms). Trace:
I20260809 23:18:50.305414 27721 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:50.305737 24542 rpcz_store.cc:269] 0809 23:18:47.060533 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.079714 (+ 19181us) service_pool.cc:227] Handling call
0809 23:18:50.304940 (+3225226us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54002706d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:50.305104 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3608421120}
I20260809 23:18:50.307009 27946 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.307471 27946 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.307834 27946 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.307883 24543 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:50.308987 24543 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.307473 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:50.311363 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2092) took 1289 ms. Trace:
I20260809 23:18:50.311653 24544 rpcz_store.cc:276] 0809 23:18:49.022451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.217517 (+1195066us) service_pool.cc:227] Handling call
0809 23:18:50.311308 (+ 93791us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.313396 24546 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:50.314121 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2051) took 1064 ms. Trace:
I20260809 23:18:50.314527 24546 rpcz_store.cc:276] 0809 23:18:49.249763 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.219837 (+970074us) service_pool.cc:227] Handling call
0809 23:18:50.314059 (+ 94222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.267076 27896 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:50.349700 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2076) took 3165 ms. Trace:
I20260809 23:18:50.351477 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:50.352624 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.295742 24422 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.356261 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1806) took 2864 ms. Trace:
I20260809 23:18:50.355949 24534 rpcz_store.cc:276] 0809 23:18:47.184376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.244254 (+3059878us) service_pool.cc:227] Handling call
0809 23:18:50.270733 (+ 26479us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.349174 (+ 78441us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.349227 (+    53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.349313 (+    86us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:50.349359 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.349452 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.349605 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.368718 24543 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260809 23:18:50.355721 27766 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.509s	user 0.012s	sys 0.000s
I20260809 23:18:50.310918 27723 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:50.370635 24428 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:50.266283 27510 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.263563 27751 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.383344 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.257077 27938 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.387324 24545 raft_consensus.cc:2362] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4. Re-sending same reply.
W20260809 23:18:50.396138 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 163740us
I20260809 23:18:50.388653 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.358289 24320 rpcz_store.cc:276] 0809 23:18:47.491932 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.839748 (+2347816us) service_pool.cc:227] Handling call
0809 23:18:49.840291 (+   543us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:49.845627 (+  5336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:49.845662 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:49.846126 (+   464us) log.cc:842] Serialized 26 byte log entry
0809 23:18:50.230444 (+384318us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:50.259871 (+ 29427us) log.cc:842] Serialized 14 byte log entry
0809 23:18:50.260672 (+   801us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.260757 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:50.355979 (+ 95222us) raft_consensus.cc:1671] finished
0809 23:18:50.355997 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.356185 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":506,"prepare.queue_time_us":29240,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":127,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":118,"thread_start_us":383551,"threads_started":1,"wal-append.queue_time_us":499182}
I20260809 23:18:50.470908 24422 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:50.402474 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.577399 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.577993 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:18:50.579429 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.580286 24542 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:50.467885 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 116537us
I20260809 23:18:50.581627 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.574766 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.584275 24546 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:50.584426 27797 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:50.481477 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.585973 24546 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:50.582479 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.598577 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2078) took 3333 ms. Trace:
I20260809 23:18:50.599242 24428 raft_consensus.cc:2362] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3. Re-sending same reply.
I20260809 23:18:50.599056 24546 rpcz_store.cc:276] 0809 23:18:47.264970 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.585106 (+3320136us) service_pool.cc:227] Handling call
0809 23:18:50.585539 (+   433us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.598346 (+ 12807us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.598509 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.600176 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:50.588136 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2079) took 3322 ms. Trace:
I20260809 23:18:50.384629 27723 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.600884 24542 rpcz_store.cc:276] 0809 23:18:47.265826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.587045 (+3321219us) service_pool.cc:227] Handling call
0809 23:18:50.587447 (+   402us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.587699 (+   252us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.587737 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.587793 (+    56us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:50.587847 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.587928 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.588047 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.602443 24544 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:50.594367 27938 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:50.362694 27945 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:50.312523 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:50.596272 27896 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:50.508386 27952 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
W20260809 23:18:50.482347 24270 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 2.150s (SENT)
I20260809 23:18:50.599013 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.615413 27941 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:50.616041 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1788) took 1606 ms. Trace:
I20260809 23:18:50.617906 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:50.618518 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.619237 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.620306 27689 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:50.620783 24435 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:50.622541 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.622711 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1220) took 3273 ms. Trace:
I20260809 23:18:50.641036 24431 rpcz_store.cc:276] 0809 23:18:49.009711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.800366 (+790655us) service_pool.cc:227] Handling call
0809 23:18:50.615802 (+815436us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:50.615964 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259657984}
I20260809 23:18:50.623376 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.643687 24423 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:50.644250 24545 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:50.644719 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.645005 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2083) took 1875 ms. Trace:
I20260809 23:18:50.623533 24534 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:50.624150 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.645417 24545 rpcz_store.cc:276] 0809 23:18:48.769719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.642455 (+1872736us) service_pool.cc:227] Handling call
0809 23:18:50.644077 (+  1622us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.644724 (+   647us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.644923 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.624266 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.646250 24425 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:18:50.626267 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 444) took 1605 ms. Trace:
I20260809 23:18:50.628396 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.629256 27649 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.631163 27953 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:50.631693 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2044) took 2865 ms. Trace:
I20260809 23:18:50.636988 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:50.649901 24422 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.601192 27723 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.647142 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.645419 24423 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:50.649999 24540 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:50.650293 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2082) took 1881 ms. Trace:
I20260809 23:18:50.653846 24546 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:50.653750 24534 rpcz_store.cc:276] 0809 23:18:48.769276 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.623000 (+1853724us) service_pool.cc:227] Handling call
0809 23:18:50.623330 (+   330us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.650065 (+ 26735us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.650244 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.654386 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 472) took 1684 ms. Trace:
I20260809 23:18:50.654759 24540 rpcz_store.cc:276] 0809 23:18:48.970056 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.223235 (+1253179us) service_pool.cc:227] Handling call
0809 23:18:50.654321 (+431086us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.646732 24424 rpcz_store.cc:276] 0809 23:18:49.020594 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.868503 (+847909us) service_pool.cc:227] Handling call
0809 23:18:49.892364 (+ 23861us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:49.902405 (+ 10041us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:49.902446 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:49.902494 (+    48us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:49.902557 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:49.902664 (+   107us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.626019 (+723355us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b54003331d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:50.626164 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54179072}
I20260809 23:18:50.643252 24313 rpcz_store.cc:276] 0809 23:18:47.349100 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.037756 (+2688656us) service_pool.cc:227] Handling call
0809 23:18:50.621890 (+584134us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:50.622599 (+   709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":350016768}
I20260809 23:18:50.651944 24542 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:50.649587 24544 rpcz_store.cc:276] 0809 23:18:47.766468 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.608361 (+2841893us) service_pool.cc:227] Handling call
0809 23:18:50.631184 (+ 22823us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.631377 (+   193us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.631400 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.631441 (+    41us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:50.631473 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.631526 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.631618 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.653252 24431 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:50.656971 27959 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:50.662094 27715 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:50.662551 27715 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.649730 27949 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.663007 27715 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.663273 27949 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:50.662264 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.664247 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.645702 27945 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:50.670194 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 183700us
I20260809 23:18:50.665313 24545 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.663697 27949 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.668133 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.664633 27941 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.668406 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.673053 24429 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:50.673235 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55427.7us
I20260809 23:18:50.669370 24427 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:50.669694 24425 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:50.669049 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:50.672011 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73105us
I20260809 23:18:50.669431 27689 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:50.669020 24544 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:50.649478 27943 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:50.679037 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.659693 25368 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:50.679529 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 477) took 1687 ms. Trace:
I20260809 23:18:50.679982 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.684147 24380 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:50.683864 24544 rpcz_store.cc:276] 0809 23:18:48.997517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.656464 (+1658947us) service_pool.cc:227] Handling call
0809 23:18:50.668856 (+ 12392us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.679275 (+ 10419us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.679467 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.684476 24380 leader_election.cc:401] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:18:50.672891 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2040) took 3064 ms. Trace:
I20260809 23:18:50.662350 27959 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.685199 24378 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:50.688746 24425 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:50.691407 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.685465 24542 rpcz_store.cc:276] 0809 23:18:47.608794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.603468 (+2994674us) service_pool.cc:227] Handling call
0809 23:18:50.648554 (+ 45086us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.667606 (+ 19052us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.672839 (+  5233us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:50.686335 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 204036us
I20260809 23:18:50.671887 27945 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:50.687703 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.687361 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 475) took 1710 ms. Trace:
I20260809 23:18:50.676964 27689 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.697184 24540 rpcz_store.cc:276] 0809 23:18:48.977270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.655803 (+1678533us) service_pool.cc:227] Handling call
0809 23:18:50.656017 (+   214us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.680232 (+ 24215us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.680263 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.686834 (+  6571us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:50.686885 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.686976 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.687305 (+   329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9947,"raft.queue_time_us":26769}
I20260809 23:18:50.686333 27649 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:50.701169 27903 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.627s	user 0.011s	sys 0.000s
I20260809 23:18:50.692865 27567 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:50.695003 27938 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:50.698400 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.716845 24545 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:50.716974 24544 raft_consensus.cc:2362] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4. Re-sending same reply.
I20260809 23:18:50.717657 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.712160 24431 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:18:50.704353 24269 leader_election.cc:398] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:18:50.719967 24269 leader_election.cc:401] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:18:50.701566 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.715108 24429 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:18:50.715744 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.717293 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 470) took 1748 ms. Trace:
I20260809 23:18:50.691200 27959 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.713380 27567 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.705394 27689 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:50.721829 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.707594 27510 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:50.713114 27822 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3.
I20260809 23:18:50.692909 27666 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260809 23:18:50.737607 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.738220 24542 raft_consensus.cc:2362] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5. Re-sending same reply.
I20260809 23:18:50.724107 27944 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:50.726078 27715 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.739071 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.739135 27944 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.727396 24427 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:50.740480 27715 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.720927 24545 rpcz_store.cc:276] 0809 23:18:48.969232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.646260 (+1677028us) service_pool.cc:227] Handling call
0809 23:18:50.658489 (+ 12229us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.717114 (+ 58625us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.717253 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.730098 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.741495 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.741820 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.742573 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.743239 24540 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260809 23:18:50.743775 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 478) took 1722 ms. Trace:
I20260809 23:18:50.744387 24540 rpcz_store.cc:276] 0809 23:18:49.021562 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.742676 (+1721114us) service_pool.cc:227] Handling call
0809 23:18:50.742975 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.743555 (+   580us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.743718 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.746136 24540 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:50.746817 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2091) took 1724 ms. Trace:
I20260809 23:18:50.747208 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.747197 24540 rpcz_store.cc:276] 0809 23:18:49.022083 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.745270 (+1723187us) service_pool.cc:227] Handling call
0809 23:18:50.745644 (+   374us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.746568 (+   924us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.746763 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.750442 24540 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:18:50.732496 27567 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.751253 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2055) took 1500 ms. Trace:
I20260809 23:18:50.751729 24540 rpcz_store.cc:276] 0809 23:18:49.251084 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.749733 (+1498649us) service_pool.cc:227] Handling call
0809 23:18:50.750076 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.750901 (+   825us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.751195 (+   294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:50.753300 24540 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:18:50.755414 24540 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:50.757443 24540 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:18:50.762683 24540 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:50.765043 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.731289 24423 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:50.711694 27649 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.732461 27764 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.735777 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:50.770479 27649 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.771082 27764 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.771546 27764 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:50.771716 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57588.3us
I20260809 23:18:50.773417 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.774163 27822 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:50.774216 27715 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:50.774329 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.774858 27715 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.691579 27943 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.775338 27822 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.775874 27822 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.776078 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:50.776250 27822 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.777007 27943 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:50.777254 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.778962 27680 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:50.779369 27941 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.780140 27680 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:18:50.780550 27909 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260809 23:18:50.782177 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2219) took 3331 ms. Trace:
I20260809 23:18:50.775384 27715 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.782572 24430 rpcz_store.cc:276] 0809 23:18:47.450537 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.450831 (+   294us) service_pool.cc:227] Handling call
0809 23:18:47.451154 (+   323us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:47.451596 (+   442us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:47.451635 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:47.458246 (+  6611us) log.cc:842] Serialized 26 byte log entry
0809 23:18:48.674898 (+1216652us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:48.701023 (+ 26125us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:48.716571 (+ 15548us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:50.522555 (+1805984us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b1800248920. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:18:50.781840 (+259285us) raft_consensus.cc:1671] finished
0809 23:18:50.781866 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.782107 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5054,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":10231,"spinlock_wait_cycles":31411456,"thread_start_us":1083024,"threads_started":1,"wal-append.queue_time_us":1103375}
I20260809 23:18:50.785742 27649 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.780552 27680 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.792706 27680 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:50.793943 27822 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.765826 24544 raft_consensus.cc:2362] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4. Re-sending same reply.
W20260809 23:18:50.733961 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61176us
I20260809 23:18:50.728094 27938 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:50.844544 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67349.2us
W20260809 23:18:50.804661 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75756.8us
I20260809 23:18:50.805150 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:50.806257 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.824020 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.841176 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.844280 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.836087 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.857401 24435 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:50.837288 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.838471 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.832010 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.850124 24544 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
W20260809 23:18:50.826316 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79267.7us
I20260809 23:18:50.866001 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.866480 25372 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:50.870291 27943 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:18:50.878486 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68690.9us
I20260809 23:18:50.856057 24423 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:50.880194 24430 raft_consensus.cc:2354] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:18:50.798908 27715 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.880002 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:50.884040 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.889087 24427 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3. Re-sending same reply.
I20260809 23:18:50.878865 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:50.891217 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.885641 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:50.843804 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119142us
I20260809 23:18:50.832234 27680 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.893754 27633 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.894394 27649 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.897365 27764 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.898487 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.900316 27938 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:50.901635 24423 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
W20260809 23:18:50.902465 27818 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.953s	user 0.012s	sys 0.002s
I20260809 23:18:50.780141 27941 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:50.853679 25365 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:50.921404 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.829146 27977 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:18:50.909862 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.929369 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.903517 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:50.930757 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.939163 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 471) took 1957 ms. Trace:
I20260809 23:18:50.927004 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:50.940618 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:50.947229 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.948187 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:50.948418 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:50.950708 24540 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:50.944213 24423 raft_consensus.cc:2362] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3. Re-sending same reply.
I20260809 23:18:50.952469 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:50.953019 24435 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 4
I20260809 23:18:50.922901 27633 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.956082 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:50.927767 27834 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:18:50.957155 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78391.4us
I20260809 23:18:50.958657 24540 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:18:50.945379 24534 rpcz_store.cc:276] 0809 23:18:48.969684 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.654942 (+1685258us) service_pool.cc:227] Handling call
0809 23:18:50.669652 (+ 14710us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.912645 (+242993us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:50.912675 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:50.926262 (+ 13587us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:50.926326 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:50.926406 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:50.926524 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5068,"raft.queue_time_us":39813,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6542,"thread_start_us":226826,"threads_started":1}
I20260809 23:18:50.939419 27649 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:50.960791 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:50.963814 25369 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:50.900372 27846 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260809 23:18:50.959919 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:50.941804 27941 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.947351 27896 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:50.968892 24314 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:18:50.926501 27943 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260809 23:18:50.974629 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:50.975446 27949 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:50.976276 27649 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:50.977548 27822 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.979178 27941 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:50.979689 27689 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:50.980955 27680 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.983219 27953 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:50.983305 27715 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.942098 27952 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:50.967347 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:50.983881 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81191.9us
W20260809 23:18:50.987612 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74152.1us
I20260809 23:18:51.016695 27680 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:51.028802 27680 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:51.028870 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.029902 27680 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:50.989316 24435 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:51.030817 27649 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.031368 27952 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:51.033763 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 354935us
I20260809 23:18:51.035205 27730 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.036063 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:51.036962 27971 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:51.038177 27976 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:51.038636 27976 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.039021 27976 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.040052 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.040302 27689 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:50.996799 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.041826 27822 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:51.009315 25370 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 23:18:51.042205 27822 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:51.042799 27822 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.042297 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.043272 27805 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.044086 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:51.023995 27896 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.044320 27805 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.044598 24321 raft_consensus.cc:2362] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4. Re-sending same reply.
I20260809 23:18:51.045508 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.046131 27896 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.046535 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:50.962838 27834 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:50.963498 27938 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:50.965073 27977 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.025874 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:51.067047 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:50.944000 27818 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.035785 27730 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.042459 25370 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 5
I20260809 23:18:51.009487 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:51.047770 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2046) took 3280 ms (client timeout 2913 ms). Trace:
I20260809 23:18:51.048086 27715 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:51.047793 27764 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:50.962574 27633 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.051282 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:51.086187 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 118922us
W20260809 23:18:51.052942 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 651) took 2185 ms (client timeout 1674 ms). Trace:
I20260809 23:18:51.053594 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:18:50.992386 27966 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:18:51.042191 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2039) took 3438 ms. Trace:
I20260809 23:18:51.058836 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.089411 24546 rpcz_store.cc:276] 0809 23:18:47.608482 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.600118 (+2991636us) service_pool.cc:227] Handling call
0809 23:18:50.653677 (+ 53559us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.657224 (+  3547us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:51.042024 (+384800us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b54001fcf50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.042141 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122211712}
I20260809 23:18:51.059689 24312 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:51.061422 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.042598 27975 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:18:51.070308 24539 rpcz_store.cc:269] 0809 23:18:47.766991 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.887060 (+2120069us) service_pool.cc:227] Handling call
0809 23:18:51.047579 (+1160519us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.047673 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42433408}
I20260809 23:18:51.069660 25370 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:18:51.094743 25370 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:18:51.073565 24314 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:51.013914 27590 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260809 23:18:51.096391 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.098220 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.100561 27730 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.101310 27945 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.102484 27715 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:51.103717 24498 leader_election.cc:398] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:18:51.104038 24498 leader_election.cc:401] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260809 23:18:51.104781 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.105875 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.073071 27953 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:51.106738 27651 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:18:51.107101 27764 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:18:51.107913 27764 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:51.108544 27764 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.106859 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:51.109267 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.109984 27846 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:51.110358 27846 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.110698 27846 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.112099 27818 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:51.112757 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.113894 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:51.013526 27721 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.708s	user 0.007s	sys 0.005s
I20260809 23:18:51.080572 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.051891 27857 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:18:51.087121 24311 rpcz_store.cc:269] 0809 23:18:48.870129 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.907072 (+ 36943us) service_pool.cc:227] Handling call
0809 23:18:51.052757 (+2145685us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.052873 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2976163072}
I20260809 23:18:51.093568 25369 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:51.122059 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.070524 27896 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:51.127444 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:51.063560 27834 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.129143 24314 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:51.130892 24496 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:51.135568 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 110618us
W20260809 23:18:51.138981 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 159049us
I20260809 23:18:51.140308 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:51.141714 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73604.8us
I20260809 23:18:51.142531 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:18:51.142800 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47307.3us
I20260809 23:18:51.144795 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:51.146806 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99050.9us
I20260809 23:18:51.107193 27953 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:51.114986 27975 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.123675 27938 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:51.114912 27943 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.019129 27969 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.133s
W20260809 23:18:51.154878 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 206518us
I20260809 23:18:51.153251 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.159380 27723 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.169749 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.132332 27896 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:51.150799 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.161478 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.171047 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:51.133400 27730 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260809 23:18:51.149464 27818 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:51.151953 27953 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:51.155453 27903 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.155318 27975 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.156745 27721 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.261737 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:51.262301 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105642us
I20260809 23:18:51.147799 27805 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:18:51.234565 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83862.5us
I20260809 23:18:51.298787 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.237604 27764 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.240703 27567 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.175966 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:51.254225 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78966.5us
I20260809 23:18:51.303911 25370 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:51.256072 24320 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:51.074050 27977 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.174412 27896 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.156150 27797 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260809 23:18:51.152812 27666 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260809 23:18:51.175128 27938 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:18:51.289350 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 149105us
I20260809 23:18:51.260476 27980 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:51.160857 27510 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
I20260809 23:18:51.292402 24542 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:51.308619 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.176807 27943 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.125s
W20260809 23:18:51.296782 27959 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
W20260809 23:18:51.311249 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 140291us
I20260809 23:18:51.316368 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.314482 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.257359 27971 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260809 23:18:51.274338 27981 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:18:51.322047 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 160637us
W20260809 23:18:51.334241 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 179716us
I20260809 23:18:51.308251 27982 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:51.312120 27949 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.334901 25371 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:51.335687 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.352671 25370 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.318308 27721 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260809 23:18:51.369491 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47062.3us
I20260809 23:18:51.370532 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.372963 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.325637 27984 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:51.351332 27982 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.377753 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.356245 27980 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:51.378676 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.368570 27748 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:51.399510 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.404417 25371 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.326406 27651 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260809 23:18:51.369212 27992 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.383189 27984 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:51.377738 28004 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.350387 27847 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.256s
I20260809 23:18:51.374845 27818 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:51.406577 27748 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.437711 27748 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.438024 27818 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.439429 27846 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.402199 27982 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.443244 28011 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:18:51.446035 27996 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260809 23:18:51.446153 27567 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.445914 27846 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:51.427251 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 112910us
W20260809 23:18:51.400585 27946 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.092s	user 0.005s	sys 0.009s
I20260809 23:18:51.447809 27797 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:51.447615 27946 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:51.452716 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102571us
W20260809 23:18:51.429983 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 90770.6us
I20260809 23:18:51.383677 27968 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:51.334813 27822 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.404922 27949 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:18:51.432427 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53853.1us
I20260809 23:18:51.346630 27959 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.403682 27680 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.469754 27680 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.472584 27680 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.449934 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.474124 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 698) took 1070 ms. Trace:
I20260809 23:18:51.474536 25375 rpcz_store.cc:276] 0809 23:18:50.403293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.403452 (+   159us) service_pool.cc:227] Handling call
0809 23:18:51.449773 (+1046321us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:51.474041 (+ 24268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11724544}
I20260809 23:18:51.476153 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.477734 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.405346 27805 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.423755 27986 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:51.454674 24496 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.424444 27994 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.481027 27994 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.427134 27980 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:51.484414 27980 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.489892 27986 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:51.473212 27680 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:51.494728 27680 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.498342 27680 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.496012 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.500360 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.501204 25375 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 1.
I20260809 23:18:51.502722 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:51.453076 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.795s (SENT) [suppressed 45 similar messages]
W20260809 23:18:51.510074 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57027us
I20260809 23:18:51.450551 27903 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:18:51.456602 27979 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.311s	user 0.003s	sys 0.006s
W20260809 23:18:51.508535 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53964.5us
I20260809 23:18:51.508663 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.450323 27723 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.404155 27941 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.491307 27986 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.463014 27949 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:51.522641 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.523142 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.434466 27689 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260809 23:18:51.455832 27846 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260809 23:18:51.542877 27748 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.563097 27723 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:18:51.563738 25370 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:51.564482 27723 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:51.524199 28024 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:51.565733 27812 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.565838 28024 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:51.536652 28006 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.566946 27748 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.566865 28024 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.534348 27949 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:51.568259 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.569422 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.570540 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:51.592538 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 219670us
I20260809 23:18:51.540251 27972 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:51.541028 27999 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.543584 27994 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:18:51.547827 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.548548 27822 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.536687 27660 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:18:51.549356 27990 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260809 23:18:51.550287 27939 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.551275 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.435271 27938 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.554272 27945 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.467437 27992 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:51.555104 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2038) took 3988 ms (client timeout 1730 ms). Trace:
I20260809 23:18:51.555462 27977 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.556972 25369 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:51.527858 28014 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.522085 28002 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260809 23:18:51.559608 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.559913 27946 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:51.560307 27638 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.561923 28001 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.576049 27633 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.577742 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.584319 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:51.585341 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:51.539765 27993 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260809 23:18:51.588186 27987 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.203s	user 0.003s	sys 0.000s
W20260809 23:18:51.588776 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55358.1us
I20260809 23:18:51.589545 27949 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.535483 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.593269 24312 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:51.542730 25371 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:51.593796 24266 leader_election.cc:398] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
W20260809 23:18:51.651814 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58146.6us
I20260809 23:18:51.652979 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.604949 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 699) took 1119 ms. Trace:
I20260809 23:18:51.654161 25373 rpcz_store.cc:276] 0809 23:18:50.485905 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.487222 (+  1317us) service_pool.cc:227] Handling call
0809 23:18:51.547624 (+1060402us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:51.604864 (+ 57240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11848576}
W20260809 23:18:51.613785 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2061) took 2361 ms (client timeout 1692 ms). Trace:
W20260809 23:18:51.655876 24541 rpcz_store.cc:269] 0809 23:18:49.252689 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.972758 (+720069us) service_pool.cc:227] Handling call
0809 23:18:51.611181 (+1638423us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b540029d954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.613687 (+  2506us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70163200}
W20260809 23:18:51.618618 24543 rpcz_store.cc:269] 0809 23:18:47.568616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:48.756487 (+1187871us) service_pool.cc:227] Handling call
0809 23:18:51.554870 (+2798383us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.555027 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3486304000}
W20260809 23:18:51.621411 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285771us
W20260809 23:18:51.628845 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0924955948721617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260809 23:18:51.657485 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0717955948784947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260809 23:18:51.657923 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0276955950539559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
I20260809 23:18:51.658677 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:18:51.637647 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105387us
W20260809 23:18:51.622257 27976 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.583s	user 0.011s	sys 0.000s
I20260809 23:18:51.613229 24545 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:18:51.663340 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:51.664158 24539 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.664939 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:51.666584 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 82357.5us
I20260809 23:18:51.667970 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.648787 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.670531 25369 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:51.673620 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.673892 25372 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:51.665768 24540 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:51.674642 25369 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:51.675196 24496 leader_election.cc:398] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:18:51.677264 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 752) took 2795 ms. Trace:
I20260809 23:18:51.678956 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.603183 27975 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.679116 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.700184 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.617918 27834 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.708694 24435 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 23:18:51.617754 27982 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.711229 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.712975 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.713410 24435 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:51.711483 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.711822 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:51.712172 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53599.1us
I20260809 23:18:51.717237 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.667550 25366 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:51.718319 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50454us
I20260809 23:18:51.720230 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.722029 24422 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.722496 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:51.723811 27813 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.724257 24428 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:18:51.724418 27813 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.725219 28031 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:51.725803 28022 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:51.728271 24424 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:51.729625 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.730336 28004 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:51.730852 27780 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:51.731402 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 695) took 1329 ms. Trace:
I20260809 23:18:51.731813 25366 rpcz_store.cc:276] 0809 23:18:50.401982 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.412138 (+ 10156us) service_pool.cc:227] Handling call
0809 23:18:51.577554 (+1165416us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:51.731338 (+153784us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11701632}
I20260809 23:18:51.732510 27834 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.734082 27982 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.673743 27976 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.735700 27797 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:51.677892 27666 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 23:18:51.736107 27797 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:51.736948 24320 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:51.737174 27797 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.738466 27797 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.708454 28006 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:51.631453 28014 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.708825 28020 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:51.743266 28020 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:51.743916 28020 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.745639 27953 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.746462 27992 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.703274 27952 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260809 23:18:51.747262 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.711813 27822 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260809 23:18:51.748760 27949 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.698287 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 696) took 1279 ms. Trace:
I20260809 23:18:51.735971 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:51.730499 24431 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:51.736179 27976 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.738771 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:51.750270 24316 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:51.758541 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:51.759307 24423 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:51.768179 24323 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:51.726661 27813 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:51.764765 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44131us
I20260809 23:18:51.773119 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:51.730336 25365 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.780s	user 0.010s	sys 0.000s
I20260809 23:18:51.641327 28001 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.763252 25363 rpcz_store.cc:276] 0809 23:18:50.402445 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.411399 (+  8954us) service_pool.cc:227] Handling call
0809 23:18:51.550958 (+1139559us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:51.682131 (+131173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7840128}
I20260809 23:18:51.787665 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.804831 24427 raft_consensus.cc:2362] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3. Re-sending same reply.
I20260809 23:18:51.788379 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.789022 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:51.791448 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.800760 24435 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260809 23:18:51.800912 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:51.801510 24313 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:51.801611 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.807292 24428 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:51.807610 24431 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.802160 24424 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:51.731372 27780 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:51.812974 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:51.813722 25370 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 4
I20260809 23:18:51.806764 24425 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:51.677711 25368 rpcz_store.cc:276] 0809 23:18:48.881868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.614652 (+1732784us) service_pool.cc:227] Handling call
0809 23:18:50.640669 (+ 26017us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:50.691005 (+ 50336us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:51.677029 (+986024us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b5400860950. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:51.677184 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56385280}
W20260809 23:18:51.819571 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112703us
I20260809 23:18:51.750123 27633 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:51.825403 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52360.8us
W20260809 23:18:51.825909 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 103483us
I20260809 23:18:51.826783 27628 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:51.826318 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109148us
W20260809 23:18:51.828330 27857 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.695s	user 0.013s	sys 0.006s
I20260809 23:18:51.869628 24317 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:51.787348 25365 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:18:51.802459 28019 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:51.794169 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:51.854909 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.832825 24422 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:51.790442 27977 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.874480 24323 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:51.767827 28022 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:51.778159 28031 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:51.873929 24430 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:51.894373 28031 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.895139 27857 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:51.897413 28015 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.897970 28015 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.900344 28015 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:51.902292 27896 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.576s	user 0.001s	sys 0.007s
I20260809 23:18:51.789628 27986 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:51.777479 27847 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260809 23:18:51.904196 27847 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:51.904582 27847 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:51.905462 27847 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.881587 27822 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260809 23:18:51.887301 27764 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.021s
I20260809 23:18:51.870326 27778 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.908280 27780 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.909134 27778 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:51.909550 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.911329 27975 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.913915 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:51.914572 24321 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.915163 27954 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:51.915563 27954 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:51.916390 27954 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.917668 27954 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:51.921006 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:51.767469 27981 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.105s
I20260809 23:18:51.720046 27680 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260809 23:18:51.939113 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:51.941567 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56536.9us
I20260809 23:18:51.942624 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:51.874993 24423 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:51.788024 27972 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.883970 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.871374 27939 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.802695 28027 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:51.787699 27834 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260809 23:18:51.891429 27903 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:18:51.891808 27953 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.892028 28022 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.881439 27628 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.893280 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 828) took 1090 ms. Trace:
I20260809 23:18:51.902737 28001 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:51.922838 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:51.920928 24541 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:18:51.859208 24539 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:51.928647 24321 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:51.902861 24312 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:18:51.930529 25370 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:51.906718 27847 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:51.946425 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:51.831944 27813 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:51.949767 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:51.887650 27992 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20260809 23:18:51.886996 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:51.997071 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:51.953951 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:52.073129 25375 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:18:51.949875 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66005.6us
I20260809 23:18:52.074239 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:51.902905 27896 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:51.886176 27573 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.884519 27968 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.048571 24309 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260809 23:18:52.048571 24323 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260809 23:18:52.052628 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102936us
I20260809 23:18:51.992475 24431 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:52.060293 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:51.992376 27715 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:52.064406 27941 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.066358 24535 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:18:52.071547 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.071723 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2071) took 1357 ms. Trace:
I20260809 23:18:52.071872 27944 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:51.935321 27977 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.992583 27764 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:18:52.080034 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 133713us
I20260809 23:18:51.994586 28004 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
W20260809 23:18:52.092622 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 338889us
I20260809 23:18:52.026861 27939 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.042867 27946 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.107872 24312 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:51.949992 27638 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.129s
I20260809 23:18:51.946223 27975 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:51.997349 27857 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.091s
I20260809 23:18:52.057893 28027 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:52.136065 24541 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:52.135880 24542 rpcz_store.cc:276] 0809 23:18:50.714530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.738806 (+ 24276us) service_pool.cc:227] Handling call
0809 23:18:52.071551 (+1332745us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54000b6154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:52.071664 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":895906560}
I20260809 23:18:52.043244 28014 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.139487 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.073792 27689 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:52.140796 24314 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:18:52.100049 27822 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.099546 27976 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:52.149535 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:52.073017 24534 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:52.152884 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.115907 28006 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.116137 27982 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:52.189834 27941 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:52.190284 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.165717 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.098097 25366 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:52.170174 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.050002 25365 rpcz_store.cc:276] 0809 23:18:50.802765 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.816493 (+ 13728us) service_pool.cc:227] Handling call
0809 23:18:50.836649 (+ 20156us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:51.874510 (+1037861us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:51.893212 (+ 18702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:52.198407 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.199136 24542 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:52.169342 27999 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.114612 27715 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:52.050756 27972 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.181622 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:52.151382 27977 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:52.181721 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 107571us
I20260809 23:18:52.183776 24312 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260809 23:18:52.185550 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 230449us
I20260809 23:18:52.146162 27813 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:52.187773 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.225035 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.225590 24423 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:52.188094 25371 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 23:18:52.189309 28006 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.195401 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.230379 24541 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:52.133186 27573 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.202504 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.183125 27689 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.206688 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.173642 27975 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.190191 27982 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:52.190336 27941 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:52.225862 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.235759 27689 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:52.238812 24542 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:52.229851 27813 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.239614 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.240710 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.241242 24542 raft_consensus.cc:2362] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4. Re-sending same reply.
I20260809 23:18:52.241297 27813 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:52.237044 28036 raft_consensus.cc:2741] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260809 23:18:52.243180 24534 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:52.244197 27573 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.244897 27573 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:52.244656 27984 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.246217 27984 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:52.204188 28002 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.253480 24423 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:52.229159 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:52.235213 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.254644 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.255265 24541 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:52.135687 27968 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.236016 24539 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:52.237111 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:52.241132 28042 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260809 23:18:52.193809 27822 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.217660 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:52.231170 28047 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 23:18:52.265985 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56957.8us
I20260809 23:18:52.255589 24423 raft_consensus.cc:2354] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4.
I20260809 23:18:52.168721 27946 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.260840 27778 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.261152 25365 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:52.261911 28006 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.263244 27977 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.264036 27896 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.270218 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.247284 27941 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:52.267040 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:18:52.205976 27999 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.298123 24431 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:52.267194 27813 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.247943 27857 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260809 23:18:52.309242 27822 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.309974 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.311205 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 675) took 1281 ms. Trace:
I20260809 23:18:52.309319 27999 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.268723 27748 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:18:52.325107 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 382556us
I20260809 23:18:52.329226 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:52.330550 24423 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4.
I20260809 23:18:52.330900 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:52.326526 24311 rpcz_store.cc:276] 0809 23:18:51.029768 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.661145 (+631377us) service_pool.cc:227] Handling call
0809 23:18:51.699183 (+ 38038us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:51.699386 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:51.699410 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:51.703939 (+  4529us) log.cc:842] Serialized 26 byte log entry
0809 23:18:51.881146 (+177207us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:51.881189 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:51.881262 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:52.310921 (+429659us) raft_consensus.cc:1671] finished
0809 23:18:52.310964 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:52.311136 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":93443,"threads_started":1,"wal-append.queue_time_us":241819}
I20260809 23:18:52.335553 25369 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:18:52.337806 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.332115 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.313470 25366 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:52.318490 27813 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.349123 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.372612 24423 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:52.373687 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:52.373886 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 192334us
I20260809 23:18:52.326149 27953 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:52.374670 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:52.338050 28020 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.590s	user 0.021s	sys 0.000s
I20260809 23:18:52.318346 27972 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.374413 24423 raft_consensus.cc:2362] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4. Re-sending same reply.
I20260809 23:18:52.394505 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:18:52.394731 25321 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.674s (SENT)
I20260809 23:18:52.275177 27715 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.399501 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.257913 28027 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.373780 28014 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:52.398571 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:52.399050 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61364.7us
I20260809 23:18:52.398463 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.405512 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.374821 27813 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:52.392652 27822 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:18:52.398422 27996 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260809 23:18:52.408227 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:52.411659 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.411841 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:52.404946 28019 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.522s	user 0.010s	sys 0.001s
I20260809 23:18:52.348181 27748 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.404709 27972 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.397966 27939 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.404462 27938 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:52.410697 27715 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:52.406623 28062 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
W20260809 23:18:52.413190 28044 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260809 23:18:52.408943 27812 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:52.437230 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.434235 28019 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:18:52.468747 28046 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260809 23:18:52.474675 27959 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.409024 27968 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:52.475414 27954 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.557s	user 0.014s	sys 0.000s
I20260809 23:18:52.475792 27946 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:52.477022 28067 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260809 23:18:52.478542 27797 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.000s
I20260809 23:18:52.419955 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.481142 24423 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:52.481968 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:18:52.440861 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46189.1us
I20260809 23:18:52.409751 27977 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:52.443323 27822 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:52.443800 28031 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:52.445884 27751 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:52.446874 28048 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260809 23:18:52.447641 28057 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.482837 24541 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:52.486279 28057 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.488745 28057 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:52.376294 27982 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:52.488983 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 85075.3us
I20260809 23:18:52.435949 27972 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.491183 27972 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:52.447932 28053 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260809 23:18:52.397500 28020 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.440690 27938 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.404731 28004 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260809 23:18:52.475051 27748 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.495666 24541 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:18:52.493682 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:18:52.571513 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76255.6us
I20260809 23:18:52.753995 27946 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:52.754572 27946 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:52.755470 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 681) took 1007 ms. Trace:
I20260809 23:18:52.755640 27946 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.498956 27812 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:52.756916 27946 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:52.755852 24321 rpcz_store.cc:276] 0809 23:18:51.748046 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.056525 (+308479us) service_pool.cc:227] Handling call
0809 23:18:52.057052 (+   527us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.107255 (+ 50203us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:52.107309 (+    54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:52.108048 (+   739us) log.cc:842] Serialized 26 byte log entry
0809 23:18:52.324178 (+216130us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:52.324221 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.324295 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:52.755256 (+430961us) raft_consensus.cc:1671] finished
0809 23:18:52.755274 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:52.755416 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49402,"raft.queue_time_us":51018,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":30688,"thread_start_us":159649,"threads_started":1,"wal-append.queue_time_us":280177}
I20260809 23:18:52.759063 28065 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260809 23:18:52.763556 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1221) took 1702 ms. Trace:
I20260809 23:18:52.763960 24312 rpcz_store.cc:276] 0809 23:18:51.061006 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.762386 (+1701380us) service_pool.cc:227] Handling call
0809 23:18:52.763472 (+  1086us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:52.765610 27689 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.527s	user 0.002s	sys 0.006s
W20260809 23:18:52.634255 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55085.7us
I20260809 23:18:52.766112 27689 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.687496 24422 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:18:52.767554 28045 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.516s	user 0.009s	sys 0.004s
I20260809 23:18:52.768632 25375 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:52.769624 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2254) took 1058 ms. Trace:
I20260809 23:18:52.769819 27689 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:52.769994 24429 rpcz_store.cc:276] 0809 23:18:51.711693 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.712519 (+   826us) service_pool.cc:227] Handling call
0809 23:18:51.727625 (+ 15106us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:51.787243 (+ 59618us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:51.787275 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:51.800437 (+ 13162us) log.cc:842] Serialized 26 byte log entry
0809 23:18:52.081790 (+281353us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:52.081827 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.081901 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:52.766986 (+685085us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:18:52.769315 (+  2329us) raft_consensus.cc:1671] finished
0809 23:18:52.769345 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:52.769562 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54519,"raft.queue_time_us":49432,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":9495424,"thread_start_us":194437,"threads_started":1,"wal-append.queue_time_us":369843}
I20260809 23:18:52.675328 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.463991 27633 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:18:52.716188 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 304204us
I20260809 23:18:52.757119 28020 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.465184 28062 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:52.717621 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 706) took 1666 ms. Trace:
I20260809 23:18:52.793124 25372 rpcz_store.cc:276] 0809 23:18:51.051251 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.051777 (+   526us) service_pool.cc:227] Handling call
0809 23:18:51.052104 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:51.679683 (+627579us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.717361 (+1037678us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540064c8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:52.717553 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429405056}
I20260809 23:18:52.587152 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:52.776764 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 283194us
W20260809 23:18:52.713521 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 242709us
I20260809 23:18:52.799111 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.801219 24429 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:52.802279 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:52.755692 25321 leader_election.cc:334] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.807s (SENT)
I20260809 23:18:52.494537 27780 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:52.606837 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 415349us
I20260809 23:18:52.471292 28047 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:52.783293 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.808459 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.759352 27812 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.491457 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:52.802866 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:52.464946 27975 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.781178 27689 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:52.803421 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.632902 27938 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:52.806245 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.824775 24539 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:52.472044 27976 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:52.785679 27748 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.809489 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:52.828155 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.803689 24534 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:52.649040 28014 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:52.642663 27941 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.687722 27813 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.821389 28083 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:52.821631 27666 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:52.813951 28020 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:52.831605 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.805249 27780 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:52.807503 27834 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:52.840852 27834 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.841279 27834 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:52.833812 28047 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:18:52.843920 28047 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:52.819720 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64138.4us
I20260809 23:18:52.830464 24429 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:18:52.836675 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.850912 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:52.859742 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.828579 27975 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260809 23:18:52.868384 24544 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:52.871197 24539 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:52.873046 24540 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:18:52.872072 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.845767 28047 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.879736 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.877465 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.852694 27689 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:52.859876 27971 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.889132 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.889523 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:52.889856 24539 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:52.886124 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:52.880523 24429 raft_consensus.cc:2362] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4. Re-sending same reply.
I20260809 23:18:52.891249 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:18:52.874225 28056 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.473s	user 0.000s	sys 0.001s
I20260809 23:18:52.887089 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.846457 27780 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.867901 27847 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.873005 28064 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.892378 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.894035 24423 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:52.876349 28077 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.834164 27748 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.881810 27822 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:52.893169 27649 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:52.903255 27649 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:52.903641 27649 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:52.904278 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.905628 24429 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:18:52.906102 27628 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.906711 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.906780 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.912985 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1836) took 1028 ms. Trace:
I20260809 23:18:52.909515 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2253) took 1198 ms. Trace:
I20260809 23:18:52.927173 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.924947 24534 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
W20260809 23:18:52.914208 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2282) took 4135 ms (client timeout 2301 ms). Trace:
I20260809 23:18:52.927529 24430 rpcz_store.cc:276] 0809 23:18:51.711378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.712101 (+   723us) service_pool.cc:227] Handling call
0809 23:18:51.712357 (+   256us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:51.894209 (+181852us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.909238 (+1015029us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002f94d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:52.909441 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":330935680}
W20260809 23:18:52.929273 24316 rpcz_store.cc:269] 0809 23:18:48.778988 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.826856 (+1047868us) service_pool.cc:227] Handling call
0809 23:18:52.914032 (+3087176us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:52.914141 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4185564544}
I20260809 23:18:52.933378 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.896739 28019 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:18:52.936651 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:52.941538 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.942607 24423 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:52.894194 27805 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:18:52.946369 25321 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.932s (SENT)
I20260809 23:18:52.946255 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:52.925493 24311 rpcz_store.cc:276] 0809 23:18:51.884528 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.372789 (+488261us) service_pool.cc:227] Handling call
0809 23:18:52.396438 (+ 23649us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.426324 (+ 29886us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:52.426357 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:52.436853 (+ 10496us) log.cc:842] Serialized 26 byte log entry
0809 23:18:52.807487 (+370634us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:52.807535 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.807642 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:52.912748 (+105106us) raft_consensus.cc:1671] finished
0809 23:18:52.912768 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:52.912915 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9720,"raft.queue_time_us":25597,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":8461,"thread_start_us":331367,"threads_started":1,"wal-append.queue_time_us":409451}
I20260809 23:18:52.888239 27975 raft_consensus.cc:2741] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 23:18:52.950192 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:52.952076 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.914412 24170 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:18:52.839663 28062 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.908581 27980 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:52.907913 27780 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:52.959745 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.963389 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.962882 24170 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:18:52.900023 27847 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:52.900283 28064 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:52.964507 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:52.965786 24164 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 237) took 6751 ms. Trace:
I20260809 23:18:53.009869 24430 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:53.012466 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:52.966593 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.013552 24424 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:18:53.014011 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2261) took 1219 ms (client timeout 1559 ms). Trace:
W20260809 23:18:53.014449 24424 rpcz_store.cc:269] 0809 23:18:51.802366 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.936028 (+1133662us) service_pool.cc:227] Handling call
0809 23:18:53.013949 (+ 77921us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.015179 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.907111 28066 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:53.016381 24539 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:52.967721 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:52.894549 27971 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.935737 28019 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:52.953533 27950 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260809 23:18:52.978220 27968 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:52.980671 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:52.945161 27903 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:52.945034 27723 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:52.964241 27805 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:53.047055 24424 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:18:52.914860 27976 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.984606 27945 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
W20260809 23:18:52.988301 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 178447us
I20260809 23:18:53.051483 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:52.935446 28077 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.989908 27689 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.972689 27980 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:52.991971 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 498) took 1078 ms. Trace:
I20260809 23:18:53.060837 24540 rpcz_store.cc:276] 0809 23:18:51.913931 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.848206 (+934275us) service_pool.cc:227] Handling call
0809 23:18:52.872786 (+ 24580us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.873553 (+   767us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.991764 (+118211us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b5400271ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:52.991899 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55667968}
I20260809 23:18:53.062225 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.063396 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:52.996127 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 678) took 1344 ms. Trace:
I20260809 23:18:53.064113 24320 rpcz_store.cc:276] 0809 23:18:51.651902 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.738495 (+ 86593us) service_pool.cc:227] Handling call
0809 23:18:52.995943 (+1257448us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54002530d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:52.996071 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2104897408}
I20260809 23:18:52.996773 27748 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:52.996251 27834 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:52.997217 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1830) took 2044 ms. Trace:
I20260809 23:18:52.998953 27972 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:52.999758 24428 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
W20260809 23:18:53.048771 27715 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.584s	user 0.005s	sys 0.005s
I20260809 23:18:53.002645 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1826) took 2051 ms. Trace:
I20260809 23:18:52.944228 28014 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:52.967214 27984 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.000s
I20260809 23:18:53.009792 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:53.024071 24534 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:18:53.096841 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.017202 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:53.014304 24170 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:18:53.098366 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81252.9us
I20260809 23:18:53.032616 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.043875 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:53.103359 25370 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:53.011374 27847 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.021704 28064 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.060166 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:53.031848 27666 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.032542 28019 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:18:53.048681 27903 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.070858 28077 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.071094 27813 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:53.070989 27805 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.043937 27723 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.073207 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.073573 27971 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.049898 25366 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:53.074582 27980 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.068185 27941 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:53.075362 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2083) took 1051 ms. Trace:
I20260809 23:18:53.021662 27975 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260809 23:18:53.077875 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1838) took 1158 ms. Trace:
W20260809 23:18:53.121222 27982 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.010s	sys 0.019s
I20260809 23:18:53.081703 24539 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:53.084043 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2258) took 1289 ms. Trace:
I20260809 23:18:53.001952 24431 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:53.134196 27723 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.085237 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.087548 24313 rpcz_store.cc:276] 0809 23:18:50.951707 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.951889 (+   182us) service_pool.cc:227] Handling call
0809 23:18:50.952540 (+   651us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:51.807238 (+854698us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.002425 (+1195187us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.002581 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1951961344}
I20260809 23:18:53.065173 27986 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:53.106595 24541 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:18:53.108594 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.110553 24534 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:53.154634 24428 rpcz_store.cc:276] 0809 23:18:51.794154 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.999279 (+1205125us) service_pool.cc:227] Handling call
0809 23:18:52.999570 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.083810 (+ 84240us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.083988 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.155324 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.161342 24539 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:53.110652 24312 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:18:53.155988 24540 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:53.157421 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.110889 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:53.163089 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:18:53.163492 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62842.5us
W20260809 23:18:53.151217 27982 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.010s	sys 0.020s
I20260809 23:18:53.116426 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.169116 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:53.129988 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.170388 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.111452 27946 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:53.106732 27715 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.079350 27797 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260809 23:18:53.114861 27628 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.079798 24317 rpcz_store.cc:276] 0809 23:18:50.953007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.026594 (+ 73587us) service_pool.cc:227] Handling call
0809 23:18:52.997056 (+1970462us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:52.997160 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1931495808}
I20260809 23:18:53.140805 24321 rpcz_store.cc:276] 0809 23:18:51.919605 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.759028 (+839423us) service_pool.cc:227] Handling call
0809 23:18:52.759382 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.759724 (+   342us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:52.759752 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:52.760144 (+   392us) log.cc:842] Serialized 26 byte log entry
0809 23:18:52.855224 (+ 95080us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:52.855279 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.855376 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:53.077682 (+222306us) raft_consensus.cc:1671] finished
0809 23:18:53.077702 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.077821 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1078,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":94647,"threads_started":1,"wal-append.queue_time_us":300643}
I20260809 23:18:53.122534 27950 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:53.137274 24546 rpcz_store.cc:276] 0809 23:18:52.023867 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.470789 (+446922us) service_pool.cc:227] Handling call
0809 23:18:52.471150 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.587511 (+116361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:52.587541 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:52.719374 (+131833us) log.cc:842] Serialized 26 byte log entry
0809 23:18:52.833522 (+114148us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:52.833971 (+   449us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:52.834045 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:53.075162 (+241117us) raft_consensus.cc:1671] finished
0809 23:18:53.075181 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.075298 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":120603,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":19044,"thread_start_us":74525,"threads_started":1,"wal-append.queue_time_us":296999}
I20260809 23:18:53.011951 24164 rpcz_store.cc:276] 0809 23:18:46.164792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:46.172819 (+  8027us) service_pool.cc:227] Handling call
0809 23:18:52.913129 (+6740310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:18:47.298039 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:18:47.304900 (+  6861us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:18:47.304958 (+    58us) write_op.cc:435] Acquired schema lock
0809 23:18:47.304974 (+    16us) tablet.cc:660] Decoding operations
0809 23:18:47.353550 (+ 48576us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:18:47.370799 (+ 17249us) write_op.cc:641] Partition lock acquired for write op
0809 23:18:47.376497 (+  5698us) tablet.cc:683] Acquiring locks for 7 operations
0809 23:18:47.723362 (+346865us) tablet.cc:699] Row locks acquired
0809 23:18:47.748961 (+ 25599us) write_op.cc:260] PREPARE: finished
0809 23:18:47.827230 (+ 78269us) write_op.cc:270] Start()
0809 23:18:47.868961 (+ 41731us) write_op.cc:276] Timestamp: P: 1786317527785636 usec, L: 0
0809 23:18:47.943868 (+ 74907us) op_driver.cc:331] REPLICATION: starting
0809 23:18:48.089585 (+145717us) log.cc:842] Serialized 2966 byte log entry
0809 23:18:48.896885 (+807300us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b5400300d20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237ee771 00007fac237f9220 00007fac237f91c8 00007fac237f9148 00007fac237f9103 00007fac237f838c 00007fac29869206 00007fac29861808 00007fac2383538b 00007fac23837e80
0809 23:18:49.147066 (+250181us) op_driver.cc:444] REPLICATION: finished
0809 23:18:50.077665 (+930599us) write_op.cc:301] APPLY: starting
0809 23:18:50.077794 (+   129us) tablet.cc:1365] starting BulkCheckPresence
0809 23:18:50.107643 (+ 29849us) tablet.cc:1368] finished BulkCheckPresence
0809 23:18:50.108954 (+  1311us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:18:50.234012 (+125058us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:18:50.234454 (+   442us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0809 23:18:50.234520 (+    66us) write_op.cc:312] APPLY: finished
0809 23:18:50.235075 (+   555us) log.cc:842] Serialized 59 byte log entry
0809 23:18:50.627081 (+392006us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:50.627321 (+   240us) write_op.cc:454] Released schema lock
0809 23:18:50.627377 (+    56us) write_op.cc:341] FINISH: Updating metrics
0809 23:18:50.627776 (+   399us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:50.627803 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":877631,"apply.run_cpu_time_us":39240,"apply.run_wall_time_us":550528,"num_ops":7,"prepare.queue_time_us":267150,"prepare.run_cpu_time_us":21415,"prepare.run_wall_time_us":956474,"raft.queue_time_us":188562,"raft.run_cpu_time_us":98984,"raft.run_wall_time_us":855786,"replication_time_us":1156882,"spinlock_wait_cycles":4485376,"thread_start_us":1069975,"threads_started":5,"wal-append.queue_time_us":534353,"wal-append.run_cpu_time_us":19592,"wal-append.run_wall_time_us":2822115}]]}
I20260809 23:18:53.164620 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:53.186820 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:53.159408 24430 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:53.188516 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:53.112403 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52324.5us
I20260809 23:18:53.160213 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.190059 24534 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:53.155598 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:53.166739 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.170487 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.133270 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.191063 24429 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.170416 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.172875 24316 raft_consensus.cc:2396] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:18:53.169667 24320 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:18:53.192592 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.192708 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.169983 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.177901 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.178043 24542 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:18:53.116976 27666 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.152545 27980 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:53.180789 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.176088 27975 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:53.193572 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.196477 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.169643 27986 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:53.174805 28077 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:53.177151 27857 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:53.206955 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.201659 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.203644 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:53.203516 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.215627 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.216977 24320 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260809 23:18:53.213994 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.195281 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.221995 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.226488 24540 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:53.222709 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.227566 24424 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:53.228777 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:53.225687 24431 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:18:53.241631 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.242550 24540 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:53.241822 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:53.242615 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:53.212682 28096 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260809 23:18:53.243993 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:53.204854 28004 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:53.180812 27715 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.202172 27950 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:18:53.249653 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 299500us
I20260809 23:18:53.251109 24540 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:53.201928 27633 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:53.214921 27857 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.248654 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:18:53.250839 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 304760us
I20260809 23:18:53.293948 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 506) took 1358 ms. Trace:
I20260809 23:18:53.205413 28083 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:53.201164 27952 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:53.255656 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.310739 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2262) took 1480 ms. Trace:
I20260809 23:18:53.256271 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.405344 24542 rpcz_store.cc:276] 0809 23:18:51.935867 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.322579 (+386712us) service_pool.cc:227] Handling call
0809 23:18:52.323012 (+   433us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.194169 (+871157us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.293668 (+ 99499us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b5400146c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.293886 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1960429952}
I20260809 23:18:53.186290 27628 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.408044 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.408818 24431 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:53.409330 27628 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:53.410475 24534 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:18:53.375168 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.222214 28100 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:53.257004 27834 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:53.349980 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 479) took 4328 ms. Trace:
I20260809 23:18:53.411515 24425 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:53.348368 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 720) took 1119 ms. Trace:
I20260809 23:18:53.411682 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:53.411962 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2260) took 1617 ms (client timeout 1892 ms). Trace:
W20260809 23:18:53.344367 28097 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260809 23:18:53.349198 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 448) took 1162 ms. Trace:
I20260809 23:18:53.376811 28026 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.297926 28108 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.103s
I20260809 23:18:53.411572 24545 rpcz_store.cc:276] 0809 23:18:49.021999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:50.743351 (+1721352us) service_pool.cc:227] Handling call
0809 23:18:51.612766 (+869415us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:51.660054 (+ 47288us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.349741 (+1689687us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540029d950. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:53.349910 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1998714752}
I20260809 23:18:53.413414 28026 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.298079 27903 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:53.291361 27666 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.415743 28026 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.259387 27633 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:53.416600 27633 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:53.283468 27941 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:18:53.387813 27812 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:53.295635 27986 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.417613 27941 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.418004 27812 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:53.418267 27941 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:18:53.416311 28026 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:53.369298 24429 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:53.422072 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:53.423441 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.424611 24429 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.247442 27999 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:53.406430 24435 rpcz_store.cc:276] 0809 23:18:51.830492 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.989141 (+1158649us) service_pool.cc:227] Handling call
0809 23:18:52.989475 (+   334us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:52.989684 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:18:53.181838 (+192154us) log.cc:842] Serialized 14 byte log entry
0809 23:18:53.310319 (+128481us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:53.310376 (+    57us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:53.310440 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.310524 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.310668 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":297911,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":20934,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11382,"thread_start_us":319991,"threads_started":2,"wal-append.queue_time_us":216677}
I20260809 23:18:53.284117 27857 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.292836 27981 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260809 23:18:53.428534 27666 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:53.391184 27945 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:53.429453 27666 consensus_queue.cc:1225] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:18:53.430022 27666 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:53.430378 27666 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:53.430289 27945 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.202550 27975 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:53.431985 27903 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:53.432472 27975 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.434773 27975 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:53.251699 28047 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.439831 28047 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.389652 27950 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.287518 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.443226 24424 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:53.444515 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.436322 27666 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:53.391805 28111 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260809 23:18:53.446894 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:53.390709 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.448411 24540 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:53.370139 27976 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.449224 27976 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.451607 27976 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:53.453683 27976 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:53.452347 24429 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:53.457523 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.461007 24429 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:53.466167 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.463675 27666 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:18:53.399637 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 150713us
W20260809 23:18:53.385176 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 174289us
I20260809 23:18:53.336679 27689 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260809 23:18:53.197965 27984 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:53.405876 28083 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.411904 25375 rpcz_store.cc:276] 0809 23:18:52.228633 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.228777 (+   144us) service_pool.cc:227] Handling call
0809 23:18:52.229123 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:52.769061 (+539938us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.348089 (+579028us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.348284 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1240273152}
I20260809 23:18:53.413412 24422 rpcz_store.cc:276] 0809 23:18:52.186978 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.187100 (+   122us) service_pool.cc:227] Handling call
0809 23:18:53.349002 (+1161902us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:53.349130 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1048741120}
W20260809 23:18:53.416443 24425 rpcz_store.cc:269] 0809 23:18:51.794721 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.372495 (+1577774us) service_pool.cc:227] Handling call
0809 23:18:53.411908 (+ 39413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.501755 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.418433 27812 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:53.424666 28026 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:53.427199 27999 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.247901 28077 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:53.193861 27938 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.204968 27813 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260809 23:18:53.445355 24424 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:53.628319 24424 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:53.628439 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.247339 27797 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:53.525626 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.328521 24430 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:53.467660 24429 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
W20260809 23:18:53.631052 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.46357e+06us
I20260809 23:18:53.464598 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.470055 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:53.632004 24435 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 2.
I20260809 23:18:53.480322 27969 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:53.405117 28004 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:53.483922 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:53.483340 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:53.404081 28094 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
W20260809 23:18:53.484113 28091 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.314s	user 0.001s	sys 0.000s
I20260809 23:18:53.566613 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:53.511157 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:53.651691 25363 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
W20260809 23:18:53.652734 28092 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.490s	user 0.002s	sys 0.000s
I20260809 23:18:53.514268 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:53.514529 24422 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:18:53.516106 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2263) took 1238 ms. Trace:
I20260809 23:18:53.655884 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 456) took 1455 ms. Trace:
I20260809 23:18:53.519382 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.656524 24316 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:53.520170 24321 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:53.519837 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.499106 27952 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.505043 28083 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.565335 28124 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:18:53.574473 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 677) took 2514 ms. Trace:
I20260809 23:18:53.565982 28113 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260809 23:18:53.579644 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 673) took 2550 ms. Trace:
I20260809 23:18:53.671329 24314 rpcz_store.cc:276] 0809 23:18:51.029141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.139291 (+110150us) service_pool.cc:227] Handling call
0809 23:18:51.149812 (+ 10521us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.182268 (+1032456us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.579197 (+1396929us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b540043bcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.579564 (+   367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2278363904}
I20260809 23:18:53.536747 27999 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:53.565757 28026 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:53.511741 27981 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:53.586140 28022 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:53.574319 27949 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:18:53.565107 28100 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:53.566727 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:53.566418 28107 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.216s	user 0.000s	sys 0.000s
W20260809 23:18:53.565833 28110 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260809 23:18:53.571705 27812 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.705133 24543 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 23:18:53.706192 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 497) took 2034 ms. Trace:
I20260809 23:18:53.615335 27941 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.572118 28136 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:53.616585 27976 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:53.575935 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:53.677297 27949 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:53.616014 28064 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.620725 27590 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.624886 27938 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.625257 28047 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:53.630034 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:53.723083 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 93163.5us
I20260809 23:18:53.252094 28020 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:18:53.628993 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 467) took 1102 ms. Trace:
I20260809 23:18:53.631349 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 455) took 1440 ms. Trace:
I20260809 23:18:53.488878 27984 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:53.569041 28117 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:18:53.636657 27969 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:18:53.651081 28033 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.747s	user 0.004s	sys 0.010s
I20260809 23:18:53.656275 24422 rpcz_store.cc:276] 0809 23:18:52.200889 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.500202 (+1299313us) service_pool.cc:227] Handling call
0809 23:18:53.514320 (+ 14118us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.655627 (+141307us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.655818 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.520435 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.656003 24425 rpcz_store.cc:276] 0809 23:18:52.268792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.500943 (+1232151us) service_pool.cc:227] Handling call
0809 23:18:53.506372 (+  5429us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.506545 (+   173us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:53.506578 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:53.506626 (+    48us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:18:53.506662 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.506726 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.506840 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.662878 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:53.739244 24540 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:18:53.516736 27649 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.613s	user 0.008s	sys 0.004s
I20260809 23:18:53.655653 25370 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:18:53.666739 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 182886us
I20260809 23:18:53.742354 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:53.499191 24431 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:18:53.744705 25375 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
W20260809 23:18:53.667838 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 419651us
I20260809 23:18:53.667740 24309 rpcz_store.cc:276] 0809 23:18:51.060598 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.922027 (+861429us) service_pool.cc:227] Handling call
0809 23:18:52.046356 (+124329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.135387 (+ 89031us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.574248 (+1438861us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b54003854d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:53.574408 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117681920}
I20260809 23:18:53.630683 27797 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.691927 28122 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.674072 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.761106 24321 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:53.582083 27847 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.672933 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:53.654912 24244 auto_rebalancer.cc:663] Could not move replica: Not found: Couldn't find leader for tablet e4ed6a400f2643da95b8f744ebab6f4c
I20260809 23:18:53.675657 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:53.663414 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 180141us
I20260809 23:18:53.775216 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:18:53.681222 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 177180us
I20260809 23:18:53.731133 27812 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:53.677560 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.650269 27723 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:53.682999 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:53.667168 27903 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:53.669227 27628 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:53.667461 28124 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:53.706552 24543 rpcz_store.cc:276] 0809 23:18:51.671642 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.704570 (+2032928us) service_pool.cc:227] Handling call
0809 23:18:53.704914 (+   344us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.705913 (+   999us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.706127 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.691154 27981 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.704350 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2084) took 1680 ms. Trace:
W20260809 23:18:53.828513 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 145636us
I20260809 23:18:53.828715 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:53.828409 24544 rpcz_store.cc:276] 0809 23:18:52.024140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.478977 (+454837us) service_pool.cc:227] Handling call
0809 23:18:52.479323 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:52.873945 (+394622us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.704077 (+830132us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.704282 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":891826560}
I20260809 23:18:53.829452 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.830078 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:53.683607 28066 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.025s	sys 0.031s
I20260809 23:18:53.724041 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.724681 24429 rpcz_store.cc:276] 0809 23:18:52.191293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.464242 (+1272949us) service_pool.cc:227] Handling call
0809 23:18:53.466551 (+  2309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.631112 (+164561us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.631300 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.724642 24424 rpcz_store.cc:276] 0809 23:18:52.526383 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.627564 (+1101181us) service_pool.cc:227] Handling call
0809 23:18:53.627978 (+   414us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.628713 (+   735us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.628936 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:53.833418 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 109464us
I20260809 23:18:53.734325 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 712) took 2084 ms. Trace:
I20260809 23:18:53.834101 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.834363 25371 rpcz_store.cc:276] 0809 23:18:51.649698 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.651010 (+  1312us) service_pool.cc:227] Handling call
0809 23:18:51.651309 (+   299us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.226137 (+574828us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.734031 (+1507894us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.734257 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1233930496}
I20260809 23:18:53.735199 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2297) took 2565 ms. Trace:
I20260809 23:18:53.835269 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.835749 27952 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:53.835893 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:53.736315 24435 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:18:53.727487 27590 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.630601 27715 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.834087 24424 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:18:53.747474 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:53.723711 27780 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.713s	user 0.009s	sys 0.002s
I20260809 23:18:53.775774 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 470) took 1013 ms. Trace:
I20260809 23:18:53.474265 27633 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.732450 27976 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:54.079599 27976 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:54.080000 27976 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:54.080194 28131 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:54.080993 27590 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:53.807304 28024 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.896s	user 0.020s	sys 0.026s
I20260809 23:18:53.742832 27969 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:53.804240 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:53.763414 27797 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:53.807734 28122 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.746410 28026 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:53.774619 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Not found: Couldn't find leader for tablet e4ed6a400f2643da95b8f744ebab6f4c
I20260809 23:18:53.817826 24314 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
W20260809 23:18:53.817792 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 468) took 6463 ms (client timeout 1701 ms). Trace:
I20260809 23:18:53.818769 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.816265 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 511) took 1006 ms. Trace:
I20260809 23:18:53.824592 24540 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:18:53.727691 28004 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.824940 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:53.693440 28022 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.825937 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2259) took 2031 ms. Trace:
I20260809 23:18:54.100234 24539 rpcz_store.cc:276] 0809 23:18:52.810226 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.406748 (+596522us) service_pool.cc:227] Handling call
0809 23:18:53.412994 (+  6246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.413243 (+   249us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:18:53.693994 (+280751us) log.cc:842] Serialized 14 byte log entry
0809 23:18:53.815933 (+121939us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:53.815980 (+    47us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:53.816030 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.816090 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.816211 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4151,"prepare.queue_time_us":161907,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":6791,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7852,"thread_start_us":386077,"threads_started":2,"wal-append.queue_time_us":145297}
I20260809 23:18:54.125470 24427 rpcz_store.cc:276] 0809 23:18:51.794418 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.997989 (+1203571us) service_pool.cc:227] Handling call
0809 23:18:52.998316 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.998608 (+   292us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:52.998646 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:53.007378 (+  8732us) log.cc:842] Serialized 26 byte log entry
0809 23:18:53.199185 (+191807us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:53.199237 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.199324 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:53.825744 (+626420us) raft_consensus.cc:1671] finished
0809 23:18:53.825764 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.825879 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113361,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":8246,"thread_start_us":137728,"threads_started":1,"wal-append.queue_time_us":299078}
I20260809 23:18:53.572311 27730 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:53.756762 28108 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:53.832358 24545 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 23:18:53.838424 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:53.839370 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:54.144487 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 305184us
W20260809 23:18:53.821789 28019 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.698s	user 0.007s	sys 0.004s
I20260809 23:18:54.145306 28019 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:53.760071 24425 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:53.701792 28077 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:53.843694 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96289.6us
I20260809 23:18:53.843257 24422 rpcz_store.cc:276] 0809 23:18:52.762757 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.738594 (+975837us) service_pool.cc:227] Handling call
0809 23:18:53.775217 (+ 36623us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.775442 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:53.775466 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:53.775519 (+    53us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:53.775554 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.775607 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:53.775715 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.835605 24323 rpcz_store.cc:276] 0809 23:18:51.169625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.948890 (+779265us) service_pool.cc:227] Handling call
0809 23:18:52.046194 (+ 97304us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:52.133116 (+ 86922us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.734979 (+1601863us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b54000dbbd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:53.735132 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55633280}
I20260809 23:18:54.147519 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 477) took 1197 ms. Trace:
I20260809 23:18:54.147909 24425 rpcz_store.cc:276] 0809 23:18:52.956023 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.743294 (+787271us) service_pool.cc:227] Handling call
0809 23:18:53.748601 (+  5307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.147181 (+398580us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.147451 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.148432 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:53.841655 24543 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:18:53.714148 28136 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.149626 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2111) took 1116 ms. Trace:
I20260809 23:18:54.150012 24543 rpcz_store.cc:276] 0809 23:18:53.033721 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.836633 (+802912us) service_pool.cc:227] Handling call
0809 23:18:53.836938 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.149373 (+312435us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.149570 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.855624 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2087) took 1053 ms. Trace:
I20260809 23:18:54.151269 24543 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:54.151897 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2118) took 1082 ms. Trace:
I20260809 23:18:54.151963 28136 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:53.859594 27971 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:18:54.152851 24423 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 2: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:54.153047 27972 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:54.153136 27971 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:54.153618 27972 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:54.153775 27971 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:54.152860 24317 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
I20260809 23:18:54.146706 28113 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:54.154899 28113 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:54.155284 28113 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:53.838043 27986 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:54.104797 27945 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:54.159255 27986 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:54.159844 27986 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:53.837519 28103 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:18:53.824069 27949 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:53.833098 28066 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.025s	sys 0.031s
I20260809 23:18:54.163169 28100 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:54.163354 28066 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:54.163807 28100 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.164574 28019 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:54.165199 28019 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:18:54.164680 28066 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.166008 28100 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.166641 24425 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:18:53.757830 27649 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.169098 27649 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:53.856719 27638 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:54.072007 24429 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 23:18:54.152235 24543 rpcz_store.cc:276] 0809 23:18:53.069311 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.150779 (+1081468us) service_pool.cc:227] Handling call
0809 23:18:54.151111 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.151698 (+   587us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.151850 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.171568 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1826) took 1115 ms. Trace:
I20260809 23:18:54.171888 24429 rpcz_store.cc:276] 0809 23:18:53.056501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.833630 (+777129us) service_pool.cc:227] Handling call
0809 23:18:53.865064 (+ 31434us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.171314 (+306250us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.171518 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.857959 27968 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:54.178783 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:53.814435 28065 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:53.843204 27903 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260809 23:18:54.183485 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:54.102414 27805 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:53.843549 28047 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:18:54.074306 24544 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:54.191874 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2101) took 1186 ms. Trace:
I20260809 23:18:54.192267 24544 rpcz_store.cc:276] 0809 23:18:53.005666 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.829579 (+823913us) service_pool.cc:227] Handling call
0809 23:18:54.073893 (+244314us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.191580 (+117687us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.191813 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:53.830722 28121 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260809 23:18:53.875468 27938 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:53.820288 27689 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:18:54.171119 27638 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:54.082393 28002 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:53.732841 27984 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:54.070246 27952 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:54.086483 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 241325us
W20260809 23:18:54.092207 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 288044us
I20260809 23:18:54.235633 24546 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:54.216812 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:54.245012 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:53.824673 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:54.099239 24535 rpcz_store.cc:269] 0809 23:18:47.312109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:47.437979 (+125870us) service_pool.cc:227] Handling call
0809 23:18:53.746281 (+6308302us) spinlock_profiling.cc:243] Waited 4.5 s on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:53.775926 (+ 29645us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10353127168}
I20260809 23:18:54.102890 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:54.092610 28024 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.181s	user 0.020s	sys 0.026s
W20260809 23:18:54.100715 27857 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.665s	user 0.014s	sys 0.008s
I20260809 23:18:54.133152 25371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:18:54.263033 25371 raft_consensus.cc:595] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:54.222019 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:54.263424 25371 raft_consensus.cc:603] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:18:54.134023 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:54.135187 24427 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:54.262022 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:54.265012 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1835) took 1200 ms. Trace:
I20260809 23:18:54.136207 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2117) took 1067 ms. Trace:
I20260809 23:18:54.265779 24539 rpcz_store.cc:276] 0809 23:18:53.068965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.126616 (+1057651us) service_pool.cc:227] Handling call
0809 23:18:54.135727 (+  9111us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.135907 (+   180us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:54.135933 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:54.135973 (+    40us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:54.136002 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.136049 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:54.136151 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.137532 28083 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.270545 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:54.144435 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 306072us
I20260809 23:18:53.828053 27628 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:54.152652 27847 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:54.151075 24541 rpcz_store.cc:276] 0809 23:18:52.802058 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.802183 (+   125us) service_pool.cc:227] Handling call
0809 23:18:52.814343 (+ 12160us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.161551 (+347208us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:53.855243 (+693692us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400116554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:53.855513 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":671534336}
W20260809 23:18:54.180339 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 367181us
I20260809 23:18:54.203269 28047 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:54.243049 28002 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:54.187276 24323 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:18:54.194968 24422 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:54.195652 24429 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:54.196563 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:54.091373 28131 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.092s
I20260809 23:18:54.151978 28064 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.129856 27730 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:54.081800 27982 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:18:54.154201 27972 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:54.202543 27813 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.211136 27941 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.212664 28122 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:54.213611 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140758us
I20260809 23:18:54.289639 27628 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:54.215781 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1824) took 1159 ms. Trace:
I20260809 23:18:54.320652 24431 rpcz_store.cc:276] 0809 23:18:53.055916 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.744199 (+688283us) service_pool.cc:227] Handling call
0809 23:18:53.744611 (+   412us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.744913 (+   302us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:53.744950 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:53.748957 (+  4007us) log.cc:842] Serialized 26 byte log entry
0809 23:18:54.100117 (+351160us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:54.100186 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.100320 (+   134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:54.215487 (+115167us) raft_consensus.cc:1671] finished
0809 23:18:54.215515 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:54.215703 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":256975,"threads_started":1,"wal-append.queue_time_us":384525}
I20260809 23:18:54.266664 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:54.265483 24427 rpcz_store.cc:276] 0809 23:18:53.065061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.126802 (+1061741us) service_pool.cc:227] Handling call
0809 23:18:54.134824 (+  8022us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.264760 (+129936us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.264959 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.261472 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:54.194361 28103 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:18:54.268491 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:54.195755 28062 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:54.195389 27968 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:54.214573 27638 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:54.186441 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:54.287544 28152 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:54.287282 28161 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:18:54.290089 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:53.813670 28020 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:54.286618 25321 connection.cc:232] RPC call timeout handler was delayed by -1.204306255094707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31120000000000003s for a response
I20260809 23:18:54.292277 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:54.242738 28128 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.294615 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1839) took 1138 ms. Trace:
I20260809 23:18:54.406008 24422 rpcz_store.cc:276] 0809 23:18:53.156265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.171373 (+1015108us) service_pool.cc:227] Handling call
0809 23:18:54.189630 (+ 18257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.294331 (+104701us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.294547 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.288600 28156 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.296924 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:54.297513 24541 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:18:54.408728 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2119) took 1339 ms. Trace:
I20260809 23:18:54.409116 24541 rpcz_store.cc:276] 0809 23:18:53.069607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.283229 (+1213622us) service_pool.cc:227] Handling call
0809 23:18:54.297232 (+ 14003us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.408419 (+111187us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.408666 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.298034 24543 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:54.300462 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1840) took 1111 ms. Trace:
I20260809 23:18:54.410207 24429 rpcz_store.cc:276] 0809 23:18:53.188689 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.172708 (+984019us) service_pool.cc:227] Handling call
0809 23:18:54.195497 (+ 22789us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.300232 (+104735us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.300409 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.311080 24535 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:18:54.306619 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260809 23:18:54.313452 27812 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:18:54.411337 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2120) took 1341 ms. Trace:
I20260809 23:18:54.411641 24535 rpcz_store.cc:276] 0809 23:18:53.069882 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.289450 (+1219568us) service_pool.cc:227] Handling call
0809 23:18:54.310908 (+ 21458us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.411100 (+100192us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.411288 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.313114 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:54.272146 27945 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:18:54.203608 27938 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.316660 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 469) took 1554 ms. Trace:
I20260809 23:18:54.271692 28019 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:54.189388 28108 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:54.318363 24316 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.000s
I20260809 23:18:54.425812 24316 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:54.427030 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:54.290805 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 107407us
I20260809 23:18:54.326609 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:54.327209 28103 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.327430 27649 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:18:54.328749 27730 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.330291 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2110) took 1297 ms. Trace:
I20260809 23:18:54.331202 27972 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.331801 27946 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
W20260809 23:18:54.335048 28082 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.536s	user 0.003s	sys 0.000s
I20260809 23:18:54.291427 27723 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:54.493309 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 208091us
I20260809 23:18:54.199994 28065 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.268539 27857 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.317711 27847 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:18:54.265415 27952 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.397146 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:54.315150 27941 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:54.398358 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:54.400195 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:54.463779 28019 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.313843 27813 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.413313 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:18:54.421949 24539 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:54.422223 24435 rpcz_store.cc:276] 0809 23:18:52.762382 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.632714 (+870332us) service_pool.cc:227] Handling call
0809 23:18:53.633116 (+   402us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.843964 (+210848us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.316334 (+472370us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:54.316590 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228771200}
I20260809 23:18:54.517664 27847 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:54.390239 28100 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:54.423048 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:54.404124 28152 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:54.497907 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:54.198436 28124 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:54.464986 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 196549us
I20260809 23:18:54.463796 24534 rpcz_store.cc:276] 0809 23:18:53.033341 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.194847 (+161506us) service_pool.cc:227] Handling call
0809 23:18:54.330107 (+1135260us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:54.330237 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424494464}
W20260809 23:18:54.503502 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 281585us
W20260809 23:18:54.424660 27980 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.220s	user 0.011s	sys 0.008s
I20260809 23:18:54.514660 28159 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:54.509644 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 183096us
I20260809 23:18:54.608875 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:54.429507 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:54.609934 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
W20260809 23:18:54.511919 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 240248us
I20260809 23:18:54.513877 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:54.428952 28161 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.613289 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:54.414085 27976 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.613874 24427 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:18:54.524240 24313 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:54.611119 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:54.614902 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2304) took 1121 ms. Trace:
I20260809 23:18:54.410431 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2121) took 1340 ms. Trace:
I20260809 23:18:54.416643 28020 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.529675 27986 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.418222 28128 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.424458 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
W20260809 23:18:54.531528 28150 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260809 23:18:54.532059 27903 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:54.532652 27628 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.542212 28022 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:54.428740 28108 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:18:54.502717 27946 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:54.423771 24541 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:54.550295 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:54.555457 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2098) took 1480 ms. Trace:
I20260809 23:18:54.726308 24539 rpcz_store.cc:276] 0809 23:18:53.075484 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.405476 (+1329992us) service_pool.cc:227] Handling call
0809 23:18:54.421720 (+ 16244us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.555134 (+133414us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.555389 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.726326 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2099) took 1603 ms. Trace:
I20260809 23:18:54.724323 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2269) took 1466 ms. Trace:
I20260809 23:18:54.727707 24541 rpcz_store.cc:276] 0809 23:18:53.123307 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.409982 (+1286675us) service_pool.cc:227] Handling call
0809 23:18:54.423612 (+ 13630us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.726088 (+302476us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.726271 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.727756 24429 rpcz_store.cc:276] 0809 23:18:53.257932 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.617405 (+1359473us) service_pool.cc:227] Handling call
0809 23:18:54.621329 (+  3924us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.672312 (+ 50983us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:54.672341 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:54.672385 (+    44us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:54.723971 (+ 51586us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.724081 (+   110us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:54.724248 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50648}
W20260809 23:18:54.509050 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97183.9us
I20260809 23:18:54.596601 25371 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:54.598352 28066 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:18:54.604347 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:54.421407 27633 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:54.611133 24378 leader_election.cc:398] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
I20260809 23:18:54.615290 24313 rpcz_store.cc:276] 0809 23:18:53.493869 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.523457 (+1029588us) service_pool.cc:227] Handling call
0809 23:18:54.523994 (+   537us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.614666 (+ 90672us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.614847 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.615506 24543 rpcz_store.cc:276] 0809 23:18:53.070148 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.297373 (+1227225us) service_pool.cc:227] Handling call
0809 23:18:54.297796 (+   423us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.410181 (+112385us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.410380 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.550273 28083 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:54.427213 28047 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.420513 28156 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.563176 27847 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:54.660373 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48945.4us
I20260809 23:18:54.645967 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:54.641896 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:54.556800 28019 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.618400 27950 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.862097 24539 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:18:54.862974 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 519) took 1402 ms. Trace:
I20260809 23:18:54.562206 27941 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.865113 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:54.831568 24431 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:54.863723 24539 rpcz_store.cc:276] 0809 23:18:53.460345 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.727204 (+1266859us) service_pool.cc:227] Handling call
0809 23:18:54.861880 (+134676us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.862678 (+   798us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.862901 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.830879 24378 leader_election.cc:401] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260809 23:18:54.835556 24316 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:54.752897 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2300) took 1260 ms. Trace:
I20260809 23:18:54.688588 24320 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:54.731328 28160 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260809 23:18:54.857518 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:18:54.745601 28100 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:18:54.686477 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:54.877444 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:54.881455 25371 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:18:54.881971 27999 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.190s	user 0.009s	sys 0.002s
I20260809 23:18:54.882660 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:54.876773 27751 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260809 23:18:54.750875 27857 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.731426 28177 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.625461 28124 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:54.615649 27945 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:54.591928 28172 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:54.854349 24422 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:54.618816 28113 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:54.691226 27666 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.202s	user 0.008s	sys 0.005s
I20260809 23:18:54.867429 24317 rpcz_store.cc:276] 0809 23:18:53.492526 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.502533 (+ 10007us) service_pool.cc:227] Handling call
0809 23:18:54.752668 (+1250135us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:54.752818 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":952374144}
I20260809 23:18:54.872484 24543 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:18:54.948827 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2103) took 1445 ms. Trace:
I20260809 23:18:54.949280 24543 rpcz_store.cc:276] 0809 23:18:53.503785 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.831526 (+1327741us) service_pool.cc:227] Handling call
0809 23:18:54.870686 (+ 39160us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.948498 (+ 77812us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.948754 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:54.687392 28122 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:54.732861 27976 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.748756 28020 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:54.881390 28154 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:54.885605 24535 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:54.731134 27980 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.886425 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:18:54.782339 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 110839us
I20260809 23:18:54.835927 24427 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:18:54.733930 28103 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:54.989483 24427 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:18:54.691684 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 780) took 1196 ms. Trace:
I20260809 23:18:54.897651 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:54.882704 27999 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:54.875921 27950 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:54.902832 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:54.999665 24539 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:54.762377 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1840) took 1667 ms. Trace:
I20260809 23:18:54.750895 27723 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
I20260809 23:18:54.860589 28131 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260809 23:18:54.958678 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 468) took 2245 ms. Trace:
I20260809 23:18:55.013206 24430 rpcz_store.cc:276] 0809 23:18:52.713617 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.636958 (+923341us) service_pool.cc:227] Handling call
0809 23:18:53.637287 (+   329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.637775 (+   488us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:53.637819 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:53.666166 (+ 28347us) log.cc:842] Serialized 26 byte log entry
0809 23:18:54.086289 (+420123us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:54.086347 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.086446 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:54.958391 (+871945us) raft_consensus.cc:1671] finished
0809 23:18:54.958437 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:54.958604 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":571136,"thread_start_us":378541,"threads_started":1,"wal-append.queue_time_us":617724}
I20260809 23:18:54.979697 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:54.990280 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43969.6us
I20260809 23:18:54.992825 24422 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:18:55.023170 24422 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:54.990073 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1845) took 1361 ms. Trace:
W20260809 23:18:54.953428 28002 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.000s
I20260809 23:18:54.744530 28166 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.203s
I20260809 23:18:55.001420 24543 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:54.641405 28161 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.000607 24311 rpcz_store.cc:276] 0809 23:18:53.094734 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.113402 (+ 18668us) service_pool.cc:227] Handling call
0809 23:18:53.121671 (+  8269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.121941 (+   270us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:18:54.389613 (+1267672us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b18002414e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:18:54.398750 (+  9137us) log.cc:842] Serialized 14 byte log entry
0809 23:18:54.761924 (+363174us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:54.761996 (+    72us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:54.762063 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.762147 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:54.762309 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114109,"prepare.queue_time_us":260398,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":114899,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":36518,"spinlock_wait_cycles":67521024,"thread_start_us":397773,"threads_started":2,"wal-append.queue_time_us":475876}
I20260809 23:18:55.008503 25368 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
W20260809 23:18:55.010162 27971 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.856s	user 0.014s	sys 0.000s
I20260809 23:18:55.024034 24427 rpcz_store.cc:276] 0809 23:18:53.628763 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.988860 (+1360097us) service_pool.cc:227] Handling call
0809 23:18:54.989152 (+   292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.989838 (+   686us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.990025 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:55.026062 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 415019us
I20260809 23:18:55.027050 24309 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 6. Ops: []
W20260809 23:18:55.026994 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 169534us
I20260809 23:18:54.993783 28066 consensus_queue.cc:1225] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:18:54.618677 28159 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.033588 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1834) took 1969 ms. Trace:
I20260809 23:18:54.989118 28154 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:55.038964 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2104) took 1426 ms. Trace:
I20260809 23:18:55.241956 24543 rpcz_store.cc:276] 0809 23:18:53.612568 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.950214 (+1337646us) service_pool.cc:227] Handling call
0809 23:18:55.001180 (+ 50966us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:55.038650 (+ 37470us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:55.038876 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:55.027982 27976 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:55.016027 28103 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:18:54.991983 25365 rpcz_store.cc:276] 0809 23:18:53.494976 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.497395 (+  2419us) service_pool.cc:227] Handling call
0809 23:18:54.312968 (+815573us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900b7c 00007fac2090073b 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:54.674231 (+361263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12329216}
I20260809 23:18:54.927093 28177 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.089995 28077 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:18:55.144677 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 104358us
I20260809 23:18:54.749197 28152 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.100289 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:55.319197 25371 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:55.035787 24422 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:55.002242 27980 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.096969 24311 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:18:55.242377 28196 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:55.030992 27938 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:55.244326 27857 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:54.945494 28108 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:55.240746 24424 rpcz_store.cc:276] 0809 23:18:53.063892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.839580 (+775688us) service_pool.cc:227] Handling call
0809 23:18:53.839978 (+   398us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:53.840375 (+   397us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:53.840412 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:54.103339 (+262927us) log.cc:842] Serialized 26 byte log entry
0809 23:18:54.339343 (+236004us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:54.339403 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.339492 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:55.033317 (+693825us) raft_consensus.cc:1671] finished
0809 23:18:55.033347 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:55.033521 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11266,"raft.queue_time_us":2084,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"thread_start_us":156518,"threads_started":1,"wal-append.queue_time_us":395137}
I20260809 23:18:55.030608 28065 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:55.091109 27971 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:55.217455 27638 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.817s	user 0.010s	sys 0.001s
I20260809 23:18:55.097445 27952 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:55.213722 27813 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:55.319949 24422 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:55.326478 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2309) took 1709 ms. Trace:
I20260809 23:18:55.036674 25368 pending_rounds.cc:76] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 5
I20260809 23:18:55.326961 24539 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:18:54.953840 28172 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.006263 27666 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.213s
W20260809 23:18:55.341346 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 336765us
I20260809 23:18:55.330204 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:55.288192 28122 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:55.034145 28002 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.290231 27976 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:55.259016 28181 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.481s
I20260809 23:18:55.247723 27945 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:56.149802 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.915s	user 0.012s	sys 0.027s
W20260809 23:18:56.259398 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.016s	sys 0.027s
W20260809 23:18:56.250244 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.0349e+06us
I20260809 23:18:55.289577 28057 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:56.272568 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:56.275884 24311 rpcz_store.cc:276] 0809 23:18:53.617451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.032212 (+1414761us) service_pool.cc:227] Handling call
0809 23:18:55.096744 (+ 64532us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:55.325544 (+228800us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:55.326334 (+   790us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:55.932231 24431 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.025s	user 0.000s	sys 0.008s
W20260809 23:18:56.284474 24431 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.378s	user 0.000s	sys 0.011s
I20260809 23:18:55.390856 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:56.250572 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 485) took 1960 ms. Trace:
I20260809 23:18:56.326427 24427 rpcz_store.cc:276] 0809 23:18:54.290547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.042198 (+751651us) service_pool.cc:227] Handling call
0809 23:18:55.097427 (+ 55229us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.250170 (+1152743us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:18:56.250214 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:56.250263 (+    49us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:56.250307 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.250390 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:56.250512 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121772,"raft.queue_time_us":1152778}
W20260809 23:18:54.622268 28014 consensus_queue.cc:1159] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 7 status { last_received { term: 4 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]" } } last_received_current_leader { term: 0 index: 0 } }
W20260809 23:18:56.445103 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.830s	user 0.014s	sys 0.081s
I20260809 23:18:56.394490 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 521) took 2580 ms. Trace:
I20260809 23:18:56.262773 25372 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:56.391623 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:56.467520 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75998.4us
I20260809 23:18:56.335768 24313 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:18:56.416020 25370 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:18:56.465294 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.850s	user 0.016s	sys 0.084s
I20260809 23:18:56.466552 24539 rpcz_store.cc:276] 0809 23:18:53.814516 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.000426 (+1185910us) service_pool.cc:227] Handling call
0809 23:18:55.026855 (+ 26429us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.275285 (+1248430us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.394220 (+118935us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:56.394413 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":689266560}
W20260809 23:18:56.393541 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 452) took 4204 ms (client timeout 3112 ms). Trace:
I20260809 23:18:56.392710 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:56.393595 24429 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
W20260809 23:18:55.745873 28183 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.564s	user 0.000s	sys 0.002s
W20260809 23:18:56.389263 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.05907e+06us
W20260809 23:18:56.392221 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 119769us
W20260809 23:18:55.594296 28184 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.589s	user 0.000s	sys 0.001s
W20260809 23:18:56.174120 27797 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 2.050s	user 0.011s	sys 0.002s
W20260809 23:18:55.592648 28169 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.640s	user 0.001s	sys 0.000s
I20260809 23:18:56.330295 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 730) took 1479 ms. Trace:
I20260809 23:18:56.506685 25371 rpcz_store.cc:276] 0809 23:18:54.850725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.339320 (+488595us) service_pool.cc:227] Handling call
0809 23:18:56.330210 (+990890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:55.324764 28103 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:18:55.600069 28187 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.502s	user 0.001s	sys 0.000s
W20260809 23:18:56.246395 28135 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.161s	user 0.007s	sys 0.000s
W20260809 23:18:55.880731 27969 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.785s	user 0.004s	sys 0.010s
W20260809 23:18:55.869791 28190 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.295s	user 0.001s	sys 0.002s
I20260809 23:18:55.835326 28128 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.392359 27971 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:55.323714 27628 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:56.480845 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 3.112s (SENT) [suppressed 22 similar messages]
W20260809 23:18:56.171455 27984 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.899s	user 0.012s	sys 0.002s
I20260809 23:18:56.358497 28122 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.447986 27651 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:55.363154 27999 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:56.470902 28192 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260809 23:18:56.323375 25368 pending_rounds.cc:97] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260809 23:18:56.483160 27986 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.322700 27638 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.604321 28165 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.484522 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:56.484928 28194 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:56.486474 28179 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260809 23:18:56.487254 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:56.486850 27730 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:56.488564 28024 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:56.492000 28108 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:56.492756 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 784) took 2990 ms. Trace:
I20260809 23:18:56.499821 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 515) took 3427 ms. Trace:
I20260809 23:18:56.501029 28002 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:18:56.501444 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2308) took 2982 ms (client timeout 1556 ms). Trace:
I20260809 23:18:56.502486 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2100) took 3108 ms. Trace:
I20260809 23:18:56.484416 28066 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:18:56.504674 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:56.392616 28134 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.309s	user 0.008s	sys 0.011s
I20260809 23:18:55.876387 28113 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:55.851200 27941 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:55.545120 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:56.579931 27984 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.597829 28203 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.603227 28207 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:56.604624 28024 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.620424 28203 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.617375 28066 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:56.523341 27969 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.627673 28066 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.589944 27981 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.477006 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 725) took 1875 ms. Trace:
I20260809 23:18:56.631632 27981 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.631922 25372 rpcz_store.cc:276] 0809 23:18:54.602049 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.602183 (+   134us) service_pool.cc:227] Handling call
0809 23:18:56.262075 (+1659892us) spinlock_profiling.cc:243] Waited 9.96 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:18:56.262333 (+   258us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.467086 (+204753us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.476929 (+  9843us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22901504}
I20260809 23:18:56.632647 27981 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:55.613368 27980 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.322239 28057 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:56.635284 28057 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:56.617061 28113 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
W20260809 23:18:55.607841 28186 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
I20260809 23:18:55.430142 28019 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:55.612591 28156 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.391659 28064 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:56.469503 24423 rpcz_store.cc:269] 0809 23:18:52.189110 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:52.948074 (+758964us) service_pool.cc:227] Handling call
0809 23:18:56.393220 (+3445146us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:56.393405 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2609025792}
I20260809 23:18:56.529502 24422 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:18:56.642886 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1844) took 3014 ms. Trace:
I20260809 23:18:56.643309 24422 rpcz_store.cc:276] 0809 23:18:53.628469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.939259 (+1310790us) service_pool.cc:227] Handling call
0809 23:18:54.979218 (+ 39959us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.642610 (+1663392us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.642820 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.644834 24422 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260809 23:18:56.509205 28124 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.583s	user 0.013s	sys 0.003s
I20260809 23:18:56.574414 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:56.645825 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 851) took 1788 ms. Trace:
I20260809 23:18:56.646242 25371 rpcz_store.cc:276] 0809 23:18:54.857319 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.515453 (+1658134us) service_pool.cc:227] Handling call
0809 23:18:56.645737 (+130284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.508510 27666 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:55.370572 27857 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.522322 28159 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.515216 27797 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:56.587754 27649 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.579s	user 0.043s	sys 0.064s
I20260809 23:18:56.572082 28177 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:56.596045 24311 raft_consensus.cc:2354] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
I20260809 23:18:56.578739 28128 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:55.575536 28154 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.599526 25375 rpcz_store.cc:276] 0809 23:18:53.502195 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.511904 (+  9709us) service_pool.cc:227] Handling call
0809 23:18:53.565511 (+ 53607us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:53.759828 (+194317us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.492533 (+2732705us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:56.492698 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411520256}
I20260809 23:18:56.600035 24535 rpcz_store.cc:276] 0809 23:18:53.394399 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.412356 (+1017957us) service_pool.cc:227] Handling call
0809 23:18:54.428177 (+ 15821us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.978695 (+550518us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.502274 (+1523579us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b54000b6154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:56.502436 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1037175680}
I20260809 23:18:56.597859 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2112) took 3550 ms. Trace:
I20260809 23:18:56.596226 24539 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
W20260809 23:18:56.600625 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 726) took 1998 ms (client timeout 1546 ms). Trace:
W20260809 23:18:56.665755 25373 rpcz_store.cc:269] 0809 23:18:54.602363 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.602474 (+   111us) service_pool.cc:227] Handling call
0809 23:18:56.504522 (+1902048us) spinlock_profiling.cc:243] Waited 6.2 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:56.600541 (+ 96019us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14270848}
I20260809 23:18:56.581933 27628 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:56.599666 24544 rpcz_store.cc:276] 0809 23:18:53.072613 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.401888 (+1329275us) service_pool.cc:227] Handling call
0809 23:18:54.423815 (+ 21927us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.545722 (+121907us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:54.545765 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:54.545819 (+    54us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:54.545868 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.499619 (+1953751us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:56.499658 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:56.499766 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279821696}
I20260809 23:18:56.658941 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:56.599905 24320 rpcz_store.cc:269] 0809 23:18:53.519447 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.643173 (+123726us) service_pool.cc:227] Handling call
0809 23:18:56.501246 (+2858073us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b540019d2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:56.501378 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2095137536}
I20260809 23:18:56.583832 25368 raft_consensus.cc:2866] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:18:56.621099 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 734) took 1725 ms (client timeout 2202 ms). Trace:
I20260809 23:18:55.545547 28077 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:18:56.664040 24540 rpcz_store.cc:276] 0809 23:18:53.047594 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.101364 (+1053770us) service_pool.cc:227] Handling call
0809 23:18:54.101692 (+   328us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.122730 (+ 21038us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:54.122759 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:54.125712 (+  2953us) log.cc:842] Serialized 26 byte log entry
0809 23:18:54.679152 (+553440us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:54.679204 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:54.679292 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:56.597552 (+1918260us) raft_consensus.cc:1671] finished
0809 23:18:56.597578 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:56.597783 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":435,"raft.queue_time_us":41321,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":489846,"threads_started":1,"wal-append.queue_time_us":708783}
W20260809 23:18:56.601491 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123423us
W20260809 23:18:56.599575 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.743s	user 0.016s	sys 0.090s
I20260809 23:18:56.649446 24321 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:56.608182 28062 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 2.249s	user 0.006s	sys 0.010s
I20260809 23:18:56.658100 24427 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260809 23:18:56.672561 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.817s	user 0.016s	sys 0.091s
I20260809 23:18:56.658598 24428 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:56.607336 28192 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:56.616729 27847 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.749s	user 0.016s	sys 0.000s
I20260809 23:18:56.587338 27651 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.673655 24312 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260809 23:18:56.674810 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:56.677439 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:56.678251 24422 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:56.675930 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 2.02989e+06us
I20260809 23:18:56.653308 28019 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.649899 28064 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:56.636461 28172 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.720124 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:56.721127 24423 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:18:56.722409 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:56.723165 24427 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:56.652124 28181 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.695382 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:56.666443 27649 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.657s	user 0.046s	sys 0.064s
I20260809 23:18:56.695837 25375 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:56.731765 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 731) took 1871 ms. Trace:
I20260809 23:18:56.732016 24311 raft_consensus.cc:2396] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:18:56.732256 25375 rpcz_store.cc:276] 0809 23:18:54.863918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.663673 (+1799755us) service_pool.cc:227] Handling call
0809 23:18:56.695391 (+ 31718us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.731453 (+ 36062us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.731689 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.704324 24540 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 23:18:56.734041 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 524) took 1956 ms. Trace:
I20260809 23:18:56.734483 24540 rpcz_store.cc:276] 0809 23:18:54.777352 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.671579 (+1894227us) service_pool.cc:227] Handling call
0809 23:18:56.704022 (+ 32443us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:56.733725 (+ 29703us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.733974 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.705756 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 852) took 1841 ms. Trace:
I20260809 23:18:56.735603 25373 rpcz_store.cc:276] 0809 23:18:54.865946 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.671090 (+1805144us) service_pool.cc:227] Handling call
0809 23:18:56.705116 (+ 34026us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.705346 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:56.705381 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:56.705434 (+    53us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:18:56.705479 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.705543 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:56.705684 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.668824 28124 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.717785 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:56.718888 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.032s	sys 0.069s
W20260809 23:18:56.745522 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 264729us
W20260809 23:18:56.745946 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.032s	sys 0.069s
I20260809 23:18:56.746122 24539 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:18:56.674521 28077 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:18:56.706396 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:56.669665 28156 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.683652 27590 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.672120 28210 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:56.600811 28165 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.726562 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:56.681778 28192 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:56.719568 28207 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.796200 24544 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:18:56.805451 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:56.806529 24539 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:56.728780 27938 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:56.773123 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66945.2us
I20260809 23:18:56.737586 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:56.772414 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:56.749797 24498 leader_election.cc:398] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
I20260809 23:18:56.654589 27666 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.789872 24540 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:56.811102 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:56.674585 28128 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.706570 28154 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:56.822566 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85083.6us
W20260809 23:18:56.674019 25365 rpcz_store.cc:269] 0809 23:18:54.895917 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.368374 (+472457us) service_pool.cc:227] Handling call
0809 23:18:56.620977 (+1252603us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.826977 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2128) took 1968 ms. Trace:
I20260809 23:18:56.680495 27628 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.817806 24498 leader_election.cc:401] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:18:56.610237 28194 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.832234 24430 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260809 23:18:56.739625 27976 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:56.837347 24427 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:18:56.826272 24311 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:18:56.840230 24540 rpcz_store.cc:276] 0809 23:18:54.858170 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.735389 (+1877219us) service_pool.cc:227] Handling call
0809 23:18:56.789469 (+ 54080us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.826700 (+ 37231us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.826892 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:56.852289 27981 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.693426 27847 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.842648 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:56.787834 28225 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.739957 28024 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.805343 28192 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:56.887939 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:56.794879 28215 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.861639 25375 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:18:56.748364 28002 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260809 23:18:56.720546 28019 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.854991 28177 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:56.884363 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 209606us
W20260809 23:18:56.843325 28196 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.465s	user 0.005s	sys 0.009s
I20260809 23:18:56.684091 28103 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.779258 28124 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.841176 27945 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:56.899459 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 82691.3us
I20260809 23:18:56.859398 28222 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:18:56.900720 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:56.903003 24541 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:18:56.958217 27969 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.961364 25369 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:18:56.961933 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 723) took 2365 ms (client timeout 1529 ms). Trace:
I20260809 23:18:56.954545 28128 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:56.962491 28128 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:56.904230 28221 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:56.963403 27628 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.963222 28221 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.964174 27628 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:56.954746 28196 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
I20260809 23:18:56.965040 28196 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.965377 28221 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:56.905198 28197 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:18:56.965432 28196 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.966298 28197 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.905735 28165 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:56.902559 28215 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.906245 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:18:56.962301 25369 rpcz_store.cc:269] 0809 23:18:54.596595 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.596711 (+   116us) service_pool.cc:227] Handling call
0809 23:18:56.900570 (+2303859us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:18:56.961844 (+ 61274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65875328}
I20260809 23:18:56.968132 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:56.968875 28215 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.835170 27938 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.743718 28181 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.906898 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:56.907292 28231 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.863875 27649 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260809 23:18:56.907505 27651 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.908283 24540 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:18:56.797673 27590 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.885054 28064 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.955265 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:56.796144 28156 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:56.886518 27638 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:56.831928 28047 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20260809 23:18:56.956386 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:18:56.815279 28207 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.892710 28203 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:57.070704 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114370us
I20260809 23:18:56.982455 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:56.910979 28177 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:56.906312 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:57.007793 27730 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:57.031668 27984 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.082305 25365 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:57.083889 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:57.085213 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 850) took 2624 ms (client timeout 3005 ms). Trace:
W20260809 23:18:57.085736 25370 rpcz_store.cc:269] 0809 23:18:54.461142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.461301 (+   159us) service_pool.cc:227] Handling call
0809 23:18:57.084904 (+2623603us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.085124 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4150151936}
I20260809 23:18:56.892748 28172 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260809 23:18:56.883697 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:18:56.969941 27941 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.958813 28024 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:56.981550 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:56.804600 28210 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.996521 28215 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:57.075945 28226 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260809 23:18:57.014305 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:57.008194 27969 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:18:56.887102 28077 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.969493 27945 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.978613 28128 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:56.893308 28225 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:57.092233 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 137047us
I20260809 23:18:57.456327 28229 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:18:57.457019 28229 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:57.457578 28229 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:18:57.082360 27984 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:18:57.321043 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1851) took 3059 ms (client timeout 3005 ms). Trace:
I20260809 23:18:57.092131 27797 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:57.463817 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 481441us
I20260809 23:18:57.117960 27857 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:57.465656 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:57.092387 24539 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
W20260809 23:18:57.316650 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2107) took 4305 ms (client timeout 1936 ms). Trace:
I20260809 23:18:57.054996 28222 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:57.463297 27715 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:57.089859 28177 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:57.092980 28165 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:18:57.469256 28222 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:57.470520 25363 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:18:57.321452 25373 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:18:57.058811 28062 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:57.466032 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2272) took 4082 ms. Trace:
I20260809 23:18:57.462320 27984 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:57.468000 24546 rpcz_store.cc:269] 0809 23:18:53.011130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.184036 (+172906us) service_pool.cc:227] Handling call
0809 23:18:57.316426 (+4132390us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.316559 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1503160192}
W20260809 23:18:57.471819 24316 rpcz_store.cc:269] 0809 23:18:54.261920 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.426767 (+164847us) service_pool.cc:227] Handling call
0809 23:18:57.320812 (+2894045us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54003c23d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.320964 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":938041088}
W20260809 23:18:57.371726 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.004s	sys 0.002s
I20260809 23:18:57.329562 24320 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
W20260809 23:18:57.398183 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 693) took 3702 ms (client timeout 3093 ms). Trace:
W20260809 23:18:57.312819 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2311) took 3693 ms (client timeout 2995 ms). Trace:
I20260809 23:18:57.395172 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1838) took 4250 ms. Trace:
I20260809 23:18:57.467700 28182 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:57.468931 27715 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:57.395973 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1828) took 4338 ms (client timeout 1720 ms). Trace:
W20260809 23:18:57.474355 24378 connection.cc:232] RPC call timeout handler was delayed by -2.5292452550493181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260809 23:18:57.478385 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 464141us
I20260809 23:18:57.478652 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:57.466527 24422 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4. Re-sending same reply.
I20260809 23:18:57.570398 25370 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:18:57.571521 24378 connection.cc:232] RPC call timeout handler was delayed by -2.4032452551182359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
W20260809 23:18:57.570470 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.004s	sys 0.008s
I20260809 23:18:57.572578 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:18:57.570644 24428 rpcz_store.cc:269] 0809 23:18:53.057065 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.164530 (+107465us) service_pool.cc:227] Handling call
0809 23:18:57.395703 (+4231173us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b54003bae54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.395864 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7952326400}
W20260809 23:18:57.573163 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 827984us
W20260809 23:18:57.524981 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48790.7us
I20260809 23:18:57.571558 24425 rpcz_store.cc:276] 0809 23:18:53.144794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.148770 (+1003976us) service_pool.cc:227] Handling call
0809 23:18:54.149098 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:54.167061 (+ 17963us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:57.394835 (+3227774us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:57.395094 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39908224}
W20260809 23:18:57.450369 28233 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.362s	user 0.000s	sys 0.001s
I20260809 23:18:57.570300 24431 rpcz_store.cc:276] 0809 23:18:53.383139 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.866459 (+1483320us) service_pool.cc:227] Handling call
0809 23:18:54.885223 (+ 18764us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:56.284971 (+1399748us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:56.285015 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:56.285561 (+   546us) log.cc:842] Serialized 26 byte log entry
0809 23:18:56.777837 (+492276us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:18:56.777891 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.777971 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:57.465821 (+687850us) raft_consensus.cc:1671] finished
0809 23:18:57.465840 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:57.465977 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1089578,"raft.queue_time_us":333901,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":76196,"thread_start_us":1641026,"threads_started":2,"wal-append.queue_time_us":611781}
W20260809 23:18:57.570855 24321 rpcz_store.cc:269] 0809 23:18:53.618973 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.131296 (+512323us) service_pool.cc:227] Handling call
0809 23:18:57.312529 (+3181233us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.312726 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5307981696}
W20260809 23:18:57.570609 24323 rpcz_store.cc:269] 0809 23:18:53.696135 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.148046 (+451911us) service_pool.cc:227] Handling call
0809 23:18:57.397944 (+3249898us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.398091 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87229696}
I20260809 23:18:57.583597 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:18:57.401582 28236 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:18:57.183162 28239 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:57.093848 28231 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:57.281210 28196 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.355582 27847 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:57.471168 28024 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:57.393838 27938 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.472612 28019 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:57.572949 24378 connection.cc:232] RPC call timeout handler was delayed by -2.1836452551651746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
W20260809 23:18:57.384846 28175 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.433s	user 0.000s	sys 0.010s
W20260809 23:18:57.220615 28113 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.006s	sys 0.009s
I20260809 23:18:57.324683 27812 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:18:58.289428 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.402s	user 0.027s	sys 0.047s
I20260809 23:18:57.402093 28163 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:18:57.619941 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:58.427513 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.540s	user 0.033s	sys 0.053s
I20260809 23:18:57.843195 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:57.618774 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:57.574062 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:57.527379 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 724) took 2925 ms. Trace:
I20260809 23:18:57.477824 27590 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.618383 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:58.479732 24323 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:57.649708 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 703) took 3029 ms (client timeout 2318 ms). Trace:
I20260809 23:18:57.616803 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:57.617151 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:57.311329 28006 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:18:57.476171 28195 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:57.530272 27941 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.425562 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:57.227090 27666 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:57.584197 27628 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.000s
W20260809 23:18:58.428855 24378 connection.cc:232] RPC call timeout handler was delayed by -2.1791452551260591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260809 23:18:57.529093 27950 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:57.537905 28215 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:58.471766 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1864) took 1014 ms. Trace:
I20260809 23:18:58.532945 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:18:58.432013 28113 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.792s	user 0.020s	sys 0.035s
W20260809 23:18:58.432696 28192 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.523s	user 0.008s	sys 0.009s
I20260809 23:18:57.529853 28128 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:58.128401 28154 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.285s	user 0.000s	sys 0.019s
W20260809 23:18:57.871376 28155 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 3.356s	user 0.007s	sys 0.045s
I20260809 23:18:58.472389 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:58.615865 25370 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:58.557262 24422 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:18:57.540861 27857 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:18:58.168673 28234 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260809 23:18:58.563931 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:58.631119 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:18:58.631299 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2317) took 1170 ms. Trace:
I20260809 23:18:58.631712 24321 rpcz_store.cc:276] 0809 23:18:57.460529 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.474144 (+1013615us) service_pool.cc:227] Handling call
0809 23:18:58.631239 (+157095us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:57.875830 28188 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.600s	user 0.008s	sys 0.003s
W20260809 23:18:58.559587 24378 connection.cc:232] RPC call timeout handler was delayed by -1.3979452550411224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28530000000000005s for a response
I20260809 23:18:58.559126 24425 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:18:58.481024 24313 rpcz_store.cc:269] 0809 23:18:54.625727 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.879927 (+254200us) service_pool.cc:227] Handling call
0809 23:18:57.649439 (+2769512us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:57.649619 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3094907136}
I20260809 23:18:58.556496 24431 rpcz_store.cc:276] 0809 23:18:57.457865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.619502 (+161637us) service_pool.cc:227] Handling call
0809 23:18:58.471686 (+852184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.473382 28181 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:18:57.858186 28108 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.251s	user 0.010s	sys 0.006s
I20260809 23:18:57.481292 28124 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:58.569883 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2320) took 1089 ms. Trace:
W20260809 23:18:58.631505 28057 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.991s	user 0.008s	sys 0.012s
I20260809 23:18:58.410946 28240 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:58.616843 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2282) took 1093 ms. Trace:
I20260809 23:18:58.616421 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 788) took 1084 ms. Trace:
I20260809 23:18:58.557837 28244 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:57.620100 28122 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:57.622231 27715 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.446005 27945 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.630785 27857 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:58.678352 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2096) took 5808 ms. Trace:
I20260809 23:18:58.678830 24545 rpcz_store.cc:276] 0809 23:18:52.860087 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.502628 (+642541us) service_pool.cc:227] Handling call
0809 23:18:53.503003 (+   375us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.143937 (+640934us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:58.668502 (+4524565us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:58.668769 (+   267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756615552}
W20260809 23:18:58.687043 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.476s	user 0.030s	sys 0.056s
W20260809 23:18:58.687580 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.477s	user 0.030s	sys 0.056s
I20260809 23:18:58.614703 28195 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.645036 28237 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
W20260809 23:18:58.565058 27797 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.033s	user 0.011s	sys 0.015s
I20260809 23:18:58.687753 27857 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:58.525039 27590 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:18:57.586650 28062 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:57.804996 28156 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:58.558305 28006 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:58.547164 28163 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:58.527220 27968 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:58.569703 27628 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
I20260809 23:18:58.445837 28194 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.561653 27984 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:57.719128 28161 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.888541 28207 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:58.665951 25370 rpcz_store.cc:276] 0809 23:18:57.531610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.471832 (+940222us) service_pool.cc:227] Handling call
0809 23:18:58.616361 (+144529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:57.536769 25365 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260809 23:18:58.641029 28103 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.672s	user 0.010s	sys 0.026s
I20260809 23:18:58.638758 28004 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:58.427570 27812 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:58.649147 28229 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.191s	user 0.000s	sys 0.015s
W20260809 23:18:58.645943 28066 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.287s	user 0.000s	sys 0.016s
I20260809 23:18:58.655876 27981 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:58.630498 28019 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260809 23:18:58.615290 28192 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:58.664806 28228 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:58.401320 28248 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:57.890060 28210 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:57.588003 28002 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 1.020s
I20260809 23:18:58.480875 25363 rpcz_store.cc:276] 0809 23:18:54.601682 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.601878 (+   196us) service_pool.cc:227] Handling call
0809 23:18:57.469448 (+2867570us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:18:57.469791 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:57.481533 (+ 11742us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:57.527275 (+ 45742us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178873216}
I20260809 23:18:58.523381 28239 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:58.707084 27797 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.180s	user 0.011s	sys 0.022s
I20260809 23:18:58.697523 28246 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:58.666833 28057 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:58.697151 28244 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.694118 28240 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:58.670326 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 5.03632e+06us
I20260809 23:18:58.639515 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:58.654269 24316 rpcz_store.cc:276] 0809 23:18:57.480861 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.582312 (+101451us) service_pool.cc:227] Handling call
0809 23:18:58.569808 (+987496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.657830 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:58.665899 24422 rpcz_store.cc:276] 0809 23:18:57.523570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.570498 (+ 46928us) service_pool.cc:227] Handling call
0809 23:18:58.616783 (+1046285us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:58.640717 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.064s	user 0.011s	sys 0.024s
I20260809 23:18:57.633503 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:58.638783 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2701452551409602s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29950000000000004s for a response
I20260809 23:18:58.665174 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:58.644773 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2279) took 1170 ms. Trace:
I20260809 23:18:58.668660 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2271) took 5286 ms. Trace:
W20260809 23:18:58.869560 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.002s	sys 0.013s
W20260809 23:18:58.870072 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.003s	sys 0.014s
I20260809 23:18:58.848338 24313 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:18:58.850029 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:18:58.872957 24316 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260809 23:18:58.700695 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2127) took 4531 ms. Trace:
I20260809 23:18:58.873494 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1865) took 2158 ms. Trace:
I20260809 23:18:58.873987 24316 rpcz_store.cc:276] 0809 23:18:56.714821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.849177 (+2134356us) service_pool.cc:227] Handling call
0809 23:18:58.873328 (+ 24151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.851282 24321 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:18:58.875833 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2315) took 1416 ms. Trace:
I20260809 23:18:58.876220 24321 rpcz_store.cc:276] 0809 23:18:57.459055 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.632515 (+1173460us) service_pool.cc:227] Handling call
0809 23:18:58.875724 (+243209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.877517 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:58.878295 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 709) took 1899 ms. Trace:
I20260809 23:18:58.878569 24321 rpcz_store.cc:276] 0809 23:18:56.979309 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.877100 (+1897791us) service_pool.cc:227] Handling call
0809 23:18:58.878237 (+  1137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.880221 25371 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:18:58.873739 24543 rpcz_store.cc:276] 0809 23:18:54.169043 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.244906 (+1075863us) service_pool.cc:227] Handling call
0809 23:18:56.259318 (+1014412us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:56.691873 (+432555us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:56.691919 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:56.715587 (+ 23668us) log.cc:842] Serialized 26 byte log entry
0809 23:18:56.854032 (+138445us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:56.854073 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:56.854502 (+   429us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:58.700432 (+1845930us) raft_consensus.cc:1671] finished
0809 23:18:58.700459 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:58.700624 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63384,"raft.queue_time_us":313332,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":12566,"thread_start_us":505876,"threads_started":2,"wal-append.queue_time_us":413390}
I20260809 23:18:58.880434 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:58.758383 28192 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:58.729516 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:18:58.863269 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.007s	sys 0.027s
W20260809 23:18:58.912675 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.007s	sys 0.028s
I20260809 23:18:58.802917 28197 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:58.913246 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 858) took 1105 ms. Trace:
I20260809 23:18:58.715150 27590 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:58.863884 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2701452551409602s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29950000000000004s for a response
I20260809 23:18:58.884044 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2314) took 1426 ms. Trace:
W20260809 23:18:58.866135 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.008s	sys 0.015s
I20260809 23:18:58.884137 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:58.865507 24431 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:58.867885 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.001s	sys 0.025s
I20260809 23:18:58.867826 24425 rpcz_store.cc:276] 0809 23:18:57.474380 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.580401 (+106021us) service_pool.cc:227] Handling call
0809 23:18:58.644700 (+1064299us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.885470 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:58.887101 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:58.862284 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:58.864393 24428 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:58.867878 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:58.740263 27981 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:58.741132 28006 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:58.905260 24540 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:18:58.780869 28244 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.727028 28215 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:58.906421 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 686) took 5227 ms. Trace:
W20260809 23:18:58.730244 28177 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.261s	user 0.004s	sys 0.013s
I20260809 23:18:58.868829 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1855) took 2223 ms. Trace:
I20260809 23:18:58.737931 28004 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.784144 28195 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.728482 28062 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.753439 28248 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:58.738761 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.010s	sys 0.018s
I20260809 23:18:58.911787 25363 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:18:58.707679 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.008s	sys 0.030s
W20260809 23:18:58.886375 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.287s	user 0.011s	sys 0.024s
I20260809 23:18:58.914088 24313 rpcz_store.cc:276] 0809 23:18:57.457343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.639232 (+1181889us) service_pool.cc:227] Handling call
0809 23:18:58.883965 (+244733us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.908142 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:18:58.901728 24323 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:18:58.716696 28163 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:18:58.729169 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 738) took 1956 ms. Trace:
I20260809 23:18:58.913754 25370 rpcz_store.cc:276] 0809 23:18:57.807252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.725158 (+917906us) service_pool.cc:227] Handling call
0809 23:18:58.912960 (+187802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:58.869223 24429 rpcz_store.cc:276] 0809 23:18:53.382301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.728868 (+1346567us) service_pool.cc:227] Handling call
0809 23:18:56.393391 (+1664523us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.470509 (+ 77118us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:58.668432 (+2197923us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540017fcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:58.668591 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3505448832}
W20260809 23:18:58.944368 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.009s	sys 0.016s
I20260809 23:18:59.023679 24539 raft_consensus.cc:2396] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260809 23:18:58.970530 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.001s	sys 0.025s
I20260809 23:18:59.036475 28197 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:59.025166 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:58.740310 28229 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.024036 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.033088 28057 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:18:59.103528 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2305) took 5609 ms. Trace:
W20260809 23:18:59.020660 28245 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.034s	sys 0.030s
W20260809 23:18:59.104470 28245 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.916s	user 0.035s	sys 0.036s
I20260809 23:18:59.025552 28269 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:59.011971 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1862) took 1545 ms (client timeout 1720 ms). Trace:
W20260809 23:18:59.106026 24428 rpcz_store.cc:269] 0809 23:18:57.457167 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.573884 (+116717us) service_pool.cc:227] Handling call
0809 23:18:58.999157 (+1425273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.105818 28269 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.107497 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:59.108299 24428 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:18:59.036021 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 168681us
W20260809 23:18:59.111197 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.009s	sys 0.002s
W20260809 23:18:59.111626 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.009s	sys 0.002s
I20260809 23:18:59.111428 25365 rpcz_store.cc:276] 0809 23:18:56.772902 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.224322 (+451420us) service_pool.cc:227] Handling call
0809 23:18:57.363462 (+139140us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:58.728884 (+1365422us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:58.729109 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.112190 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.53806e+06us
I20260809 23:18:59.113456 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:59.113969 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:59.114370 25365 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:18:59.040146 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:18:59.115145 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 861351us
I20260809 23:18:59.062664 24427 rpcz_store.cc:276] 0809 23:18:56.645191 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.855772 (+210581us) service_pool.cc:227] Handling call
0809 23:18:56.856099 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.856294 (+   195us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:18:57.210396 (+354102us) log.cc:842] Serialized 14 byte log entry
0809 23:18:58.702451 (+1492055us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:58.868089 (+165638us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:18:58.868247 (+   158us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:58.868388 (+   141us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:58.868642 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6500,"prepare.queue_time_us":1627102,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":33115,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":2042,"thread_start_us":1800480,"threads_started":2,"wal-append.queue_time_us":1551884}
W20260809 23:18:59.116707 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76814us
I20260809 23:18:59.117189 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.117795 24378 leader_election.cc:398] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5.
I20260809 23:18:59.118115 24427 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:59.014731 24422 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 4
I20260809 23:18:59.119313 28237 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:59.119695 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1859) took 2473 ms. Trace:
I20260809 23:18:59.121071 28237 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:59.122478 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.013s	sys 0.021s
I20260809 23:18:58.726440 28225 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.105568 28229 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.763108 27812 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.913719 28214 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:59.123850 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:59.120433 24423 rpcz_store.cc:276] 0809 23:18:56.646301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.890891 (+244590us) service_pool.cc:227] Handling call
0809 23:18:56.891253 (+   362us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:57.126733 (+235480us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:57.126777 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:57.139302 (+ 12525us) log.cc:842] Serialized 26 byte log entry
0809 23:18:59.118604 (+1979302us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:59.118816 (+   212us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.119129 (+   313us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:18:59.119249 (+   120us) raft_consensus.cc:1671] finished
0809 23:18:59.119265 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:59.119553 (+   288us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":573017,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":143,"thread_start_us":1740015,"threads_started":2,"wal-append.queue_time_us":1244091}
I20260809 23:18:59.191803 28215 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:18:59.197232 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.162539 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.120321 24316 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:18:58.729252 28103 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.760s	user 0.011s	sys 0.026s
I20260809 23:18:59.062626 24321 rpcz_store.cc:276] 0809 23:18:53.679102 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.904271 (+5225169us) service_pool.cc:227] Handling call
0809 23:18:58.905043 (+   772us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:58.905680 (+   637us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:58.905728 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:58.905836 (+   108us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:18:58.905886 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:58.905991 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:58.906252 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.023037 27797 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:18:58.738379 28066 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.407s	user 0.000s	sys 0.016s
I20260809 23:18:59.208040 24429 raft_consensus.cc:2354] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4.
W20260809 23:18:59.118829 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2291) took 1592 ms (client timeout 1956 ms). Trace:
W20260809 23:18:59.210810 24427 rpcz_store.cc:269] 0809 23:18:57.526790 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.116742 (+1589952us) service_pool.cc:227] Handling call
0809 23:18:59.118751 (+  2009us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.212680 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.010s	sys 0.035s
I20260809 23:18:59.213068 28222 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:18:59.213878 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.215056 28274 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.215857 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 735) took 4287 ms. Trace:
W20260809 23:18:59.216082 28271 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260809 23:18:59.216392 25375 rpcz_store.cc:276] 0809 23:18:54.927856 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.733281 (+1805425us) service_pool.cc:227] Handling call
0809 23:18:56.790909 (+ 57628us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.894813 (+103904us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.215186 (+2320373us) spinlock_profiling.cc:243] Waited 70.3 ms on lock 0x7b5400631354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:59.215603 (+   417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161660288}
I20260809 23:18:59.049104 27857 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.219120 28197 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.219416 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:18:59.219969 28197 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.221171 27857 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:59.221969 27857 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:18:59.222407 28197 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.194350 27812 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:18:59.225839 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0673954079393297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260809 23:18:59.226308 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0142860321793705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2036s for a response
I20260809 23:18:59.215107 28257 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.228005 28257 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.228694 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:59.229869 24539 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:59.230911 27834 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:59.216105 28274 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.238382 27949 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:59.104131 24309 rpcz_store.cc:276] 0809 23:18:53.494166 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.610679 (+1116513us) service_pool.cc:227] Handling call
0809 23:18:54.671882 (+ 61203us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:55.241182 (+569300us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.102976 (+3861794us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:59.103316 (+   340us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":804465408}
I20260809 23:18:59.239470 24425 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:18:59.239106 27628 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:18:59.016021 28141 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 4.857s	user 0.006s	sys 0.043s
I20260809 23:18:59.240041 27952 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.038285 27590 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.239521 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:18:59.240870 28268 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:59.243134 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 484) took 4952 ms. Trace:
I20260809 23:18:59.244123 28064 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:59.245579 24545 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:18:59.227028 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109514us
W20260809 23:18:59.208398 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2318) took 1738 ms (client timeout 1921 ms). Trace:
I20260809 23:18:59.103930 24544 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:18:59.019057 28006 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:18:59.250247 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2319) took 1737 ms. Trace:
W20260809 23:18:58.756373 28182 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.134s	user 0.014s	sys 0.000s
I20260809 23:18:59.110814 28004 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:59.187798 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 853) took 1586 ms (client timeout 2087 ms). Trace:
I20260809 23:18:59.270763 24431 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:18:59.257261 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2283) took 1722 ms (client timeout 2034 ms). Trace:
I20260809 23:18:59.265064 24422 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
W20260809 23:18:59.083976 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 787) took 1619 ms (client timeout 2135 ms). Trace:
I20260809 23:18:59.174746 28057 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:18:59.269865 24316 rpcz_store.cc:276] 0809 23:18:57.473437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.907561 (+1434124us) service_pool.cc:227] Handling call
0809 23:18:59.210753 (+303192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.268755 24323 rpcz_store.cc:269] 0809 23:18:57.462778 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:57.588730 (+125952us) service_pool.cc:227] Handling call
0809 23:18:59.188168 (+1599438us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.266860 24430 rpcz_store.cc:276] 0809 23:18:54.290230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:55.024994 (+734764us) service_pool.cc:227] Handling call
0809 23:18:55.025327 (+   333us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:56.858767 (+1833440us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.218658 (+2359891us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:18:59.242916 (+ 24258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4135388544}
I20260809 23:18:59.273185 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:18:59.272759 28214 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.227131 28272 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:18:59.290907 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:59.275800 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:18:59.262652 28222 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:18:59.238406 28274 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.760390 28240 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:59.282002 24429 rpcz_store.cc:269] 0809 23:18:57.523931 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.123403 (+1599472us) service_pool.cc:227] Handling call
0809 23:18:59.246225 (+122822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.290345 24323 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
I20260809 23:18:58.719275 27968 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:58.755537 28239 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:58.717026 28231 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.234368 27946 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:59.286433 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1861) took 1825 ms (client timeout 1820 ms). Trace:
I20260809 23:18:59.286195 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.297690 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:59.292008 24428 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 23:18:58.742969 28024 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:18:59.297233 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2290) took 1765 ms (client timeout 1737 ms). Trace:
W20260809 23:18:59.303709 24422 rpcz_store.cc:269] 0809 23:18:57.456824 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.860763 (+1403939us) service_pool.cc:227] Handling call
0809 23:18:59.282294 (+421531us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.306897 24431 rpcz_store.cc:269] 0809 23:18:57.526402 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.639483 (+1113081us) service_pool.cc:227] Handling call
0809 23:18:59.287613 (+648130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.305475 28224 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:18:59.305827 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 689) took 5620 ms. Trace:
I20260809 23:18:59.310809 24323 rpcz_store.cc:276] 0809 23:18:53.679988 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.285048 (+5605060us) service_pool.cc:227] Handling call
0809 23:18:59.290025 (+  4977us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:59.296930 (+  6905us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.300152 (+  3222us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.312999 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2278) took 1834 ms (client timeout 1921 ms). Trace:
I20260809 23:18:59.316116 28269 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.316956 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:18:58.762017 28246 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:18:59.281461 25365 rpcz_store.cc:269] 0809 23:18:57.574804 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.112871 (+1538067us) service_pool.cc:227] Handling call
0809 23:18:59.160796 (+ 47925us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.320127 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.320993 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:18:59.321703 25365 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:18:59.323437 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.323272 28177 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:59.324211 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2287) took 1799 ms (client timeout 2119 ms). Trace:
W20260809 23:18:59.324633 24427 rpcz_store.cc:269] 0809 23:18:57.525251 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.271756 (+1746505us) service_pool.cc:227] Handling call
0809 23:18:59.324130 (+ 52374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.324635 25370 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:18:59.318593 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:59.321014 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2288) took 1795 ms (client timeout 2280 ms). Trace:
W20260809 23:18:59.327782 24422 rpcz_store.cc:269] 0809 23:18:57.525620 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.319759 (+1794139us) service_pool.cc:227] Handling call
0809 23:18:59.320928 (+  1169us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.313403 24428 rpcz_store.cc:269] 0809 23:18:57.474109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.107012 (+1632903us) service_pool.cc:227] Handling call
0809 23:18:59.308153 (+201141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.336426 28272 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.338580 28272 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:59.339178 28274 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:18:59.323302 28057 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.340584 24321 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260809 23:18:59.318812 24545 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:18:58.748441 28228 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.310173 27952 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.310433 28268 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.308598 27797 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:18:59.283536 25363 rpcz_store.cc:269] 0809 23:18:57.462126 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:58.879590 (+1417464us) service_pool.cc:227] Handling call
0809 23:18:59.081795 (+202205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.308038 28229 raft_consensus.cc:2741] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 23:18:59.552762 24539 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:18:59.384151 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.379320 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:59.310192 28224 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:18:59.374497 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:18:59.588868 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:59.381115 24423 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:18:59.472820 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 692) took 5781 ms. Trace:
W20260809 23:18:59.371162 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95577.2us
W20260809 23:18:59.507853 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 234766us
I20260809 23:18:59.380000 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:18:59.933038 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.001s	sys 0.024s
I20260809 23:18:59.381567 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.340975 24321 pending_rounds.cc:76] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 6
W20260809 23:18:59.815330 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.006s	sys 0.020s
W20260809 23:18:59.946491 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.006s	sys 0.023s
I20260809 23:18:59.943894 24316 rpcz_store.cc:276] 0809 23:18:53.691715 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.343435 (+5651720us) service_pool.cc:227] Handling call
0809 23:18:59.472125 (+128690us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:59.472424 (+   299us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:18:59.472451 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:59.472513 (+    62us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:59.472570 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:59.472622 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:18:59.472740 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.424574 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 695) took 5721 ms (client timeout 3195 ms). Trace:
W20260809 23:18:59.940721 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2289) took 2414 ms (client timeout 2522 ms). Trace:
W20260809 23:18:59.947959 24431 rpcz_store.cc:269] 0809 23:18:57.526026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.334044 (+1808018us) service_pool.cc:227] Handling call
0809 23:18:59.940653 (+606609us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.948498 24545 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:18:59.945429 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.001s	sys 0.024s
I20260809 23:18:59.950949 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:18:59.884063 28006 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.595s	user 0.012s	sys 0.000s
W20260809 23:18:59.893441 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 385704us
W20260809 23:18:59.947376 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 734876us
I20260809 23:18:59.945770 24321 pending_rounds.cc:97] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.7
W20260809 23:18:59.954617 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2280) took 2472 ms (client timeout 2087 ms). Trace:
I20260809 23:18:59.534240 24435 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:18:59.947891 24314 rpcz_store.cc:269] 0809 23:18:53.703310 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.261196 (+557886us) service_pool.cc:227] Handling call
0809 23:18:59.423997 (+5162801us) spinlock_profiling.cc:243] Waited 4.31 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:18:59.424266 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9907169024}
I20260809 23:18:59.386106 28171 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:59.385839 27968 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.379709 28057 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:18:59.949290 27834 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.718s	user 0.003s	sys 0.017s
I20260809 23:18:59.955466 24321 raft_consensus.cc:2866] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:18:59.955554 24423 rpcz_store.cc:269] 0809 23:18:57.482267 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.284067 (+1801800us) service_pool.cc:227] Handling call
0809 23:18:59.954490 (+670423us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:18:59.954200 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.009s	sys 0.022s
I20260809 23:18:59.466849 28280 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
W20260809 23:18:59.968950 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.009s	sys 0.022s
I20260809 23:18:59.769487 28161 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.380541 28246 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:59.381829 28225 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:18:59.714088 28177 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:18:59.336480 28182 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:18:59.979553 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:59.960840 28237 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.007s	sys 0.031s
I20260809 23:18:59.981832 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:18:59.984952 28237 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.007s	sys 0.032s
I20260809 23:18:59.949941 28136 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:18:59.985623 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:18:59.913826 28163 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.985800 28136 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:18:59.987061 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 607166us
I20260809 23:18:59.988058 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:18:59.987785 28136 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.988921 24428 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260809 23:18:59.989398 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2285) took 2464 ms (client timeout 3062 ms). Trace:
I20260809 23:18:59.986291 24540 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
W20260809 23:18:59.989841 24428 rpcz_store.cc:269] 0809 23:18:57.524551 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.344667 (+1820116us) service_pool.cc:227] Handling call
0809 23:18:59.989334 (+644667us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:18:59.989432 28136 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:18:59.990617 24309 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 23:18:59.958865 28171 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:59.992247 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.981667 27834 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:18:59.995909 28198 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:18:59.996415 27952 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:18:59.997125 28272 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:18:59.985543 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:18:59.991093 24309 pending_rounds.cc:76] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:18:59.998435 24309 pending_rounds.cc:97] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:18:59.998804 24309 raft_consensus.cc:2866] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:18:59.996536 28198 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:18:59.383389 28062 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.594s
I20260809 23:18:59.999373 24427 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:00.006174 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.004185 24429 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:18:59.973537 28280 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:00.008669 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 688) took 6322 ms. Trace:
I20260809 23:19:00.009562 24539 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:18:59.999691 27812 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.776s	user 0.006s	sys 0.007s
I20260809 23:19:00.005506 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.012392 24321 rpcz_store.cc:276] 0809 23:18:53.679691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.270089 (+5590398us) service_pool.cc:227] Handling call
0809 23:18:59.340078 (+ 69989us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.001426 (+661348us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.001612 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.013891 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.382601 28222 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:18:59.997622 28272 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:18:59.994675 28171 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:00.009577 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:00.016848 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:18:59.989801 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:00.021387 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 865) took 1348 ms (client timeout 1592 ms). Trace:
W20260809 23:19:00.022100 25363 rpcz_store.cc:269] 0809 23:18:58.673110 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.989382 (+1316272us) service_pool.cc:227] Handling call
0809 23:19:00.021245 (+ 31863us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.012135 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.006s	sys 0.008s
W20260809 23:19:00.023306 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.007s	sys 0.008s
I20260809 23:18:59.477025 28278 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.533s
W20260809 23:19:00.023759 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 745337us
I20260809 23:19:00.027402 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:00.029978 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46128.9us
W20260809 23:19:00.030340 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44943.6us
I20260809 23:19:00.023234 28272 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:00.036556 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2102) took 6562 ms. Trace:
I20260809 23:19:00.037074 24541 rpcz_store.cc:276] 0809 23:18:53.460714 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.728634 (+1267920us) service_pool.cc:227] Handling call
0809 23:18:54.740792 (+ 12158us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.961272 (+2220480us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.022333 (+3061061us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.022961 (+   628us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4972109696}
I20260809 23:19:00.017562 28161 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.040853 24314 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:00.041941 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1852) took 5779 ms. Trace:
I20260809 23:19:00.042397 24314 rpcz_store.cc:276] 0809 23:18:54.262207 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.019162 (+5756955us) service_pool.cc:227] Handling call
0809 23:19:00.040449 (+ 21287us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:00.041405 (+   956us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.041863 (+   458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.042869 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:00.044843 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:00.018666 28214 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:00.045619 28161 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:00.046279 24423 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:19:00.032606 28246 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:00.045312 27949 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:00.041998 28280 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.047847 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:00.050976 28283 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:00.052686 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:00.053072 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.054311 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.083037 24540 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:00.084091 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.084306 24546 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:00.084877 24540 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:00.055637 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.056170 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 685) took 6379 ms. Trace:
I20260809 23:19:00.085752 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.060669 24314 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:19:00.045218 28182 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260809 23:19:00.087532 25365 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.003s
I20260809 23:19:00.062052 24422 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:00.091594 25365 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:00.091759 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2292) took 2405 ms (client timeout 2747 ms). Trace:
I20260809 23:19:00.092039 28163 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:00.092100 24422 rpcz_store.cc:269] 0809 23:18:57.686738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.334462 (+1647724us) service_pool.cc:227] Handling call
0809 23:19:00.091685 (+757223us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.092517 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 863) took 1522 ms (client timeout 1833 ms). Trace:
W20260809 23:19:00.092989 25363 rpcz_store.cc:269] 0809 23:18:58.569969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.034362 (+1464393us) service_pool.cc:227] Handling call
0809 23:19:00.092461 (+ 58099us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.093384 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:00.094137 24422 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:00.094074 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:00.068945 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.070346 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2306) took 6575 ms. Trace:
I20260809 23:19:00.070966 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:00.074177 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 518) took 6638 ms. Trace:
I20260809 23:19:00.079402 28224 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:00.094691 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 488) took 1533 ms. Trace:
I20260809 23:19:00.095250 24317 rpcz_store.cc:276] 0809 23:18:53.494445 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.958457 (+1464012us) service_pool.cc:227] Handling call
0809 23:18:57.398104 (+2439647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:57.398349 (+   245us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:57.398397 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:57.398452 (+    55us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:57.398505 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:18:57.398575 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.065046 (+2666471us) spinlock_profiling.cc:243] Waited 2.4 s on lock 0x7b54003a43d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.070164 (+  5118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5517865728}
I20260809 23:19:00.095800 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.096824 24423 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:00.069191 28246 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:00.097553 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.097960 28246 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:00.098701 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:00.077747 24269 connection.cc:232] RPC call timeout handler was delayed by -1.9357105421368033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20260809 23:19:00.099280 24269 connection.cc:232] RPC call timeout handler was delayed by -1.9204105420503765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
I20260809 23:19:00.081557 28228 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:00.081745 24311 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:00.100212 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1856) took 5830 ms. Trace:
I20260809 23:19:00.095486 24534 rpcz_store.cc:276] 0809 23:18:53.435449 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.560178 (+1124729us) service_pool.cc:227] Handling call
0809 23:18:54.602344 (+ 42166us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:55.010230 (+407886us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:55.010276 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:55.012866 (+  2590us) log.cc:842] Serialized 26 byte log entry
0809 23:18:59.470079 (+4457213us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b18003add40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:18:59.899088 (+429009us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:18:59.899132 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.073738 (+174606us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b540029d954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.073789 (+    51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:00.073840 (+    51us) raft_consensus.cc:1671] finished
0809 23:19:00.073853 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.074004 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":172379,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":149057,"spinlock_wait_cycles":653729536,"thread_start_us":564813,"threads_started":2,"wal-append.queue_time_us":4726922}
I20260809 23:19:00.100612 24311 rpcz_store.cc:276] 0809 23:18:54.269318 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.080883 (+5811565us) service_pool.cc:227] Handling call
0809 23:19:00.081318 (+   435us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.099962 (+ 18644us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.100148 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.101179 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.101882 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 849) took 5641 ms. Trace:
I20260809 23:19:00.077921 27981 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:00.088040 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.031121 28062 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:00.086107 24320 rpcz_store.cc:276] 0809 23:18:53.678675 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.680830 (+3002155us) service_pool.cc:227] Handling call
0809 23:18:57.329317 (+648487us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:57.575552 (+246235us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.055766 (+2480214us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b540007e250. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.055981 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1360921984}
I20260809 23:19:00.088537 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:00.105646 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60956.3us
I20260809 23:19:00.106554 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:00.107216 24320 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:00.107944 28237 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:00.089058 28214 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:19:00.109045 28229 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:00.109529 28229 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:00.109920 28229 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:00.117506 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1855) took 5818 ms. Trace:
I20260809 23:19:00.118003 24314 rpcz_store.cc:276] 0809 23:18:54.269007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.043641 (+5774634us) service_pool.cc:227] Handling call
0809 23:19:00.044126 (+   485us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:00.086673 (+ 42547us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.086925 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.090637 24430 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2286) took 2565 ms (client timeout 2245 ms). Trace:
W20260809 23:19:00.119581 24430 rpcz_store.cc:269] 0809 23:18:57.524882 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.090414 (+2565532us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:00.090559 (+   145us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260809 23:19:00.119910 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50115.3us
I20260809 23:19:00.096460 24422 rpcz_store.cc:276] 0809 23:18:58.561298 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.093090 (+1531792us) service_pool.cc:227] Handling call
0809 23:19:00.094618 (+  1528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.035441 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:00.099617 24269 connection.cc:232] RPC call timeout handler was delayed by -1.8232105420902371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
I20260809 23:19:00.096450 28224 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:00.097728 24317 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 23:19:00.108096 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1860) took 3848 ms. Trace:
I20260809 23:19:00.108466 28237 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:00.122762 24269 connection.cc:232] RPC call timeout handler was delayed by -1.7350105422083288s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
I20260809 23:19:00.088222 24541 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:19:00.099589 24539 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:00.095032 27797 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:00.058744 27941 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:00.102267 25368 rpcz_store.cc:276] 0809 23:18:54.460668 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.463415 (+  2747us) service_pool.cc:227] Handling call
0809 23:18:54.543382 (+ 79967us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.710217 (+2166835us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.101427 (+3391210us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.101690 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1005368704}
I20260809 23:19:00.101130 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 974) took 6000 ms. Trace:
W20260809 23:19:00.149951 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119103us
I20260809 23:19:00.154889 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 523) took 5609 ms. Trace:
I20260809 23:19:00.175341 24311 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:19:00.183743 24320 rpcz_store.cc:276] 0809 23:18:56.260054 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.106528 (+3846474us) service_pool.cc:227] Handling call
0809 23:19:00.107023 (+   495us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.107696 (+   673us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.108018 (+   322us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.179773 24269 connection.cc:232] RPC call timeout handler was delayed by -1.7305105421692133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
I20260809 23:19:00.139922 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.181880 27949 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:19:00.185527 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.185802 27949 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:00.187106 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 697) took 6453 ms. Trace:
I20260809 23:19:00.187668 24314 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260809 23:19:00.188133 24314 pending_rounds.cc:76] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 1
I20260809 23:19:00.187639 24309 rpcz_store.cc:276] 0809 23:18:53.733597 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.335397 (+5601800us) service_pool.cc:227] Handling call
0809 23:18:59.468145 (+132748us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.030478 (+562333us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.186821 (+156343us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b54000dc5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.187036 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1150165760}
I20260809 23:19:00.188508 24314 pending_rounds.cc:97] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 23:19:00.188824 24314 raft_consensus.cc:2866] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:00.189492 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1867) took 3455 ms. Trace:
I20260809 23:19:00.189795 24314 rpcz_store.cc:276] 0809 23:18:56.734325 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.186963 (+3452638us) service_pool.cc:227] Handling call
0809 23:19:00.187338 (+   375us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.189269 (+  1931us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.189441 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.151095 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130842us
I20260809 23:19:00.187165 27949 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:00.135692 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.080040 28231 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.243s
I20260809 23:19:00.135556 28268 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:00.147264 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1863) took 2689 ms (client timeout 1904 ms). Trace:
I20260809 23:19:00.182492 24424 rpcz_store.cc:276] 0809 23:18:54.100933 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.329711 (+2228778us) service_pool.cc:227] Handling call
0809 23:19:00.099887 (+3770176us) spinlock_profiling.cc:243] Waited 3.63 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.100874 (+   987us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8337509248}
I20260809 23:19:00.182098 24427 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
W20260809 23:19:00.092165 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 861) took 1607 ms (client timeout 1737 ms). Trace:
W20260809 23:19:00.435139 24425 rpcz_store.cc:269] 0809 23:18:57.457460 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.024740 (+1567280us) service_pool.cc:227] Handling call
0809 23:19:00.147080 (+1122340us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.147188 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6323584}
I20260809 23:19:00.250854 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.201442 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1857) took 5913 ms. Trace:
I20260809 23:19:00.440699 24317 rpcz_store.cc:276] 0809 23:18:54.269727 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.096894 (+5827167us) service_pool.cc:227] Handling call
0809 23:19:00.097374 (+   480us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:00.179930 (+ 82556us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.183472 (+  3542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.182844 24535 rpcz_store.cc:276] 0809 23:18:54.545371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.664105 (+2118734us) service_pool.cc:227] Handling call
0809 23:18:56.709034 (+ 44929us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:56.890036 (+181002us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:56.890079 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:18:56.890136 (+    57us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:56.890188 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.154429 (+3264241us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.154506 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.154672 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415620480}
W20260809 23:19:00.293428 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93830.8us
I20260809 23:19:00.199739 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1859) took 5918 ms. Trace:
W20260809 23:19:00.316852 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46812.2us
W20260809 23:19:00.303628 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49050.4us
W20260809 23:19:00.185011 24269 connection.cc:232] RPC call timeout handler was delayed by -1.6738105420954525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18200000000000002s for a response
W20260809 23:19:00.307559 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67944.9us
I20260809 23:19:00.225037 24422 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:00.442751 24269 connection.cc:232] RPC call timeout handler was delayed by -1.4004319270607084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260809 23:19:00.261691 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 481) took 5972 ms (client timeout 3093 ms). Trace:
W20260809 23:19:00.262830 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 857) took 2616 ms (client timeout 2036 ms). Trace:
I20260809 23:19:00.224849 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 514) took 7329 ms. Trace:
W20260809 23:19:00.443943 24269 connection.cc:232] RPC call timeout handler was delayed by -1.3840105421841145s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2142s for a response
W20260809 23:19:00.443466 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2296) took 1802 ms (client timeout 2244 ms). Trace:
W20260809 23:19:00.445832 24269 connection.cc:232] RPC call timeout handler was delayed by -1.3282105422113091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
I20260809 23:19:00.436801 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
W20260809 23:19:00.446208 24269 connection.cc:232] RPC call timeout handler was delayed by -1.3021105420775712s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22330000000000003s for a response
I20260809 23:19:00.446452 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2297) took 1226 ms. Trace:
W20260809 23:19:00.446746 24269 connection.cc:232] RPC call timeout handler was delayed by -1.2958105420693755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
I20260809 23:19:00.391500 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
W20260809 23:19:00.445930 24422 rpcz_store.cc:269] 0809 23:18:58.641478 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.122149 (+1480671us) service_pool.cc:227] Handling call
0809 23:19:00.443115 (+320966us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.335248 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.256311 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 704) took 3430 ms. Trace:
I20260809 23:19:00.447350 24430 rpcz_store.cc:276] 0809 23:18:59.220568 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.135351 (+914783us) service_pool.cc:227] Handling call
0809 23:19:00.446399 (+311048us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:00.390470 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2307) took 6883 ms (client timeout 1736 ms). Trace:
W20260809 23:19:00.436923 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 487) took 1876 ms (client timeout 1607 ms). Trace:
W20260809 23:19:00.448331 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1865) took 2990 ms (client timeout 3339 ms). Trace:
I20260809 23:19:00.109088 28214 consensus_queue.cc:1225] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 23:19:00.238519 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.354383 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.359886 25372 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
W20260809 23:19:00.444831 25370 rpcz_store.cc:269] 0809 23:18:57.646749 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.239076 (+1592327us) service_pool.cc:227] Handling call
0809 23:19:00.262588 (+1023512us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.262746 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101650688}
W20260809 23:19:00.450587 24427 rpcz_store.cc:269] 0809 23:18:58.560943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.374087 (+813144us) service_pool.cc:227] Handling call
0809 23:19:00.436850 (+1062763us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.334450 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 737) took 3561 ms. Trace:
W20260809 23:19:00.451108 24423 rpcz_store.cc:269] 0809 23:18:57.458183 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.070563 (+2612380us) service_pool.cc:227] Handling call
0809 23:19:00.448158 (+377595us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.254964 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:00.460266 25373 rpcz_store.cc:276] 0809 23:18:56.772547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.799378 (+ 26831us) service_pool.cc:227] Handling call
0809 23:18:56.834813 (+ 35435us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:57.483487 (+648674us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.333046 (+2849559us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54004a5720. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:00.333980 (+   934us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.334278 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4032122496}
W20260809 23:19:00.461159 24543 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.003s	sys 0.013s
I20260809 23:19:00.461577 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.182143 28237 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:00.444528 24435 rpcz_store.cc:269] 0809 23:18:54.289216 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.550035 (+260819us) service_pool.cc:227] Handling call
0809 23:19:00.261412 (+5711377us) spinlock_profiling.cc:243] Waited 4.81 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.261585 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11056879488}
W20260809 23:19:00.463049 28295 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260809 23:19:00.463172 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.464493 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.392030 28246 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.465166 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:00.466034 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.466235 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:00.450484 24312 rpcz_store.cc:269] 0809 23:18:53.506474 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.525329 (+ 18855us) service_pool.cc:227] Handling call
0809 23:19:00.389971 (+6864642us) spinlock_profiling.cc:243] Waited 3.04 s on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.390232 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6990285696}
I20260809 23:19:00.449383 24309 rpcz_store.cc:276] 0809 23:18:56.826210 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.188749 (+3362539us) service_pool.cc:227] Handling call
0809 23:19:00.189152 (+   403us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.255858 (+ 66706us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:00.255907 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:00.255964 (+    57us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:00.256018 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.256087 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.256247 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10341,"raft.queue_time_us":39895,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":57464}
I20260809 23:19:00.392659 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:00.460791 24534 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:00.440853 24539 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:00.461705 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:00.360945 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:00.136570 28276 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:00.442167 24311 rpcz_store.cc:276] 0809 23:18:54.273164 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.103359 (+5830195us) service_pool.cc:227] Handling call
0809 23:19:00.174843 (+ 71484us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.182971 (+  8128us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.191415 (+  8444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.327226 28178 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:00.406313 28197 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:19:00.254202 27976 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:00.475730 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.256559 28280 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:00.471083 24423 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:00.469458 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:00.149660 28062 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:00.546330 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:00.557619 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79866.4us
I20260809 23:19:00.257498 27946 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:00.191212 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 868) took 1407 ms (client timeout 1739 ms). Trace:
I20260809 23:19:00.264024 28077 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:19:00.353231 27941 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.184s
I20260809 23:19:00.536572 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:00.472402 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.335616 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:00.443634 28006 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:00.560395 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 727) took 5943 ms. Trace:
I20260809 23:19:00.557339 28231 consensus_queue.cc:1225] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:00.478066 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:00.182302 28198 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:00.610800 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45741.8us
I20260809 23:19:00.611068 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2302) took 1388 ms. Trace:
W20260809 23:19:00.611227 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.013s	sys 0.031s
I20260809 23:19:00.611425 24425 rpcz_store.cc:276] 0809 23:18:59.222415 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.451617 (+1229202us) service_pool.cc:227] Handling call
0809 23:19:00.611007 (+159390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.564158 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:00.492084 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 701) took 6644 ms. Trace:
I20260809 23:19:00.561215 24541 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:00.566815 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.565781 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:00.557475 24422 raft_consensus.cc:2362] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:00.402559 28257 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:00.174012 28272 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:00.558314 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.012s	sys 0.027s
I20260809 23:19:00.573891 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.579046 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.445339 24542 rpcz_store.cc:276] 0809 23:18:52.895645 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.731624 (+835979us) service_pool.cc:227] Handling call
0809 23:18:53.732731 (+  1107us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:54.335737 (+603006us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:54.335774 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:54.336173 (+   399us) log.cc:842] Serialized 26 byte log entry
0809 23:18:54.853793 (+517620us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:18:54.853829 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.156474 (+5302645us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.156734 (+   260us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:00.178716 (+ 21982us) raft_consensus.cc:1671] finished
0809 23:19:00.178737 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.224666 (+ 45929us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13748,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":73,"spinlock_wait_cycles":912469376,"thread_start_us":643036,"threads_started":2,"wal-append.queue_time_us":642821,"wal-append.run_cpu_time_us":44815,"wal-append.run_wall_time_us":2812374}
I20260809 23:19:00.255714 28229 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.392601 28171 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.583623 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.583881 28161 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:00.584225 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.587219 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.477499 28197 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:00.587379 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:00.469854 28159 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:00.591226 24430 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 23:19:00.592381 24535 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:00.593616 28246 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:00.595192 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.595460 27857 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:00.474756 28178 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:00.599239 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:00.603238 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:00.477391 27976 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:00.577566 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:00.595194 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 121585us
W20260809 23:19:00.439857 25365 rpcz_store.cc:269] 0809 23:18:58.484455 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.320593 (+836138us) service_pool.cc:227] Handling call
0809 23:19:00.092085 (+771492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.479110 25373 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:00.606307 24534 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:19:00.597890 24269 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.529s (SENT)
W20260809 23:19:00.563560 24321 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.002s	sys 0.023s
I20260809 23:19:00.604532 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:00.605456 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:00.611779 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.013s	sys 0.031s
I20260809 23:19:00.484572 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 687) took 6793 ms. Trace:
W20260809 23:19:00.723960 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 363382us
I20260809 23:19:00.604406 25372 rpcz_store.cc:276] 0809 23:18:54.617280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.633378 (+2016098us) service_pool.cc:227] Handling call
0809 23:18:56.633914 (+   536us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.359512 (+3725598us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b540063f220. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:19:00.470741 (+111229us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.560344 (+ 89603us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38400384}
I20260809 23:19:00.883740 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 728) took 6224 ms. Trace:
W20260809 23:19:00.975431 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 855) took 3329 ms (client timeout 3062 ms). Trace:
I20260809 23:19:00.975592 28159 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:00.848330 24314 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260809 23:19:00.976295 28159 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:00.976883 27945 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:19:00.977687 27945 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:00.784807 28222 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.009s	sys 0.014s
I20260809 23:19:00.978760 27945 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:00.980098 28222 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.009s	sys 0.019s
W20260809 23:19:00.816774 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 461684us
W20260809 23:19:00.785188 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.003s	sys 0.018s
I20260809 23:19:00.981098 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:00.982417 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.983475 24320 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:00.983930 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.617255 28276 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:00.984712 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.984870 24311 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:00.888839 28278 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:00.985510 27857 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:00.649502 25368 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
W20260809 23:19:00.986009 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 871) took 1720 ms (client timeout 1523 ms). Trace:
I20260809 23:19:00.986192 28276 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:00.986424 24321 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.002s	sys 0.037s
I20260809 23:19:00.985921 27976 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:00.892380 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2276) took 3432 ms (client timeout 2204 ms). Trace:
I20260809 23:19:00.721516 27689 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.267s
I20260809 23:19:00.895969 28248 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:00.988734 24546 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:00.988339 28177 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:00.988370 28178 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:00.604128 28006 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:00.989215 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 402011us
I20260809 23:19:00.700054 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.617530 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:00.976719 24314 pending_rounds.cc:76] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:00.700546 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:00.613025 24539 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
W20260809 23:19:00.990224 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.016s	sys 0.031s
I20260809 23:19:00.984354 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1867) took 3525 ms. Trace:
W20260809 23:19:00.995648 28237 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.002s
W20260809 23:19:00.981106 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.003s	sys 0.023s
I20260809 23:19:00.987447 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:00.975904 25372 rpcz_store.cc:269] 0809 23:18:57.645787 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.975202 (+3329415us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:00.975344 (+   142us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:19:00.982668 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:00.989385 28171 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:00.888139 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 856) took 3241 ms (client timeout 2245 ms). Trace:
I20260809 23:19:01.226460 24435 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:01.076156 24431 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:01.234834 25375 rpcz_store.cc:269] 0809 23:18:57.646358 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.887855 (+3241497us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:00.888042 (+   187us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260809 23:19:01.215054 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 617188us
I20260809 23:19:00.960175 24164 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.239640 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1860) took 4588 ms. Trace:
I20260809 23:19:01.106995 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2144) took 1886 ms. Trace:
I20260809 23:19:01.215340 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:01.228119 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:00.998561 24317 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:00.816206 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 358418us
I20260809 23:19:00.612537 24316 rpcz_store.cc:276] 0809 23:18:53.827236 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.953692 (+6126456us) service_pool.cc:227] Handling call
0809 23:19:00.012340 (+ 58648us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.087211 (+ 74871us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:00.135081 (+ 47870us) log.cc:842] Serialized 14 byte log entry
0809 23:19:00.459331 (+324250us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:00.459395 (+    64us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:00.471457 (+ 12062us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.471529 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.471634 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23144,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":30160,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":83701,"thread_start_us":298978,"threads_started":1,"wal-append.queue_time_us":221477}
I20260809 23:19:00.882256 24170 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 369) took 10977 ms. Trace:
I20260809 23:19:01.236819 24425 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:01.241228 24435 rpcz_store.cc:276] 0809 23:18:56.646582 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.140400 (+4493818us) service_pool.cc:227] Handling call
0809 23:19:01.226002 (+ 85602us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.235089 (+  9087us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.235291 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.192399 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:01.119657 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 139508us
W20260809 23:19:01.054293 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 455437us
W20260809 23:19:00.992388 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 431586us
I20260809 23:19:01.240969 24423 rpcz_store.cc:276] 0809 23:18:57.458965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.564249 (+3105284us) service_pool.cc:227] Handling call
0809 23:19:00.984295 (+420046us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.976272 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:01.218165 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:01.248382 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 201102us
I20260809 23:19:00.999327 24312 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:19:00.976045 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.240967 24309 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:19:01.014621 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 864) took 2369 ms (client timeout 1956 ms). Trace:
I20260809 23:19:00.985095 24535 raft_consensus.cc:2396] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:19:01.249701 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2327) took 1014 ms. Trace:
W20260809 23:19:01.249992 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 491) took 2623 ms (client timeout 2133 ms). Trace:
I20260809 23:19:01.242447 24314 pending_rounds.cc:97] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260809 23:19:01.250437 24431 rpcz_store.cc:269] 0809 23:18:58.627067 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.999270 (+1372203us) service_pool.cc:227] Handling call
0809 23:19:01.249796 (+1250526us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:01.249923 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2420771584}
I20260809 23:19:01.090693 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:01.251226 24314 raft_consensus.cc:2866] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:01.243408 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2143) took 2050 ms. Trace:
W20260809 23:19:01.244107 24429 rpcz_store.cc:269] 0809 23:18:57.473464 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.313146 (+1839682us) service_pool.cc:227] Handling call
0809 23:19:00.892157 (+1579011us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b540018d154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:00.892289 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34049792}
I20260809 23:19:01.251977 24427 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:01.253873 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2317) took 1027 ms. Trace:
I20260809 23:19:01.254271 24427 rpcz_store.cc:276] 0809 23:19:00.226183 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.997818 (+771635us) service_pool.cc:227] Handling call
0809 23:19:01.253811 (+255993us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:00.975214 25371 rpcz_store.cc:276] 0809 23:18:54.659590 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.646979 (+1987389us) service_pool.cc:227] Handling call
0809 23:18:56.662219 (+ 15240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:58.910795 (+2248576us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.883459 (+1972664us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.883677 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5099309952}
W20260809 23:19:01.249702 25373 rpcz_store.cc:269] 0809 23:18:58.645573 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.461346 (+1815773us) service_pool.cc:227] Handling call
0809 23:19:01.014525 (+553179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.256752 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:01.250334 24309 rpcz_store.cc:276] 0809 23:19:00.235157 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.987179 (+752022us) service_pool.cc:227] Handling call
0809 23:19:01.249632 (+262453us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.248544 24422 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:01.236550 28272 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:01.191529 28065 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260809 23:19:01.259480 24164 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.260778 28159 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:00.877281 28066 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:01.259512 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 538) took 1201 ms. Trace:
I20260809 23:19:01.262056 24164 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 5 to 7, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 7 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:19:00.877436 28224 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.695s	user 0.008s	sys 0.000s
I20260809 23:19:01.265147 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 490) took 2703 ms. Trace:
W20260809 23:19:01.280134 24269 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.546s (SENT)
I20260809 23:19:01.127878 28222 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:19:01.241752 24543 rpcz_store.cc:276] 0809 23:18:59.201570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.288335 (+ 86765us) service_pool.cc:227] Handling call
0809 23:18:59.358677 (+ 70342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:59.470718 (+112041us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:00.461649 (+990931us) log.cc:842] Serialized 14 byte log entry
0809 23:19:01.050578 (+588929us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:01.050843 (+   265us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:01.087267 (+ 36424us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.087391 (+   124us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.087581 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115431,"prepare.queue_time_us":517310,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":279474,"spinlock_wait_cycles":59823488,"thread_start_us":901456,"threads_started":2}
I20260809 23:19:01.076157 25370 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:19:01.252473 24544 rpcz_store.cc:276] 0809 23:18:59.192603 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.286094 (+ 93491us) service_pool.cc:227] Handling call
0809 23:18:59.299785 (+ 13691us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:18:59.937709 (+637924us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:18:59.937749 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:18:59.973028 (+ 35279us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.236715 (+1263687us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:01.236753 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.236818 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:01.236876 (+    58us) raft_consensus.cc:1671] finished
0809 23:19:01.236887 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.243342 (+  6455us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36755,"raft.queue_time_us":630964,"raft.run_cpu_time_us":420,"raft.run_wall_time_us":45303,"thread_start_us":1249264,"threads_started":1,"wal-append.queue_time_us":722089}
W20260809 23:19:01.281515 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 859) took 2854 ms (client timeout 2119 ms). Trace:
I20260809 23:19:01.267722 24542 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
W20260809 23:19:01.281876 25370 rpcz_store.cc:269] 0809 23:18:58.427223 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.502213 (+2074990us) service_pool.cc:227] Handling call
0809 23:19:01.281451 (+779238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.262719 28066 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:01.287569 28066 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.289572 28066 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:01.192126 28197 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.291571 24431 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:01.291819 28197 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:19:01.292244 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2281) took 3769 ms. Trace:
I20260809 23:19:01.292609 24431 rpcz_store.cc:276] 0809 23:18:57.523117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.271718 (+3748601us) service_pool.cc:227] Handling call
0809 23:19:01.291297 (+ 19579us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.291992 (+   695us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.292165 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.293978 24431 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:19:01.294575 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 489) took 2733 ms. Trace:
I20260809 23:19:01.263203 24546 rpcz_store.cc:276] 0809 23:19:00.057759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.085446 (+ 27687us) service_pool.cc:227] Handling call
0809 23:19:01.259276 (+1173830us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:01.259429 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2074197888}
W20260809 23:19:01.266034 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 862) took 2701 ms (client timeout 2280 ms). Trace:
I20260809 23:19:01.255371 28237 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:01.299782 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 317199us
I20260809 23:19:01.294874 24431 rpcz_store.cc:276] 0809 23:18:58.561615 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.293368 (+2731753us) service_pool.cc:227] Handling call
0809 23:19:01.293764 (+   396us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.294321 (+   557us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.294507 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.307155 25375 rpcz_store.cc:269] 0809 23:18:58.565033 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.248742 (+2683709us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:01.265956 (+ 17214us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260809 23:19:01.266476 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 860) took 2807 ms (client timeout 2522 ms). Trace:
W20260809 23:19:01.308204 25372 rpcz_store.cc:269] 0809 23:18:58.459043 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.249346 (+2790303us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:01.266401 (+ 17055us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:19:01.255676 28177 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:00.990057 28161 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:00.575892 25363 rpcz_store.cc:269] 0809 23:18:58.727333 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.093719 (+1366386us) service_pool.cc:227] Handling call
0809 23:19:00.134725 (+ 41006us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.272886 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 105142us
I20260809 23:19:01.318750 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:01.274744 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 870) took 2105 ms (client timeout 1884 ms). Trace:
W20260809 23:19:01.322613 25373 rpcz_store.cc:269] 0809 23:18:59.163713 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.257575 (+2093862us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:01.268859 (+ 11284us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:19:01.273798 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.274411 24541 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:19:01.323589 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2129) took 1022 ms. Trace:
I20260809 23:19:01.324045 24541 rpcz_store.cc:276] 0809 23:19:00.301270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.617243 (+315973us) service_pool.cc:227] Handling call
0809 23:19:01.323519 (+706276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.276998 24545 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:01.275594 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 867) took 2551 ms (client timeout 1812 ms). Trace:
I20260809 23:19:01.277479 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:01.283278 28178 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:01.242586 28278 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:01.279685 24430 rpcz_store.cc:276] 0809 23:18:58.561907 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.076650 (+2514743us) service_pool.cc:227] Handling call
0809 23:19:01.265077 (+188427us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.284934 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 706) took 4458 ms. Trace:
I20260809 23:19:01.286166 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:01.335549 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 875) took 1284 ms (client timeout 1696 ms). Trace:
I20260809 23:19:01.335161 24314 rpcz_store.cc:276] 0809 23:18:56.826919 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.190468 (+3363549us) service_pool.cc:227] Handling call
0809 23:19:00.295593 (+105125us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.266380 (+970787us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.284635 (+ 18255us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:01.284860 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1270185728}
W20260809 23:19:01.243795 25365 rpcz_store.cc:269] 0809 23:18:59.265999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.985811 (+1719812us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:00.985944 (+   133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:19:01.311866 24428 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:01.295661 28083 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:01.334033 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.321012 28177 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:01.274782 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.317902 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:00.982996 24313 rpcz_store.cc:276] 0809 23:18:53.679401 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.163780 (+5484379us) service_pool.cc:227] Handling call
0809 23:18:59.164344 (+   564us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:18:59.241052 (+ 76708us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:18:59.961369 (+720317us) log.cc:842] Serialized 14 byte log entry
0809 23:19:00.472980 (+511611us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:00.473020 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:00.473053 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:00.473092 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:00.473162 (+    70us) inbound_call.cc:173] Queueing success response
0809 23:19:00.872420 (+399258us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
Metrics: {"mutex_wait_us":5881,"prepare.queue_time_us":807733,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":20776,"raft.run_cpu_time_us":2724,"raft.run_wall_time_us":394317,"spinlock_wait_cycles":13272832,"thread_start_us":1154643,"threads_started":2,"wal-append.queue_time_us":469819}
I20260809 23:19:01.325562 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 877) took 1100 ms. Trace:
I20260809 23:19:01.350458 25375 rpcz_store.cc:276] 0809 23:19:00.224748 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.318492 (+1093744us) service_pool.cc:227] Handling call
0809 23:19:01.325497 (+  7005us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.351536 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.330797 24435 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:01.278474 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:01.331719 24544 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:19:01.329066 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:01.336092 25370 rpcz_store.cc:269] 0809 23:19:00.051345 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.285866 (+1234521us) service_pool.cc:227] Handling call
0809 23:19:01.335493 (+ 49627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.258450 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2313) took 1209 ms. Trace:
I20260809 23:19:01.356052 24422 rpcz_store.cc:276] 0809 23:19:00.049423 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.976027 (+926604us) service_pool.cc:227] Handling call
0809 23:19:01.258375 (+282348us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.337236 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80532.7us
I20260809 23:19:01.299474 28159 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:01.339396 24311 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:19:01.327198 28222 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:01.308895 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.362685 24425 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:01.365293 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.367687 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2275) took 3895 ms. Trace:
I20260809 23:19:01.368217 24435 rpcz_store.cc:276] 0809 23:18:57.459585 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.255804 (+3796219us) service_pool.cc:227] Handling call
0809 23:19:01.323685 (+ 67881us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.354264 (+ 30579us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.354491 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.328549 24498 leader_election.cc:334] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.523s (SENT)
W20260809 23:19:01.280458 27946 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.000s
I20260809 23:19:01.346140 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:01.321964 28276 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:19:01.346966 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2300) took 2125 ms (client timeout 1542 ms). Trace:
I20260809 23:19:01.380740 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:01.380970 24428 rpcz_store.cc:269] 0809 23:18:59.221781 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.990751 (+768970us) service_pool.cc:227] Handling call
0809 23:19:01.346776 (+1356025us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:01.346877 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2971694336}
I20260809 23:19:01.381371 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:01.347851 24378 connection.cc:232] RPC call timeout handler was delayed by -1.264157054014504s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260809 23:19:01.348204 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.323107 28161 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:01.337076 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.383667 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:01.352734 28257 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:01.346505 24164 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 3, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:01.358901 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:01.347783 28083 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:01.361184 28278 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.244433 24170 rpcz_store.cc:276] 0809 23:18:49.905184 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:49.905528 (+   344us) service_pool.cc:227] Handling call
0809 23:18:51.039539 (+1134011us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b5400002dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac297bad6c 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec 00007fac27bc60a2 00007fac21c8a94c 00007fac21c89d86
0809 23:19:00.881988 (+9842449us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:18:53.762702 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:18:53.836360 (+ 73658us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:18:53.836416 (+    56us) write_op.cc:435] Acquired schema lock
0809 23:18:53.836432 (+    16us) tablet.cc:660] Decoding operations
0809 23:18:54.173204 (+336772us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:18:54.197487 (+ 24283us) write_op.cc:641] Partition lock acquired for write op
0809 23:18:54.199841 (+  2354us) tablet.cc:683] Acquiring locks for 3 operations
0809 23:18:54.310391 (+110550us) tablet.cc:699] Row locks acquired
0809 23:18:54.324951 (+ 14560us) write_op.cc:260] PREPARE: finished
0809 23:18:54.403984 (+ 79033us) write_op.cc:270] Start()
0809 23:18:54.460108 (+ 56124us) write_op.cc:276] Timestamp: P: 1786317534346263 usec, L: 0
0809 23:18:54.464330 (+  4222us) op_driver.cc:331] REPLICATION: starting
0809 23:18:54.602749 (+138419us) log.cc:842] Serialized 1303 byte log entry
0809 23:18:58.187602 (+3584853us) op_driver.cc:444] REPLICATION: finished
0809 23:18:59.047906 (+860304us) write_op.cc:301] APPLY: starting
0809 23:18:59.048139 (+   233us) tablet.cc:1365] starting BulkCheckPresence
0809 23:18:59.048396 (+   257us) tablet.cc:1368] finished BulkCheckPresence
0809 23:18:59.048611 (+   215us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:18:59.049419 (+   808us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:18:59.084412 (+ 34993us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0809 23:18:59.112566 (+ 28154us) write_op.cc:312] APPLY: finished
0809 23:18:59.113186 (+   620us) log.cc:842] Serialized 35 byte log entry
0809 23:18:59.343987 (+230801us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:59.382835 (+ 38848us) write_op.cc:454] Released schema lock
0809 23:18:59.382903 (+    68us) write_op.cc:341] FINISH: Updating metrics
0809 23:18:59.992883 (+609980us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:18:59.993776 (+   893us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":907569,"reported_tablets":12,"spinlock_wait_cycles":123279104,"child_traces":[["op",{"apply.queue_time_us":859688,"apply.run_cpu_time_us":41033,"apply.run_wall_time_us":967528,"num_ops":3,"prepare.queue_time_us":388360,"prepare.run_cpu_time_us":39622,"prepare.run_wall_time_us":1160176,"raft.queue_time_us":1454955,"raft.run_cpu_time_us":71518,"raft.run_wall_time_us":932490,"replication_time_us":3723089,"thread_start_us":1794858,"threads_started":5,"wal-append.queue_time_us":1202468,"wal-append.run_cpu_time_us":17385,"wal-append.run_wall_time_us":3746532}]]}
I20260809 23:19:01.362141 27857 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:01.363461 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.370385 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2274) took 3906 ms. Trace:
W20260809 23:19:01.331475 25368 rpcz_store.cc:269] 0809 23:18:58.720541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.308363 (+1587822us) service_pool.cc:227] Handling call
0809 23:19:01.272284 (+963921us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.371333 24429 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 23:19:01.398977 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2284) took 3874 ms. Trace:
I20260809 23:19:01.375725 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:01.375942 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68998.5us
I20260809 23:19:01.376961 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:01.387090 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.005s	sys 0.042s
W20260809 23:19:01.401686 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.005s	sys 0.042s
W20260809 23:19:01.374012 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 108115us
I20260809 23:19:01.334370 28224 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:01.379884 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 744) took 2159 ms (client timeout 2142 ms). Trace:
W20260809 23:19:01.382711 24378 connection.cc:232] RPC call timeout handler was delayed by -1.128257054137066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260809 23:19:01.352185 25375 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:19:01.403940 25371 rpcz_store.cc:269] 0809 23:18:59.213112 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.272277 (+2059165us) service_pool.cc:227] Handling call
0809 23:19:01.372835 (+100558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.404443 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 873) took 1402 ms. Trace:
I20260809 23:19:01.404770 25373 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:01.387614 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:01.387359 24425 rpcz_store.cc:276] 0809 23:18:57.459263 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.255611 (+3796348us) service_pool.cc:227] Handling call
0809 23:19:01.334437 (+ 78826us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.365391 (+ 30954us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.365639 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.353070 25372 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:19:01.375185 28077 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.765s	user 0.006s	sys 0.005s
I20260809 23:19:01.389731 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:01.378206 27949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.011s	sys 0.032s
I20260809 23:19:01.411271 24541 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:01.393559 24435 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:01.389251 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
W20260809 23:19:01.401968 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 677379us
I20260809 23:19:01.417193 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2303) took 2136 ms. Trace:
I20260809 23:19:01.404785 25375 rpcz_store.cc:276] 0809 23:19:00.002241 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.351269 (+1349028us) service_pool.cc:227] Handling call
0809 23:19:01.404381 (+ 53112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.419144 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.384464 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:01.374532 28229 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:01.408063 24428 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 1. Current term is 2. Ops: []
I20260809 23:19:01.425053 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:01.410254 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:01.399801 24429 rpcz_store.cc:276] 0809 23:18:57.524248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.256722 (+3732474us) service_pool.cc:227] Handling call
0809 23:19:01.325666 (+ 68944us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.396584 (+ 70918us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.398690 (+  2106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.426885 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.420804 24435 rpcz_store.cc:276] 0809 23:18:59.280515 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.376655 (+2096140us) service_pool.cc:227] Handling call
0809 23:19:01.393326 (+ 16671us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.416899 (+ 23573us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.417128 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.428125 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:01.415817 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.394008 28083 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:01.423588 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.425238 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2314) took 1375 ms. Trace:
I20260809 23:19:01.430419 24428 rpcz_store.cc:276] 0809 23:19:00.049679 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.382068 (+1332389us) service_pool.cc:227] Handling call
0809 23:19:01.403753 (+ 21685us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.424936 (+ 21183us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.425174 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.431474 28276 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:19:01.432744 28276 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
W20260809 23:19:01.415829 27949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.011s	sys 0.032s
I20260809 23:19:01.386027 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.393329 27857 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:01.427548 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2301) took 2205 ms. Trace:
I20260809 23:19:01.409327 27976 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:01.440403 25375 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:01.444272 24309 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:19:01.444475 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 795) took 1116 ms. Trace:
I20260809 23:19:01.445024 25375 rpcz_store.cc:276] 0809 23:19:00.327836 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.418606 (+1090770us) service_pool.cc:227] Handling call
0809 23:19:01.444387 (+ 25781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.403719 28159 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:01.433090 28276 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:01.438213 25323 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.318s (SENT)
I20260809 23:19:01.439991 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2315) took 1350 ms. Trace:
I20260809 23:19:01.453164 24425 rpcz_store.cc:276] 0809 23:19:00.089778 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.406340 (+1316562us) service_pool.cc:227] Handling call
0809 23:19:01.427744 (+ 21404us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.434101 (+  6357us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.434128 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:01.439585 (+  5457us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:01.439747 (+   162us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.439830 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.439934 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.435494 24164 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 4, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.455118 24311 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:01.442687 24422 rpcz_store.cc:276] 0809 23:18:59.222110 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.356792 (+2134682us) service_pool.cc:227] Handling call
0809 23:19:01.375568 (+ 18776us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.427206 (+ 51638us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.427238 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:01.427282 (+    44us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:01.427319 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.427395 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.427496 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.443696 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 115172us
I20260809 23:19:01.407343 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 793) took 1217 ms. Trace:
I20260809 23:19:01.456928 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2328) took 1159 ms. Trace:
I20260809 23:19:01.457124 25372 rpcz_store.cc:276] 0809 23:19:00.190255 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.319435 (+1129180us) service_pool.cc:227] Handling call
0809 23:19:01.407208 (+ 87773us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.457427 24309 rpcz_store.cc:276] 0809 23:19:00.297969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.430237 (+1132268us) service_pool.cc:227] Handling call
0809 23:19:01.444051 (+ 13814us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.456629 (+ 12578us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.456865 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.438993 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.394680 28257 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.474423 24314 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:01.460292 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.461026 25371 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:01.477277 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 878) took 1252 ms. Trace:
I20260809 23:19:01.477725 25371 rpcz_store.cc:276] 0809 23:19:00.225041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.435254 (+1210213us) service_pool.cc:227] Handling call
0809 23:19:01.477209 (+ 41955us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.453497 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78188.8us
I20260809 23:19:01.455864 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2324) took 1264 ms. Trace:
I20260809 23:19:01.479058 24311 rpcz_store.cc:276] 0809 23:19:00.191095 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.427953 (+1236858us) service_pool.cc:227] Handling call
0809 23:19:01.438732 (+ 10779us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.455566 (+ 16834us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.455793 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.451797 28237 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:01.463318 28159 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:01.436331 27945 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:01.463488 28065 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:19:01.493834 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.226s (SENT) [suppressed 51 similar messages]
I20260809 23:19:01.494380 28159 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.495088 28065 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:01.495699 28065 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.496394 27689 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:01.464773 24164 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 4, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 4 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.467034 24544 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:01.467728 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:01.497825 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69900us
I20260809 23:19:01.471125 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 758) took 1111 ms. Trace:
I20260809 23:19:01.471493 28276 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.471724 28077 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:01.436247 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 748) took 1099 ms. Trace:
I20260809 23:19:01.429765 28272 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:01.386660 28006 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.484069 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.412503 28224 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.484894 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 869) took 2322 ms. Trace:
I20260809 23:19:01.509805 25375 rpcz_store.cc:276] 0809 23:18:59.163108 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.459654 (+2296546us) service_pool.cc:227] Handling call
0809 23:19:01.484825 (+ 25171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.444938 27857 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:01.503963 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2325) took 1279 ms. Trace:
I20260809 23:19:01.499508 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:01.463235 27633 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:01.458810 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.510239 24164 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.504914 24316 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 23:19:01.506443 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:01.511262 24320 rpcz_store.cc:276] 0809 23:19:00.225016 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.427447 (+1202431us) service_pool.cc:227] Handling call
0809 23:19:01.503153 (+ 75706us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:01.503360 (+   207us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.503642 (+   282us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.503669 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:01.503713 (+    44us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:01.503749 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.503815 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.503912 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1979,"spinlock_wait_cycles":2833664}
I20260809 23:19:01.508194 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 759) took 1148 ms. Trace:
I20260809 23:19:01.515738 25372 rpcz_store.cc:276] 0809 23:19:00.360188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.458141 (+1097953us) service_pool.cc:227] Handling call
0809 23:19:01.508131 (+ 49990us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.513096 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108889us
I20260809 23:19:01.513098 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.507153 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.514485 24546 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:01.501047 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.518589 25371 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:01.519403 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 792) took 1329 ms. Trace:
I20260809 23:19:01.519598 28065 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:01.520067 25371 rpcz_store.cc:276] 0809 23:19:00.189868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.478481 (+1288613us) service_pool.cc:227] Handling call
0809 23:19:01.519303 (+ 40822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.487195 28278 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:01.521279 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2331) took 1129 ms. Trace:
I20260809 23:19:01.521730 24316 rpcz_store.cc:276] 0809 23:19:00.392418 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.440066 (+1047648us) service_pool.cc:227] Handling call
0809 23:19:01.473926 (+ 33860us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.521010 (+ 47084us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.521226 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.522215 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.093s (SENT)
I20260809 23:19:01.522884 24314 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:19:01.524281 24546 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:19:01.526914 24545 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:01.528926 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 547) took 1226 ms. Trace:
I20260809 23:19:01.530371 24544 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
W20260809 23:19:01.504715 28307 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260809 23:19:01.528963 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 750) took 1183 ms. Trace:
I20260809 23:19:01.532181 25375 rpcz_store.cc:276] 0809 23:19:00.334007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.510551 (+1176544us) service_pool.cc:227] Handling call
0809 23:19:01.517754 (+  7203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.529546 24545 rpcz_store.cc:276] 0809 23:19:00.302231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.617300 (+315069us) service_pool.cc:227] Handling call
0809 23:19:01.528842 (+911542us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.533138 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.502907 28276 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:01.536149 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.537258 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 747) took 1204 ms. Trace:
I20260809 23:19:01.535511 24164 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 3, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.503759 28077 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:01.537612 25370 rpcz_store.cc:276] 0809 23:19:00.333028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.355792 (+1022764us) service_pool.cc:227] Handling call
0809 23:19:01.483930 (+128138us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:19:01.537203 (+ 53273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4439808}
I20260809 23:19:01.501075 28214 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:01.537433 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:19:01.541677 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 879) took 1316 ms. Trace:
I20260809 23:19:01.542062 25372 rpcz_store.cc:276] 0809 23:19:00.225321 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.516617 (+1291296us) service_pool.cc:227] Handling call
0809 23:19:01.541625 (+ 25008us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.543274 25323 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.195s (SENT)
I20260809 23:19:01.545461 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:01.545521 24164 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 1 to 2, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:01.542965 28077 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:01.547449 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.554229 25375 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:01.557350 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 880) took 1320 ms. Trace:
I20260809 23:19:01.557781 25375 rpcz_store.cc:276] 0809 23:19:00.237460 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.532860 (+1295400us) service_pool.cc:227] Handling call
0809 23:19:01.557297 (+ 24437us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.556675 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 145084us
I20260809 23:19:01.559615 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.502154 25365 rpcz_store.cc:276] 0809 23:19:00.359733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.354795 (+995062us) service_pool.cc:227] Handling call
0809 23:19:01.471053 (+116258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.563076 25371 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:01.563426 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.562737 24164 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 3, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.564213 25375 raft_consensus.cc:2396] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:01.461656 28222 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.504230 25363 rpcz_store.cc:276] 0809 23:19:00.333369 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.342183 (+1008814us) service_pool.cc:227] Handling call
0809 23:19:01.433200 (+ 91017us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.565896 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 791) took 1376 ms. Trace:
I20260809 23:19:01.567710 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.569237 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.568697 25371 rpcz_store.cc:276] 0809 23:19:00.189421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.521944 (+1332523us) service_pool.cc:227] Handling call
0809 23:19:01.565810 (+ 43866us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.562479 28136 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260809 23:19:01.535952 27633 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.574259 25366 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:01.579018 25368 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:01.579523 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 876) took 1527 ms. Trace:
I20260809 23:19:01.579937 25368 rpcz_store.cc:276] 0809 23:19:00.051719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.410889 (+1359170us) service_pool.cc:227] Handling call
0809 23:19:01.579443 (+168554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.583334 24541 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:19:01.587014 28224 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:01.587276 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 535) took 1530 ms. Trace:
I20260809 23:19:01.587752 24540 rpcz_store.cc:276] 0809 23:19:00.057034 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.104091 (+ 47057us) service_pool.cc:227] Handling call
0809 23:19:00.109695 (+  5604us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:00.110192 (+   497us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:00.110220 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:00.110768 (+   548us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.241686 (+1130918us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:01.241720 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.241786 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:01.587025 (+345239us) raft_consensus.cc:1671] finished
0809 23:19:01.587050 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.587205 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1102,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":46665,"thread_start_us":1058196,"threads_started":1,"wal-append.queue_time_us":1434542}
W20260809 23:19:01.592504 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47141.2us
I20260809 23:19:01.592172 28077 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.593463 28077 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:01.594390 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:01.601819 25370 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:01.602451 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 794) took 1298 ms. Trace:
I20260809 23:19:01.602797 25370 rpcz_store.cc:276] 0809 23:19:00.303890 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.538382 (+1234492us) service_pool.cc:227] Handling call
0809 23:19:01.602381 (+ 63999us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.598771 28257 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:01.507498 27945 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.607306 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 789) took 1495 ms. Trace:
I20260809 23:19:01.609041 25375 rpcz_store.cc:276] 0809 23:19:00.112072 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.559274 (+1447202us) service_pool.cc:227] Handling call
0809 23:19:01.607181 (+ 47907us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.614495 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:01.614591 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:01.610998 24164 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.617414 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.618317 25369 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:19:01.619531 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.619763 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.619526 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:01.620267 28006 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:01.620427 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 866) took 2903 ms (client timeout 3226 ms). Trace:
I20260809 23:19:01.623205 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:01.536667 28272 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.599843 28246 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260809 23:19:01.624364 28171 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:01.616043 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2333) took 1159 ms. Trace:
W20260809 23:19:01.627364 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58173.1us
I20260809 23:19:01.628705 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:01.639243 25373 rpcz_store.cc:269] 0809 23:18:58.712783 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.329065 (+2616282us) service_pool.cc:227] Handling call
0809 23:19:01.616229 (+287164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.645241 24312 rpcz_store.cc:276] 0809 23:19:00.456528 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.571007 (+114479us) service_pool.cc:227] Handling call
0809 23:19:01.615971 (+1044964us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.611461 28237 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:01.639737 24545 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:01.646575 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 745) took 1314 ms. Trace:
I20260809 23:19:01.651932 25370 rpcz_store.cc:276] 0809 23:19:00.332342 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.603430 (+1271088us) service_pool.cc:227] Handling call
0809 23:19:01.646507 (+ 43077us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.653607 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.656473 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.657387 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.627848 28065 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:01.659252 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64954.7us
I20260809 23:19:01.659631 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2299) took 2430 ms. Trace:
I20260809 23:19:01.636400 28004 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:01.655663 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:01.654212 25375 raft_consensus.cc:2362] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:01.667536 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:01.656376 24164 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 5, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 5 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:01.675508 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.677709 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.662079 24430 rpcz_store.cc:276] 0809 23:18:59.221409 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.342130 (+2120721us) service_pool.cc:227] Handling call
0809 23:19:01.374802 (+ 32672us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.567822 (+193020us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.567865 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:01.567909 (+    44us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:01.567941 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.651264 (+ 83323us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:01.651310 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:01.651431 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9012,"raft.queue_time_us":9120,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":32762,"spinlock_wait_cycles":372592256}
I20260809 23:19:01.678387 25375 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 2.
I20260809 23:19:01.687409 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.658167 27689 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:01.625859 27949 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260809 23:19:01.695286 25371 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:01.696429 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 746) took 1363 ms. Trace:
I20260809 23:19:01.696805 25371 rpcz_store.cc:276] 0809 23:19:00.332735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.649547 (+1316812us) service_pool.cc:227] Handling call
0809 23:19:01.696360 (+ 46813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.660216 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.694517 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.661736 28065 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:01.705684 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.707159 25375 raft_consensus.cc:2396] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:19:01.710103 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.673849 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 753) took 1330 ms. Trace:
I20260809 23:19:01.683148 28237 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:01.716024 24378 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:19:01.724411 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.724805 24378 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:01.716522 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 749) took 1382 ms. Trace:
I20260809 23:19:01.723835 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 757) took 1364 ms. Trace:
I20260809 23:19:01.725243 25371 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:01.725353 25372 rpcz_store.cc:276] 0809 23:19:00.333706 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.688473 (+1354767us) service_pool.cc:227] Handling call
0809 23:19:01.716443 (+ 27970us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.695720 27689 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:01.726724 25375 rpcz_store.cc:276] 0809 23:19:00.359301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.680447 (+1321146us) service_pool.cc:227] Handling call
0809 23:19:01.723773 (+ 43326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.727257 27689 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:01.698804 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 872) took 1716 ms. Trace:
I20260809 23:19:01.728718 25368 rpcz_store.cc:276] 0809 23:18:59.976221 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.619208 (+1642987us) service_pool.cc:227] Handling call
0809 23:19:01.691663 (+ 72455us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:01.729709 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 434020us
I20260809 23:19:01.734882 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.735194 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.709236 28065 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:01.677148 28004 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:01.702658 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 722) took 7106 ms (client timeout 2233 ms). Trace:
W20260809 23:19:01.745237 24498 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.592s (SENT)
W20260809 23:19:01.760279 24498 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.592s (SENT)
I20260809 23:19:01.743534 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:01.760694 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:01.676931 28171 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.730094 28237 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:01.725162 24164 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:01.763121 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 882) took 1176 ms. Trace:
I20260809 23:19:02.303403 25372 rpcz_store.cc:276] 0809 23:19:00.586996 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.726549 (+1139553us) service_pool.cc:227] Handling call
0809 23:19:01.763047 (+ 36498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:01.769366 25375 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:01.771672 28177 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:01.690356 27857 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:01.699504 28224 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260809 23:19:02.317386 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 751) took 1983 ms. Trace:
I20260809 23:19:02.317926 25375 rpcz_store.cc:276] 0809 23:19:00.334350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.732317 (+1397967us) service_pool.cc:227] Handling call
0809 23:19:02.317107 (+584790us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b5400717854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:02.317307 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76682496}
I20260809 23:19:02.060405 27949 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:02.161422 28159 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:01.869702 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 743) took 5014 ms (client timeout 1546 ms). Trace:
I20260809 23:19:01.959765 28272 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:02.128757 27689 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.060672 28214 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:01.759944 25366 rpcz_store.cc:269] 0809 23:18:54.596308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:54.596579 (+   271us) service_pool.cc:227] Handling call
0809 23:19:01.691464 (+7094885us) spinlock_profiling.cc:243] Waited 6.69 s on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:01.702298 (+ 10834us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15379614080}
W20260809 23:19:01.949805 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 449113us
W20260809 23:19:01.794314 24269 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.757s (SENT)
I20260809 23:19:02.335879 27857 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:02.335763 28214 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.339923 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:02.341255 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:02.342217 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:01.794922 28083 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:01.903728 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.014s	sys 0.019s
W20260809 23:19:02.352540 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.029s	sys 0.030s
I20260809 23:19:02.167194 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 700) took 8342 ms. Trace:
W20260809 23:19:02.173246 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 412949us
I20260809 23:19:01.768433 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 797) took 1180 ms. Trace:
I20260809 23:19:01.811554 28237 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:01.742063 28246 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4.
I20260809 23:19:01.808475 28197 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260809 23:19:02.361438 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 549) took 1959 ms. Trace:
I20260809 23:19:02.363828 24543 rpcz_store.cc:276] 0809 23:19:00.401859 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.282498 (+880639us) service_pool.cc:227] Handling call
0809 23:19:01.332061 (+ 49563us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.332293 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:01.349367 (+ 17074us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.361049 (+1011682us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.361107 (+    58us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:02.361161 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.361229 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.361373 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16896,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":1003249,"threads_started":1}
W20260809 23:19:02.093956 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.018s	sys 0.032s
W20260809 23:19:02.365865 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.021s	sys 0.034s
W20260809 23:19:01.927371 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.008s	sys 0.052s
I20260809 23:19:01.780704 28077 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:19:02.228364 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2277) took 4734 ms. Trace:
W20260809 23:19:01.799424 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94191us
I20260809 23:19:01.916965 28278 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:02.378675 28278 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:02.347311 27949 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.379981 28278 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.381464 27949 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:02.381501 28278 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:02.061901 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:02.383661 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 790) took 2202 ms. Trace:
I20260809 23:19:02.384106 25371 rpcz_store.cc:276] 0809 23:19:00.180948 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.734818 (+1553870us) service_pool.cc:227] Handling call
0809 23:19:02.061721 (+326903us) spinlock_profiling.cc:243] Waited 5.27 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900bce 000000000034aea9 00007fac2872ca30 00007fac287f27b4 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:19:02.383573 (+321852us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12110720}
I20260809 23:19:02.385542 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:01.778638 27945 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:01.839805 28259 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:01.795162 28156 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:01.805436 27976 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:02.386415 25371 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:02.157994 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 472421us
I20260809 23:19:01.715095 25365 rpcz_store.cc:276] 0809 23:19:00.352398 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.601541 (+1249143us) service_pool.cc:227] Handling call
0809 23:19:01.665811 (+ 64270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.388510 28197 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:02.391302 28197 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:02.391572 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:02.392529 25365 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260809 23:19:02.392238 28197 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:02.392966 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 752) took 2058 ms. Trace:
I20260809 23:19:02.393292 25365 rpcz_store.cc:276] 0809 23:19:00.334673 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.391230 (+2056557us) service_pool.cc:227] Handling call
0809 23:19:02.392910 (+  1680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.394147 28077 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:02.394333 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:02.214282 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.006s	sys 0.011s
W20260809 23:19:02.395092 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.006s	sys 0.012s
W20260809 23:19:02.395408 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 727924us
I20260809 23:19:02.396018 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:02.296320 28323 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20260809 23:19:02.297075 28319 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260809 23:19:02.295358 28318 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.562s	user 0.002s	sys 0.000s
I20260809 23:19:02.320704 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:02.410352 25375 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:02.323690 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:02.214179 28276 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.000s
W20260809 23:19:02.304718 28161 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.888s	user 0.011s	sys 0.000s
W20260809 23:19:01.885021 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.022s	sys 0.033s
W20260809 23:19:02.414394 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.022s	sys 0.070s
I20260809 23:19:02.218457 28229 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:19:02.328647 25369 rpcz_store.cc:269] 0809 23:18:56.855353 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:56.967907 (+112554us) service_pool.cc:227] Handling call
0809 23:19:01.869340 (+4901433us) spinlock_profiling.cc:243] Waited 4.45 s on lock 0x7b540073e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:01.869614 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10228303616}
I20260809 23:19:02.416495 25372 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:02.417129 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:02.229840 28171 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:02.334290 25370 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:19:02.353050 24323 rpcz_store.cc:276] 0809 23:18:53.824430 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:59.341946 (+5517516us) service_pool.cc:227] Handling call
0809 23:18:59.982959 (+641013us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:00.031303 (+ 48344us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:00.990255 (+958952us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.164968 (+1174713us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.165066 (+    98us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:02.166123 (+  1057us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.166228 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.167123 (+   895us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":9347,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":425289,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":66,"thread_start_us":1939838,"threads_started":3,"wal-append.queue_time_us":1039217}
W20260809 23:19:02.332640 28178 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.987s	user 0.012s	sys 0.000s
W20260809 23:19:02.366400 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.011s	sys 0.063s
W20260809 23:19:02.427954 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.05029e+06us
I20260809 23:19:02.429579 24431 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260809 23:19:02.430019 24431 pending_rounds.cc:76] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 4
I20260809 23:19:02.430262 24431 pending_rounds.cc:97] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260809 23:19:02.430558 24431 raft_consensus.cc:2866] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:02.276468 28065 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:02.324278 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:02.432380 25368 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:02.433380 28065 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:02.433825 25368 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:02.434476 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 786) took 4973 ms. Trace:
I20260809 23:19:02.341902 25366 raft_consensus.cc:2362] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:02.436010 25366 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260809 23:19:02.436443 25366 pending_rounds.cc:76] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 6
I20260809 23:19:02.436708 25366 pending_rounds.cc:97] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260809 23:19:02.437043 25366 raft_consensus.cc:2866] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:02.371336 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 577028us
W20260809 23:19:02.371613 28066 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.081s	user 0.010s	sys 0.003s
I20260809 23:19:02.372118 28177 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:19:02.374770 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 876050us
I20260809 23:19:02.367961 27857 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:02.375272 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2294) took 4054 ms. Trace:
I20260809 23:19:02.376322 27633 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:02.377161 27689 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:02.157672 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 443445us
I20260809 23:19:02.444722 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:02.388962 25373 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:02.368487 24423 rpcz_store.cc:276] 0809 23:18:57.473782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.266248 (+3792466us) service_pool.cc:227] Handling call
0809 23:19:01.323541 (+ 57293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.348828 (+ 25287us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:01.843464 (+494636us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.208162 (+364698us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.208236 (+    74us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:02.208317 (+    81us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.208412 (+    95us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.208570 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":805537,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":121454,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4861,"thread_start_us":785351,"threads_started":2}
I20260809 23:19:02.451942 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.452617 24423 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:02.453768 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.393438 28197 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:02.311314 24164 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 3 to 5, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:02.414325 28159 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:02.419080 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:02.420399 28004 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:02.419029 24435 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260809 23:19:02.421269 28077 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.421156 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2293) took 4215 ms. Trace:
W20260809 23:19:02.422024 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 713) took 1858 ms (client timeout 2318 ms). Trace:
I20260809 23:19:02.442082 24424 rpcz_store.cc:276] 0809 23:18:58.322618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.278158 (+2955540us) service_pool.cc:227] Handling call
0809 23:19:01.342912 (+ 64754us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.387025 (+ 44113us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.387053 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:01.427555 (+ 40502us) log.cc:842] Serialized 26 byte log entry
0809 23:19:02.009226 (+581671us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:02.009273 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.009349 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:02.375000 (+365651us) raft_consensus.cc:1671] finished
0809 23:19:02.375029 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.375195 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":470396,"threads_started":1,"wal-append.queue_time_us":891131}
I20260809 23:19:02.447222 28348 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:19:02.456524 25372 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 2. Current term is 4. Ops: []
I20260809 23:19:02.456904 24430 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260809 23:19:02.457846 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 800) took 1579 ms. Trace:
I20260809 23:19:02.471243 24423 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:02.359745 25363 rpcz_store.cc:276] 0809 23:19:00.582861 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.608971 (+1026110us) service_pool.cc:227] Handling call
0809 23:19:01.762441 (+153470us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.472031 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.571220 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2128) took 2270 ms. Trace:
I20260809 23:19:02.556110 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:19:02.516588 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.455s (SENT)
I20260809 23:19:02.408769 25365 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
W20260809 23:19:02.490372 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71358.7us
I20260809 23:19:02.572870 24425 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260809 23:19:02.511068 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 121386us
I20260809 23:19:02.649057 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 543) took 2491 ms. Trace:
I20260809 23:19:02.654551 24539 rpcz_store.cc:276] 0809 23:19:00.157353 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.255528 (+1098175us) service_pool.cc:227] Handling call
0809 23:19:01.277553 (+ 22025us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.312715 (+ 35162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.312758 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:01.345221 (+ 32463us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.870277 (+525056us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:01.870319 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.870582 (+   263us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:02.648825 (+778243us) raft_consensus.cc:1671] finished
0809 23:19:02.648846 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.648997 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":470779,"threads_started":1,"wal-append.queue_time_us":1054669}
W20260809 23:19:02.656934 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140388us
I20260809 23:19:02.491289 28159 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:02.659042 25373 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260809 23:19:02.649248 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77041.1us
I20260809 23:19:02.571730 24535 rpcz_store.cc:276] 0809 23:19:00.300991 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.274625 (+973634us) service_pool.cc:227] Handling call
0809 23:19:01.313980 (+ 39355us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.349170 (+ 35190us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.349211 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:01.372010 (+ 22799us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.829948 (+457938us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:01.829984 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.830056 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:02.571029 (+740973us) raft_consensus.cc:1671] finished
0809 23:19:02.571047 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.571166 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10760,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":19848,"thread_start_us":423094,"threads_started":1,"wal-append.queue_time_us":1034919}
W20260809 23:19:02.588040 24317 rpcz_store.cc:269] 0809 23:19:00.563984 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.255870 (+691886us) service_pool.cc:227] Handling call
0809 23:19:02.421825 (+1165955us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:02.421963 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2526941440}
I20260809 23:19:02.637174 24429 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:02.660337 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 219352us
I20260809 23:19:02.660952 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 542) took 2590 ms. Trace:
W20260809 23:19:02.572125 24269 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.961s (SENT)
I20260809 23:19:02.665400 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:02.667989 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.668623 24422 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:19:02.669709 25371 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:02.668629 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:02.670218 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 893) took 1059 ms. Trace:
I20260809 23:19:02.670539 25371 rpcz_store.cc:276] 0809 23:19:01.611312 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.385186 (+773874us) service_pool.cc:227] Handling call
0809 23:19:02.670162 (+284976us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.667572 24534 rpcz_store.cc:276] 0809 23:19:00.074631 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.987196 (+912565us) service_pool.cc:227] Handling call
0809 23:19:00.987567 (+   371us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.168421 (+180854us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.168455 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:01.208929 (+ 40474us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.748902 (+539973us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:01.748953 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.749050 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:02.598561 (+849511us) raft_consensus.cc:1671] finished
0809 23:19:02.598582 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.660870 (+ 62288us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131691,"raft.queue_time_us":151997,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":30174,"thread_start_us":484865,"threads_started":1,"wal-append.queue_time_us":839701}
I20260809 23:19:02.564884 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 550) took 2162 ms. Trace:
I20260809 23:19:02.671327 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.571451 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:02.671721 24542 rpcz_store.cc:276] 0809 23:19:00.402179 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.285895 (+883716us) service_pool.cc:227] Handling call
0809 23:19:01.327337 (+ 41442us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:01.327584 (+   247us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:01.327619 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:01.349940 (+ 22321us) log.cc:842] Serialized 26 byte log entry
0809 23:19:01.837157 (+487217us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:01.837202 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:01.837271 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:02.564639 (+727368us) raft_consensus.cc:1671] finished
0809 23:19:02.564663 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.564804 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":435960,"threads_started":1,"wal-append.queue_time_us":1157562}
W20260809 23:19:02.673157 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 101781us
W20260809 23:19:02.481429 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 874) took 2457 ms (client timeout 3005 ms). Trace:
I20260809 23:19:02.673714 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:02.570147 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2295) took 4008 ms. Trace:
W20260809 23:19:02.674181 25370 rpcz_store.cc:269] 0809 23:19:00.023814 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.653009 (+1629195us) service_pool.cc:227] Handling call
0809 23:19:02.469773 (+816764us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:02.481350 (+ 11577us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1482609920}
I20260809 23:19:02.674819 24431 rpcz_store.cc:276] 0809 23:18:58.561972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.306884 (+2744912us) service_pool.cc:227] Handling call
0809 23:19:01.353518 (+ 46634us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.431064 (+1077546us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.431748 (+   684us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:02.570066 (+138318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2473545984}
W20260809 23:19:02.650306 28346 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260809 23:19:02.691063 24424 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:02.482645 28171 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:02.696339 24539 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:19:02.498178 27689 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:02.714663 25371 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260809 23:19:02.715994 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:02.717360 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.722620 24431 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:02.434824 25368 rpcz_store.cc:276] 0809 23:18:57.461326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.433085 (+4971759us) service_pool.cc:227] Handling call
0809 23:19:02.433469 (+   384us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.434216 (+   747us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.434400 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.726593 28159 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:02.714486 28358 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:02.728188 28358 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:02.728545 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 881) took 2141 ms. Trace:
I20260809 23:19:02.729135 25363 rpcz_store.cc:276] 0809 23:19:00.586648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.660270 (+2073622us) service_pool.cc:227] Handling call
0809 23:19:02.727649 (+ 67379us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.728030 (+   381us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:02.728074 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.728140 (+    66us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:02.728219 (+    79us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.728300 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.728446 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.730494 28358 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:02.680337 28178 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:02.731473 28178 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:02.733593 28178 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:02.715165 25371 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 7
I20260809 23:19:02.739748 25371 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260809 23:19:02.740087 25371 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:02.747187 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 318438us
I20260809 23:19:02.674641 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 756) took 2305 ms. Trace:
I20260809 23:19:02.748528 25373 rpcz_store.cc:276] 0809 23:19:00.358772 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.451037 (+2092265us) service_pool.cc:227] Handling call
0809 23:19:02.571893 (+120856us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.664252 (+ 92359us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.664446 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.588493 25369 rpcz_store.cc:276] 0809 23:19:00.878518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.416839 (+1538321us) service_pool.cc:227] Handling call
0809 23:19:02.457773 (+ 40934us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.681823 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:02.675021 24427 rpcz_store.cc:276] 0809 23:18:58.206333 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.275733 (+3069400us) service_pool.cc:227] Handling call
0809 23:19:01.286251 (+ 10518us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:01.286475 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:19:01.727171 (+440696us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.420584 (+693413us) spinlock_profiling.cc:243] Waited 57.1 ms on lock 0x7b1800464320. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:19:02.420769 (+   185us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.420814 (+    45us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:19:02.420888 (+    74us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.420976 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.421101 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":817808,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":1047,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":50,"spinlock_wait_cycles":131286400,"thread_start_us":870954,"threads_started":2,"wal-append.queue_time_us":693502}
W20260809 23:19:02.646351 28326 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260809 23:19:02.749567 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:19:02.436010 27946 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:02.750245 27946 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:02.750806 27946 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:02.694401 24423 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:02.691069 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 755) took 2322 ms. Trace:
I20260809 23:19:02.680596 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:02.681067 28224 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:02.755378 28237 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:02.635706 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 754) took 2277 ms. Trace:
I20260809 23:19:02.673895 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:02.425498 28161 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:02.758847 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 86270.5us
I20260809 23:19:02.534385 28065 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.203s
I20260809 23:19:02.712620 28171 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:02.763391 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 798) took 2057 ms. Trace:
W20260809 23:19:02.793874 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 120032us
I20260809 23:19:02.794119 25371 rpcz_store.cc:276] 0809 23:19:00.709896 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.680271 (+1970375us) service_pool.cc:227] Handling call
0809 23:19:02.680580 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.763037 (+ 82457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.763310 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:02.738606 28197 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.017s
W20260809 23:19:02.764940 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 318557us
I20260809 23:19:02.760555 27689 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:02.900861 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49654.1us
I20260809 23:19:03.002470 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 741) took 6168 ms. Trace:
I20260809 23:19:02.942090 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 708) took 6114 ms. Trace:
I20260809 23:19:02.902774 24425 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:03.083694 24313 rpcz_store.cc:276] 0809 23:18:56.827501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.416202 (+4588701us) service_pool.cc:227] Handling call
0809 23:19:01.453245 (+ 37043us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.083993 (+630748us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:02.308860 (+224867us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.939289 (+630429us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.939346 (+    57us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:02.939405 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.941792 (+  2387us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540019dcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:02.941854 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.942017 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":510242,"prepare.queue_time_us":336835,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":513654,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":161,"spinlock_wait_cycles":1449510784,"thread_start_us":289719,"threads_started":2,"wal-append.queue_time_us":361975}
I20260809 23:19:02.948900 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2337) took 2479 ms. Trace:
I20260809 23:19:02.796285 28171 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:02.996965 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 301248us
W20260809 23:19:03.081638 25375 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.668s	user 0.007s	sys 0.006s
I20260809 23:19:03.092178 25375 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:03.092820 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 891) took 1560 ms. Trace:
I20260809 23:19:03.093274 25375 rpcz_store.cc:276] 0809 23:19:01.532381 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.318922 (+786541us) service_pool.cc:227] Handling call
0809 23:19:03.092750 (+773828us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:03.079890 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:02.972711 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 400579us
I20260809 23:19:02.934233 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.083401 25372 rpcz_store.cc:276] 0809 23:18:56.810576 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.418480 (+5607904us) service_pool.cc:227] Handling call
0809 23:19:02.456338 (+ 37858us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.588711 (+132373us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.978194 (+389483us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b5400550850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:02.978406 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86289152}
I20260809 23:19:02.796057 28358 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:02.927563 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 886) took 1557 ms. Trace:
I20260809 23:19:03.098696 25370 rpcz_store.cc:276] 0809 23:19:01.369666 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.680942 (+1311276us) service_pool.cc:227] Handling call
0809 23:19:02.924713 (+243771us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:02.925099 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.926996 (+  1897us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:02.927039 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.927103 (+    64us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:02.927160 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.927253 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.927460 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22985088}
I20260809 23:19:03.095162 24535 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 2.
W20260809 23:19:03.000250 28324 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260809 23:19:03.103972 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.106230 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:03.107658 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:03.115628 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.086895 24309 rpcz_store.cc:276] 0809 23:19:00.469963 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:01.458600 (+988637us) service_pool.cc:227] Handling call
0809 23:19:01.491712 (+ 33112us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.008397 (+516685us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:02.592526 (+584129us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.948488 (+355962us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:02.948551 (+    63us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:02.948617 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.948699 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:02.948834 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":114065,"prepare.queue_time_us":452505,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":401250,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":51093,"thread_start_us":1097993,"threads_started":3}
I20260809 23:19:03.117534 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:02.996874 28214 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.122112 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:02.781603 25365 rpcz_store.cc:276] 0809 23:19:00.357975 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.394057 (+2036082us) service_pool.cc:227] Handling call
0809 23:19:02.627635 (+233578us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:03.038165 28246 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:02.948069 28278 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:03.121498 24269 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.142s (SENT)
I20260809 23:19:02.803755 28259 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:02.763949 25366 rpcz_store.cc:276] 0809 23:19:00.358420 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.435338 (+2076918us) service_pool.cc:227] Handling call
0809 23:19:02.435647 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:02.437854 (+  2207us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.680796 (+242942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:03.123524 24546 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260809 23:19:03.038800 28178 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20260809 23:19:02.928102 28257 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:02.930804 28229 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:02.994982 28197 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:02.802740 28161 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:02.805500 28237 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.130877 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:03.137113 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:03.151645 28197 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:03.118336 28066 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:03.163112 28197 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.045807 28272 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.164685 28272 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.139741 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:03.159049 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56003.9us
I20260809 23:19:03.154870 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.164012 24270 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:19:03.122478 27857 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:03.155879 28083 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.168424 27857 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:03.167243 24539 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.168983 27857 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.130344 28336 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:03.170292 28336 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:03.170717 28336 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.145795 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1854) took 8883 ms. Trace:
I20260809 23:19:03.138713 28259 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:02.802153 28347 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.332s
I20260809 23:19:03.168896 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:02.930208 28156 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.211s
I20260809 23:19:03.174856 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:03.139178 28246 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.176222 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:03.147938 28327 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:03.176324 28083 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:03.177222 28327 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:03.177569 28161 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:03.154816 28359 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:03.038233 28341 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
I20260809 23:19:03.152365 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.153304 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.179435 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:03.171497 24321 rpcz_store.cc:276] 0809 23:18:54.268680 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:00.023534 (+5754854us) service_pool.cc:227] Handling call
0809 23:19:00.039409 (+ 15875us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:00.040152 (+   743us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:19:01.243457 (+1203305us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.936031 (+1692574us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b18004d7100. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:19:02.936187 (+   156us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:02.936658 (+   471us) log.cc:842] Serialized 26 byte log entry
0809 23:19:02.936820 (+   162us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:02.937052 (+   232us) log.cc:842] Serialized 14 byte log entry
0809 23:19:02.937206 (+   154us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.937278 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:03.145591 (+208313us) raft_consensus.cc:1671] finished
0809 23:19:03.145614 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:03.145735 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58886,"prepare.queue_time_us":13536,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":523197,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":271,"spinlock_wait_cycles":19119616,"thread_start_us":2295090,"threads_started":2,"wal-append.queue_time_us":1143425}
I20260809 23:19:03.154663 28237 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260809 23:19:03.179828 28341 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:03.155260 27946 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.168049 24270 leader_election.cc:401] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260809 23:19:03.180289 28341 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:03.180413 28276 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:03.151700 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.166700 28272 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.180946 28341 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:03.181008 28276 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:03.181382 28335 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4.
I20260809 23:19:03.178838 28083 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:03.177663 28327 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:03.182492 28327 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:03.182118 28276 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.183631 28276 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.184182 28246 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.138288 28371 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.184762 25366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:03.185853 25366 raft_consensus.cc:595] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:03.186232 25366 raft_consensus.cc:603] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:03.152030 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.130164 28214 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.184947 28197 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:03.188000 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260809 23:19:03.189239 28214 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.194634 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.190265 27946 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.155433 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.200155 24323 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:03.202500 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:03.191136 28335 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:03.098444 28358 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:03.189615 28359 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:19:03.212570 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.213004 28359 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:03.213198 24323 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:03.213403 28359 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:19:03.214357 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.215370 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.215432 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.216323 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:03.216580 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.217716 28347 consensus_queue.cc:1225] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:03.218318 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.219169 24323 raft_consensus.cc:2396] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:03.219491 28065 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:03.219872 28065 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:03.221457 28065 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.224349 28066 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:03.229583 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.227725 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:03.230368 24312 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.231810 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.232322 24312 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:03.233243 28065 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:03.150781 28257 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:03.235860 28257 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.237004 28257 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:03.238858 24312 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:02.905606 27689 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:03.200040 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93067.9us
I20260809 23:19:03.203050 27857 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.184731 28083 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.208155 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.266633 24545 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.267297 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.204872 24539 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:03.211824 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.269690 24542 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.270915 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.226270 28347 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:19:03.226766 24317 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:03.211867 28335 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:03.227707 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.227910 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.272692 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:03.187666 28371 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.256431 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:03.294251 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 145714us
I20260809 23:19:03.296332 24535 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:03.152040 28229 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:03.259514 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 892) took 1658 ms. Trace:
I20260809 23:19:03.225837 24323 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:03.318007 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.272783 24539 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.202143 28341 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:03.294905 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130241us
I20260809 23:19:03.240211 28358 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.308669 28327 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.311471 24534 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:03.312014 28159 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:03.313287 27857 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:03.314275 28369 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.563s	user 0.021s	sys 0.036s
W20260809 23:19:03.305327 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 177571us
I20260809 23:19:03.295621 28335 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:03.336464 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:03.353235 24545 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:19:03.358773 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 105932us
I20260809 23:19:03.380360 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:03.297845 28375 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260809 23:19:03.385176 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:03.385501 24269 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.233s (SENT)
W20260809 23:19:03.385255 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 252479us
W20260809 23:19:03.386196 24498 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.737s (SENT)
I20260809 23:19:03.384711 27946 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
W20260809 23:19:03.395540 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.739s (SENT)
I20260809 23:19:03.394773 24539 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:03.397078 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.397667 24323 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4. Re-sending same reply.
I20260809 23:19:03.397671 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:03.402876 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.739s (SENT)
I20260809 23:19:03.403364 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:03.404636 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68271.4us
I20260809 23:19:03.377517 28159 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:03.294749 28259 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.411892 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:03.280184 28379 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.144s
I20260809 23:19:03.416997 24534 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:03.419792 28004 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:03.485873 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.487054 24323 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.511394 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:03.511831 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 126353us
I20260809 23:19:03.512137 24309 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:19:03.489290 28323 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:19:03.413470 27946 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:03.521344 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:03.521705 24269 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.240s (SENT)
W20260809 23:19:03.524183 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 107885us
I20260809 23:19:03.525246 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
W20260809 23:19:03.527451 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 121997us
I20260809 23:19:03.530890 24539 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:03.530191 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.512727 25366 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:03.532051 25375 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.534296 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.537122 24534 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.515164 28398 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:19:03.539239 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.544919 24323 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:03.529963 28224 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.774s	user 0.009s	sys 0.006s
I20260809 23:19:03.551744 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.552297 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.555035 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1872) took 1155 ms. Trace:
I20260809 23:19:03.519606 28257 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:03.546173 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.552544 25369 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.545553 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.559165 25375 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.418186 28381 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.120s
W20260809 23:19:03.564337 24378 connection.cc:232] RPC call timeout handler was delayed by -1.8834959997329861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
I20260809 23:19:03.564456 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.574592 24535 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:03.565039 25372 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.555518 24428 rpcz_store.cc:276] 0809 23:19:02.399538 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.407350 (+  7812us) service_pool.cc:227] Handling call
0809 23:19:02.418689 (+ 11339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:02.419147 (+   458us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:02.419177 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:02.419634 (+   457us) log.cc:842] Serialized 26 byte log entry
0809 23:19:02.674216 (+254582us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:02.674260 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:02.674321 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:03.313576 (+639255us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:03.537833 (+224257us) raft_consensus.cc:1671] finished
0809 23:19:03.537883 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:03.554924 (+ 17041us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2556,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":7238400,"thread_start_us":254234,"threads_started":1,"wal-append.queue_time_us":331316}
I20260809 23:19:03.576825 24423 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:03.577564 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.553135 24317 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:03.572970 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.578517 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:03.579293 24378 connection.cc:232] RPC call timeout handler was delayed by -1.8429959998466074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260809 23:19:03.579563 24312 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.579252 25371 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260809 23:19:03.579871 24378 connection.cc:232] RPC call timeout handler was delayed by -1.7637959998100996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260809 23:19:03.585964 24378 connection.cc:232] RPC call timeout handler was delayed by -1.5900959998834878s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260809 23:19:03.586395 24378 connection.cc:232] RPC call timeout handler was delayed by -1.5189959998242557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
I20260809 23:19:03.585232 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:03.587025 24378 connection.cc:232] RPC call timeout handler was delayed by -1.4172959998250008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260809 23:19:03.587414 24378 connection.cc:232] RPC call timeout handler was delayed by -1.3281959998421371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260809 23:19:03.587711 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2993959998711944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260809 23:19:03.587786 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:03.588019 24378 connection.cc:232] RPC call timeout handler was delayed by -1.1292959998827428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24230000000000002s for a response
I20260809 23:19:03.599442 24309 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260809 23:19:03.600822 24435 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260809 23:19:03.596609 25369 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:03.601845 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.602523 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.605553 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.571161 28224 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:03.605574 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:03.590495 28376 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.279s	user 0.001s	sys 0.001s
I20260809 23:19:03.610566 24317 raft_consensus.cc:2354] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8.
I20260809 23:19:03.608055 25373 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:03.582089 28257 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:03.616047 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.617242 24312 raft_consensus.cc:2396] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:03.600694 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.615293 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:03.624500 24535 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:03.632252 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.634330 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.634660 24534 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:03.634963 24309 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:03.585204 28214 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260809 23:19:03.636673 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:03.639338 28397 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260809 23:19:03.639585 24314 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260809 23:19:03.639986 24314 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 1
I20260809 23:19:03.640200 24314 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260809 23:19:03.640426 24314 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:03.641604 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.623201 28419 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.642365 24314 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.538354 28335 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.643219 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.643723 28335 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:03.644148 24314 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.645393 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.646476 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.647056 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:03.647432 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.647809 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.648358 24314 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 6
I20260809 23:19:03.637928 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:03.627554 28383 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.225s	user 0.001s	sys 0.001s
I20260809 23:19:03.612627 28371 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:03.652733 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.653445 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.661572 24312 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:03.619158 28341 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.663486 25373 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:03.664646 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.564476 28159 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.673684 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:03.642377 28419 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.682278 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.683197 24429 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.683560 28419 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.684209 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.684326 28418 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:03.674834 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:03.680783 28393 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260809 23:19:03.686131 24423 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:03.674976 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.687026 24422 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:03.687202 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.675187 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.675748 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.687808 24311 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:03.635690 28344 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260809 23:19:03.688709 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.675974 24424 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:19:03.664475 28214 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:03.689801 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.690541 24311 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:03.315505 25368 rpcz_store.cc:276] 0809 23:19:01.600853 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.723922 (+1123069us) service_pool.cc:227] Handling call
0809 23:19:02.974683 (+250761us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:02.974975 (+   292us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:03.038600 (+ 63625us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:03.038636 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:03.039152 (+   516us) log.cc:842] Serialized 26 byte log entry
0809 23:19:03.144814 (+105662us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:03.152195 (+  7381us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:03.189147 (+ 36952us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:03.243180 (+ 54033us) raft_consensus.cc:1671] finished
0809 23:19:03.252043 (+  8863us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:03.259429 (+  7386us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3553,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":2359,"spinlock_wait_cycles":37581824,"thread_start_us":85218,"threads_started":1,"wal-append.queue_time_us":180585}
I20260809 23:19:03.677735 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.691731 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.692234 24320 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:03.692869 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.693269 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.693496 24317 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:03.694443 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.695123 25365 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.678571 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.695976 24323 raft_consensus.cc:2362] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7. Re-sending same reply.
I20260809 23:19:03.649044 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.631871 27946 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.699559 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.679415 28257 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.700235 24312 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:03.680145 28278 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:03.703486 28278 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.680375 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:03.680757 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.681057 28415 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:03.709455 28415 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:03.709507 24321 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:03.681593 28159 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:03.710947 28278 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.689833 28214 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:03.712291 28214 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.713492 28214 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:03.711234 28259 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:03.658111 28066 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:19:03.675956 28399 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260809 23:19:03.665210 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:03.606232 28359 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.715623 24534 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:03.717727 24542 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:03.718536 24317 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:03.732327 24314 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:03.709928 28415 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:03.607359 28336 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.741721 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.749962 24424 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:03.741843 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.746079 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:03.747263 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:03.750980 24435 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:03.752306 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 78731.7us
I20260809 23:19:03.753248 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.755613 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:03.756325 24269 connection.cc:232] RPC call timeout handler was delayed by -2.111331244930625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4813s for a response
W20260809 23:19:03.756815 24269 connection.cc:232] RPC call timeout handler was delayed by -1.9561092648655176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
W20260809 23:19:03.757292 24269 connection.cc:232] RPC call timeout handler was delayed by -1.7536092649679631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260809 23:19:03.757547 24269 connection.cc:232] RPC call timeout handler was delayed by -1.5007092650048435s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
W20260809 23:19:03.757778 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0090488300193101s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
I20260809 23:19:03.766013 24428 raft_consensus.cc:2354] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3.
I20260809 23:19:03.766871 24534 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:03.770601 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.770884 24424 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:03.771341 24422 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:03.772421 24313 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:03.768210 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.693518 28358 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.756758 28278 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:03.761919 28327 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.783192 24427 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:03.783855 28255 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:03.784905 28358 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.785284 28410 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:03.764362 28341 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:03.790684 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.777220 28004 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:03.793063 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.799309 28004 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:03.803180 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50004.9us
I20260809 23:19:03.805287 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:03.806143 28358 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:03.806756 28358 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:03.807575 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.806704 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.777702 28398 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:03.820086 28255 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:03.821245 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.822129 25366 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.808692 28341 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.823174 25370 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
W20260809 23:19:03.821519 28407 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260809 23:19:03.824000 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:03.792697 28410 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:03.825238 24435 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:03.837212 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.794667 28344 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:03.839990 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:03.852792 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46153.6us
I20260809 23:19:03.853785 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.818261 28358 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.858978 24534 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:03.861305 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.862248 24435 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:03.823423 28416 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:03.822535 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:03.869853 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:03.871410 28237 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:03.872951 28359 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.854761 28420 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:03.867024 28344 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.883706 28398 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.884895 28344 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:03.885803 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2161) took 1224 ms. Trace:
I20260809 23:19:03.890479 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.840233 28410 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:03.888545 24427 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:03.895833 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.897403 28432 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:03.904069 24534 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:03.886243 24543 rpcz_store.cc:276] 0809 23:19:02.661738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:02.661873 (+   135us) service_pool.cc:227] Handling call
0809 23:19:02.689138 (+ 27265us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:02.752485 (+ 63347us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:02.752515 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:02.753079 (+   564us) log.cc:842] Serialized 52 byte log entry
0809 23:19:02.831906 (+ 78827us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:03.127441 (+295535us) log.cc:842] Serialized 14 byte log entry
0809 23:19:03.127637 (+   196us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:03.127710 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:03.885517 (+757807us) raft_consensus.cc:1671] finished
0809 23:19:03.885545 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:03.885716 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8930,"prepare.queue_time_us":391030,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":138730,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":184,"thread_start_us":391446,"threads_started":3,"wal-append.queue_time_us":544583}
I20260809 23:19:03.907075 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:03.859553 28278 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:03.908571 28083 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:19:03.908525 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.910236 28391 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:03.929879 24422 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260809 23:19:03.910386 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 507484us
W20260809 23:19:03.931878 24498 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.812s (SENT)
W20260809 23:19:03.932518 24498 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.812s (SENT)
I20260809 23:19:03.932893 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.910012 28415 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:03.911082 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 496764us
I20260809 23:19:03.913522 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:03.823439 28255 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:03.915030 25368 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:19:03.915668 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:03.935148 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.986s (SENT)
I20260809 23:19:03.916384 28341 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.916486 28422 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:03.902915 28420 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:03.937980 25365 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:03.939738 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.947266 24431 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:03.948020 28415 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.928246 28083 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:03.951253 28083 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:19:03.952018 28229 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.587s	user 0.009s	sys 0.000s
I20260809 23:19:03.952411 28229 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:03.954356 28229 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:03.956696 28404 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.556s	user 0.003s	sys 0.000s
I20260809 23:19:03.962284 24542 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260809 23:19:03.962749 24542 pending_rounds.cc:76] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 2
I20260809 23:19:03.945401 28419 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:03.937201 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:03.944554 24534 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260809 23:19:03.964416 28391 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:03.917944 28336 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:03.944689 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.963058 24542 pending_rounds.cc:97] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260809 23:19:03.964396 24534 pending_rounds.cc:76] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 5
W20260809 23:19:03.942730 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.833s (SENT)
W20260809 23:19:04.013227 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 218266us
W20260809 23:19:04.036078 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 93380.5us
I20260809 23:19:03.966465 24534 pending_rounds.cc:97] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 7.6
W20260809 23:19:04.041090 24498 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.036s (SENT)
I20260809 23:19:04.041085 24534 raft_consensus.cc:2866] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:04.041693 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.042514 28391 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:03.966392 24542 raft_consensus.cc:2866] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:04.039000 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.997604 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:03.968614 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.047899 24428 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:04.049062 24425 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
W20260809 23:19:04.052888 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84376.6us
I20260809 23:19:04.053817 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:04.055044 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:03.954798 28395 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:04.055822 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.056949 24430 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260809 23:19:03.965596 28171 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.845s	user 0.012s	sys 0.000s
I20260809 23:19:04.055935 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:04.067611 24498 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.087s (SENT)
I20260809 23:19:04.059453 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:04.067736 24422 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:04.064013 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:04.068198 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:03.981287 28336 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.997833 28341 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:03.976089 28419 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:04.002127 28425 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.074019 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.097121 24435 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
W20260809 23:19:04.086707 24498 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.119s (SENT)
I20260809 23:19:04.099807 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.073516 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:04.080240 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 82748.5us
I20260809 23:19:03.946009 28224 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.126s
I20260809 23:19:04.081058 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:03.965394 28083 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.010972 28379 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260809 23:19:04.065204 28422 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.066373 28395 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.066084 28420 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:04.105286 24424 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:04.104456 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:04.106235 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.107859 28394 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:04.110080 24428 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:04.071952 28171 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.076324 27857 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260809 23:19:04.125834 24431 pending_rounds.cc:76] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 4
I20260809 23:19:04.126271 24431 pending_rounds.cc:97] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 23:19:04.126596 24431 raft_consensus.cc:2866] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:04.077464 28398 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:04.049021 28278 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.111402 24542 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260809 23:19:04.120821 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:04.121260 24498 leader_election.cc:334] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.245s (SENT)
I20260809 23:19:04.122974 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.113698 28178 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:04.101133 28347 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.131032 24542 pending_rounds.cc:76] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 4
I20260809 23:19:04.132519 24542 pending_rounds.cc:97] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260809 23:19:04.132634 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:04.131600 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.078400 28348 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260809 23:19:04.104786 28437 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:04.078011 28391 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260809 23:19:04.144244 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.156240 24435 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:04.132795 24542 raft_consensus.cc:2866] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:04.079177 28336 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.082016 28341 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:04.169890 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.757s (SENT)
I20260809 23:19:04.160130 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:04.164058 24428 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:04.159866 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:04.204092 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:04.120474 28229 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:04.209106 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:04.210330 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.804s (SENT)
I20260809 23:19:04.205243 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:04.206156 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61969.3us
I20260809 23:19:04.210632 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.213925 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:04.207661 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89104.6us
I20260809 23:19:04.226099 24422 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:04.228840 24317 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:04.232383 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.130309 28398 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:04.234443 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:04.155279 28394 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:04.242900 28443 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260809 23:19:04.243062 28375 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.243554 28375 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.188576 28197 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:04.236774 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.133116 27857 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.147116 28422 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:04.240092 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:04.240645 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.160168 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:04.159250 28437 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:04.154624 28420 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:04.126710 28435 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.134s
I20260809 23:19:04.101820 28442 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.154s
I20260809 23:19:04.250618 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.284140 24428 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:04.164709 28178 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:04.230150 28276 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:04.263386 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.290140 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.291222 24378 leader_election.cc:398] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8.
I20260809 23:19:04.291446 24378 leader_election.cc:401] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260809 23:19:04.235183 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.292764 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.293824 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:04.266479 24311 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:04.262065 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:04.118151 28425 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.272311 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:04.244796 28398 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:04.238852 28396 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:04.316687 24422 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:04.261163 28214 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.000s
W20260809 23:19:04.324306 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 89203.4us
I20260809 23:19:04.275668 28437 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:04.252310 28394 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.299701 28278 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.302820 24534 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:04.304318 28083 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:04.305619 28171 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:04.312496 28272 raft_consensus.cc:2741] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 23:19:04.309417 28178 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.284955 28197 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:04.270447 28419 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:04.147787 28358 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.230008 28335 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260809 23:19:04.326990 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.334103 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.343029 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:04.346114 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:04.250777 28395 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.465206 24540 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:19:04.465505 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:04.443420 24546 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:04.446687 24435 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:04.307230 28336 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:04.295423 28359 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.329545 28237 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:04.286716 27857 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.465668 24430 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260809 23:19:04.488686 24430 pending_rounds.cc:76] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 6
I20260809 23:19:04.488917 24430 pending_rounds.cc:97] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260809 23:19:04.489125 24430 raft_consensus.cc:2866] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:04.358973 25323 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.487s (SENT)
I20260809 23:19:04.317934 28224 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.143s
I20260809 23:19:04.323603 28442 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:04.489702 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 130793us
I20260809 23:19:04.472805 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:04.329587 27633 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.333160 28416 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8.
W20260809 23:19:04.514245 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 187385us
I20260809 23:19:04.335016 28065 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.334851 28381 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:04.331159 28391 raft_consensus.cc:2741] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260809 23:19:04.486743 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 272902us
I20260809 23:19:04.539340 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.344202 28156 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.159s
I20260809 23:19:04.517558 24430 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:04.518299 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:04.548128 24540 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:04.527194 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:04.527897 24542 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
W20260809 23:19:04.536377 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 296634us
I20260809 23:19:04.550628 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:04.482111 24422 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:04.487579 24427 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:04.483016 28197 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:04.541539 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:04.551456 24423 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:04.482786 28083 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260809 23:19:04.555624 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:04.560587 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:04.560914 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:04.523317 28436 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:04.561616 24542 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:19:04.536481 28237 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:04.566960 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:04.565013 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 230983us
I20260809 23:19:04.570760 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:04.571640 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55091us
I20260809 23:19:04.571943 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.573277 24498 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
I20260809 23:19:04.573763 24498 leader_election.cc:401] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260809 23:19:04.575807 24423 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:04.558183 28359 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.555024 27857 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:04.577989 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.578708 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.566962 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.579177 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:04.580120 28359 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.554896 28396 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:04.582810 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:04.516904 28418 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260809 23:19:04.590158 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.594511 24543 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260809 23:19:04.595120 24544 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:04.514981 28442 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.523434 27633 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.567824 28436 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:04.567010 28336 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:04.575300 28278 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:04.582386 28065 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:04.633594 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.597576 28004 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:04.604758 28432 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:04.577041 28381 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.578629 28341 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:04.587734 28335 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:04.593288 28396 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:04.642541 24542 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:04.515841 28420 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.596714 28077 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:19:04.596527 28347 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:04.576838 28435 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.635730 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:04.739217 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 134959us
I20260809 23:19:04.637004 28259 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:04.632768 28436 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:04.604206 28156 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5.
W20260809 23:19:04.699633 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 128940us
I20260809 23:19:04.701524 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.635162 28375 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:04.605052 28276 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
W20260809 23:19:04.730548 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 158652us
W20260809 23:19:04.732791 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66636.3us
I20260809 23:19:04.631675 28214 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20260809 23:19:04.653270 28237 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:04.632284 28224 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:04.740079 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 104460us
I20260809 23:19:04.746196 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.596586 28471 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.118s
I20260809 23:19:04.676177 28432 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:04.751344 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.700883 28065 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.762511 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:04.555063 28394 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.657210 28391 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260809 23:19:04.698326 28475 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:04.734827 28347 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.776420 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:04.788144 24435 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260809 23:19:04.749219 28335 raft_consensus.cc:2741] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 23:19:04.746003 28077 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:04.751277 28358 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:04.789628 24542 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
W20260809 23:19:04.789662 28450 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
W20260809 23:19:04.817618 28468 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260809 23:19:04.762627 28259 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:04.818095 28478 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:04.818991 28347 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:04.821650 28391 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Stepping down as leader of term 3
W20260809 23:19:04.870416 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122914us
I20260809 23:19:04.837728 28471 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260809 23:19:04.810737 28278 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20260809 23:19:04.772727 28237 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:04.822098 28466 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:04.823521 28442 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:04.824682 28464 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260809 23:19:04.749008 28425 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:04.807085 28474 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:04.826038 28359 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:04.786298 28432 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:19:04.817464 28457 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260809 23:19:04.697065 28463 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
W20260809 23:19:04.854166 28462 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.349s	user 0.000s	sys 0.001s
W20260809 23:19:04.879457 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87489.7us
W20260809 23:19:04.894484 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148365us
I20260809 23:19:04.828171 28336 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
W20260809 23:19:04.884773 24311 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.590s	user 0.017s	sys 0.001s
I20260809 23:19:04.857409 27857 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:04.857499 28478 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:04.906546 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 134193us
I20260809 23:19:04.859017 28391 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:04.904832 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.008s	sys 0.008s
I20260809 23:19:04.896391 28347 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:04.927376 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.893590 28237 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:04.892704 28335 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:04.912508 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:04.879096 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102768us
I20260809 23:19:04.752127 28435 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.918515 24311 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:04.827811 28077 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:04.920171 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.008s	sys 0.008s
I20260809 23:19:04.905544 28474 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:04.892807 28375 consensus_queue.cc:1159] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 5 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: [4.2-4.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:04.911290 28359 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:04.955593 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:04.958879 24429 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:04.959332 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:04.961598 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49202.6us
I20260809 23:19:04.966763 24435 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:19:04.961709 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:04.935345 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:04.910426 28486 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:19:04.940042 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 350104us
I20260809 23:19:04.946449 27633 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:04.938702 28335 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:05.013918 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78713.3us
I20260809 23:19:04.998154 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:05.001140 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:04.971455 24320 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260809 23:19:05.002548 24423 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:04.858076 28381 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:05.004570 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:04.945843 28474 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:05.022698 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:05.022848 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:05.022400 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:05.027382 24534 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260809 23:19:05.028407 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:05.004465 28425 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 23:19:05.003027 28410 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.097s	user 0.019s	sys 0.005s
I20260809 23:19:05.027331 24323 pending_rounds.cc:76] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:05.034876 24323 pending_rounds.cc:97] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:05.035171 24323 raft_consensus.cc:2866] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:05.034093 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 381999us
W20260809 23:19:05.037449 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74855us
W20260809 23:19:05.037621 28422 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.002s
I20260809 23:19:05.038400 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:05.038568 28422 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:05.010985 28486 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:05.040458 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:05.040958 28486 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:05.021907 28336 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:05.019712 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:05.049165 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:05.049011 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.050880 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:04.951191 28065 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:04.954238 28259 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:05.124006 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85701.4us
I20260809 23:19:04.929378 28478 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:05.019809 28501 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:05.019930 28278 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260809 23:19:05.127664 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:05.045678 28463 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:05.130869 28425 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:05.050195 28394 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.082s
I20260809 23:19:05.167446 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:05.167991 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:05.124958 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:05.013657 28391 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:05.125480 28481 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.236s	user 0.000s	sys 0.003s
W20260809 23:19:05.065994 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61522.2us
I20260809 23:19:05.045984 28410 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:05.167178 28344 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:05.100837 28358 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:19:05.051959 28422 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:05.116567 28178 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:05.050486 28255 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:05.159207 28463 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:05.085086 28336 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:05.192035 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.192771 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:05.199167 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:05.165086 28505 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:05.203600 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 166862us
I20260809 23:19:05.204082 25321 leader_election.cc:398] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3.
I20260809 23:19:05.218578 25321 leader_election.cc:401] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:05.050313 28480 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.134s
I20260809 23:19:05.134127 28359 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
W20260809 23:19:05.221323 24498 connection.cc:232] RPC call timeout handler was delayed by -1.7030836341436952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2522s for a response
W20260809 23:19:05.222491 24498 connection.cc:232] RPC call timeout handler was delayed by -1.6888714770320803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260809 23:19:05.224135 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.542s (SENT)
W20260809 23:19:05.225087 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.884s (SENT)
I20260809 23:19:05.227286 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:05.184333 28435 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:05.231467 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:05.218107 28463 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:05.217801 28335 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:05.241925 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:05.242049 28516 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:05.217898 28358 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:05.242174 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:05.242863 28336 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:05.214532 28422 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:05.242956 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.843333 24316 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:05.314589 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.640s (SENT)
W20260809 23:19:05.844264 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 658412us
W20260809 23:19:05.844511 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 529914us
W20260809 23:19:05.828730 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.009s	sys 0.008s
W20260809 23:19:05.766166 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.011s	sys 0.032s
W20260809 23:19:05.845717 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.011s	sys 0.038s
I20260809 23:19:05.254679 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2355) took 1099 ms. Trace:
W20260809 23:19:05.736987 24425 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.002s
I20260809 23:19:05.273710 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.301178 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:05.857795 24425 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:19:05.842908 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 617786us
I20260809 23:19:05.575014 25373 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:19:05.851953 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 585) took 1106 ms. Trace:
I20260809 23:19:05.858433 24314 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:05.859090 25373 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:05.859373 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.244s (SENT)
I20260809 23:19:05.846659 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:05.855705 24435 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260809 23:19:05.860018 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:05.860298 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.005s (SENT)
I20260809 23:19:05.846513 24317 rpcz_store.cc:276] 0809 23:19:04.155511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.156277 (+   766us) service_pool.cc:227] Handling call
0809 23:19:05.254496 (+1098219us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b54003c32d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:05.254612 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55280000}
I20260809 23:19:05.858354 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:05.749831 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:05.845254 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.009s	sys 0.009s
W20260809 23:19:05.863054 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 827995us
W20260809 23:19:05.293944 28461 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.758s	user 0.001s	sys 0.000s
W20260809 23:19:05.455487 28396 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.712s	user 0.006s	sys 0.005s
W20260809 23:19:05.262804 28512 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260809 23:19:05.261299 28493 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
W20260809 23:19:05.264439 28444 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.977s	user 0.004s	sys 0.000s
I20260809 23:19:05.844762 25370 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260809 23:19:05.783638 28259 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.007s	sys 0.010s
I20260809 23:19:05.859226 24544 rpcz_store.cc:276] 0809 23:19:04.745505 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.745649 (+   144us) service_pool.cc:227] Handling call
0809 23:19:04.745961 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.858081 (+112120us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.858200 (+   119us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.896248 (+ 38048us) log.cc:842] Serialized 26 byte log entry
0809 23:19:05.077447 (+181199us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:05.077498 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:05.077590 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:05.266804 (+189214us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:05.843877 (+577073us) raft_consensus.cc:1671] finished
0809 23:19:05.843897 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:05.851883 (+  7986us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":16822,"raft.run_cpu_time_us":1946,"raft.run_wall_time_us":496867,"spinlock_wait_cycles":31328000,"thread_start_us":268229,"threads_started":2,"wal-append.queue_time_us":329984}
I20260809 23:19:05.871563 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:05.274673 28398 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.941s	user 0.013s	sys 0.007s
W20260809 23:19:05.262181 28500 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260809 23:19:05.873605 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:05.874447 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:05.239948 28255 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 1 FOLLOWER]: Advancing to term 3
I20260809 23:19:05.444780 28156 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:05.841997 28528 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:05.258590 28507 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260809 23:19:05.874151 25373 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:05.879383 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:05.881146 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.977716 24317 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:05.265246 28437 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.824s	user 0.015s	sys 0.007s
W20260809 23:19:05.855410 28432 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.936s	user 0.011s	sys 0.000s
I20260809 23:19:05.266033 28197 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:05.746770 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:05.886514 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:05.255517 28495 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:05.888345 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:05.843199 25363 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:05.855201 28278 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:05.261791 28410 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:05.903160 24314 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 3: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:05.997598 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:05.964550 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90189.1us
I20260809 23:19:05.293900 28237 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:05.971155 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 109782us
W20260809 23:19:06.002388 24498 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.062s (SENT)
W20260809 23:19:05.904158 28259 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.010s	sys 0.011s
W20260809 23:19:05.923074 28501 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.001s	sys 0.027s
I20260809 23:19:05.261162 28510 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.848s
I20260809 23:19:05.999783 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:06.007898 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.002414 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.008685 24316 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:06.009171 24314 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:06.002408 25366 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:05.956849 28398 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:05.975129 27633 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:06.020397 28463 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.021958 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.024175 24314 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:05.232839 28505 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:06.009631 28419 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.535s	user 0.013s	sys 0.011s
W20260809 23:19:06.007880 28501 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.001s	sys 0.028s
I20260809 23:19:06.023165 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:06.022912 24498 connection.cc:232] RPC call timeout handler was delayed by -1.50058363401331s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2747s for a response
I20260809 23:19:06.025496 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2354) took 1870 ms. Trace:
W20260809 23:19:06.029657 24498 connection.cc:232] RPC call timeout handler was delayed by -1.3309919938910753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260809 23:19:06.024860 24429 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:06.030105 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0860229020472616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2897s for a response
I20260809 23:19:06.029707 24313 rpcz_store.cc:276] 0809 23:19:04.155191 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.156195 (+  1004us) service_pool.cc:227] Handling call
0809 23:19:04.208696 (+ 52501us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.633309 (+424613us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.633363 (+    54us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.634077 (+   714us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.855254 (+221177us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:04.855301 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:04.855403 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:06.025304 (+1169901us) raft_consensus.cc:1671] finished
0809 23:19:06.025322 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.025440 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7023,"raft.queue_time_us":55807,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":30492,"thread_start_us":546680,"threads_started":2,"wal-append.queue_time_us":282814}
I20260809 23:19:05.979895 28077 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260809 23:19:05.976465 28156 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:06.031031 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:05.854133 28516 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.032579 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2174) took 2496 ms. Trace:
W20260809 23:19:06.071058 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102289us
I20260809 23:19:06.033969 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:06.032567 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2146) took 1900 ms. Trace:
I20260809 23:19:06.023782 27633 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:06.031644 28472 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260809 23:19:05.886132 28396 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.024014 28335 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.075026 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.025144 28534 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:05.984179 28437 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:06.081300 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50365.5us
I20260809 23:19:06.083056 24314 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:05.989477 28432 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.071190 24545 rpcz_store.cc:276] 0809 23:19:03.535682 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:03.535874 (+   192us) service_pool.cc:227] Handling call
0809 23:19:03.545555 (+  9681us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:03.914448 (+368893us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:03.914479 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:03.914946 (+   467us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.007068 (+ 92122us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:19:04.007105 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:04.007176 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:05.263718 (+1256542us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:06.026897 (+763179us) raft_consensus.cc:1671] finished
0809 23:19:06.026921 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.032523 (+  5602us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":350372,"raft.queue_time_us":351778,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":18521472,"thread_start_us":91704,"threads_started":1,"wal-append.queue_time_us":296646}
I20260809 23:19:06.004609 28278 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:06.091863 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2331) took 1757 ms. Trace:
W20260809 23:19:06.095352 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 513) took 1907 ms (client timeout 1776 ms). Trace:
I20260809 23:19:06.141179 24427 rpcz_store.cc:276] 0809 23:19:04.334243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.334422 (+   179us) service_pool.cc:227] Handling call
0809 23:19:04.455313 (+120891us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:04.551457 (+ 96144us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.091571 (+1540114us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:06.091791 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73630592}
I20260809 23:19:06.098138 28463 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:06.099255 28532 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260809 23:19:06.033552 28419 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 23:19:06.100028 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 515) took 1828 ms (client timeout 2226 ms). Trace:
W20260809 23:19:06.155287 24422 rpcz_store.cc:269] 0809 23:19:04.271316 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.271994 (+   678us) service_pool.cc:227] Handling call
0809 23:19:06.099811 (+1827817us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:06.099959 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11012352}
I20260809 23:19:06.102996 28425 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:06.158896 24323 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:06.103227 28422 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:06.079321 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 205834us
W20260809 23:19:06.159168 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 136071us
I20260809 23:19:06.109639 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.080649 24428 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260809 23:19:06.129410 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2327) took 1979 ms. Trace:
I20260809 23:19:06.084636 28543 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.131210 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 728) took 1106 ms. Trace:
W20260809 23:19:06.132072 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132373us
I20260809 23:19:06.072767 24539 rpcz_store.cc:276] 0809 23:19:04.131776 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.131930 (+   154us) service_pool.cc:227] Handling call
0809 23:19:04.132255 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.141074 (+  8819us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.141113 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.141568 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.266016 (+124448us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:04.266055 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:04.266119 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:05.293245 (+1027126us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:06.024787 (+731542us) raft_consensus.cc:1671] finished
0809 23:19:06.024820 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.032504 (+  7684us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9509,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":33382272,"thread_start_us":124003,"threads_started":1,"wal-append.queue_time_us":309557}
W20260809 23:19:06.141273 24425 rpcz_store.cc:269] 0809 23:19:04.187570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.187710 (+   140us) service_pool.cc:227] Handling call
0809 23:19:06.095121 (+1907411us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:06.095254 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857130240}
I20260809 23:19:06.095935 28156 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:06.144907 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71406.4us
I20260809 23:19:05.990411 28358 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.079775 27633 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:06.198748 24430 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:06.087616 28501 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:05.990134 28528 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:06.138121 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.006s	sys 0.009s
I20260809 23:19:06.209733 24427 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:06.210022 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.006s	sys 0.010s
I20260809 23:19:06.210379 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2336) took 1163 ms. Trace:
I20260809 23:19:06.210704 24427 rpcz_store.cc:276] 0809 23:19:05.046593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.154707 (+1108114us) service_pool.cc:227] Handling call
0809 23:19:06.209571 (+ 54864us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.210159 (+   588us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.210328 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:06.198729 24321 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:06.213260 28555 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260809 23:19:06.171674 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96915.6us
I20260809 23:19:06.151667 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:06.200250 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:06.215660 24313 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260809 23:19:06.216743 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.002506 28495 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:06.205981 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.000s	sys 0.047s
I20260809 23:19:06.207226 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:06.202194 28510 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260809 23:19:06.144572 28083 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:06.215600 28555 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:06.226977 28555 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:06.226884 28083 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:06.162132 24314 rpcz_store.cc:276] 0809 23:19:05.024533 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.021668 (+997135us) service_pool.cc:227] Handling call
0809 23:19:06.131131 (+109463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:06.228010 28083 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:06.228469 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.229079 24314 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5. Re-sending same reply.
I20260809 23:19:06.229910 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:06.231411 24314 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:06.231833 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 732) took 1206 ms. Trace:
I20260809 23:19:06.217337 28560 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.232218 24314 rpcz_store.cc:276] 0809 23:19:05.027339 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.229659 (+1202320us) service_pool.cc:227] Handling call
0809 23:19:06.231785 (+  2126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:06.232539 28560 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:06.233460 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:06.235189 24314 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
W20260809 23:19:06.235498 28540 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260809 23:19:06.220101 28538 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:06.218760 28197 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:06.158789 28375 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.227515 28555 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:06.236925 28375 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.160166 24424 rpcz_store.cc:276] 0809 23:19:04.150471 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.150650 (+   179us) service_pool.cc:227] Handling call
0809 23:19:04.151053 (+   403us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.164952 (+ 13899us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.164981 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.165432 (+   451us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.243784 (+ 78352us) spinlock_profiling.cc:243] Waited 4.45 ms on lock 0x7b48001701d8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:04.937664 (+693880us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:04.937701 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:04.937780 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:06.067712 (+1129932us) raft_consensus.cc:1671] finished
0809 23:19:06.067733 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.129306 (+ 61573us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13444,"raft.queue_time_us":36028,"raft.run_cpu_time_us":1395,"raft.run_wall_time_us":39691,"spinlock_wait_cycles":10226432,"thread_start_us":729020,"threads_started":1,"wal-append.queue_time_us":825566}
I20260809 23:19:06.235771 28560 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:06.217837 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:06.218312 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.000s	sys 0.047s
I20260809 23:19:06.220494 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:06.142733 28344 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:06.235532 24314 pending_rounds.cc:76] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:06.265882 24314 pending_rounds.cc:97] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260809 23:19:06.146080 28549 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:06.266178 24314 raft_consensus.cc:2866] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:06.269575 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49134.3us
I20260809 23:19:06.274058 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:06.270506 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70354.9us
I20260809 23:19:06.218318 24317 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:06.275105 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 738) took 1189 ms. Trace:
I20260809 23:19:06.163009 28425 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:06.130451 28335 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.275444 24317 rpcz_store.cc:276] 0809 23:19:05.085667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.870352 (+784685us) service_pool.cc:227] Handling call
0809 23:19:06.275046 (+404694us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:06.242214 28513 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.061s	user 0.003s	sys 0.006s
I20260809 23:19:06.284209 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:06.284413 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 736) took 1238 ms. Trace:
I20260809 23:19:06.290285 24314 rpcz_store.cc:276] 0809 23:19:05.045755 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.234581 (+1188826us) service_pool.cc:227] Handling call
0809 23:19:06.234827 (+   246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.284170 (+ 49343us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.284361 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:06.293100 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 149978us
I20260809 23:19:06.294170 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:06.245409 28171 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:06.238914 28375 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.243603 28490 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:06.290428 28442 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.168s	user 0.014s	sys 0.029s
W20260809 23:19:06.310902 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.001s	sys 0.013s
I20260809 23:19:06.306674 28171 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:06.217020 25366 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:06.219551 28156 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260809 23:19:06.273283 28549 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:06.283351 28197 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:06.283540 28492 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:06.327536 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.005s	sys 0.013s
W20260809 23:19:06.327556 28442 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.212s	user 0.016s	sys 0.029s
I20260809 23:19:06.327620 28490 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.313467 28237 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:06.329882 28474 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.295s	user 0.004s	sys 0.007s
I20260809 23:19:06.331260 28432 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.331741 28523 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:06.306850 28344 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260809 23:19:06.335098 28561 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260809 23:19:06.245129 28501 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.326315 28375 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:06.328372 27689 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260809 23:19:06.339259 28398 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.340718 27633 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.336279 28565 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election won for term 4
I20260809 23:19:06.292735 28077 consensus_queue.cc:1225] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260809 23:19:06.353632 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 148893us
I20260809 23:19:06.145074 28534 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:06.336635 28436 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.567s	user 0.013s	sys 0.000s
I20260809 23:19:06.387216 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 586) took 1514 ms. Trace:
W20260809 23:19:06.389663 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 108746us
I20260809 23:19:06.353796 28549 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:06.398404 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:06.401232 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.1247e+06us
I20260809 23:19:06.407605 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:06.401930 24542 rpcz_store.cc:276] 0809 23:19:04.872405 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.872589 (+   184us) service_pool.cc:227] Handling call
0809 23:19:04.873006 (+   417us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.873417 (+   411us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.873455 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.873898 (+   443us) log.cc:842] Serialized 26 byte log entry
0809 23:19:05.003696 (+129798us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:05.003738 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:05.013169 (+  9431us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:06.241512 (+1228343us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:06.386835 (+145323us) raft_consensus.cc:1671] finished
0809 23:19:06.386866 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.387136 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":972,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":13296,"spinlock_wait_cycles":6524160,"thread_start_us":129488,"threads_started":1,"wal-append.queue_time_us":247556}
I20260809 23:19:06.444254 24539 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:06.206035 28543 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:06.394992 28375 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:06.383843 28492 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:06.385430 28521 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:06.480026 28415 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:06.487529 28511 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.182s	user 0.012s	sys 0.025s
I20260809 23:19:06.487709 28415 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.491364 24425 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:06.495944 24544 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:06.494839 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:06.497380 24544 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:06.501789 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:06.510612 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:06.533886 24539 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:06.536767 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:06.479300 28375 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:06.540098 24380 connection.cc:232] RPC call timeout handler was delayed by -1.1200079231057316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260809 23:19:06.542734 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 258578us
I20260809 23:19:06.407989 28495 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:06.543951 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:06.450074 28538 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:06.545322 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2362) took 1448 ms. Trace:
I20260809 23:19:06.553646 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:06.627599 24534 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:06.626791 24317 rpcz_store.cc:276] 0809 23:19:05.096735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.287522 (+1190787us) service_pool.cc:227] Handling call
0809 23:19:06.287935 (+   413us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.288184 (+   249us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:06.288477 (+   293us) log.cc:842] Serialized 14 byte log entry
0809 23:19:06.544876 (+256399us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:06.544946 (+    70us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:06.545026 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.545111 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.545250 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":107,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"thread_start_us":256003,"threads_started":1}
W20260809 23:19:06.571571 24378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.610s (SENT) [suppressed 23 similar messages]
I20260809 23:19:06.628650 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:06.629185 24317 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:06.504501 28521 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:06.637844 28521 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:06.499348 28437 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.388055 28474 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.644376 24425 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260809 23:19:06.644902 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 185412us
I20260809 23:19:06.626547 24424 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:06.642472 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.655053 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:06.655131 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 361035us
I20260809 23:19:06.655298 24427 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
W20260809 23:19:06.652056 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108216us
I20260809 23:19:06.469717 28565 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:06.657104 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:06.662971 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119607us
W20260809 23:19:06.661300 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 89786.1us
I20260809 23:19:06.630890 28480 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:06.531450 28510 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.631168 28410 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.635856 28419 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.631700 28516 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.675192 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.636518 28422 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.634912 28156 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260809 23:19:06.674489 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2337) took 1550 ms. Trace:
I20260809 23:19:06.504794 28083 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.564064 28278 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:06.637156 28505 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:06.677718 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:06.642985 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:06.650758 28237 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.681838 24539 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:06.682094 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:06.449558 28436 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.448625 28171 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.714236 24317 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:06.741338 24424 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:06.742684 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:06.740677 24422 rpcz_store.cc:276] 0809 23:19:05.123998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.169332 (+1045334us) service_pool.cc:227] Handling call
0809 23:19:06.169731 (+   399us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.169972 (+   241us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:06.352098 (+182126us) log.cc:842] Serialized 14 byte log entry
0809 23:19:06.674166 (+322068us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:06.674213 (+    47us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:06.674257 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.674319 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:06.674428 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81264,"prepare.queue_time_us":169875,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":360781,"threads_started":3,"wal-append.queue_time_us":387271}
I20260809 23:19:06.682237 28432 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.692552 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.684988 28415 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.709895 25365 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:06.793967 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:06.684734 28486 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.643s	user 0.014s	sys 0.000s
W20260809 23:19:06.738297 28571 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260809 23:19:06.678612 28437 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260809 23:19:06.830477 24313 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:06.687027 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:06.715022 28335 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:06.736532 28083 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:06.683347 28510 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.688889 28419 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.685210 28410 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:06.790342 25366 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:06.803507 28237 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:06.877599 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190359us
I20260809 23:19:07.037822 25371 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:06.793583 28505 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:06.861244 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.877703 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:07.042714 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:06.792229 28504 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.038618 24424 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:07.056485 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.692961 28492 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.021066 28566 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:06.691748 28422 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.199465 28566 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.052042 24425 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:07.157142 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:07.202500 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45490.8us
I20260809 23:19:07.071557 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:07.203769 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:07.079942 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 722) took 2068 ms. Trace:
W20260809 23:19:06.877960 28582 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
W20260809 23:19:07.062700 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 405608us
I20260809 23:19:07.214915 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.874670 28539 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:07.216362 28535 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.149s	user 0.001s	sys 0.009s
I20260809 23:19:07.216650 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:07.216754 28539 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.217476 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 723) took 2205 ms. Trace:
I20260809 23:19:07.217801 28345 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:06.839468 28486 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.071666 28348 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:07.021636 28415 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.846235 28341 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:07.198709 28171 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:06.893013 28565 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.073663 28335 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.219550 24425 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:07.217865 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 816) took 3057 ms. Trace:
I20260809 23:19:06.877377 27689 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:07.217876 24312 rpcz_store.cc:276] 0809 23:19:05.011898 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.012013 (+   115us) service_pool.cc:227] Handling call
0809 23:19:05.025918 (+ 13905us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:05.218693 (+192775us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:05.218736 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:05.222466 (+  3730us) log.cc:842] Serialized 26 byte log entry
0809 23:19:05.846427 (+623961us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:05.846482 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:05.846572 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:07.215704 (+1369132us) spinlock_profiling.cc:243] Waited 8.72 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:07.217202 (+  1498us) raft_consensus.cc:1671] finished
0809 23:19:07.217220 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.217421 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":60461,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":14707,"spinlock_wait_cycles":20049792,"thread_start_us":789906,"threads_started":2,"wal-append.queue_time_us":757659}
I20260809 23:19:07.164450 28528 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.230208 24317 pending_rounds.cc:76] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 7
I20260809 23:19:07.041311 28501 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.232251 28566 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.232890 28560 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:07.071803 28229 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.059700 28505 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:07.073098 28504 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:07.205597 28495 consensus_queue.cc:1159] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 7 status { last_received { term: 6 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:07.234742 28475 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.234867 28580 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.941391 28523 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.368s
W20260809 23:19:07.240123 28499 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.081s	user 0.003s	sys 0.017s
I20260809 23:19:07.243579 28593 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:07.244392 28391 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:07.244956 28486 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:07.242566 28555 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.005s	user 0.002s	sys 0.010s
I20260809 23:19:07.176677 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:06.803368 28432 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.021405 28516 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:07.441295 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 721) took 2430 ms. Trace:
I20260809 23:19:07.443439 24321 rpcz_store.cc:276] 0809 23:19:05.011232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.011403 (+   171us) service_pool.cc:227] Handling call
0809 23:19:05.034045 (+ 22642us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.217977 (+1183932us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.440983 (+1223006us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:07.441220 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2603127168}
W20260809 23:19:07.365834 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1896) took 2409 ms (client timeout 3158 ms). Trace:
W20260809 23:19:07.444587 24429 rpcz_store.cc:269] 0809 23:19:04.956874 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.961447 (+  4573us) service_pool.cc:227] Handling call
0809 23:19:07.365586 (+2404139us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:07.365738 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2427511552}
I20260809 23:19:07.251724 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:06.983901 27633 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.254110 24317 pending_rounds.cc:97] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260809 23:19:07.446696 24427 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:07.446965 24317 raft_consensus.cc:2866] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:07.075510 28581 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.236912 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2329) took 2903 ms. Trace:
I20260809 23:19:07.461599 24428 rpcz_store.cc:276] 0809 23:19:04.333528 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.335648 (+  2120us) service_pool.cc:227] Handling call
0809 23:19:04.336093 (+   445us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:06.163187 (+1827094us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.236561 (+1073374us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:07.236835 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3905648896}
I20260809 23:19:07.463650 24428 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:06.742666 28575 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:07.249833 25375 rpcz_store.cc:276] 0809 23:19:04.160535 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.203925 (+ 43390us) service_pool.cc:227] Handling call
0809 23:19:04.204268 (+   343us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.268515 (+ 64247us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.268547 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.269075 (+   528us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.548234 (+279159us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:04.633381 (+ 85147us) log.cc:842] Serialized 14 byte log entry
0809 23:19:05.270102 (+636721us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b18001e3fa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:07.217386 (+1947284us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b18001e3fa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7
0809 23:19:07.217507 (+   121us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.217596 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:07.217650 (+    54us) raft_consensus.cc:1671] finished
0809 23:19:07.217664 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.217807 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115221,"prepare.queue_time_us":240476,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":109016,"raft.run_cpu_time_us":527,"raft.run_wall_time_us":40576,"spinlock_wait_cycles":19196160,"thread_start_us":404209,"threads_started":3,"wal-append.queue_time_us":409640,"wal-append.run_cpu_time_us":3721,"wal-append.run_wall_time_us":1826469}
I20260809 23:19:07.247175 28474 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.478055 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:07.457141 28229 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.457948 28505 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:07.362848 28422 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.458645 25373 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:19:07.491954 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 276311us
I20260809 23:19:07.440042 28471 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.460014 24422 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:07.460290 28504 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.229559 28341 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:07.255833 28543 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:07.476148 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 414170us
W20260809 23:19:07.285084 28375 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.649s	user 0.008s	sys 0.002s
W20260809 23:19:07.390022 28425 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.093s	user 0.013s	sys 0.000s
W20260809 23:19:07.283010 28573 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260809 23:19:07.483108 24428 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:07.250358 28415 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.486171 25371 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260809 23:19:07.287853 28528 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:06.941916 28420 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.075012 28436 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.495128 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:07.307936 28419 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:07.496809 28465 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.748s	user 0.000s	sys 0.021s
I20260809 23:19:07.283700 28592 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.458673 28566 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.498307 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.565452 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.439556 28534 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.204917 24309 rpcz_store.cc:276] 0809 23:19:05.011593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.011716 (+   123us) service_pool.cc:227] Handling call
0809 23:19:05.026149 (+ 14433us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:05.237229 (+211080us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:05.237263 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:05.238035 (+   772us) log.cc:842] Serialized 26 byte log entry
0809 23:19:06.355161 (+1117126us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:06.355200 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.355269 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:07.079678 (+724409us) raft_consensus.cc:1671] finished
0809 23:19:07.079715 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.079871 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":80043,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":7968,"thread_start_us":943645,"threads_started":2,"wal-append.queue_time_us":861099}
W20260809 23:19:07.492108 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.026s	sys 0.027s
W20260809 23:19:07.584429 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.027s	sys 0.027s
I20260809 23:19:07.253172 28490 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.336800 25366 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:07.402474 28539 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.460886 28396 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:07.492658 28521 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.854s	user 0.012s	sys 0.000s
I20260809 23:19:07.485450 28083 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260809 23:19:07.479388 28593 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:07.486100 25369 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:07.460743 28560 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.520442 25372 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:19:07.254662 28335 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.526518 25373 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:19:07.529011 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:07.535050 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:07.615410 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 819) took 2308 ms. Trace:
I20260809 23:19:07.552986 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:07.615882 25373 rpcz_store.cc:276] 0809 23:19:05.307036 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.428746 (+121710us) service_pool.cc:227] Handling call
0809 23:19:05.574820 (+146074us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.458242 (+1883422us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400520ba0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237ee40d 00007fac237dc050 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9
0809 23:19:07.615112 (+156870us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.615348 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":961678592}
I20260809 23:19:07.506325 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.617244 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.253511 28348 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.514716 28543 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:07.491750 28516 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.467074 28555 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:07.582749 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.008s	sys 0.033s
I20260809 23:19:07.530581 28510 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.605259 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:07.622745 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.009s	sys 0.033s
I20260809 23:19:07.609591 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:07.489615 28475 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.474720 28580 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.509948 28375 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.619139 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.513290 28425 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.513602 27689 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260809 23:19:07.627048 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:07.486994 28575 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:07.605489 28573 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260809 23:19:07.623795 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 584996us
I20260809 23:19:07.612185 28593 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.625618 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:07.628108 25370 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:19:07.633131 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.644933 25375 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260809 23:19:07.645368 25372 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
W20260809 23:19:07.622381 28255 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.648s	user 0.009s	sys 0.005s
I20260809 23:19:07.636801 28425 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.640264 28335 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.509850 28471 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.546078 28420 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.643561 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:07.645393 25371 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:07.588986 28566 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:07.634058 28549 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.161s	user 0.012s	sys 0.000s
I20260809 23:19:07.661578 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:07.613368 28539 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:07.553960 28528 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.650305 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2149) took 3066 ms. Trace:
I20260809 23:19:07.583343 28592 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.650988 24314 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:07.652418 24540 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:07.621218 28523 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.612761 28415 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:07.654699 28598 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
I20260809 23:19:07.654914 28432 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.655010 28603 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.651525 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 511) took 3572 ms. Trace:
I20260809 23:19:07.623027 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 821) took 1113 ms. Trace:
I20260809 23:19:07.658118 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:07.481478 28581 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.663614 24535 rpcz_store.cc:276] 0809 23:19:04.584349 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.584518 (+   169us) service_pool.cc:227] Handling call
0809 23:19:04.584910 (+   392us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.638363 (+ 53453us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:04.667051 (+ 28688us) log.cc:842] Serialized 14 byte log entry
0809 23:19:04.934966 (+267915us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.939662 (+  4696us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.939792 (+   130us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:04.939821 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:04.939885 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:07.239326 (+2299441us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:07.625625 (+386299us) raft_consensus.cc:1671] finished
0809 23:19:07.650103 (+ 24478us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.650246 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55973,"prepare.queue_time_us":33225,"prepare.run_cpu_time_us":6140,"prepare.run_wall_time_us":29159,"raft.queue_time_us":60379,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":16509,"spinlock_wait_cycles":34257792,"thread_start_us":161827,"threads_started":1,"wal-append.queue_time_us":342078}
I20260809 23:19:07.657199 28255 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3.
I20260809 23:19:07.603143 28341 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.614344 28521 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.707576 24321 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 2. Current term is 3. Ops: []
W20260809 23:19:07.853111 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.187s	user 0.018s	sys 0.057s
I20260809 23:19:07.741919 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2333) took 3173 ms. Trace:
W20260809 23:19:07.947119 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.281s	user 0.018s	sys 0.064s
I20260809 23:19:07.947934 24430 rpcz_store.cc:276] 0809 23:19:04.568438 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.568583 (+   145us) service_pool.cc:227] Handling call
0809 23:19:04.568855 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:06.202145 (+1633290us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.732172 (+1530027us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:07.741834 (+  9662us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3666553088}
I20260809 23:19:07.684052 24425 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:07.944952 28560 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.959287 28555 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.960671 28178 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.944510 27689 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.964811 28575 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.970422 28420 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.781910 28471 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:07.961656 28585 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.157s	user 0.015s	sys 0.000s
I20260809 23:19:07.740089 28566 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:07.972714 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 735) took 2939 ms (client timeout 3223 ms). Trace:
I20260809 23:19:07.971901 28178 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:07.812891 28358 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.010s	sys 0.017s
I20260809 23:19:07.951833 28528 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:07.973394 28358 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.013s	sys 0.019s
W20260809 23:19:07.973191 24313 rpcz_store.cc:269] 0809 23:19:05.045440 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.217612 (+1172172us) service_pool.cc:227] Handling call
0809 23:19:07.972535 (+1754923us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:07.972660 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1336097152}
I20260809 23:19:07.965119 28580 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.974479 28178 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:07.951269 28272 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:07.976120 28272 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:07.974565 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:07.971868 27689 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:07.968665 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 726) took 2944 ms (client timeout 2487 ms). Trace:
W20260809 23:19:07.980600 24323 rpcz_store.cc:269] 0809 23:19:05.023995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.871308 (+847313us) service_pool.cc:227] Handling call
0809 23:19:07.968353 (+2097045us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:07.968548 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401992960}
I20260809 23:19:07.981724 28197 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:07.898782 28592 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.982134 28197 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:07.982532 27689 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.979928 24313 raft_consensus.cc:2362] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6. Re-sending same reply.
I20260809 23:19:07.981648 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:07.985184 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 743) took 2799 ms. Trace:
I20260809 23:19:07.986063 24313 rpcz_store.cc:276] 0809 23:19:05.185417 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.984340 (+2798923us) service_pool.cc:227] Handling call
0809 23:19:07.984718 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.984896 (+   178us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.984920 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:07.984958 (+    38us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:07.984993 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.985041 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.985142 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:07.988787 24313 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260809 23:19:07.813182 28416 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.178s	user 0.038s	sys 0.043s
I20260809 23:19:07.676036 28475 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.978770 28272 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.983951 28197 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.949955 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:08.158269 28558 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.487s	user 0.014s	sys 0.009s
I20260809 23:19:08.219321 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 744) took 3006 ms. Trace:
I20260809 23:19:08.219875 24313 rpcz_store.cc:276] 0809 23:19:05.212840 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.988051 (+2775211us) service_pool.cc:227] Handling call
0809 23:19:07.988398 (+   347us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:08.218900 (+230502us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.219154 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:08.221285 24313 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
W20260809 23:19:08.176389 28496 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.020s	sys 0.020s
I20260809 23:19:07.781654 28516 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:08.105954 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.225s	user 0.015s	sys 0.046s
I20260809 23:19:07.741250 28171 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.506s
W20260809 23:19:08.249840 28602 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.400s	user 0.011s	sys 0.000s
W20260809 23:19:08.190541 28590 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.014s	user 0.001s	sys 0.023s
W20260809 23:19:08.251327 28590 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.085s	user 0.001s	sys 0.024s
I20260809 23:19:08.252146 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2187) took 1759 ms. Trace:
I20260809 23:19:08.249002 28171 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:08.224093 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:08.100473 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.005s	sys 0.022s
W20260809 23:19:08.287484 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.428s	user 0.009s	sys 0.024s
W20260809 23:19:08.158042 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.007s	sys 0.009s
W20260809 23:19:08.288702 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.008s	sys 0.010s
W20260809 23:19:08.224962 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 232474us
I20260809 23:19:07.975298 28395 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:08.102039 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.294656 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.296335 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:07.945297 24431 rpcz_store.cc:276] 0809 23:19:04.079465 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.079603 (+   138us) service_pool.cc:227] Handling call
0809 23:19:04.113103 (+ 33500us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.128190 (+ 15087us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.128251 (+    61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:04.128766 (+   515us) log.cc:842] Serialized 26 byte log entry
0809 23:19:04.496497 (+367731us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:04.496533 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:05.294965 (+798432us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:05.295015 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:07.495973 (+2200958us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:07.651313 (+155340us) raft_consensus.cc:1671] finished
0809 23:19:07.651337 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:07.651468 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":57434240,"thread_start_us":342971,"threads_started":1,"wal-append.queue_time_us":481929}
W20260809 23:19:08.157981 28159 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.006s	sys 0.015s
W20260809 23:19:08.221089 28416 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.589s	user 0.048s	sys 0.066s
W20260809 23:19:08.228857 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2358) took 4070 ms (client timeout 1610 ms). Trace:
I20260809 23:19:08.047444 28592 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:07.782761 28348 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:08.158151 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.953s	user 0.000s	sys 0.025s
W20260809 23:19:08.308596 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.104s	user 0.002s	sys 0.025s
I20260809 23:19:07.679837 28375 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:08.024499 28505 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.000s
I20260809 23:19:08.005067 28581 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:07.671245 28549 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:07.707576 28229 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:08.157763 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.958s	user 0.001s	sys 0.019s
W20260809 23:19:08.247138 28496 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.024s	sys 0.024s
I20260809 23:19:07.989900 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:08.247875 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.367s	user 0.016s	sys 0.050s
I20260809 23:19:08.305313 28516 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.033685 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1899) took 3012 ms. Trace:
W20260809 23:19:08.219471 28558 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.548s	user 0.017s	sys 0.011s
I20260809 23:19:07.678076 28523 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.252600 24545 rpcz_store.cc:276] 0809 23:19:06.493139 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.495095 (+  1956us) service_pool.cc:227] Handling call
0809 23:19:06.495398 (+   303us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:06.537718 (+ 42320us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:06.537764 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:06.662474 (+124710us) log.cc:842] Serialized 26 byte log entry
0809 23:19:07.585291 (+922817us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:19:07.585356 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.585454 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:08.251887 (+666433us) raft_consensus.cc:1671] finished
0809 23:19:08.251907 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:08.252074 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":93706,"spinlock_wait_cycles":14720,"thread_start_us":892548,"threads_started":1,"wal-append.queue_time_us":1171599}
W20260809 23:19:08.030642 28583 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.085s	user 0.002s	sys 0.000s
I20260809 23:19:08.259613 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2338) took 1720 ms. Trace:
W20260809 23:19:08.259693 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2344) took 1569 ms (client timeout 1805 ms). Trace:
I20260809 23:19:08.265065 28237 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:08.268923 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 742) took 3127 ms. Trace:
I20260809 23:19:07.952896 28391 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.004484 24323 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
W20260809 23:19:08.297775 24311 rpcz_store.cc:269] 0809 23:19:04.158914 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.159101 (+   187us) service_pool.cc:227] Handling call
0809 23:19:08.228689 (+4069588us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:08.228791 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135201664}
I20260809 23:19:08.231359 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:08.369129 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 379325us
W20260809 23:19:08.309367 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.10577e+06us
W20260809 23:19:08.339922 24541 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.002s	sys 0.021s
W20260809 23:19:08.302532 28159 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.006s	sys 0.020s
I20260809 23:19:08.283246 28478 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:19:08.324759 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.010s	sys 0.014s
I20260809 23:19:08.386157 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 745) took 3157 ms. Trace:
I20260809 23:19:08.245373 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 748) took 2800 ms. Trace:
W20260809 23:19:08.104218 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.309s	user 0.015s	sys 0.015s
W20260809 23:19:08.386375 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.010s	sys 0.014s
I20260809 23:19:08.334343 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:08.319799 28492 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.013s	sys 0.016s
I20260809 23:19:08.386780 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:08.386806 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.592s	user 0.019s	sys 0.016s
W20260809 23:19:08.334627 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.135s	user 0.003s	sys 0.020s
W20260809 23:19:08.398521 24498 connection.cc:232] RPC call timeout handler was delayed by -1.3376197889447212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20190000000000002s for a response
W20260809 23:19:08.399142 24498 connection.cc:232] RPC call timeout handler was delayed by -1.2341197889763862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2134s for a response
I20260809 23:19:08.400514 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:08.341969 28065 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.549s	user 0.017s	sys 0.059s
I20260809 23:19:08.342494 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.373313 24535 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:08.402874 24545 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:08.403959 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:08.373974 24541 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.003s	sys 0.021s
I20260809 23:19:08.343102 24425 rpcz_store.cc:276] 0809 23:19:06.539759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.253735 (+713976us) service_pool.cc:227] Handling call
0809 23:19:07.338701 (+ 84966us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.986117 (+647416us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.259389 (+273272us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:08.259556 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":793186560}
I20260809 23:19:08.405901 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:08.406745 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:08.408861 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:08.410452 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:08.343652 24321 rpcz_store.cc:276] 0809 23:19:05.159593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.444540 (+2284947us) service_pool.cc:227] Handling call
0809 23:19:07.487000 (+ 42460us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.945608 (+458608us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.268576 (+322968us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:08.268849 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":506839936}
I20260809 23:19:08.318742 28521 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:07.947041 25366 rpcz_store.cc:276] 0809 23:19:06.496672 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.528864 (+ 32192us) service_pool.cc:227] Handling call
0809 23:19:07.610099 (+1081235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:07.989639 28603 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.425818 25371 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:08.397434 28492 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.013s	sys 0.016s
I20260809 23:19:08.385869 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:08.285069 28171 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:08.386863 24313 rpcz_store.cc:276] 0809 23:19:05.449349 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.220763 (+2771414us) service_pool.cc:227] Handling call
0809 23:19:08.221124 (+   361us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:08.245060 (+ 23936us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.245304 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:08.386523 24323 rpcz_store.cc:276] 0809 23:19:05.228753 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.991678 (+2762925us) service_pool.cc:227] Handling call
0809 23:19:08.004261 (+ 12583us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:08.385869 (+381608us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.386092 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:08.387997 24431 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:08.333918 28276 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.332351 28505 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.403538 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.300678 24430 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:08.336376 28612 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.338201 24423 rpcz_store.cc:276] 0809 23:19:05.021136 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.138118 (+116982us) service_pool.cc:227] Handling call
0809 23:19:05.177653 (+ 39535us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:05.184821 (+  7168us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:19:05.213901 (+ 29080us) log.cc:842] Serialized 14 byte log entry
0809 23:19:06.240842 (+1026941us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b1800463960. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:06.339844 (+ 99002us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b1800463960. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938799 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:06.382402 (+ 42558us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b1800463960. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac27708bcf 00007fac209387b6 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d
0809 23:19:06.487743 (+105341us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:06.488363 (+   620us) log.cc:842] Serialized 26 byte log entry
0809 23:19:06.488524 (+   161us) raft_consensus.cc:1631] Marking committed up to 2
0809 23:19:06.488572 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:06.488662 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:08.033441 (+1544779us) raft_consensus.cc:1671] finished
0809 23:19:08.033471 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:08.033628 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":109128,"prepare.queue_time_us":25486,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":109676,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":7299,"spinlock_wait_cycles":29961344,"thread_start_us":143633,"threads_started":1,"wal-append.queue_time_us":1446757}
I20260809 23:19:08.414795 24316 raft_consensus.cc:2354] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9.
I20260809 23:19:08.367921 28516 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:08.337541 28589 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.000s	sys 0.035s
I20260809 23:19:08.415881 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:08.419951 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:08.426822 25373 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:08.782251 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:08.336557 28278 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:08.312707 28592 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:08.380832 28561 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.004s	sys 0.028s
I20260809 23:19:08.367481 28410 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:08.343223 24422 rpcz_store.cc:269] 0809 23:19:06.690089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.804691 (+114602us) service_pool.cc:227] Handling call
0809 23:19:08.259546 (+1454855us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:08.259639 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":939122944}
I20260809 23:19:08.852746 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:08.853824 24422 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 23:19:08.854894 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:08.855945 24422 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:08.856711 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:08.857309 24422 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:08.433983 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:08.387745 28531 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:08.402360 28495 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.406877 28394 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:08.440194 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.413489 28229 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:08.565598 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.827s	user 0.004s	sys 0.032s
I20260809 23:19:08.784921 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.785597 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2346) took 1132 ms. Trace:
I20260809 23:19:08.785805 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:08.416798 28065 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.673s	user 0.018s	sys 0.060s
I20260809 23:19:08.786434 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:08.897221 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.308s	user 0.029s	sys 0.024s
W20260809 23:19:08.897513 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.309s	user 0.029s	sys 0.024s
I20260809 23:19:08.897604 24323 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:19:08.402865 28437 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.366s	user 0.000s	sys 0.025s
I20260809 23:19:08.787189 28580 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.787079 28276 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:08.420387 28359 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.192s	user 0.007s	sys 0.016s
I20260809 23:19:08.788709 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:08.516023 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:08.427451 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:08.574975 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.008s	sys 0.028s
W20260809 23:19:08.943457 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.439s	user 0.008s	sys 0.037s
I20260809 23:19:08.784054 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:08.420938 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:08.788604 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2347) took 1066 ms. Trace:
I20260809 23:19:08.701860 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:08.830664 28609 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:08.425839 28474 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.405s
I20260809 23:19:08.830883 28595 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:08.831332 28423 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:08.830505 28255 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:08.833154 28278 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:08.386075 28391 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.835294 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:08.835592 28603 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:08.943995 28359 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.723s	user 0.009s	sys 0.021s
I20260809 23:19:08.839516 28549 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:08.495528 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.025239 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.025985 24425 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:19:09.026342 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 530962us
I20260809 23:19:09.027926 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:08.841440 28592 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.032258 24321 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:08.841380 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 425595us
W20260809 23:19:08.574170 24546 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.046s	user 0.011s	sys 0.025s
W20260809 23:19:09.033887 24546 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.506s	user 0.018s	sys 0.032s
I20260809 23:19:08.842542 28004 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:08.842804 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 817) took 4682 ms. Trace:
I20260809 23:19:08.782333 25363 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
W20260809 23:19:08.425149 24429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.001s	sys 0.033s
W20260809 23:19:09.035422 24429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.479s	user 0.008s	sys 0.043s
I20260809 23:19:08.407881 28348 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:08.894783 24431 rpcz_store.cc:276] 0809 23:19:07.653470 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.309931 (+656461us) service_pool.cc:227] Handling call
0809 23:19:08.785426 (+475495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.006273 28255 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:08.895401 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.275s	user 0.033s	sys 0.034s
W20260809 23:19:09.037672 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.417s	user 0.033s	sys 0.034s
I20260809 23:19:08.861491 25370 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:08.898906 24423 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:08.891304 24535 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:08.926083 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:08.930601 28437 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.890s	user 0.010s	sys 0.043s
I20260809 23:19:08.942548 28580 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.933573 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:08.932895 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.046192 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:08.935012 25371 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260809 23:19:09.047545 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.337s	user 0.019s	sys 0.069s
I20260809 23:19:08.830410 28614 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:08.939571 24543 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:08.783304 28589 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.530s	user 0.000s	sys 0.038s
I20260809 23:19:08.415293 28575 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:08.418708 28259 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:08.407239 28566 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.507s
I20260809 23:19:08.665604 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:08.959749 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:08.505752 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 818) took 3484 ms (client timeout 2208 ms). Trace:
W20260809 23:19:08.975937 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.47146e+06us
I20260809 23:19:08.977550 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:08.977941 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 276151us
I20260809 23:19:09.052687 28611 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260809 23:19:09.079990 28516 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.081831 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.083243 28505 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:09.023742 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.467s	user 0.054s	sys 0.087s
W20260809 23:19:09.088871 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.532s	user 0.054s	sys 0.087s
I20260809 23:19:08.893100 28161 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:09.006686 28595 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:09.004810 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 583953us
I20260809 23:19:09.009155 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 739) took 3908 ms. Trace:
I20260809 23:19:08.425379 28547 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.553s
I20260809 23:19:08.833069 28171 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.068054 28587 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:09.122073 28587 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.124182 28587 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:09.072541 25365 rpcz_store.cc:269] 0809 23:19:04.941024 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.945303 (+  4279us) service_pool.cc:227] Handling call
0809 23:19:08.395537 (+3450234us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:08.421612 (+ 26075us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3573345536}
I20260809 23:19:08.975008 28471 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.139038 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.044866 28255 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:09.037777 28615 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:09.015843 28593 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.352s	user 0.020s	sys 0.067s
I20260809 23:19:09.003784 28569 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:08.914451 28394 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:08.936887 28495 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:08.895133 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.157s	user 0.013s	sys 0.039s
I20260809 23:19:09.014037 28065 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:19:08.860810 28561 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.008s	sys 0.031s
I20260809 23:19:09.042548 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.159921 25372 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 4: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:09.046893 24535 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:09.161000 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:09.162216 24535 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:09.038877 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.040596 24545 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:09.164511 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.036476 28466 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:09.036310 28004 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:08.921411 28612 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.145308 28171 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:09.035175 25368 rpcz_store.cc:276] 0809 23:19:04.160863 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:04.161253 (+   390us) service_pool.cc:227] Handling call
0809 23:19:04.161607 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:04.238398 (+ 76791us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:04.244779 (+  6381us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:05.167000 (+922221us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b18001e90a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:19:07.199131 (+2032131us) spinlock_profiling.cc:243] Waited 6.68 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:07.199777 (+   646us) log.cc:842] Serialized 26 byte log entry
0809 23:19:08.418603 (+1218826us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:08.463578 (+ 44975us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.515038 (+ 51460us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:08.839018 (+323980us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:08.842555 (+  3537us) raft_consensus.cc:1671] finished
0809 23:19:08.842586 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:08.842737 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13916,"raft.queue_time_us":56157,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10174,"spinlock_wait_cycles":40904960,"thread_start_us":1195915,"threads_started":1,"wal-append.queue_time_us":1315957}
I20260809 23:19:09.069020 28276 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.071772 28259 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:09.145952 28505 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:09.146857 28624 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260809 23:19:09.137586 28161 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.001055 28609 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.046221 28592 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:09.073922 28580 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.163244 25370 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
W20260809 23:19:09.150254 28593 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s	user 0.021s	sys 0.067s
I20260809 23:19:09.043454 28348 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.148077 28278 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:09.157074 28565 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.421s	user 0.003s	sys 0.054s
I20260809 23:19:09.155006 28569 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.166327 28065 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.077085 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:09.043989 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.531s	user 0.010s	sys 0.032s
I20260809 23:19:09.030644 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.072402 24313 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.127664 24314 rpcz_store.cc:276] 0809 23:19:05.100664 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.291371 (+1190707us) service_pool.cc:227] Handling call
0809 23:19:06.327073 (+ 35702us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.677548 (+1350475us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.008849 (+1331301us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:09.009081 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3037288192}
I20260809 23:19:09.046759 24323 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.074054 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.514027 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1921) took 1084 ms. Trace:
I20260809 23:19:09.514734 24422 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:09.515588 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 438376us
I20260809 23:19:09.516400 28214 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:09.517028 28214 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:09.088939 24425 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260809 23:19:09.047873 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.338s	user 0.019s	sys 0.069s
I20260809 23:19:09.560024 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.561097 24425 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:08.977284 24430 rpcz_store.cc:276] 0809 23:19:07.722444 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.223793 (+501349us) service_pool.cc:227] Handling call
0809 23:19:08.788456 (+564663us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.079401 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1901) took 4044 ms. Trace:
I20260809 23:19:09.562582 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.564049 24430 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:09.561998 24435 rpcz_store.cc:276] 0809 23:19:05.034544 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.865400 (+830856us) service_pool.cc:227] Handling call
0809 23:19:05.865747 (+   347us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:06.495627 (+629880us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:06.495658 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:06.531097 (+ 35439us) log.cc:842] Serialized 26 byte log entry
0809 23:19:07.032821 (+501724us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:07.034677 (+  1856us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:07.038939 (+  4262us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.076855 (+2037916us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b2400415958. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:19:09.079028 (+  2173us) raft_consensus.cc:1671] finished
0809 23:19:09.079057 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.079318 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":602969,"raft.queue_time_us":620988,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"spinlock_wait_cycles":47232256,"thread_start_us":96740,"threads_started":1,"wal-append.queue_time_us":613614}
I20260809 23:19:09.071482 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.072167 25366 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
W20260809 23:19:09.342111 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.663s	user 0.033s	sys 0.024s
W20260809 23:19:09.590333 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.911s	user 0.040s	sys 0.027s
W20260809 23:19:09.589136 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 515804us
I20260809 23:19:09.360281 24424 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:09.343942 24311 raft_consensus.cc:2362] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4. Re-sending same reply.
W20260809 23:19:09.393539 24499 connection.cc:232] RPC call timeout handler was delayed by -1.9552983429748565s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260809 23:19:09.591459 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.394606 28161 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.396394 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 614) took 1841 ms. Trace:
W20260809 23:19:09.169467 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.000s	sys 0.012s
I20260809 23:19:09.593190 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:09.593619 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.189s	user 0.001s	sys 0.013s
I20260809 23:19:09.395476 28004 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.595347 28004 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.596263 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.597633 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:09.168799 28156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.369s	user 0.015s	sys 0.035s
W20260809 23:19:09.599449 28156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.857s	user 0.016s	sys 0.035s
I20260809 23:19:09.599572 28523 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:09.600719 28156 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.601542 28156 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.603067 28381 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:09.219432 28391 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.384435 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 609) took 1859 ms. Trace:
I20260809 23:19:09.604238 28391 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.604117 28492 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:09.605197 28391 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.181902 28259 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:09.151904 28471 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.388636 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.749s	user 0.015s	sys 0.038s
W20260809 23:19:09.607357 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.968s	user 0.015s	sys 0.038s
I20260809 23:19:09.603916 24539 rpcz_store.cc:276] 0809 23:19:07.525024 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.526385 (+  1361us) service_pool.cc:227] Handling call
0809 23:19:07.526691 (+   306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.526966 (+   275us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:07.526995 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.559495 (+ 32500us) log.cc:842] Serialized 26 byte log entry
0809 23:19:09.038104 (+1478609us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:09.038146 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.038211 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.383620 (+345409us) raft_consensus.cc:1671] finished
0809 23:19:09.383677 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.384143 (+   466us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1417760,"threads_started":1,"wal-append.queue_time_us":1687974}
I20260809 23:19:09.188500 28505 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.194099 28565 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.459s	user 0.003s	sys 0.054s
I20260809 23:19:09.609009 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.197984 28569 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:09.610081 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:19:09.610510 28478 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.215s	user 0.010s	sys 0.012s
W20260809 23:19:09.278761 28586 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.431s	user 0.002s	sys 0.011s
W20260809 23:19:09.237600 28590 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.985s	user 0.032s	sys 0.044s
W20260809 23:19:09.639624 28590 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.387s	user 0.032s	sys 0.049s
I20260809 23:19:09.195505 28592 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.641415 24320 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:09.642043 28554 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:09.145629 24470 heartbeater.cc:644] Failed to heartbeat to 127.21.255.62:37921 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.255.62:37921 timed out after 15.000s (SENT)
I20260809 23:19:09.298898 28609 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.298367 28612 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.651727 28523 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:09.085041 28614 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.653594 28523 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:09.654063 28523 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:09.650655 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54643.7us
W20260809 23:19:09.267558 28632 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.437s	user 0.003s	sys 0.000s
W20260809 23:19:09.387174 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.002s	sys 0.005s
W20260809 23:19:09.656653 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.174s	user 0.002s	sys 0.005s
W20260809 23:19:09.657156 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.17519e+06us
I20260809 23:19:09.658419 28612 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:09.294591 28647 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
W20260809 23:19:09.387396 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.000s	sys 0.044s
W20260809 23:19:09.659654 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.370s	user 0.000s	sys 0.045s
I20260809 23:19:09.659317 28612 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.391875 25375 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.388782 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.369805 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1905) took 2715 ms. Trace:
I20260809 23:19:09.661463 24545 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:09.661345 24428 rpcz_store.cc:276] 0809 23:19:06.653977 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.512388 (+858411us) service_pool.cc:227] Handling call
0809 23:19:07.512799 (+   411us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.520497 (+  7698us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.520542 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.531835 (+ 11293us) log.cc:842] Serialized 26 byte log entry
0809 23:19:08.897871 (+1366036us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:08.897911 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:08.897975 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.368976 (+471001us) raft_consensus.cc:1671] finished
0809 23:19:09.369064 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.369514 (+   450us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6082,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":9439,"thread_start_us":1309093,"threads_started":1,"wal-append.queue_time_us":1749981}
I20260809 23:19:09.662680 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:09.342566 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.058s	user 0.016s	sys 0.026s
W20260809 23:19:09.442004 28480 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.389s	user 0.023s	sys 0.032s
I20260809 23:19:09.663517 24545 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:09.663427 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.017s	sys 0.030s
W20260809 23:19:09.281738 28519 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.292s	user 0.024s	sys 0.059s
I20260809 23:19:09.663362 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.664692 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:09.664774 28519 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.721s	user 0.025s	sys 0.060s
I20260809 23:19:09.665597 24545 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:09.288609 28595 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.452355 28065 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:19:09.473592 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1906) took 3581 ms (client timeout 2019 ms). Trace:
I20260809 23:19:09.510216 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.039628 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.685014 24423 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:09.523909 28475 consensus_queue.cc:1159] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 6 status { last_received { term: 5 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:09.526448 28336 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:19:09.527036 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.967s	user 0.011s	sys 0.032s
I20260809 23:19:09.528338 28651 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.529048 28341 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.529255 28394 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:09.531391 28555 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.014s	sys 0.040s
W20260809 23:19:09.531250 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 500519us
W20260809 23:19:09.537487 28395 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.205s	user 0.010s	sys 0.003s
I20260809 23:19:09.538272 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:09.539266 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.540380 28587 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:09.548053 28348 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.548790 28495 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:09.504520 28159 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.027s	sys 0.032s
W20260809 23:19:09.503229 28496 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.046s	user 0.041s	sys 0.026s
I20260809 23:19:09.518585 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:09.517050 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.514593 24323 rpcz_store.cc:276] 0809 23:19:08.429121 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.898387 (+469266us) service_pool.cc:227] Handling call
0809 23:19:09.513760 (+615373us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.558921 28410 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:09.561000 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1906) took 2898 ms. Trace:
I20260809 23:19:09.687445 28336 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:09.565433 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.565177 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:09.342111 24317 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.000s	sys 0.006s
I20260809 23:19:09.590593 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 826) took 1160 ms. Trace:
I20260809 23:19:09.603495 28381 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:09.594190 24425 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:09.090411 28575 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.738592 28575 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.732029 28336 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.741503 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:09.742307 24323 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:09.743034 28159 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.201s	user 0.028s	sys 0.032s
I20260809 23:19:09.743269 24435 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:09.744066 28159 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:09.744560 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.745438 24435 raft_consensus.cc:2362] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5. Re-sending same reply.
I20260809 23:19:09.746383 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.615644 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.746837 24430 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:09.747423 24539 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:09.747915 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.748667 24430 raft_consensus.cc:2362] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6. Re-sending same reply.
I20260809 23:19:09.749338 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2363) took 4626 ms. Trace:
I20260809 23:19:09.749577 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:09.750748 24430 raft_consensus.cc:2354] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:09.592597 24534 rpcz_store.cc:276] 0809 23:19:07.555089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.555299 (+   210us) service_pool.cc:227] Handling call
0809 23:19:07.555680 (+   381us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.556073 (+   393us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.556110 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.556671 (+   561us) log.cc:842] Serialized 26 byte log entry
0809 23:19:09.089565 (+1532894us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:09.089664 (+    99us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.089759 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.395622 (+305863us) raft_consensus.cc:1671] finished
0809 23:19:09.395667 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.396152 (+   485us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78238,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":1532359,"threads_started":1,"wal-append.queue_time_us":1740071}
I20260809 23:19:09.632232 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2339) took 3090 ms. Trace:
I20260809 23:19:09.638764 28567 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:09.572269 28214 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.637287 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2340) took 3096 ms. Trace:
W20260809 23:19:09.188591 28619 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
W20260809 23:19:09.664038 28480 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.719s	user 0.023s	sys 0.032s
I20260809 23:19:09.763137 28480 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:09.762915 28381 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.237440 28517 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.967s	user 0.002s	sys 0.043s
I20260809 23:19:09.664543 28523 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.764479 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 608) took 3110 ms. Trace:
W20260809 23:19:09.764984 28517 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.494s	user 0.002s	sys 0.044s
I20260809 23:19:09.765164 28381 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.510504 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1916) took 1100 ms. Trace:
I20260809 23:19:09.765901 28523 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:09.766178 24321 rpcz_store.cc:276] 0809 23:19:08.409843 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.411819 (+  1976us) service_pool.cc:227] Handling call
0809 23:19:09.510000 (+1098181us) spinlock_profiling.cc:243] Waited 89.3 ms on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.510269 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":205395840}
I20260809 23:19:09.767047 28471 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.767752 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.768613 24321 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:09.769688 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.770449 24321 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:09.740440 24313 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:09.771378 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1924) took 1276 ms. Trace:
I20260809 23:19:09.771729 24313 rpcz_store.cc:276] 0809 23:19:08.494961 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.959419 (+464458us) service_pool.cc:227] Handling call
0809 23:19:09.771299 (+811880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.772874 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.773699 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1919) took 1363 ms. Trace:
I20260809 23:19:09.774048 24313 rpcz_store.cc:276] 0809 23:19:08.410708 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.772504 (+1361796us) service_pool.cc:227] Handling call
0809 23:19:09.773638 (+  1134us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:09.667349 24316 rpcz_store.cc:269] 0809 23:19:05.892257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.109317 (+217060us) service_pool.cc:227] Handling call
0809 23:19:09.473021 (+3363704us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.473269 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5104299776}
I20260809 23:19:09.775704 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.776767 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1923) took 1282 ms. Trace:
I20260809 23:19:09.777184 24316 rpcz_store.cc:276] 0809 23:19:08.494410 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.775300 (+1280890us) service_pool.cc:227] Handling call
0809 23:19:09.776689 (+  1389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.777752 28478 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:09.778237 28478 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:09.778287 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:09.662523 28612 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:09.670910 28416 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.007s	sys 0.019s
I20260809 23:19:09.642633 28554 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.701540 28555 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.015s	sys 0.041s
W20260809 23:19:09.152400 24544 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.920s	user 0.008s	sys 0.033s
W20260809 23:19:09.810736 24544 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.579s	user 0.009s	sys 0.033s
I20260809 23:19:09.686896 28505 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:09.720436 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 2.20729e+06us
I20260809 23:19:09.734094 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:09.813004 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79055.1us
I20260809 23:19:09.814407 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:09.815140 28171 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.010s
I20260809 23:19:09.816001 28171 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:09.816100 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 351) took 16004 ms (client timeout 15000 ms). Trace:
I20260809 23:19:09.816445 28171 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.816942 28171 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:09.816659 28505 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:09.818415 28505 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.735456 28410 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.735397 28651 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.818887 28505 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:09.820559 28651 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:09.618263 28229 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.691s	user 0.002s	sys 0.016s
I20260809 23:19:09.821270 28599 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:09.821750 28229 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:09.614495 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.824417 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.825853 24431 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.751526 24435 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.827266 28567 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.751498 24317 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.364s	user 0.000s	sys 0.016s
I20260809 23:19:09.827946 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:09.829226 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 740) took 4726 ms. Trace:
I20260809 23:19:09.829649 28567 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:09.759231 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 240921us
I20260809 23:19:09.830782 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:09.832701 28575 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:09.833240 28575 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.833613 28575 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.740336 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:09.668222 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.384s	user 0.034s	sys 0.049s
I20260809 23:19:09.665865 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:09.679315 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.018s	sys 0.031s
I20260809 23:19:09.834856 28259 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.816571 24164 rpcz_store.cc:269] 0809 23:18:53.811483 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:53.811722 (+   239us) service_pool.cc:227] Handling call
0809 23:19:09.815803 (+16004081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:19:03.315290 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:19:03.595893 (+280603us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:19:03.609120 (+ 13227us) write_op.cc:435] Acquired schema lock
0809 23:19:03.609144 (+    24us) tablet.cc:660] Decoding operations
0809 23:19:03.785318 (+176174us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:19:03.785423 (+   105us) write_op.cc:641] Partition lock acquired for write op
0809 23:19:03.785453 (+    30us) tablet.cc:683] Acquiring locks for 14 operations
0809 23:19:03.879613 (+ 94160us) tablet.cc:699] Row locks acquired
0809 23:19:03.879667 (+    54us) write_op.cc:260] PREPARE: finished
0809 23:19:03.879880 (+   213us) write_op.cc:270] Start()
0809 23:19:03.880064 (+   184us) write_op.cc:276] Timestamp: P: 1786317543879801 usec, L: 0
0809 23:19:03.880091 (+    27us) op_driver.cc:331] REPLICATION: starting
0809 23:19:03.881010 (+   919us) log.cc:842] Serialized 5524 byte log entry
0809 23:19:04.637363 (+756353us) op_driver.cc:444] REPLICATION: finished
0809 23:19:05.003371 (+366008us) write_op.cc:301] APPLY: starting
0809 23:19:05.012372 (+  9001us) tablet.cc:1365] starting BulkCheckPresence
0809 23:19:05.099028 (+ 86656us) tablet.cc:1368] finished BulkCheckPresence
0809 23:19:05.155313 (+ 56285us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:19:05.239062 (+ 83749us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:19:05.448722 (+209660us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0809 23:19:05.573242 (+124520us) write_op.cc:312] APPLY: finished
0809 23:19:05.973682 (+400440us) log.cc:842] Serialized 101 byte log entry
0809 23:19:06.089616 (+115934us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:06.106431 (+ 16815us) write_op.cc:454] Released schema lock
0809 23:19:06.143555 (+ 37124us) write_op.cc:341] FINISH: Updating metrics
0809 23:19:06.162049 (+ 18494us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:06.198186 (+ 36137us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1304,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":298317,"apply.run_cpu_time_us":54392,"apply.run_wall_time_us":1248363,"num_ops":14,"prepare.queue_time_us":434059,"prepare.run_cpu_time_us":16100,"prepare.run_wall_time_us":769869,"raft.queue_time_us":333447,"raft.run_cpu_time_us":34415,"raft.run_wall_time_us":272007,"replication_time_us":724300,"thread_start_us":854560,"threads_started":5,"wal-append.queue_time_us":510588,"wal-append.run_cpu_time_us":21031,"wal-append.run_wall_time_us":2449834}]]}
I20260809 23:19:09.835690 24428 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.836244 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 611) took 2310 ms. Trace:
I20260809 23:19:09.836743 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.837183 28259 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.837440 28622 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.837623 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:09.778898 28478 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:09.838150 28622 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.839111 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.55522e+06us
I20260809 23:19:09.834525 28505 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:09.840307 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 113522us
I20260809 23:19:09.841444 28505 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:09.842123 28612 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.842922 28612 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.843497 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.779106 28347 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:09.844358 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:09.844808 28347 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.845242 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.845359 28612 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.845604 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.847138 28229 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:09.847846 28229 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:09.780560 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.850090 28229 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.755085 24424 rpcz_store.cc:276] 0809 23:19:06.540086 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.444186 (+904100us) service_pool.cc:227] Handling call
0809 23:19:07.490686 (+ 46500us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.615561 (+2124875us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.618710 (+  3149us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:09.630144 (+ 11434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4297734912}
I20260809 23:19:09.850129 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.852020 24534 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:19:09.852784 24534 raft_consensus.cc:595] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.853134 24534 raft_consensus.cc:603] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:09.836633 24546 rpcz_store.cc:276] 0809 23:19:07.525687 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.527527 (+  1840us) service_pool.cc:227] Handling call
0809 23:19:07.527869 (+   342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.528088 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:09.036719 (+1508631us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.835684 (+798965us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.835787 (+   103us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:09.835881 (+    94us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.835968 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.836186 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1021510,"prepare.run_cpu_time_us":6519,"prepare.run_wall_time_us":36608,"raft.queue_time_us":2299,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"thread_start_us":2303310,"threads_started":2,"wal-append.queue_time_us":698503}
I20260809 23:19:09.855118 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.765172 24542 rpcz_store.cc:276] 0809 23:19:06.653836 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:06.654032 (+   196us) service_pool.cc:227] Handling call
0809 23:19:06.654343 (+   311us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.247549 (+593206us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:08.289259 (+1041710us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.660261 (+1371002us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.660378 (+   117us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:09.660494 (+   116us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.764091 (+103597us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:09.764131 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.764294 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3373,"prepare.queue_time_us":1294692,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":851,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":178,"spinlock_wait_cycles":1363729792,"thread_start_us":2406607,"threads_started":2,"wal-append.queue_time_us":1015472}
I20260809 23:19:09.838629 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.820559 28410 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:09.850319 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69935us
I20260809 23:19:09.829557 24317 rpcz_store.cc:276] 0809 23:19:05.103086 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.203851 (+2100765us) service_pool.cc:227] Handling call
0809 23:19:07.229622 (+ 25771us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.447711 (+218089us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.447754 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.465065 (+ 17311us) log.cc:842] Serialized 26 byte log entry
0809 23:19:08.387052 (+921987us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:09.828358 (+1441306us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.828776 (+   418us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.828940 (+   164us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.828984 (+    44us) raft_consensus.cc:1671] finished
0809 23:19:09.828999 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.829157 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1201515,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":101,"thread_start_us":2362520,"threads_started":2,"wal-append.queue_time_us":1386313}
I20260809 23:19:09.861182 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.861927 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2378) took 1477 ms. Trace:
I20260809 23:19:09.864127 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:09.770879 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1920) took 1342 ms. Trace:
I20260809 23:19:09.865765 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.726843 24427 rpcz_store.cc:276] 0809 23:19:06.662540 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.615864 (+953324us) service_pool.cc:227] Handling call
0809 23:19:07.616187 (+   323us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.631980 (+ 15793us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.632024 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.652988 (+ 20964us) log.cc:842] Serialized 26 byte log entry
0809 23:19:09.560289 (+1907301us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:09.560356 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.560472 (+   116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.560530 (+    58us) raft_consensus.cc:1671] finished
0809 23:19:09.560540 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.560766 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10202,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13661,"thread_start_us":1849662,"threads_started":1,"wal-append.queue_time_us":1644381}
W20260809 23:19:09.780570 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2163) took 1551 ms (client timeout 1736 ms). Trace:
I20260809 23:19:09.790704 25372 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:09.740207 28156 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.836768 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:09.846827 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:09.702818 28492 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:09.801653 24422 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:09.850678 28161 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:09.804549 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 613) took 2278 ms. Trace:
I20260809 23:19:09.805058 28618 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:09.807551 28416 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s	user 0.008s	sys 0.019s
I20260809 23:19:09.875945 24323 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:09.749733 24312 rpcz_store.cc:276] 0809 23:19:05.122553 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.251744 (+2129191us) service_pool.cc:227] Handling call
0809 23:19:07.253209 (+  1465us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.665675 (+412466us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:07.678667 (+ 12992us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.638662 (+1959995us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.638790 (+   128us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:09.638947 (+   157us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.748883 (+109936us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b540043add4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:09.748932 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.749213 (+   281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32519,"prepare.queue_time_us":13536,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":51282,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":221,"spinlock_wait_cycles":947996032,"thread_start_us":1911900,"threads_started":1,"wal-append.queue_time_us":1622760}
I20260809 23:19:09.734824 28614 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:09.758033 24429 rpcz_store.cc:276] 0809 23:19:06.540380 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.446036 (+905656us) service_pool.cc:227] Handling call
0809 23:19:07.511093 (+ 65057us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.527180 (+ 16087us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:09.035771 (+1508591us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.625002 (+589231us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.636692 (+ 11690us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:09.636801 (+   109us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.636930 (+   129us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.637108 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42911,"prepare.queue_time_us":436681,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":49798,"raft.run_cpu_time_us":646,"raft.run_wall_time_us":24676,"thread_start_us":1977094,"threads_started":2,"wal-append.queue_time_us":537028}
I20260809 23:19:09.819341 24168 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 4 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:09.740326 28391 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.827642 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.885383 24435 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
W20260809 23:19:09.761401 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 909) took 1349 ms (client timeout 1775 ms). Trace:
I20260809 23:19:09.731894 28394 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:09.732149 25366 rpcz_store.cc:276] 0809 23:19:08.429872 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.539046 (+109174us) service_pool.cc:227] Handling call
0809 23:19:09.590331 (+1051285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.760876 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.846741 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:09.751852 28496 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.364s	user 0.041s	sys 0.026s
I20260809 23:19:09.847932 28423 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:09.782117 28214 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:09.856864 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:09.891211 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 734) took 4863 ms (client timeout 1765 ms). Trace:
I20260809 23:19:09.865814 24321 rpcz_store.cc:276] 0809 23:19:08.428691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.769333 (+1340642us) service_pool.cc:227] Handling call
0809 23:19:09.770835 (+  1502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.883427 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:09.862252 24317 rpcz_store.cc:276] 0809 23:19:08.384554 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.860815 (+1476261us) service_pool.cc:227] Handling call
0809 23:19:09.861868 (+  1053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.873535 24542 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
W20260809 23:19:09.882570 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.226s (SENT)
W20260809 23:19:09.867022 24543 rpcz_store.cc:269] 0809 23:19:08.229356 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.295942 (+ 66586us) service_pool.cc:227] Handling call
0809 23:19:09.780069 (+1484127us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.780332 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1451160832}
W20260809 23:19:09.896756 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50283.3us
I20260809 23:19:09.892021 24316 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:09.886284 25375 rpcz_store.cc:269] 0809 23:19:08.412451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.427210 (+ 14759us) service_pool.cc:227] Handling call
0809 23:19:09.761197 (+1333987us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.761320 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1030598912}
I20260809 23:19:09.874573 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:09.871582 24423 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:09.876709 28471 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:09.877748 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:09.803478 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2376) took 1471 ms (client timeout 1610 ms). Trace:
I20260809 23:19:09.900640 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.900987 28004 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:09.901721 24542 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:09.902160 28547 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:09.808518 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:09.903082 28547 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.903895 28471 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:09.903779 28569 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.902109 28618 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:09.900417 24311 rpcz_store.cc:269] 0809 23:19:08.331987 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.385568 (+ 53581us) service_pool.cc:227] Handling call
0809 23:19:09.799158 (+1413590us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.803395 (+  4237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1276974464}
I20260809 23:19:09.904743 24422 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:09.905396 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2351) took 1648 ms. Trace:
I20260809 23:19:09.905539 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.905743 24422 rpcz_store.cc:276] 0809 23:19:08.257028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.895912 (+1638884us) service_pool.cc:227] Handling call
0809 23:19:09.904361 (+  8449us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.905145 (+   784us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.905328 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.906450 28569 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:09.901721 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:09.878325 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:09.887447 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:09.850680 28229 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:09.874179 24313 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:09.901669 28004 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.700246 28587 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:09.891873 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2380) took 1458 ms. Trace:
I20260809 23:19:09.880388 24541 rpcz_store.cc:276] 0809 23:19:07.526249 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.544448 (+ 18199us) service_pool.cc:227] Handling call
0809 23:19:07.544802 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:07.545047 (+   245us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:08.404821 (+859774us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.732091 (+1327270us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.790427 (+ 58336us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:09.804159 (+ 13732us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.804276 (+   117us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.804478 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1237949,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":11943,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":227,"thread_start_us":2117156,"threads_started":3,"wal-append.queue_time_us":890940}
W20260809 23:19:09.894407 24320 rpcz_store.cc:269] 0809 23:19:05.027883 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:05.859825 (+831942us) service_pool.cc:227] Handling call
0809 23:19:09.880584 (+4020759us) spinlock_profiling.cc:243] Waited 3.65 s on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:09.891136 (+ 10552us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8391637248}
I20260809 23:19:09.882772 28161 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:09.913384 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2349) took 2013 ms. Trace:
I20260809 23:19:09.941295 24427 rpcz_store.cc:276] 0809 23:19:07.899910 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.879928 (+1980018us) service_pool.cc:227] Handling call
0809 23:19:09.912698 (+ 32770us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.912956 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:09.912993 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:09.913046 (+    53us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:09.913098 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.913161 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.913286 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.942397 24422 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:09.913892 24539 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:09.914023 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.902820 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:09.935166 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106212us
I20260809 23:19:09.923625 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 779) took 2417 ms. Trace:
I20260809 23:19:09.939141 24323 rpcz_store.cc:276] 0809 23:19:08.426582 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.740969 (+1314387us) service_pool.cc:227] Handling call
0809 23:19:09.884512 (+143543us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.906478 24311 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260809 23:19:09.943200 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67392.6us
I20260809 23:19:09.947805 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:09.943284 24312 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:09.954157 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1914) took 1536 ms. Trace:
I20260809 23:19:09.954562 24311 rpcz_store.cc:276] 0809 23:19:08.409273 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.905173 (+1495900us) service_pool.cc:227] Handling call
0809 23:19:09.945699 (+ 40526us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.949872 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 524) took 1038 ms. Trace:
I20260809 23:19:09.956067 24422 rpcz_store.cc:276] 0809 23:19:08.911672 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.906549 (+994877us) service_pool.cc:227] Handling call
0809 23:19:09.933728 (+ 27179us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:09.949583 (+ 15855us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.949818 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.890527 28391 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:09.949257 24535 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.559s	user 0.016s	sys 0.002s
I20260809 23:19:09.952538 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:09.903911 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:09.958320 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 621) took 1081 ms. Trace:
I20260809 23:19:09.764504 28480 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.910907 28592 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:09.945945 28161 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:09.972040 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70556.9us
I20260809 23:19:09.956542 24430 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:09.937918 28004 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:09.974241 25372 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260809 23:19:09.974607 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.955799 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:09.956948 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2374) took 1622 ms. Trace:
I20260809 23:19:09.949136 28471 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:09.971889 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.975587 24312 rpcz_store.cc:276] 0809 23:19:08.329626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.896555 (+1566929us) service_pool.cc:227] Handling call
0809 23:19:09.952405 (+ 55850us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.963694 24539 rpcz_store.cc:276] 0809 23:19:08.870555 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.756310 (+885755us) service_pool.cc:227] Handling call
0809 23:19:09.913629 (+157319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.951611 (+ 37982us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.951830 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.978118 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.978099 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.969050 24168 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:09.944217 25369 rpcz_store.cc:276] 0809 23:19:07.481935 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:07.637570 (+155635us) service_pool.cc:227] Handling call
0809 23:19:07.637901 (+   331us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:07.638262 (+   361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:07.638294 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:07.638721 (+   427us) log.cc:842] Serialized 26 byte log entry
0809 23:19:09.607833 (+1969112us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:09.878753 (+270920us) log.cc:842] Serialized 14 byte log entry
0809 23:19:09.882806 (+  4053us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.891793 (+  8987us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:09.899101 (+  7308us) raft_consensus.cc:1671] finished
0809 23:19:09.899119 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:09.899278 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":47321,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":55,"raft.queue_time_us":7642,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":4238,"thread_start_us":2238961,"threads_started":2,"wal-append.queue_time_us":1924521}
I20260809 23:19:09.987638 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.979751 24435 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:09.979688 24535 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:09.953706 28229 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:09.976717 28471 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:09.995051 24323 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:10.000239 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2383) took 1214 ms. Trace:
I20260809 23:19:10.000826 24323 rpcz_store.cc:276] 0809 23:19:08.786004 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.950227 (+1164223us) service_pool.cc:227] Handling call
0809 23:19:10.000184 (+ 49957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.979487 28614 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:10.000281 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1914) took 1211 ms. Trace:
I20260809 23:19:10.001770 24435 rpcz_store.cc:276] 0809 23:19:08.782135 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.897888 (+1115753us) service_pool.cc:227] Handling call
0809 23:19:09.970387 (+ 72499us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.993649 (+ 23262us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.993837 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:09.994776 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:09.989496 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.907338 28641 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.083s
W20260809 23:19:10.019999 28682 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260809 23:19:10.023675 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:10.024639 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2375) took 1694 ms. Trace:
I20260809 23:19:10.025012 24312 rpcz_store.cc:276] 0809 23:19:08.329962 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.984788 (+1654826us) service_pool.cc:227] Handling call
0809 23:19:10.024567 (+ 39779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.026230 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:10.027792 28587 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.029026 28587 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.033255 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:09.988786 28496 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:09.967525 28492 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.018221 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:09.973709 28336 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.037396 24425 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
W20260809 23:19:10.091552 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75326.6us
I20260809 23:19:09.976969 28391 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.049415 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:10.077606 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 164410us
W20260809 23:19:10.107728 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72172.5us
I20260809 23:19:10.067378 24320 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260809 23:19:10.109758 24378 connection.cc:232] RPC call timeout handler was delayed by -1.4238215221557766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
I20260809 23:19:09.976428 28394 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.111613 24312 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:19:10.012028 28614 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:10.110785 24313 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:10.087466 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:10.114408 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.115062 24541 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:10.116004 28615 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.957s	user 0.007s	sys 0.011s
I20260809 23:19:10.116091 28683 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:10.116549 28615 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:10.116627 28683 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:10.117121 28683 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:10.117230 24311 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:10.114660 24168 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: leader changed from <none> to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 3 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:10.025259 28575 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.121222 28496 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:10.122529 28496 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.123656 24422 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260809 23:19:10.125078 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 533) took 1071 ms. Trace:
I20260809 23:19:10.125494 24425 rpcz_store.cc:276] 0809 23:19:09.041184 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.948985 (+907801us) service_pool.cc:227] Handling call
0809 23:19:09.992022 (+ 43037us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.111844 (+119822us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.112111 (+   267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.126230 24542 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:10.127094 28197 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.127439 24317 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:10.127925 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 284949us
I20260809 23:19:10.129221 24539 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:10.131965 24542 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 23:19:10.132792 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 750) took 4285 ms. Trace:
I20260809 23:19:10.133203 24320 rpcz_store.cc:276] 0809 23:19:05.825833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.034709 (+4208876us) service_pool.cc:227] Handling call
0809 23:19:10.037510 (+  2801us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.110452 (+ 72942us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.110670 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.133872 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:10.136222 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2381) took 1351 ms. Trace:
I20260809 23:19:10.136686 24309 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 23:19:10.136653 24317 rpcz_store.cc:276] 0809 23:19:08.784553 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.914842 (+1130289us) service_pool.cc:227] Handling call
0809 23:19:10.136000 (+221158us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54003859d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:10.136154 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":347578624}
W20260809 23:19:10.137795 24248 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61228.6us
I20260809 23:19:10.143042 28614 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:10.117511 24314 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260809 23:19:10.146178 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1913) took 1737 ms (client timeout 2144 ms). Trace:
W20260809 23:19:10.146641 24314 rpcz_store.cc:269] 0809 23:19:08.408954 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.509383 (+1100429us) service_pool.cc:227] Handling call
0809 23:19:10.145941 (+636558us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:10.146110 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012181376}
I20260809 23:19:10.148785 24314 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260809 23:19:10.149744 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 753) took 4126 ms. Trace:
I20260809 23:19:10.155185 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:10.150278 24314 rpcz_store.cc:276] 0809 23:19:06.023054 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.147623 (+4124569us) service_pool.cc:227] Handling call
0809 23:19:10.148047 (+   424us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.149332 (+  1285us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.149623 (+   291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.159052 28565 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:10.107442 28478 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.159946 28565 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.161906 28590 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:19:10.162849 28590 consensus_queue.cc:1225] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260809 23:19:10.163451 28156 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:10.137302 24309 pending_rounds.cc:76] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:10.163836 28156 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:10.164161 24309 pending_rounds.cc:97] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:10.164582 24309 raft_consensus.cc:2866] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:10.165067 28156 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.166718 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2364) took 4931 ms. Trace:
I20260809 23:19:10.062016 25365 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:10.176082 24312 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260809 23:19:10.184716 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195469us
I20260809 23:19:10.190623 24435 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.104990 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:10.191370 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1922) took 1619 ms (client timeout 2157 ms). Trace:
I20260809 23:19:10.012306 28569 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:10.197209 28569 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:10.197762 28569 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.017135 28599 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:10.184371 24309 rpcz_store.cc:276] 0809 23:19:05.235500 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.000826 (+2765326us) service_pool.cc:227] Handling call
0809 23:19:08.085265 (+ 84439us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.165480 (+2080215us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.166435 (+   955us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:10.166648 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4379695488}
I20260809 23:19:10.012213 28471 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:10.177608 24168 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 2, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:10.195861 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2366) took 3705 ms. Trace:
I20260809 23:19:10.207376 24312 rpcz_store.cc:276] 0809 23:19:06.490528 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.175379 (+3684851us) service_pool.cc:227] Handling call
0809 23:19:10.175781 (+   402us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.194980 (+ 19199us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.195638 (+   658us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:10.206276 24313 rpcz_store.cc:269] 0809 23:19:08.493938 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.774794 (+1280856us) service_pool.cc:227] Handling call
0809 23:19:10.112930 (+338136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:10.174568 28255 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.017s	user 0.015s	sys 0.001s
I20260809 23:19:10.166782 28156 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:10.222410 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91229.1us
I20260809 23:19:10.251420 24323 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 23:19:10.251169 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.255981 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2345) took 2593 ms. Trace:
I20260809 23:19:10.175022 28394 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:10.259083 24428 rpcz_store.cc:276] 0809 23:19:07.644130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.857434 (+2213304us) service_pool.cc:227] Handling call
0809 23:19:09.865111 (+  7677us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.888427 (+ 23316us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:10.103005 (+214578us) log.cc:842] Serialized 14 byte log entry
0809 23:19:10.237403 (+134398us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:10.237459 (+    56us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:10.237525 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.237609 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:10.237755 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12995,"thread_start_us":268161,"threads_started":2}
I20260809 23:19:10.260134 24313 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 23:19:10.192306 28615 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:10.173314 28347 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:10.172905 28622 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:10.261000 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106006us
I20260809 23:19:10.175038 28547 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:10.267733 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56284.3us
I20260809 23:19:10.267997 24539 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.273504 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:10.269537 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2372) took 3580 ms. Trace:
I20260809 23:19:10.237578 28478 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:10.278067 24313 rpcz_store.cc:276] 0809 23:19:06.680074 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.211895 (+3531821us) service_pool.cc:227] Handling call
0809 23:19:10.259819 (+ 47924us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.260533 (+   714us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.260715 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.281332 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.282300 28336 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:10.284022 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:10.286427 24541 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:10.291543 25321 leader_election.cc:398] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:10.287761 24321 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:10.268431 24168 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 3, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:10.254683 28575 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:10.297976 25321 leader_election.cc:401] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260809 23:19:10.267505 28590 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:10.292912 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2162) took 2064 ms. Trace:
I20260809 23:19:10.271499 28615 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:10.301302 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 196206us
I20260809 23:19:10.292088 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2172) took 1968 ms. Trace:
I20260809 23:19:10.287035 28341 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:10.308493 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.314272 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.272882 28229 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:10.266601 28255 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.314816 24309 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:19:10.316613 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:10.640514 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2369) took 4005 ms. Trace:
W20260809 23:19:10.641117 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 353395us
I20260809 23:19:10.641031 24309 rpcz_store.cc:276] 0809 23:19:06.635357 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.210858 (+3575501us) service_pool.cc:227] Handling call
0809 23:19:10.287358 (+ 76500us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.640187 (+352829us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.640428 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.314919 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 523) took 1403 ms. Trace:
I20260809 23:19:10.642146 24429 rpcz_store.cc:276] 0809 23:19:08.911324 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.909137 (+997813us) service_pool.cc:227] Handling call
0809 23:19:09.909519 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:09.909981 (+   462us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:09.910012 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:09.910517 (+   505us) log.cc:842] Serialized 26 byte log entry
0809 23:19:10.077157 (+166640us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:10.077193 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.077258 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:10.314719 (+237461us) raft_consensus.cc:1671] finished
0809 23:19:10.314739 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:10.314863 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12346,"raft.queue_time_us":25005,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":15730,"thread_start_us":166269,"threads_started":1,"wal-append.queue_time_us":339792}
I20260809 23:19:10.518940 24168 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 5, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 5 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:19:10.643239 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 384315us
I20260809 23:19:10.434966 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.376679 24435 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
W20260809 23:19:10.433508 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57555.8us
I20260809 23:19:10.656944 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:10.577301 24539 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:10.678556 24543 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6.
W20260809 23:19:10.680294 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 396417us
I20260809 23:19:10.520341 24313 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:10.402287 24312 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 23:19:10.304567 24544 rpcz_store.cc:276] 0809 23:19:08.228926 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.230634 (+  1708us) service_pool.cc:227] Handling call
0809 23:19:08.230979 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:08.231258 (+   279us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:09.811347 (+1580089us) log.cc:842] Serialized 14 byte log entry
0809 23:19:10.192305 (+380958us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:10.209647 (+ 17342us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:10.268358 (+ 58711us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.292714 (+ 24356us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:10.292850 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21917,"prepare.queue_time_us":664298,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":157,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":17060,"thread_start_us":1877662,"threads_started":2}
I20260809 23:19:10.707090 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2373) took 4016 ms. Trace:
I20260809 23:19:10.293427 28651 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:10.719377 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:10.658061 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:10.719982 24541 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.707144 24312 pending_rounds.cc:76] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
W20260809 23:19:10.720655 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49629.8us
I20260809 23:19:10.721843 24309 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:10.722011 28255 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:10.722467 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1912) took 2314 ms. Trace:
I20260809 23:19:10.723212 28681 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:19:10.724049 28698 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260809 23:19:10.725657 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:10.727029 28575 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:10.311415 24545 rpcz_store.cc:276] 0809 23:19:08.303390 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.666492 (+1363102us) service_pool.cc:227] Handling call
0809 23:19:09.694809 (+ 28317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:09.793655 (+ 98846us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:10.063835 (+270180us) log.cc:842] Serialized 14 byte log entry
0809 23:19:10.270310 (+206475us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:10.270586 (+   276us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:10.270750 (+   164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.271090 (+   340us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:10.271436 (+   346us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12694,"raft.queue_time_us":18711,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":13731,"thread_start_us":279177,"threads_started":2}
I20260809 23:19:10.728705 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:10.627954 28523 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.861s	user 0.014s	sys 0.001s
I20260809 23:19:10.306262 28478 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:10.720711 28547 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:10.730042 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:10.722775 24309 rpcz_store.cc:276] 0809 23:19:08.408601 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.721222 (+2312621us) service_pool.cc:227] Handling call
0809 23:19:10.721620 (+   398us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.722234 (+   614us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.722418 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.732933 24312 pending_rounds.cc:97] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:10.744613 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:10.745687 25372 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:10.746472 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.385496 28554 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:10.309327 28590 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:10.707759 24313 rpcz_store.cc:276] 0809 23:19:06.691057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.294970 (+3603913us) service_pool.cc:227] Handling call
0809 23:19:10.436817 (+141847us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.706737 (+269920us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.707019 (+   282us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:10.743100 24316 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.828s	user 0.012s	sys 0.001s
I20260809 23:19:10.413691 28517 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:10.550115 28423 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:10.711018 28347 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:10.753130 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95166.1us
I20260809 23:19:10.750100 24312 raft_consensus.cc:2866] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:10.317898 28615 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:10.739341 24539 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:10.710436 28341 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:10.717172 28565 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:10.759997 24316 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:10.365181 28618 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:10.742081 28171 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.924s	user 0.014s	sys 0.000s
W20260809 23:19:10.799602 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2379) took 2403 ms (client timeout 2486 ms). Trace:
I20260809 23:19:10.777746 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.738672 28478 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:10.784220 28347 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:10.317037 28592 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.449s
I20260809 23:19:10.745241 28612 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:10.778323 28394 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:10.755676 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:10.800273 24316 rpcz_store.cc:269] 0809 23:19:08.396198 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.777913 (+1381715us) service_pool.cc:227] Handling call
0809 23:19:10.799515 (+1021602us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.760424 28684 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:10.805465 24317 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 3. Current term is 5. Ops: [3.2-3.2]
I20260809 23:19:10.500115 28381 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.277s
I20260809 23:19:10.801400 24544 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
W20260809 23:19:10.810506 24269 connection.cc:232] RPC call timeout handler was delayed by -2.191897930810228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
I20260809 23:19:10.813198 28255 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:10.792193 28391 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:10.308087 28416 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.482s
I20260809 23:19:10.769095 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:10.769466 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:10.770164 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2386) took 1982 ms. Trace:
W20260809 23:19:10.769538 28161 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.785s	user 0.007s	sys 0.003s
I20260809 23:19:10.851311 24323 rpcz_store.cc:276] 0809 23:19:08.787945 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.063010 (+1275065us) service_pool.cc:227] Handling call
0809 23:19:10.769975 (+706965us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002535d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:10.770104 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":238216064}
I20260809 23:19:10.770754 25371 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:10.822343 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.750396 28410 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:10.769522 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:10.761783 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2377) took 2418 ms (client timeout 2628 ms). Trace:
W20260809 23:19:10.854447 24321 rpcz_store.cc:269] 0809 23:19:08.343230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.946778 (+1603548us) service_pool.cc:227] Handling call
0809 23:19:10.761609 (+814831us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:10.761719 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688186624}
I20260809 23:19:10.769233 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:10.796972 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.775781 28492 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:10.834645 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2371) took 4150 ms. Trace:
W20260809 23:19:10.765412 28065 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.067s	user 0.013s	sys 0.000s
I20260809 23:19:10.831480 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40854 (request call id 1223) took 4933 ms. Trace:
W20260809 23:19:10.770467 28471 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.508s	user 0.009s	sys 0.001s
I20260809 23:19:10.819695 28478 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:10.848583 24269 connection.cc:232] RPC call timeout handler was delayed by -2.1432979309465736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1976s for a response
I20260809 23:19:10.824249 28347 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:10.768889 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:10.761752 28590 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:10.775197 28547 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.860093 24312 rpcz_store.cc:276] 0809 23:19:06.675906 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.252062 (+3576156us) service_pool.cc:227] Handling call
0809 23:19:10.292793 (+ 40731us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.825565 (+532772us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.825891 (+   326us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:10.871959 24311 rpcz_store.cc:276] 0809 23:19:05.890814 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.118176 (+4227362us) service_pool.cc:227] Handling call
0809 23:19:10.803156 (+684980us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:19:10.824532 (+ 21376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":209691520}
I20260809 23:19:10.879616 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:10.880532 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65385.8us
I20260809 23:19:10.890515 24313 pending_rounds.cc:76] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 3
I20260809 23:19:10.906695 24313 pending_rounds.cc:97] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260809 23:19:10.906842 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:10.907279 24313 raft_consensus.cc:2866] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:10.899883 24541 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
W20260809 23:19:10.909054 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69131.1us
W20260809 23:19:10.901010 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 132042us
W20260809 23:19:10.892936 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.009s	sys 0.067s
I20260809 23:19:10.911202 24544 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:10.912981 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.009s	sys 0.067s
I20260809 23:19:10.816946 28171 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.916834 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.764899 28651 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.925308 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.938462 24323 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:10.939126 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2387) took 2019 ms. Trace:
I20260809 23:19:10.939405 24323 rpcz_store.cc:276] 0809 23:19:08.920140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.881803 (+1961663us) service_pool.cc:227] Handling call
0809 23:19:10.938120 (+ 56317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.938871 (+   751us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.939072 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:10.940289 28689 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.576s	user 0.000s	sys 0.002s
I20260809 23:19:10.860677 28161 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.941064 28341 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.925479 28523 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:10.948276 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:10.956369 25369 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:10.963640 24269 connection.cc:232] RPC call timeout handler was delayed by -2.0253979309927672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
W20260809 23:19:10.964532 24269 connection.cc:232] RPC call timeout handler was delayed by -1.7807799999136478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
W20260809 23:19:10.965257 24269 connection.cc:232] RPC call timeout handler was delayed by -1.3318946841172874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1962s for a response
W20260809 23:19:10.965651 24269 connection.cc:232] RPC call timeout handler was delayed by -1.1560299170669168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260809 23:19:10.965987 24269 connection.cc:232] RPC call timeout handler was delayed by -1.017678553937003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260809 23:19:10.839999 28684 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.957077 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.890277 28065 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:10.973762 28065 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:10.974049 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:10.974210 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1909) took 4294 ms. Trace:
W20260809 23:19:10.973780 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 160898us
I20260809 23:19:10.976524 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1908) took 5073 ms. Trace:
I20260809 23:19:10.978179 24317 rpcz_store.cc:276] 0809 23:19:05.904193 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.137732 (+4233539us) service_pool.cc:227] Handling call
0809 23:19:10.799476 (+661744us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.835986 (+ 36510us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.976334 (+140348us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b540019d2d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:10.976467 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1520798080}
I20260809 23:19:10.974877 24539 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:19:10.975443 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:10.975404 28640 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.502s	user 0.001s	sys 0.000s
I20260809 23:19:10.985106 24542 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: []
W20260809 23:19:10.996843 24431 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.006s	sys 0.022s
W20260809 23:19:10.998399 24431 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.006s	sys 0.023s
I20260809 23:19:10.974623 24424 rpcz_store.cc:276] 0809 23:19:06.679778 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.851386 (+3171608us) service_pool.cc:227] Handling call
0809 23:19:09.851947 (+   561us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:09.852450 (+   503us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:09.852478 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:09.852938 (+   460us) log.cc:842] Serialized 26 byte log entry
0809 23:19:09.936631 (+ 83693us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:09.936670 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:09.936737 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:10.973999 (+1037262us) raft_consensus.cc:1671] finished
0809 23:19:10.974018 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:10.974146 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4316,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":143,"thread_start_us":83316,"threads_started":1,"wal-append.queue_time_us":1094239}
I20260809 23:19:10.995182 25372 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
W20260809 23:19:11.009568 24534 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.012s	sys 0.022s
W20260809 23:19:11.009938 24534 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.012s	sys 0.022s
I20260809 23:19:11.011130 24546 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260809 23:19:10.975200 28587 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.946s	user 0.011s	sys 0.005s
I20260809 23:19:10.887023 28471 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:10.954456 28523 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:10.955013 28589 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:10.977046 28391 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:11.019153 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:10.942461 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.033051 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2194) took 1038 ms. Trace:
I20260809 23:19:11.025293 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:10.984822 28554 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:11.041428 24542 rpcz_store.cc:276] 0809 23:19:09.994444 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.132696 (+138252us) service_pool.cc:227] Handling call
0809 23:19:10.133082 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.995893 (+862811us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.032811 (+ 36918us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:11.032990 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1941990272}
I20260809 23:19:11.043394 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:11.044521 24323 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:11.035230 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:10.925246 28410 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:11.040791 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.009s	sys 0.024s
W20260809 23:19:11.053937 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.009s	sys 0.024s
W20260809 23:19:11.033994 28701 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260809 23:19:11.026341 28700 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
W20260809 23:19:11.054149 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 465972us
I20260809 23:19:11.054759 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.090037 24544 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:11.093482 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 539) took 1435 ms. Trace:
I20260809 23:19:11.095415 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:11.064405 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:11.035778 28523 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:11.099865 24428 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:11.101560 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66991.3us
I20260809 23:19:11.040884 28381 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:11.018950 28336 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:11.096946 24311 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:11.024479 28171 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:11.047016 28554 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:11.107805 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:11.113006 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:11.118734 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2177) took 1990 ms. Trace:
I20260809 23:19:11.119240 24546 rpcz_store.cc:276] 0809 23:19:09.128028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.855915 (+727887us) service_pool.cc:227] Handling call
0809 23:19:09.865111 (+  9196us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:11.016291 (+1151180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.118437 (+102146us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:11.118659 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2627544832}
W20260809 23:19:11.121313 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 372721us
W20260809 23:19:11.096074 28683 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.978s	user 0.013s	sys 0.000s
I20260809 23:19:11.132503 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:11.129446 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 72499us
I20260809 23:19:11.126538 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.025684 28587 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260809 23:19:11.046698 28589 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.056402 28391 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:11.138335 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 832) took 2116 ms. Trace:
W20260809 23:19:11.124537 28640 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260809 23:19:11.144062 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:11.116583 28554 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:11.138387 24429 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:11.142918 25371 rpcz_store.cc:276] 0809 23:19:09.022479 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.055311 (+ 32832us) service_pool.cc:227] Handling call
0809 23:19:11.117799 (+2062488us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:11.138258 (+ 20459us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3654501632}
I20260809 23:19:11.159364 24311 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:11.136840 28161 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.188423 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.144838 28517 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:11.199920 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.200663 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:11.118651 28336 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.130708 28381 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.209698 24311 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:11.217177 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.218331 24541 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:11.213129 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 531) took 2275 ms. Trace:
I20260809 23:19:11.142489 28683 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.208148 25368 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:11.195941 28706 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260809 23:19:11.224390 24323 raft_consensus.cc:2396] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:19:11.188774 28711 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:11.226903 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.237711 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:11.219615 24431 rpcz_store.cc:276] 0809 23:19:08.937353 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.938859 (+1001506us) service_pool.cc:227] Handling call
0809 23:19:09.984673 (+ 45814us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.120828 (+136155us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.120862 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:10.121359 (+   497us) log.cc:842] Serialized 26 byte log entry
0809 23:19:10.263494 (+142135us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:10.998836 (+735342us) log.cc:842] Serialized 14 byte log entry
0809 23:19:10.999053 (+   217us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.999123 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:11.212880 (+213757us) raft_consensus.cc:1671] finished
0809 23:19:11.212906 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.213058 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":545512,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":71617,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":186,"thread_start_us":852454,"threads_started":2,"wal-append.queue_time_us":686543}
I20260809 23:19:11.285676 24535 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:11.205291 24423 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.945s	user 0.020s	sys 0.020s
I20260809 23:19:11.244746 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.290831 24541 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:11.222162 28547 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.273865 24542 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
W20260809 23:19:11.280279 28708 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260809 23:19:11.304239 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2360) took 1486 ms. Trace:
W20260809 23:19:11.292062 24423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.020s	sys 0.020s
I20260809 23:19:11.296778 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 754) took 5153 ms. Trace:
I20260809 23:19:11.298108 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.299247 28517 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.300968 28381 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:11.281824 28569 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.083s	user 0.010s	sys 0.006s
I20260809 23:19:11.266844 28684 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.304731 24422 rpcz_store.cc:276] 0809 23:19:09.818177 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.124571 (+306394us) service_pool.cc:227] Handling call
0809 23:19:10.124896 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.125318 (+   422us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.125364 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:10.188070 (+ 62706us) log.cc:842] Serialized 26 byte log entry
0809 23:19:10.913612 (+725542us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:10.913665 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:10.913755 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:11.303990 (+390235us) raft_consensus.cc:1671] finished
0809 23:19:11.304014 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.304172 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3132,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":625331,"threads_started":1,"wal-append.queue_time_us":951902}
I20260809 23:19:11.321472 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.340134 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1222) took 5482 ms. Trace:
I20260809 23:19:11.272516 28391 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.272071 28592 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260809 23:19:11.271342 28612 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:11.317824 28700 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260809 23:19:11.351912 24312 rpcz_store.cc:276] 0809 23:19:05.876632 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.978664 (+5102032us) service_pool.cc:227] Handling call
0809 23:19:11.340017 (+361353us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:11.375371 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 911) took 1906 ms. Trace:
I20260809 23:19:11.375597 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:11.377635 28714 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260809 23:19:11.389896 24429 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:11.392082 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:11.387358 25368 rpcz_store.cc:276] 0809 23:19:09.468999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.478892 (+  9893us) service_pool.cc:227] Handling call
0809 23:19:11.142178 (+1663286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:11.240060 (+ 97882us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.375071 (+135011us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b5400403b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:11.375272 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3739792768}
I20260809 23:19:11.317049 28681 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:11.321426 28171 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:11.309613 24314 rpcz_store.cc:276] 0809 23:19:06.140348 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.159489 (+4019141us) service_pool.cc:227] Handling call
0809 23:19:10.209580 (+ 50091us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.299801 (+ 90221us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:10.815908 (+516107us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.293024 (+477116us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.293077 (+    53us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:11.293123 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.293212 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.293323 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":589216,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"thread_start_us":725745,"threads_started":2}
I20260809 23:19:11.328502 28471 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:11.409407 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 265972us
I20260809 23:19:11.411834 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:11.390198 28614 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.247s	user 0.013s	sys 0.002s
I20260809 23:19:11.413959 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:11.416280 24422 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:11.340452 28683 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:11.409569 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:11.424603 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:11.388867 28410 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.329058 28569 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.407233 28684 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:11.433693 24312 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:11.403574 28681 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:11.436867 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:11.442898 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 464454us
I20260809 23:19:11.429275 28336 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.454128 24312 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:11.450084 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:11.450722 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 468161us
I20260809 23:19:11.314442 28711 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.466248 28711 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:11.460235 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 422120us
I20260809 23:19:11.462030 24316 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
W20260809 23:19:11.451812 28197 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.322s	user 0.012s	sys 0.004s
W20260809 23:19:11.462507 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 349595us
I20260809 23:19:11.483883 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:11.484690 24270 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6.
I20260809 23:19:11.485030 24270 leader_election.cc:401] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 23:19:11.485951 24540 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:11.486063 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:11.486596 24539 raft_consensus.cc:595] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:11.471603 24311 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 23:19:11.426390 28171 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.476044 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.489238 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:11.490113 28171 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:11.454425 28004 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.464s	user 0.008s	sys 0.010s
I20260809 23:19:11.487119 24539 raft_consensus.cc:603] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:11.491151 28004 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.492713 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.493398 28004 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.495146 28539 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:11.495710 28539 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.497638 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:11.499504 24498 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 3.023s (SENT)
I20260809 23:19:11.429157 28614 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.456363 28410 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:11.458518 28684 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:11.468178 28589 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.506582 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:11.511970 28590 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.570s	user 0.008s	sys 0.003s
I20260809 23:19:11.519452 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.559607 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:11.510051 28229 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:11.511750 28410 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:11.563228 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.566107 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1907) took 5673 ms. Trace:
W20260809 23:19:11.575059 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74570.9us
W20260809 23:19:11.583364 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69909.3us
I20260809 23:19:11.578293 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.573671 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.510579 28336 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260809 23:19:11.596336 28336 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:11.564721 28229 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:11.599453 28721 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:11.600142 28721 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:11.600610 28721 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:11.601073 28478 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.663s	user 0.011s	sys 0.000s
I20260809 23:19:11.601539 28478 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.564977 28171 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:11.603236 28684 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:11.604015 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2175) took 2516 ms. Trace:
I20260809 23:19:11.604449 24534 rpcz_store.cc:276] 0809 23:19:09.087981 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.855401 (+767420us) service_pool.cc:227] Handling call
0809 23:19:09.855753 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.257320 (+401567us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:11.010272 (+752952us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.460432 (+450160us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.460480 (+    48us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:11.460529 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.603772 (+143243us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:11.603821 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.603955 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8446,"prepare.queue_time_us":616199,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":9111,"spinlock_wait_cycles":885984896,"thread_start_us":1136421,"threads_started":2,"wal-append.queue_time_us":588230}
I20260809 23:19:11.606449 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:11.609599 24317 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.008s	sys 0.028s
I20260809 23:19:11.589671 25371 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
W20260809 23:19:11.611742 24317 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.008s	sys 0.028s
I20260809 23:19:11.607482 24544 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:11.564378 28410 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:11.574007 24320 rpcz_store.cc:276] 0809 23:19:05.902693 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.134009 (+4231316us) service_pool.cc:227] Handling call
0809 23:19:10.134454 (+   445us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.250608 (+116154us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.250655 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:10.260722 (+ 10067us) log.cc:842] Serialized 26 byte log entry
0809 23:19:10.641328 (+380606us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800455b00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:11.094317 (+452989us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:11.094962 (+   645us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.095047 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:11.561530 (+466483us) raft_consensus.cc:1671] finished
0809 23:19:11.561558 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.565994 (+  4436us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83933,"raft.queue_time_us":115678,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":21128,"spinlock_wait_cycles":25320320,"thread_start_us":452784,"threads_started":1,"wal-append.queue_time_us":916031}
I20260809 23:19:11.608086 24534 raft_consensus.cc:2362] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7. Re-sending same reply.
I20260809 23:19:11.618413 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:11.625830 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 327718us
I20260809 23:19:11.628484 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.630532 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:11.599448 28229 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:11.633467 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:11.638181 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 138713us
I20260809 23:19:11.641474 25371 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 3.
I20260809 23:19:11.562896 28590 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.575058 28554 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:11.639685 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.133s (SENT) [suppressed 41 similar messages]
W20260809 23:19:11.643290 28156 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.384s	user 0.009s	sys 0.006s
I20260809 23:19:11.666421 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.671437 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:11.671893 24543 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:11.645990 28214 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:11.630708 28592 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260809 23:19:11.672222 24544 raft_consensus.cc:2362] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5. Re-sending same reply.
W20260809 23:19:11.701985 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 216897us
W20260809 23:19:11.691059 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 206495us
I20260809 23:19:11.693559 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:11.700503 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 197737us
I20260809 23:19:11.704910 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.726411 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.664522 28569 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.736346 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:11.737124 24429 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:11.674139 28612 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:11.741153 28612 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.741750 28004 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260809 23:19:11.727795 25368 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:11.743177 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.743830 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:11.744740 28480 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:19:11.745407 28480 consensus_queue.cc:1225] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260809 23:19:11.746071 28336 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6.
I20260809 23:19:11.768602 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.761010 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.733733 28614 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:11.691438 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:11.771914 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2391) took 1793 ms. Trace:
I20260809 23:19:11.774010 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:11.767313 28478 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.769066 28753 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260809 23:19:11.795850 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64304.3us
I20260809 23:19:11.797719 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:11.798202 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51310.2us
I20260809 23:19:11.799935 28737 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.162s
W20260809 23:19:11.804872 28732 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260809 23:19:11.812309 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101432us
I20260809 23:19:11.786883 28589 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:11.835527 24429 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:11.844590 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.844911 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:11.813567 28423 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.025s	user 0.015s	sys 0.000s
W20260809 23:19:11.841818 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.019s (SENT)
I20260809 23:19:11.807956 28590 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.693126 28735 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.159s
I20260809 23:19:11.823570 28592 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:11.854708 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:11.839589 28214 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:11.844259 28347 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.900s	user 0.003s	sys 0.010s
I20260809 23:19:11.863198 28347 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.863744 28768 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260809 23:19:11.863765 28214 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.864689 28768 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:11.865067 28768 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:11.865571 28347 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.865880 28768 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.867340 24545 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:11.867497 28731 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260809 23:19:11.860939 28590 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:11.841166 28753 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:11.871297 28754 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260809 23:19:11.872135 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111244us
I20260809 23:19:11.875430 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.877965 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.855930 28423 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:11.881305 24431 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
W20260809 23:19:11.881336 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.500s (SENT)
W20260809 23:19:11.876354 28729 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.282s	user 0.001s	sys 0.001s
I20260809 23:19:11.858532 28471 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:11.926537 24429 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:11.935892 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57194.3us
I20260809 23:19:11.926923 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:11.929232 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73773us
W20260809 23:19:11.929615 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88572.5us
W20260809 23:19:11.939401 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.664s (SENT)
I20260809 23:19:11.880477 28756 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:11.940302 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:11.939527 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.939916 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:11.947233 28727 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:19:11.948725 28746 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260809 23:19:11.951391 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:11.952179 24544 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:11.952994 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:11.953534 24544 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:11.954370 24542 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:11.954016 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:11.957733 24422 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.541s	user 0.008s	sys 0.000s
I20260809 23:19:11.958056 24422 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:11.958832 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:11.935958 28768 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:11.972535 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:11.972611 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:11.973502 24424 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:11.973881 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:11.974253 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:11.975615 28751 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260809 23:19:11.094113 24430 rpcz_store.cc:276] 0809 23:19:09.657844 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.973127 (+315283us) service_pool.cc:227] Handling call
0809 23:19:10.000922 (+ 27795us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.138657 (+137735us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:10.309382 (+170725us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.092906 (+783524us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.092979 (+    73us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:11.093043 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.093214 (+   171us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.093404 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":669893,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":40363,"thread_start_us":770886,"threads_started":2}
I20260809 23:19:11.977065 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:11.977617 28759 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:19:11.969540 24431 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6.
I20260809 23:19:11.978538 24430 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260809 23:19:11.979048 28747 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:11.979231 28777 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:11.979517 28747 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:11.979750 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2363) took 1022 ms. Trace:
W20260809 23:19:11.954483 28748 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260809 23:19:11.980582 28747 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:11.982388 24428 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260809 23:19:11.983466 28336 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:19:11.983909 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44480.7us
I20260809 23:19:11.985083 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1917) took 3575 ms. Trace:
I20260809 23:19:11.985925 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.992146 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:11.930114 28554 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:11.988756 28727 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:11.994706 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.986755 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:11.980156 24435 rpcz_store.cc:276] 0809 23:19:10.957552 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.958276 (+   724us) service_pool.cc:227] Handling call
0809 23:19:10.958683 (+   407us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:11.011567 (+ 52884us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:11.011601 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:11.033368 (+ 21767us) log.cc:842] Serialized 26 byte log entry
0809 23:19:11.441382 (+408014us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:11.441434 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.441519 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:11.979532 (+538013us) raft_consensus.cc:1671] finished
0809 23:19:11.979551 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.979670 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":407621,"threads_started":1,"wal-append.queue_time_us":614175}
I20260809 23:19:11.985476 24313 rpcz_store.cc:276] 0809 23:19:08.410141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.790784 (+2380643us) service_pool.cc:227] Handling call
0809 23:19:10.873489 (+ 82705us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.947815 (+ 74326us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.947854 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:10.963548 (+ 15694us) log.cc:842] Serialized 26 byte log entry
0809 23:19:11.432483 (+468935us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:11.440013 (+  7530us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.440403 (+   390us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.440669 (+   266us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:11.984880 (+544211us) raft_consensus.cc:1671] finished
0809 23:19:11.984899 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.985017 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6994,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":155320,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":456886,"threads_started":1,"wal-append.queue_time_us":585493}
I20260809 23:19:12.012682 24542 raft_consensus.cc:2396] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 23:19:11.975750 28229 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.017022 28229 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:11.996057 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:11.990007 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.943734 28756 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:11.987702 28575 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:11.982652 28747 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:11.943236 28615 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.113s	user 0.014s	sys 0.010s
I20260809 23:19:12.033020 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.032641 28615 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.034438 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.029309 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:11.963689 28471 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.035293 28780 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.035792 28780 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.036113 28780 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.036218 28471 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.037561 28554 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:12.038568 28554 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260809 23:19:12.019547 24498 leader_election.cc:334] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.727s (SENT)
W20260809 23:19:12.010789 28681 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.554s	user 0.001s	sys 0.010s
I20260809 23:19:12.047811 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.050740 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.051672 24429 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:12.050136 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.040284 28575 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:12.061674 24498 leader_election.cc:334] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.775s (SENT)
I20260809 23:19:12.061898 28575 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.062244 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.062481 28575 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.063060 28347 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:12.063251 28615 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.064600 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.064761 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.064841 28711 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.065364 24314 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.065934 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.066278 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.066609 24313 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:12.059815 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:12.060163 28721 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.067381 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.067555 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.067881 28229 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:12.068212 28772 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260809 23:19:12.068642 24312 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:12.088429 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:12.068862 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 129506us
I20260809 23:19:12.093647 24429 raft_consensus.cc:2396] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260809 23:19:12.069228 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.069682 28480 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:12.069836 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.096673 24435 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.070086 28762 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.070588 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:12.048043 28735 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260809 23:19:12.076218 24544 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:12.079130 28391 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.105309 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.106040 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:12.079412 28756 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:12.081197 28736 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.482s	user 0.001s	sys 0.005s
I20260809 23:19:12.061709 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 760) took 1061 ms. Trace:
I20260809 23:19:12.082118 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:12.120076 28214 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:12.120637 28214 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.121564 28214 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.009668 28727 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:12.123854 28727 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.125694 28727 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:12.126260 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2185) took 1825 ms (client timeout 1839 ms). Trace:
W20260809 23:19:12.126649 24535 rpcz_store.cc:269] 0809 23:19:10.301194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.372879 (+ 71685us) service_pool.cc:227] Handling call
0809 23:19:12.126073 (+1753194us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:12.126202 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1473751296}
I20260809 23:19:12.127149 28229 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:12.127640 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.128832 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.129578 24534 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
W20260809 23:19:12.083453 28523 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.953s	user 0.014s	sys 0.000s
I20260809 23:19:12.041859 28612 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260809 23:19:12.130386 28523 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.057058 28681 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.131488 28612 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:12.131880 28612 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.057405 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.132613 28523 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.132746 28681 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:12.087716 28589 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.018328 28416 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:12.107795 24422 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.110234 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.112166 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.139739 24544 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:12.119438 28754 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:12.127774 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:12.133399 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 78556.4us
I20260809 23:19:12.085495 24425 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:12.134204 24312 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 7. Ops: []
I20260809 23:19:12.137272 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:12.139757 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70655.2us
I20260809 23:19:12.139183 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.136169 24534 pending_rounds.cc:76] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 1
I20260809 23:19:12.145533 24535 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:12.144840 24534 pending_rounds.cc:97] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260809 23:19:12.145964 24534 raft_consensus.cc:2866] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:12.146629 24544 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:12.142994 24435 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:12.143594 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.135143 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.150313 24313 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.150260 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.154814 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.155169 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.156177 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.156462 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.157382 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.157671 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.158464 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.158833 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.119297 24317 rpcz_store.cc:276] 0809 23:19:11.000242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:11.010914 (+ 10672us) service_pool.cc:227] Handling call
0809 23:19:11.023995 (+ 13081us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:11.613101 (+589106us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:11.613134 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:11.638847 (+ 25713us) log.cc:842] Serialized 26 byte log entry
0809 23:19:11.772794 (+133947us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:11.803612 (+ 30818us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.803858 (+   246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.803967 (+   109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:12.061288 (+257321us) raft_consensus.cc:1671] finished
0809 23:19:12.061319 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:12.061612 (+   293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14280,"prepare.queue_time_us":30592,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":236760,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":24611,"thread_start_us":650937,"threads_started":2,"wal-append.queue_time_us":394873}
I20260809 23:19:12.159435 24430 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:12.153878 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.160238 24320 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:12.143154 28756 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:12.138984 28416 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:12.144325 28592 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:12.188010 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:12.188956 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49835us
I20260809 23:19:12.152421 28772 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:12.190433 28592 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:12.193424 28416 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:12.169036 28768 raft_consensus.cc:2741] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260809 23:19:12.207074 28768 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:12.207258 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2210) took 1376 ms. Trace:
I20260809 23:19:12.207587 28768 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:12.207644 24546 rpcz_store.cc:276] 0809 23:19:10.831117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:11.120291 (+289174us) service_pool.cc:227] Handling call
0809 23:19:11.147858 (+ 27567us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:11.672165 (+524307us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:11.672197 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.672237 (+    40us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:11.675945 (+  3708us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:12.207031 (+531086us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54001d6854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:12.207082 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:12.207203 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":725,"raft.queue_time_us":280,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":17288,"spinlock_wait_cycles":1204940032}
I20260809 23:19:12.208608 28768 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.210482 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 763) took 1151 ms. Trace:
I20260809 23:19:12.165606 28480 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:12.210837 24323 rpcz_store.cc:276] 0809 23:19:11.058714 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:11.058874 (+   160us) service_pool.cc:227] Handling call
0809 23:19:12.210273 (+1151399us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:12.210425 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293963136}
I20260809 23:19:12.211314 28480 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:19:12.212423 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.213164 24317 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.213970 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.202993 28592 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:12.204813 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.165674 28391 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 23:19:12.206449 28410 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.000s
I20260809 23:19:12.217265 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.216614 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2382) took 3431 ms. Trace:
I20260809 23:19:12.221452 24316 rpcz_store.cc:276] 0809 23:19:08.785257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.847592 (+2062335us) service_pool.cc:227] Handling call
0809 23:19:10.873865 (+ 26273us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:11.483621 (+609756us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:12.216412 (+732791us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:12.216560 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1346886016}
I20260809 23:19:12.223062 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.225200 28721 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.201579 28547 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:12.228693 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:12.243527 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 85925.4us
W20260809 23:19:12.229312 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 135523us
I20260809 23:19:12.132759 28612 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.246431 24320 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.203709 28772 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:12.248785 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 104260us
I20260809 23:19:12.251562 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.252323 24316 raft_consensus.cc:2362] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:12.253849 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.253952 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.186386 28347 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260809 23:19:12.254601 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.255067 24317 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.259668 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:12.251585 28341 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.005s	sys 0.010s
I20260809 23:19:12.261803 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.271648 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.274041 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.267558 24323 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 23:19:12.275653 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.276902 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.277105 24313 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:12.278275 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:12.275223 28341 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.005s	sys 0.010s
I20260809 23:19:12.288414 28612 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:12.291391 28791 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260809 23:19:12.298130 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 537) took 2641 ms. Trace:
W20260809 23:19:12.298893 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2169) took 4029 ms (client timeout 2506 ms). Trace:
W20260809 23:19:12.299325 24540 rpcz_store.cc:269] 0809 23:19:08.269922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:08.784657 (+514735us) service_pool.cc:227] Handling call
0809 23:19:12.298665 (+3514008us) spinlock_profiling.cc:243] Waited 2.59 s on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:12.298798 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5956771072}
I20260809 23:19:12.298686 24427 rpcz_store.cc:276] 0809 23:19:09.656989 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.952859 (+295870us) service_pool.cc:227] Handling call
0809 23:19:09.969502 (+ 16643us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:10.072930 (+103428us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.072976 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:10.188541 (+115565us) log.cc:842] Serialized 52 byte log entry
0809 23:19:10.582612 (+394071us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:11.034113 (+451501us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.035803 (+  1690us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.035891 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:12.297152 (+1261261us) spinlock_profiling.cc:243] Waited 7.12 ms on lock 0x7b2400415b08. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:19:12.297727 (+   575us) raft_consensus.cc:1671] finished
0809 23:19:12.297873 (+   146us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:12.298054 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53757,"prepare.queue_time_us":226583,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":103957,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":44012,"spinlock_wait_cycles":16375680,"thread_start_us":598820,"threads_started":2,"wal-append.queue_time_us":742440}
W20260809 23:19:12.301383 28688 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.074s	user 0.001s	sys 0.001s
I20260809 23:19:12.253201 28772 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:12.310127 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.311110 28651 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:12.311510 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.227456 28410 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.315267 24312 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:12.330135 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.330296 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.315929 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.330856 24541 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:12.277989 28547 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:12.334688 24323 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:12.324117 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64513.8us
I20260809 23:19:12.335690 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.327217 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.328285 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.337971 24313 raft_consensus.cc:2362] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6. Re-sending same reply.
I20260809 23:19:12.328755 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.275699 28347 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:12.307582 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:12.331306 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190583us
W20260809 23:19:12.331748 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 169480us
W20260809 23:19:12.267963 28554 consensus_queue.cc:1159] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 6 status { last_received { term: 6 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 6 index: 7 } }
I20260809 23:19:12.305548 28471 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:12.339476 24320 raft_consensus.cc:2362] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6. Re-sending same reply.
I20260809 23:19:12.330866 24312 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.348243 24312 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 23:19:12.306171 28229 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:12.352101 28229 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.356622 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:12.334939 28341 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:12.343004 28547 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.361783 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.261370 28575 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.364298 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.365403 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.369065 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.371145 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.374157 24313 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.369859 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:12.382740 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.384619 24431 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.355549 28471 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.386214 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.391510 24320 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.362520 28754 raft_consensus.cc:2741] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 23:19:12.382081 28336 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:12.433593 28752 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.628s	user 0.001s	sys 0.000s
I20260809 23:19:12.433328 28336 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.434387 24535 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:12.361657 28341 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.436959 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.437815 28341 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.440229 28416 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.442303 24424 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:12.398857 28569 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.399310 28539 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:12.401028 28804 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.395776 28764 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:12.405328 28565 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:19:12.406795 28703 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.140s	user 0.000s	sys 0.002s
I20260809 23:19:12.397306 28471 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:12.411473 28517 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:12.424839 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.425765 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.426103 28737 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:12.426491 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.427515 24320 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:12.474606 24323 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.429391 24311 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:12.430629 28575 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.430962 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2205) took 1705 ms. Trace:
I20260809 23:19:12.479277 24545 rpcz_store.cc:276] 0809 23:19:10.725756 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.873413 (+147657us) service_pool.cc:227] Handling call
0809 23:19:12.430731 (+1557318us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:12.430885 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2171684224}
I20260809 23:19:12.431614 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.432278 28547 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.442458 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.448596 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:12.454432 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:12.457710 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76107.2us
I20260809 23:19:12.408217 28711 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:12.459017 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.459685 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.475046 24320 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:12.477017 25323 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.312s (SENT)
I20260809 23:19:12.488481 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.474712 24313 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.451668 28336 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.455905 28764 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:12.457424 28565 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:12.473989 28727 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 23:19:12.490377 28777 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.002s
I20260809 23:19:12.473452 28517 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:12.477260 28737 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.494920 24320 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.453431 28780 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.490816 28539 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:12.508574 24320 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:12.487419 28711 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:12.511795 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:12.511104 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203650us
I20260809 23:19:12.512674 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.496374 28764 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:12.500067 28727 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:12.518124 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.532068 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 535) took 2893 ms. Trace:
I20260809 23:19:12.512458 25369 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
W20260809 23:19:12.533421 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74558.4us
I20260809 23:19:12.491633 28651 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.018s
I20260809 23:19:12.503314 28575 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:12.507062 28587 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:12.511294 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.504511 28777 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.532434 24423 rpcz_store.cc:276] 0809 23:19:09.639230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:09.950747 (+311517us) service_pool.cc:227] Handling call
0809 23:19:09.951190 (+   443us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:10.176319 (+225129us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:10.176348 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:10.176814 (+   466us) log.cc:842] Serialized 26 byte log entry
0809 23:19:10.210016 (+ 33202us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:11.312120 (+1102104us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.312464 (+   344us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:12.212685 (+900221us) spinlock_profiling.cc:243] Waited 7.86 ms on lock 0x7b1800415ae0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:19:12.213149 (+   464us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:12.213195 (+    46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:12.406513 (+193318us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b2400415b08. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:19:12.531850 (+125337us) raft_consensus.cc:1671] finished
0809 23:19:12.531878 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:12.532009 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":598780,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":242,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":6845,"spinlock_wait_cycles":365122944,"thread_start_us":1079966,"threads_started":2,"wal-append.queue_time_us":1024196}
I20260809 23:19:12.539978 24266 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6.
I20260809 23:19:12.511562 28539 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:12.511514 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.535075 28727 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:12.516687 28416 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.543601 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.545115 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.569554 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.570259 24316 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:12.572346 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.580380 24535 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:12.580525 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:12.563736 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51132.2us
I20260809 23:19:12.571820 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.570622 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.582592 25375 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:12.584811 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.586755 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.569690 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.575485 24534 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:12.518775 28004 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:12.576402 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:12.541742 28739 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.004s	sys 0.009s
I20260809 23:19:12.589218 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.590341 25372 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:12.543471 28410 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.570890 28777 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:12.596101 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.571964 28684 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.599131 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:12.599038 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53980.9us
I20260809 23:19:12.608325 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.609973 24425 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:12.610426 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.548931 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.587797 28539 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:12.596122 28739 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.005s	sys 0.012s
I20260809 23:19:12.614121 24424 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:12.616752 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.594396 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.618357 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.614918 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:12.619590 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.619786 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.621310 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.599462 28618 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:12.576236 28587 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260809 23:19:12.628368 24534 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:12.626366 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:12.625813 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76989.4us
I20260809 23:19:12.615249 28214 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.541512 28565 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.612617 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.635883 24430 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 23:19:12.642724 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:12.638000 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54663.2us
I20260809 23:19:12.622292 28569 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.640949 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.660626 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.624598 28681 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.663159 28681 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:12.663484 28832 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:12.663740 28681 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.664004 28832 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.664407 28832 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.667122 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.667738 24535 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:12.668442 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1915) took 4258 ms. Trace:
I20260809 23:19:12.668790 24309 rpcz_store.cc:276] 0809 23:19:08.409556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.742531 (+2332975us) service_pool.cc:227] Handling call
0809 23:19:10.796882 (+ 54351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.832338 (+ 35456us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:11.195748 (+363410us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.509801 (+314053us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.509853 (+    52us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:11.509915 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:12.668233 (+1158318us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:12.668276 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:12.668388 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":199028,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":6464,"raft.queue_time_us":111220,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":20751,"spinlock_wait_cycles":80613632,"thread_start_us":634520,"threads_started":2,"wal-append.queue_time_us":476292}
I20260809 23:19:12.671069 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.624028 28341 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:12.620365 28336 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:12.646693 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:12.647408 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:12.639606 28575 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
W20260809 23:19:12.569727 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:12.599180 28711 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.655267 25365 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:12.655207 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.626684 25366 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:12.703480 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.599300 28004 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.656086 28835 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:12.642270 28756 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.015s
I20260809 23:19:12.657667 28592 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.689971 28214 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260809 23:19:12.716912 28592 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.717084 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.717465 28416 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:12.717886 25375 raft_consensus.cc:2362] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7. Re-sending same reply.
I20260809 23:19:12.719909 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.721006 25368 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.625169 28554 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260809 23:19:12.643342 28618 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:12.670750 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.671149 24540 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:12.673504 24320 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260809 23:19:12.723398 24320 pending_rounds.cc:76] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:12.723667 24320 pending_rounds.cc:97] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260809 23:19:12.723951 24320 raft_consensus.cc:2866] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:12.673951 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.674307 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:12.675156 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55611.1us
I20260809 23:19:12.679404 25369 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:12.537662 28816 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:12.696388 28809 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260809 23:19:12.698710 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 187477us
I20260809 23:19:12.663095 28822 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.699205 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.617436 28777 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.747012 28822 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.747575 28822 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.748570 28777 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.750120 25366 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:19:12.686247 28807 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260809 23:19:12.578761 28517 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.753069 28517 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.630462 28768 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260809 23:19:12.626596 28171 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.723512 28569 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 23:19:12.755620 28171 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:12.756126 28171 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.755787 28539 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.585018 28737 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.757527 28737 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.758311 28539 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.759407 28618 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.760628 28004 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.704696 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:12.761260 28410 consensus_queue.cc:1159] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 5 status { last_received { term: 2 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: [4.3-4.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20260809 23:19:12.762120 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57505.1us
I20260809 23:19:12.762614 28554 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.704301 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:12.700815 28804 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:12.696283 28341 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:12.763554 28783 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260809 23:19:12.763621 25373 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:12.708043 28711 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.763823 28822 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.764125 28341 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:12.764889 28754 consensus_queue.cc:1159] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 5 status { last_received { term: 5 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 5 index: 6 } }
I20260809 23:19:12.709138 28780 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.712100 28772 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:12.715170 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84575.4us
I20260809 23:19:12.725198 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:12.725178 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54518.2us
W20260809 23:19:12.771885 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46757.9us
I20260809 23:19:12.772603 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.776736 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.776974 28727 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:12.779253 28587 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.779675 28587 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.780081 28587 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.782018 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.730535 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.630782 28819 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.132s
W20260809 23:19:12.738572 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64346us
I20260809 23:19:12.774290 28681 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.785826 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:12.724253 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 379715us
W20260809 23:19:12.739683 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106322us
I20260809 23:19:12.739806 24546 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:12.792197 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.721836 28336 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.800015 28764 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.791851 28004 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
W20260809 23:19:12.801204 28845 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260809 23:19:12.746304 28742 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.746814 25375 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:12.802281 28764 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.810801 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.769521 28822 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:12.740310 28575 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:12.739269 24534 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:12.787153 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.645079 28229 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260809 23:19:12.795120 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.785737 24314 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:12.797045 28772 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.797669 28735 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:12.752244 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.802413 28780 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.803418 28492 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.767414 28835 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260809 23:19:12.805280 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.809514 28832 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:12.859104 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 106953us
I20260809 23:19:12.859341 28836 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.859980 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.860725 28832 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:12.810726 28565 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.862040 28768 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:12.862736 28768 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:12.863114 28768 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:12.863421 28804 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:12.782497 28721 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:12.870383 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75346.7us
I20260809 23:19:12.870378 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.819065 24424 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:12.821352 24544 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:12.821584 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:38878 (request call id 25) took 1853 ms. Trace:
I20260809 23:19:12.875763 24539 rpcz_store.cc:276] 0809 23:19:10.964537 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.974737 (+ 10200us) service_pool.cc:227] Handling call
0809 23:19:12.817439 (+1842702us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:12.817552 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165576576}
I20260809 23:19:12.793176 28539 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.822005 25368 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:12.894582 24422 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:12.820670 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.820726 24540 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:12.895339 25369 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:12.859715 24316 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.738694 24535 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:12.894752 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84030.2us
I20260809 23:19:12.897189 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.800040 28471 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.898947 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:12.899837 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.902109 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.903200 24424 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:12.864809 28804 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
W20260809 23:19:12.915591 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 110427us
W20260809 23:19:12.916833 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 132229us
I20260809 23:19:12.917527 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:12.919374 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47937.4us
I20260809 23:19:12.820329 28684 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:12.926856 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56694.4us
I20260809 23:19:12.864787 28835 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260809 23:19:12.935418 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:12.936291 24534 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:12.920070 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.920302 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:12.915725 28804 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:12.847549 28727 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:12.857255 28819 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.918810 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:12.857790 28837 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260809 23:19:12.950129 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:12.951016 24309 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:12.953454 24312 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:12.933316 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:12.934022 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.956161 24535 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:12.827210 28575 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:12.856110 28336 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:12.948047 24435 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:12.948012 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.967808 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:12.969120 24424 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260809 23:19:12.925853 28478 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:12.916508 28780 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.919039 28827 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.944835 28004 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:12.919131 28735 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:12.969780 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:12.958048 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.899791 28651 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:12.968253 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.972553 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:13.001178 24428 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:13.001323 24431 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:12.977885 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44622.4us
I20260809 23:19:13.001920 25375 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:12.978842 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:12.979352 25368 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:19:12.917116 28737 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:12.985658 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.986305 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:12.942276 28065 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:12.989087 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:12.991945 28478 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.992583 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:12.996893 25363 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260809 23:19:12.919765 28171 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:12.969210 28575 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:12.919037 28822 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260809 23:19:13.008117 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:13.039172 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:13.010682 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:13.011780 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:12.939904 28539 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260809 23:19:12.939718 28840 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
W20260809 23:19:13.025695 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56002.9us
W20260809 23:19:13.026821 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 90836.9us
I20260809 23:19:13.027727 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:12.985201 28336 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:12.960394 28569 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260809 23:19:13.005570 28742 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:13.040024 24427 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260809 23:19:12.988495 28684 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:12.959259 28161 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260809 23:19:13.045280 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:13.059196 24320 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:13.046691 24428 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:13.060981 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:13.037667 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:13.049618 24423 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 23:19:13.063927 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.064116 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.066063 24545 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:13.065934 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:13.067400 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:13.068462 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148222us
I20260809 23:19:13.070915 24428 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 23:19:13.072204 24546 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:13.041051 28841 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:12.972798 28836 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.072s
I20260809 23:19:13.031720 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:13.076591 24535 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 23:19:13.080686 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.109251 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:13.064769 28711 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:13.024701 28651 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.030128 28065 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.075863 25366 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:13.114100 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:13.110103 24539 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:13.070555 28681 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.123117 24320 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:13.120487 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:13.125031 28847 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260809 23:19:13.125337 24427 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:13.023635 28804 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.126266 28804 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:13.126648 28416 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:13.127833 28171 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.128079 28618 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:13.128453 28171 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:13.128610 28618 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:13.120625 28780 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260809 23:19:13.129552 28618 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.130131 28780 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:13.130487 28780 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:13.120685 28478 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.132416 28065 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:13.133488 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.134125 25366 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:13.134634 28471 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.136606 24309 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:13.138417 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:13.076093 28841 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:13.108577 28547 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:13.150557 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:13.119418 28711 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:13.171128 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:13.073817 28744 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
W20260809 23:19:13.175997 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164308us
I20260809 23:19:13.176438 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:13.180019 24427 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:13.181295 28711 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.182392 28711 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:13.141681 28777 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:13.186383 24428 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:13.141069 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.141535 28841 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:13.193315 24428 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:13.193602 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1929) took 1128 ms. Trace:
I20260809 23:19:13.193035 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:13.194012 24311 rpcz_store.cc:276] 0809 23:19:12.065248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.069115 (+  3867us) service_pool.cc:227] Handling call
0809 23:19:12.069477 (+   362us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:12.477934 (+408457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:13.193378 (+715444us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:13.193544 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":827142528}
I20260809 23:19:13.195093 24320 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:13.137569 28612 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.046s
I20260809 23:19:13.197671 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:13.200213 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:13.204016 24427 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:13.207995 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:13.209054 24424 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:13.214169 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.191877 28777 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:13.217506 24431 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260809 23:19:13.215408 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 152245us
W20260809 23:19:13.218964 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 191302us
I20260809 23:19:13.220201 24320 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:13.223798 24311 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:13.220523 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.219020 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 791) took 1546 ms. Trace:
I20260809 23:19:13.225893 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:13.199589 28547 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:13.222627 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 155324us
I20260809 23:19:13.227702 24320 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 23:19:13.225625 25371 rpcz_store.cc:276] 0809 23:19:11.672365 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:11.672638 (+   273us) service_pool.cc:227] Handling call
0809 23:19:13.218845 (+1546207us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:13.218966 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":490961792}
I20260809 23:19:13.212208 28822 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:13.229640 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:13.234503 24535 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:13.220306 28777 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:13.179584 28336 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.220551 28861 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:13.250730 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:13.184351 28780 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.183514 28684 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:13.275974 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61875.3us
I20260809 23:19:13.237249 28822 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:13.255666 28861 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:13.255460 28517 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:13.216187 28753 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260809 23:19:13.278704 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.286412 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:13.281747 28850 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.113s	user 0.000s	sys 0.002s
I20260809 23:19:13.276669 28835 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:13.229879 28471 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260809 23:19:13.275131 28171 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 23:19:13.356338 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 81003.6us
I20260809 23:19:13.277232 28065 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:13.281047 28822 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:13.417702 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 135396us
I20260809 23:19:13.283308 28565 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:13.278606 28857 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:13.282392 28861 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:13.359398 28780 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:13.353729 28517 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:13.430637 25363 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:13.438705 28651 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.430878 28547 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.443102 25371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:13.456149 25371 raft_consensus.cc:595] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:13.444105 28884 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260809 23:19:13.445578 28867 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.209s	user 0.000s	sys 0.000s
I20260809 23:19:13.447340 28873 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:19:13.282538 28840 consensus_queue.cc:1159] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:13.451278 24431 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:13.447707 28777 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.436220 28835 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:13.439486 28857 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:13.498733 24323 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:13.458822 28565 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:13.504776 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 75562.1us
W20260809 23:19:13.511559 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 438715us
I20260809 23:19:13.285085 28862 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.182s
W20260809 23:19:13.513274 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87893.6us
I20260809 23:19:13.515311 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.517623 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.498960 28738 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:13.524312 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.504339 28857 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
W20260809 23:19:13.531541 28587 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.751s	user 0.014s	sys 0.000s
I20260809 23:19:13.532110 28587 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.532701 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:13.533653 24313 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:13.538534 25370 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260809 23:19:13.538831 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 260629us
I20260809 23:19:13.542733 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.546028 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.548620 28575 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.550081 28873 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:13.521881 28738 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:13.555696 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.519198 28753 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260809 23:19:13.493036 28854 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.318s
I20260809 23:19:13.522336 28744 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260809 23:19:13.574987 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.512509 28065 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260809 23:19:13.578747 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:13.535382 25366 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
W20260809 23:19:13.592942 28812 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260809 23:19:13.625015 28897 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:13.551513 28565 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.625160 28882 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:13.580246 28575 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.625375 28744 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
W20260809 23:19:13.619239 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44399.9us
I20260809 23:19:13.554543 28884 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:13.684785 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.628474 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:13.631531 25371 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:13.526839 28881 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260809 23:19:13.528187 28517 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:13.625620 28856 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20260809 23:19:13.627555 28587 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:13.639065 28876 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
I20260809 23:19:13.584398 28892 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:13.626785 28171 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
W20260809 23:19:13.724758 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96393.2us
I20260809 23:19:13.582947 28868 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20260809 23:19:13.697528 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.836097 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:13.691083 28858 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.410s	user 0.000s	sys 0.001s
W20260809 23:19:13.798305 28768 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.935s	user 0.011s	sys 0.003s
I20260809 23:19:13.684510 28744 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:13.739307 28804 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.613s	user 0.006s	sys 0.006s
I20260809 23:19:13.808806 28885 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:13.777745 28881 consensus_queue.cc:1225] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:13.794270 28882 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:13.734570 28856 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:13.680929 28859 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.195s
W20260809 23:19:13.888391 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 190933us
W20260809 23:19:13.881454 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45474.9us
I20260809 23:19:13.779947 28587 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:13.681253 28897 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.797839 28777 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:13.893703 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.894191 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:13.910319 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.004s	sys 0.000s
W20260809 23:19:13.923902 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.004s	sys 0.004s
I20260809 23:19:13.890650 28871 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:13.891315 28881 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:13.850459 28822 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.925340 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:13.690608 28884 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:13.926436 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 642448us
I20260809 23:19:13.902736 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.937389 24535 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:13.942102 24534 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 23:19:13.935577 25372 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:13.903301 28912 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:13.939564 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:13.943200 24424 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:13.877933 28768 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.940718 24423 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260809 23:19:13.980782 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:13.932716 28896 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:13.928285 28744 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:13.980824 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:13.980288 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:14.018675 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58320.2us
I20260809 23:19:13.889587 28885 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.027693 28684 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:14.027952 28827 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.028s	user 0.012s	sys 0.000s
I20260809 23:19:14.028374 28827 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:14.028637 28684 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:14.029033 28684 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:14.029488 28684 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:13.932581 28866 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260809 23:19:14.029084 24422 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:13.924584 28587 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:14.015758 28898 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.192s	user 0.001s	sys 0.001s
I20260809 23:19:14.045910 25371 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260809 23:19:14.046283 25371 pending_rounds.cc:76] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 5
I20260809 23:19:14.046504 25371 pending_rounds.cc:97] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:14.046746 25371 raft_consensus.cc:2866] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:14.016980 28841 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.811s	user 0.008s	sys 0.004s
I20260809 23:19:14.045925 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:14.026752 28171 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:14.054136 24427 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
W20260809 23:19:14.051004 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.006s	sys 0.007s
W20260809 23:19:14.058799 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.007s	sys 0.007s
W20260809 23:19:14.059641 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 544399us
W20260809 23:19:14.050701 28711 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.868s	user 0.013s	sys 0.000s
I20260809 23:19:14.035773 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:14.061805 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.071321 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.086576 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 559) took 1152 ms. Trace:
I20260809 23:19:14.087318 24429 rpcz_store.cc:276] 0809 23:19:12.934600 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.937842 (+  3242us) service_pool.cc:227] Handling call
0809 23:19:12.961991 (+ 24149us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:12.962203 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:12.962227 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:13.004123 (+ 41896us) log.cc:842] Serialized 26 byte log entry
0809 23:19:13.152459 (+148336us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:13.152511 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:13.152600 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:14.086238 (+933638us) raft_consensus.cc:1671] finished
0809 23:19:14.086258 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:14.086509 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":94035,"threads_started":1,"wal-append.queue_time_us":245584}
I20260809 23:19:14.088997 24424 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260809 23:19:14.075235 28903 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260809 23:19:14.074988 28899 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260809 23:19:14.068835 28873 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:14.051155 28868 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260809 23:19:14.068571 28912 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:14.075336 28911 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260809 23:19:14.099951 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 174724us
I20260809 23:19:14.076505 28711 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.053284 28841 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:14.060549 28827 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260809 23:19:14.083788 28871 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.105468 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:14.102648 28909 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260809 23:19:14.102450 28587 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:14.126420 28925 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.113s
W20260809 23:19:14.129465 28901 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260809 23:19:14.130383 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59135us
I20260809 23:19:14.137172 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.138875 28517 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:14.138952 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:14.139284 28900 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:14.140352 28912 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:14.140594 28873 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:14.140744 28912 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:14.141171 28912 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:14.133050 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:14.142453 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:14.143246 28927 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:14.143743 28927 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:14.144691 28927 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.101338 28915 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
W20260809 23:19:14.134579 28906 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260809 23:19:14.134804 28884 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:14.126765 28905 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.146s	user 0.002s	sys 0.000s
W20260809 23:19:14.147665 28780 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.687s	user 0.012s	sys 0.000s
I20260809 23:19:14.124487 28711 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.141144 28873 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:14.149262 28841 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.148942 28873 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.149475 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:14.150149 24539 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 23:19:14.129549 28929 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:14.154455 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.156244 28768 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.152561 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:14.069437 28862 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260809 23:19:14.149600 28711 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:14.111016 28822 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.160595 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.164480 28915 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:14.166229 28882 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:14.161172 28937 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:14.172904 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:14.167522 28857 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.625s	user 0.005s	sys 0.006s
I20260809 23:19:14.179766 24535 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:14.109664 28918 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260809 23:19:14.174875 28868 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:14.161614 28884 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election won for term 4
W20260809 23:19:14.186565 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 125778us
I20260809 23:19:14.196485 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:14.197319 24429 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:14.199146 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:14.200234 24427 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260809 23:19:14.202291 28917 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260809 23:19:14.202487 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:14.203182 24323 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:14.204303 24431 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:14.205127 28923 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:14.205528 28923 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:14.205888 28923 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:14.211153 28884 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:14.212047 28861 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.754s	user 0.011s	sys 0.000s
I20260809 23:19:14.212214 28884 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.212471 28861 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.214385 28861 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.214977 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:14.131193 28871 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.215597 24540 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:14.216552 28912 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.217247 28871 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.219437 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 804) took 1551 ms. Trace:
I20260809 23:19:14.219534 28885 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.220291 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:14.197878 24539 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:14.219858 25369 rpcz_store.cc:276] 0809 23:19:12.667630 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.739243 (+ 71613us) service_pool.cc:227] Handling call
0809 23:19:12.895004 (+155761us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:12.895914 (+   910us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:14.219247 (+1323333us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400443c50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:14.219381 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":357449088}
I20260809 23:19:14.223858 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.224514 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:14.224651 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:14.225752 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:14.173130 28915 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:14.227269 25372 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:14.228039 24323 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:14.231570 24323 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260809 23:19:14.231817 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:14.222225 28912 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.227331 28410 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.189525 28857 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.229725 28897 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.178567 28684 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:14.268792 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:14.233283 24423 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:14.270200 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 784) took 1220 ms. Trace:
I20260809 23:19:14.202334 25365 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:14.270414 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:14.229239 28871 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:14.281746 25369 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:14.273646 24428 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:14.274613 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:14.283680 25372 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:14.231063 28565 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:14.277586 24323 rpcz_store.cc:276] 0809 23:19:13.049725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.228880 (+1179155us) service_pool.cc:227] Handling call
0809 23:19:14.229172 (+   292us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.260093 (+ 30921us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:14.270124 (+ 10031us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:14.279907 24539 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:14.281251 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 141232us
I20260809 23:19:14.285643 24429 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:14.267458 28587 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.281731 28871 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:14.292140 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:14.292447 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:14.265944 28915 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:14.295122 25323 leader_election.cc:334] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.893s (SENT)
I20260809 23:19:14.305584 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:14.299734 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.300320 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:14.279322 28881 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:14.304451 25369 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:14.308790 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.308817 25372 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:14.309846 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:14.311221 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:14.312176 25372 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:14.313054 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:14.274420 28711 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260809 23:19:14.326249 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:14.327379 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:14.297648 28862 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.328773 28517 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:14.331241 28862 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:14.298702 28873 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.334520 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:14.276578 28410 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:14.305528 28871 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:14.283946 28925 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260809 23:19:14.356956 24539 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:14.327256 28822 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:14.317381 28897 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:14.334578 28881 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:14.313905 28863 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260809 23:19:14.369915 28857 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.297032 28565 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.327548 28939 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:14.335944 28915 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:14.368825 28410 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.314842 28587 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.357656 28936 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:14.341306 28517 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:14.383147 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57865.3us
I20260809 23:19:14.363061 28885 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:14.387881 28881 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:14.373965 28822 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.396763 25365 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260809 23:19:14.408855 24424 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:14.409179 28587 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.410684 28943 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:14.411363 28882 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:14.412639 28517 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:14.414140 28938 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260809 23:19:14.418083 28936 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:14.423813 28939 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:14.426059 28565 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:14.390399 28863 consensus_queue.cc:1225] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260809 23:19:14.339808 28873 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:14.428287 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70701.7us
I20260809 23:19:14.378894 28923 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.464407 25369 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:14.475692 28923 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:14.454442 28862 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.465343 28684 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:14.472548 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:14.456856 28882 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:14.450397 28885 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:14.482856 24424 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:14.523871 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 785) took 1473 ms. Trace:
I20260809 23:19:14.525031 28885 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:14.498073 24423 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:14.524312 24323 rpcz_store.cc:276] 0809 23:19:13.050066 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.290571 (+1240505us) service_pool.cc:227] Handling call
0809 23:19:14.291002 (+   431us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.315852 (+ 24850us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:14.315900 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:14.335148 (+ 19248us) log.cc:842] Serialized 26 byte log entry
0809 23:19:14.459930 (+124782us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:14.470077 (+ 10147us) log.cc:842] Serialized 14 byte log entry
0809 23:19:14.470347 (+   270us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:14.470441 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:14.523651 (+ 53210us) raft_consensus.cc:1671] finished
0809 23:19:14.523673 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:14.523798 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13664,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":36287,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":6295,"thread_start_us":89167,"threads_started":1,"wal-append.queue_time_us":109589}
I20260809 23:19:14.527040 28822 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:14.516767 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:14.528514 24427 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:14.527241 28684 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260809 23:19:14.531950 28881 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:14.493366 28948 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260809 23:19:14.535332 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.536058 24534 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:14.537022 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.537690 24534 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:14.538900 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:14.559336 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.155s (SENT)
I20260809 23:19:14.559980 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:14.559630 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:14.569518 24534 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:14.610208 25372 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:14.647586 24323 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:14.648770 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2405) took 1589 ms. Trace:
I20260809 23:19:14.649309 24323 rpcz_store.cc:276] 0809 23:19:13.059497 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.526640 (+1467143us) service_pool.cc:227] Handling call
0809 23:19:14.528606 (+  1966us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.648364 (+119758us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:14.648701 (+   337us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:14.471099 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:14.713667 24428 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:14.523837 24424 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:14.724189 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:14.791689 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:14.792805 24427 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:14.805810 24429 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:14.794063 28897 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:14.826131 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 266646us
I20260809 23:19:14.545451 28937 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.269s
I20260809 23:19:14.794436 28915 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:14.843153 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 119080us
I20260809 23:19:14.976854 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:14.810248 28955 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:14.502036 28923 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:14.817145 28857 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:14.518297 28936 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.977280 28897 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.953478 28885 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.133s
I20260809 23:19:14.979813 28943 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:15.252056 24534 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:14.975863 28868 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.287032 24534 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:15.371364 28873 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.053856 28923 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.377146 28897 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.378412 28868 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.286515 28943 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:15.382452 28517 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.071951 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:15.387449 28959 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:15.061399 28857 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.048813 28955 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:14.491585 28939 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.389943 24534 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.471917 24534 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:15.328146 28915 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:15.364259 28844 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.011s	sys 0.034s
I20260809 23:19:15.363068 24424 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:15.404126 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.876s	user 0.032s	sys 0.071s
W20260809 23:19:15.475859 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.948s	user 0.033s	sys 0.071s
W20260809 23:19:15.408034 28881 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.019s	sys 0.085s
W20260809 23:19:15.408408 28966 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.395s	user 0.001s	sys 0.000s
W20260809 23:19:15.469204 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 419760us
I20260809 23:19:15.475973 24424 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:15.465833 28943 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:15.470454 28959 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:15.475557 28844 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.011s	sys 0.035s
W20260809 23:19:15.478446 28881 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.020s	sys 0.086s
W20260809 23:19:15.492760 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.004s	sys 0.027s
W20260809 23:19:15.493182 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.004s	sys 0.027s
W20260809 23:19:15.493458 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 678186us
I20260809 23:19:15.482787 25365 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:15.505023 24424 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260809 23:19:15.508579 24424 pending_rounds.cc:76] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 5
I20260809 23:19:15.508961 24424 pending_rounds.cc:97] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260809 23:19:15.490836 28959 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:15.509378 24424 raft_consensus.cc:2866] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:15.511662 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 574) took 1029 ms. Trace:
I20260809 23:19:15.470077 28517 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.480532 28915 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.512238 24424 rpcz_store.cc:276] 0809 23:19:14.482034 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.482352 (+   318us) service_pool.cc:227] Handling call
0809 23:19:14.482706 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.511351 (+1028645us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.511596 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30976}
W20260809 23:19:15.512521 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.008s	sys 0.029s
I20260809 23:19:15.523396 28978 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:15.527897 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.009s	sys 0.029s
I20260809 23:19:15.531950 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2251) took 1388 ms. Trace:
I20260809 23:19:15.532485 24539 rpcz_store.cc:276] 0809 23:19:14.143802 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.379063 (+235261us) service_pool.cc:227] Handling call
0809 23:19:14.379371 (+   308us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.385191 (+  5820us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:14.385242 (+    51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:14.393367 (+  8125us) log.cc:842] Serialized 26 byte log entry
0809 23:19:14.456008 (+ 62641us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:19:14.456061 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:14.456147 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:15.531658 (+1075511us) raft_consensus.cc:1671] finished
0809 23:19:15.531687 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.531873 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5359,"raft.queue_time_us":5465,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4804,"thread_start_us":45509,"threads_started":1,"wal-append.queue_time_us":412372}
I20260809 23:19:15.549253 24539 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:15.549980 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 665) took 1019 ms. Trace:
I20260809 23:19:15.515671 28844 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.550350 24539 rpcz_store.cc:276] 0809 23:19:14.530821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.548403 (+1017582us) service_pool.cc:227] Handling call
0809 23:19:15.548815 (+   412us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.549698 (+   883us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.549914 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:15.570782 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2211) took 4684 ms (client timeout 3632 ms). Trace:
W20260809 23:19:15.573273 24543 rpcz_store.cc:269] 0809 23:19:10.879136 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.892049 (+ 12913us) service_pool.cc:227] Handling call
0809 23:19:15.562845 (+4670796us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:15.563009 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1685463424}
W20260809 23:19:15.613056 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 106329us
W20260809 23:19:15.624519 28970 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260809 23:19:15.624764 28988 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.625408 28988 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.628126 28988 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.619745 28868 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.594412 28844 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.623200 28923 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.635210 28857 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.641343 28915 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:15.644876 28871 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.274s	user 0.006s	sys 0.007s
I20260809 23:19:15.643008 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.650015 28844 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.650764 28955 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.546829 28978 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.651398 28955 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.649919 28871 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.652032 28955 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.653959 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.655333 28727 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:15.655743 28727 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:15.656580 28727 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.657532 28727 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:15.657913 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:15.658607 24543 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:15.646085 28962 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260809 23:19:15.644101 28968 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260809 23:19:15.647146 28517 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.665506 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 573) took 1184 ms. Trace:
I20260809 23:19:15.663386 28871 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.665946 24435 rpcz_store.cc:276] 0809 23:19:14.480813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.480998 (+   185us) service_pool.cc:227] Handling call
0809 23:19:14.481398 (+   400us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.497330 (+ 15932us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:14.497363 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:14.527343 (+ 29980us) log.cc:842] Serialized 26 byte log entry
0809 23:19:15.483231 (+955888us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:15.483281 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.483550 (+   269us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:15.665188 (+181638us) raft_consensus.cc:1671] finished
0809 23:19:15.665224 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.665410 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15501,"raft.queue_time_us":17306,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":948508,"threads_started":1,"wal-append.queue_time_us":946070}
I20260809 23:19:15.668602 24543 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:15.676548 24534 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:15.677269 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2254) took 1450 ms. Trace:
I20260809 23:19:15.677665 24534 rpcz_store.cc:276] 0809 23:19:14.226780 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.570307 (+343527us) service_pool.cc:227] Handling call
0809 23:19:15.251881 (+681574us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.676951 (+425070us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.677202 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.631942 28969 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.662295 28959 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.692627 28871 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:15.695761 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:15.696331 28912 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260809 23:19:15.696395 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 786) took 2568 ms. Trace:
I20260809 23:19:15.696909 24323 rpcz_store.cc:276] 0809 23:19:13.127574 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.650419 (+1522845us) service_pool.cc:227] Handling call
0809 23:19:14.925426 (+275007us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:14.925666 (+   240us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:14.925704 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:14.926244 (+   540us) log.cc:842] Serialized 26 byte log entry
0809 23:19:15.494772 (+568528us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:15.494807 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.494891 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:15.696181 (+201290us) raft_consensus.cc:1671] finished
0809 23:19:15.696202 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.696327 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":313929,"threads_started":1,"wal-append.queue_time_us":632495}
I20260809 23:19:15.698316 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.698750 24323 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:15.699843 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 857) took 1405 ms. Trace:
I20260809 23:19:15.700407 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 787) took 2498 ms. Trace:
I20260809 23:19:15.700194 25372 rpcz_store.cc:276] 0809 23:19:14.294556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.312814 (+ 18258us) service_pool.cc:227] Handling call
0809 23:19:15.699683 (+1386869us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:15.699789 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":473839872}
I20260809 23:19:15.702234 28983 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:15.702450 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.700948 24323 rpcz_store.cc:276] 0809 23:19:13.201681 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.698192 (+2496511us) service_pool.cc:227] Handling call
0809 23:19:15.698508 (+   316us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:15.700102 (+  1594us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.700360 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.703191 25372 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:15.700894 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.704993 24543 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:15.706139 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.652774 28978 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.712638 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 788) took 2500 ms. Trace:
I20260809 23:19:15.713101 24323 rpcz_store.cc:276] 0809 23:19:13.212572 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.703862 (+2491290us) service_pool.cc:227] Handling call
0809 23:19:15.712011 (+  8149us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.712241 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:15.712279 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:15.712343 (+    64us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:15.712392 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.712455 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.712577 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.710345 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.716255 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 789) took 2429 ms. Trace:
I20260809 23:19:15.716717 24323 rpcz_store.cc:276] 0809 23:19:13.286885 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.714333 (+2427448us) service_pool.cc:227] Handling call
0809 23:19:15.714686 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.714886 (+   200us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:15.714921 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:15.715963 (+  1042us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:15.716012 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.716080 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.716199 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.718312 24323 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:15.718977 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2406) took 2206 ms. Trace:
I20260809 23:19:15.719439 24323 rpcz_store.cc:276] 0809 23:19:13.512085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.717777 (+2205692us) service_pool.cc:227] Handling call
0809 23:19:15.718113 (+   336us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.718720 (+   607us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.718903 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:15.724337 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54764.7us
I20260809 23:19:15.734804 28844 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:15.735452 28844 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.709719 28912 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:15.737815 28912 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.707716 28969 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.738628 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.831728 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1970) took 2311 ms. Trace:
W20260809 23:19:15.746371 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 775) took 3208 ms (client timeout 2241 ms). Trace:
I20260809 23:19:15.833545 24323 rpcz_store.cc:276] 0809 23:19:13.520679 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.720466 (+2199787us) service_pool.cc:227] Handling call
0809 23:19:15.720774 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.720996 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:15.721029 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:15.721080 (+    51us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:15.721128 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.721199 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.831654 (+110455us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:15.834105 24313 rpcz_store.cc:269] 0809 23:19:12.537743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.532338 (+994595us) service_pool.cc:227] Handling call
0809 23:19:15.742908 (+2210570us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.793787 28971 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.839939 28971 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.841509 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 808) took 1949 ms. Trace:
I20260809 23:19:15.721691 28955 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.842398 28844 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.712061 28983 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.843366 28971 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.845479 28983 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.844842 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1225) took 2645 ms. Trace:
I20260809 23:19:15.846560 24313 rpcz_store.cc:276] 0809 23:19:13.199773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.843633 (+2643860us) service_pool.cc:227] Handling call
0809 23:19:15.844760 (+  1127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.740171 28959 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.752940 28836 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:15.849884 28836 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.738489 28943 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.842005 25371 rpcz_store.cc:276] 0809 23:19:13.892537 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.896487 (+  3950us) service_pool.cc:227] Handling call
0809 23:19:13.920705 (+ 24218us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:14.050261 (+129556us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.841265 (+1791004us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:15.841445 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":287044864}
I20260809 23:19:15.852406 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 844) took 2880 ms. Trace:
I20260809 23:19:15.852809 25375 rpcz_store.cc:276] 0809 23:19:12.972255 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.972724 (+   469us) service_pool.cc:227] Handling call
0809 23:19:12.997890 (+ 25166us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:13.002327 (+  4437us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.852178 (+2849851us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:15.852340 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8820992}
I20260809 23:19:15.853794 28683 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:15.854418 28683 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.857092 28683 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.859889 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.860333 24541 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:15.860915 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 646) took 3196 ms. Trace:
I20260809 23:19:15.861061 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:15.883718 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96551.6us
I20260809 23:19:15.852833 28969 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.890247 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:15.886807 24541 rpcz_store.cc:276] 0809 23:19:12.664536 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.859863 (+3195327us) service_pool.cc:227] Handling call
0809 23:19:15.860157 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.860689 (+   532us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.860851 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.820354 25365 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
W20260809 23:19:15.903831 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 951) took 1424 ms (client timeout 1559 ms). Trace:
I20260809 23:19:15.904081 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.904451 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.904711 24543 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:15.897007 28959 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:15.904268 25365 rpcz_store.cc:269] 0809 23:19:14.479501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.953274 (+473773us) service_pool.cc:227] Handling call
0809 23:19:15.903754 (+950480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.900477 28836 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.902199 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:15.903157 28480 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:15.907755 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:15.908170 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 949) took 1499 ms. Trace:
I20260809 23:19:15.915175 25371 rpcz_store.cc:276] 0809 23:19:14.399287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.886776 (+1487489us) service_pool.cc:227] Handling call
0809 23:19:15.897959 (+ 11183us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.898112 (+   153us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:15.898126 (+    14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:15.898153 (+    27us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:15.898176 (+    23us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.898209 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:15.898283 (+    74us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:15.911489 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:15.916635 28683 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:15.912750 24541 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:15.912734 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:15.915299 24534 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:15.905532 28959 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.920938 28993 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:15.921366 28805 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.920435 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:15.924149 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:15.917229 28683 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:15.931010 28943 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:15.886910 28727 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260809 23:19:15.956000 28683 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.915014 28480 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:15.959298 25365 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260809 23:19:15.959738 28480 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:15.911252 28969 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:15.960938 28969 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:15.961616 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.933087 28836 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:15.969134 28836 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:15.969743 28836 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:15.970031 28969 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:15.952466 28993 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.971688 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:15.972993 28993 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.974205 28927 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:15.962420 28805 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:15.976397 28959 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.981045 28912 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.978401 28805 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:15.987885 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.992561 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:15.993415 25371 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:15.995740 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 650) took 3330 ms. Trace:
I20260809 23:19:15.996379 24544 rpcz_store.cc:276] 0809 23:19:12.665741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.685871 (+ 20130us) service_pool.cc:227] Handling call
0809 23:19:12.686191 (+   320us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:12.899405 (+213214us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.995505 (+3096100us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:15.995682 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310243200}
W20260809 23:19:15.997865 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73770.6us
I20260809 23:19:15.999661 28943 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:15.999975 28727 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:19:15.998498 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 78138.7us
I20260809 23:19:16.000332 28943 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:16.004156 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.000603 28727 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.089787 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1971) took 2568 ms. Trace:
W20260809 23:19:16.076555 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73491.9us
I20260809 23:19:16.094008 24541 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:16.005604 24544 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.115658 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:16.090056 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 860) took 1794 ms. Trace:
I20260809 23:19:16.007732 25371 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:16.119027 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 859) took 1823 ms. Trace:
I20260809 23:19:16.120433 25372 rpcz_store.cc:276] 0809 23:19:14.295211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.723297 (+1428086us) service_pool.cc:227] Handling call
0809 23:19:15.723662 (+   365us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:15.723927 (+   265us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:15.897339 (+173412us) log.cc:842] Serialized 14 byte log entry
0809 23:19:16.118657 (+221318us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:16.118713 (+    56us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:16.118761 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:16.118824 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:16.118971 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2271,"prepare.queue_time_us":107835,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":287958,"threads_started":2}
I20260809 23:19:16.085887 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.138388 24543 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:16.113590 24323 rpcz_store.cc:276] 0809 23:19:13.521089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.834725 (+2313636us) service_pool.cc:227] Handling call
0809 23:19:15.835085 (+   360us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:15.835423 (+   338us) raft_consensus.cc:1520] Early marking committed up to index 3
0809 23:19:15.835441 (+    18us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:15.835751 (+   310us) log.cc:842] Serialized 26 byte log entry
0809 23:19:15.917404 (+ 81653us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:15.917482 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.917539 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:16.089582 (+172043us) raft_consensus.cc:1671] finished
0809 23:19:16.089601 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:16.089728 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87,"thread_start_us":31159,"threads_started":1,"wal-append.queue_time_us":202115}
I20260809 23:19:15.998978 28805 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:16.153683 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:16.076020 28966 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260809 23:19:16.084661 28852 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.864s	user 0.010s	sys 0.004s
I20260809 23:19:16.007395 28943 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.116437 25375 rpcz_store.cc:276] 0809 23:19:14.295456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:15.854101 (+1558645us) service_pool.cc:227] Handling call
0809 23:19:15.854550 (+   449us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:15.861675 (+  7125us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:15.861709 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:15.888683 (+ 26974us) log.cc:842] Serialized 26 byte log entry
0809 23:19:15.920003 (+ 31320us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:15.920050 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.920122 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:16.089780 (+169658us) raft_consensus.cc:1671] finished
0809 23:19:16.089808 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:16.089989 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4275,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":141,"thread_start_us":19734,"threads_started":1,"wal-append.queue_time_us":101636}
I20260809 23:19:16.156860 25372 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:16.081311 29010 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.160815 28835 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:16.120311 28727 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:16.157142 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.084435 28754 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.171478 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1953) took 3402 ms. Trace:
I20260809 23:19:16.001423 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:16.172708 24544 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:16.080317 28993 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:16.164543 28868 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.002s
I20260809 23:19:16.005391 28959 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.160s
I20260809 23:19:16.175578 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.176640 25371 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:16.172820 24430 rpcz_store.cc:276] 0809 23:19:12.769222 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.769368 (+   146us) service_pool.cc:227] Handling call
0809 23:19:12.775360 (+  5992us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:12.775741 (+   381us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:12.897109 (+121368us) log.cc:842] Serialized 14 byte log entry
0809 23:19:13.038606 (+141497us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:13.734541 (+695935us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:13.734984 (+   443us) log.cc:842] Serialized 26 byte log entry
0809 23:19:13.735161 (+   177us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:13.735206 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:13.735304 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:16.016283 (+2280979us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:16.171259 (+154976us) raft_consensus.cc:1671] finished
0809 23:19:16.171282 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:16.171419 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24432,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":9275,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":9640,"spinlock_wait_cycles":9766016,"thread_start_us":204737,"threads_started":2,"wal-append.queue_time_us":264354}
I20260809 23:19:16.166853 28835 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:16.162460 28996 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:16.211179 28984 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.181250 29003 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:16.211967 28984 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.213341 29003 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.214862 29003 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.159916 28805 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.161123 28927 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20260809 23:19:16.227870 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74308.2us
I20260809 23:19:16.178707 28754 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.166515 29010 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.208082 28835 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.232439 28927 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:16.217427 28996 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.240530 28836 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.227962 28739 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:16.181495 28993 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.229635 28959 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:16.243916 28835 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:16.248797 28739 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:16.254652 28959 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.255887 28923 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:16.276700 28959 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:16.276081 28996 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.293092 25372 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:16.295511 28754 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.303153 29022 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.305889 28805 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.308930 28480 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.310258 29010 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.314391 28996 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.314682 28943 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.315191 28996 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.316272 28480 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.285571 28745 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.319995 28868 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:16.321118 28836 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.321290 28868 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:16.324453 28993 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.329160 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.329628 25372 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:16.279685 28923 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.332832 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:16.333036 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.241s (SENT)
I20260809 23:19:16.333405 25372 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:16.333612 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:16.336369 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 160859us
I20260809 23:19:16.337059 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.339227 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.339496 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.340122 24539 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:16.341406 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.341364 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.342986 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.343487 24544 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:16.345402 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 853) took 3305 ms (client timeout 1994 ms). Trace:
W20260809 23:19:16.345906 25368 rpcz_store.cc:269] 0809 23:19:13.031671 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.064791 (+ 33120us) service_pool.cc:227] Handling call
0809 23:19:16.336378 (+3271587us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:16.336504 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1701772544}
I20260809 23:19:16.348667 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.349471 24539 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:16.328745 28871 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.352213 24317 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:16.303925 29022 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.353322 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.319738 28984 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:16.362110 29020 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260809 23:19:16.319677 28745 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.363551 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.345559 29003 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:16.352559 29023 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:16.310983 28739 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.365118 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.366541 25372 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:16.381848 24541 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:16.365100 29023 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.387172 28943 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.418879 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.422369 28739 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:16.388127 28996 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.381610 29003 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.427217 29003 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.427762 28914 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:16.381217 28871 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.433387 24539 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:16.434537 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.435113 24539 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.439563 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:16.440169 29018 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260809 23:19:16.421808 28727 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.386880 29023 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:16.474114 24543 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
W20260809 23:19:16.469461 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 159189us
I20260809 23:19:16.475003 24543 raft_consensus.cc:595] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:16.472971 28969 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.487754 28969 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:16.491360 29035 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.060s	user 0.002s	sys 0.000s
I20260809 23:19:16.476752 28943 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:16.495504 28943 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.496055 28923 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.497052 28943 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.499431 28727 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.412143 28997 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.185s
I20260809 23:19:16.514426 24541 pending_rounds.cc:76] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 4
I20260809 23:19:16.510586 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.516095 24541 pending_rounds.cc:97] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260809 23:19:16.516480 24541 raft_consensus.cc:2866] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:16.512835 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1936) took 4417 ms (client timeout 1540 ms). Trace:
W20260809 23:19:16.517776 24314 rpcz_store.cc:269] 0809 23:19:12.095451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.106207 (+ 10756us) service_pool.cc:227] Handling call
0809 23:19:16.512645 (+4406438us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b540043bcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:16.512742 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1268381056}
I20260809 23:19:16.528393 24544 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:16.533329 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.533478 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.535025 28969 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:16.537824 25368 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:16.538517 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2402) took 3514 ms. Trace:
I20260809 23:19:16.538977 24311 rpcz_store.cc:276] 0809 23:19:13.024036 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.195458 (+171422us) service_pool.cc:227] Handling call
0809 23:19:13.223548 (+ 28090us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:13.224405 (+   857us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:16.538304 (+3313899us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b54000dc0d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:16.538449 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36734848}
I20260809 23:19:16.539912 28871 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.530735 24539 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:16.547937 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.562322 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.568397 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.572150 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
I20260809 23:19:16.542027 28745 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.571187 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:16.579356 29029 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:16.579856 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 780) took 4006 ms (client timeout 2155 ms). Trace:
I20260809 23:19:16.550657 28969 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.060s
I20260809 23:19:16.590539 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.591226 28416 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.603298 28923 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.633730 28416 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.633827 28923 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.596688 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1952) took 3876 ms. Trace:
I20260809 23:19:16.557024 28754 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.639309 28754 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.644691 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:16.646369 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:16.647569 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:16.615221 24316 rpcz_store.cc:269] 0809 23:19:12.573056 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.580277 (+  7221us) service_pool.cc:227] Handling call
0809 23:19:16.579607 (+3999330us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:16.579780 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":566374144}
I20260809 23:19:16.573913 29022 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.659583 29022 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:16.661078 28416 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.642594 28754 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.619952 28959 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.668949 28959 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.669065 28835 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.594287 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.623991 29031 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:16.626348 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 647) took 3961 ms. Trace:
I20260809 23:19:16.628422 29025 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:16.671109 24545 rpcz_store.cc:276] 0809 23:19:12.664804 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.685072 (+ 20268us) service_pool.cc:227] Handling call
0809 23:19:12.685425 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:13.070283 (+384858us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:16.625910 (+3555627us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:16.626256 (+   346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":858202880}
I20260809 23:19:16.672202 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.672838 24543 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:16.671999 29025 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.673452 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:16.674178 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:16.675158 24316 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:16.675606 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 790) took 3140 ms. Trace:
I20260809 23:19:16.675962 24316 rpcz_store.cc:276] 0809 23:19:13.535650 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:16.673853 (+3138203us) service_pool.cc:227] Handling call
0809 23:19:16.675555 (+  1702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:16.637256 24425 rpcz_store.cc:276] 0809 23:19:12.720295 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.720480 (+   185us) service_pool.cc:227] Handling call
0809 23:19:12.741283 (+ 20803us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:12.767233 (+ 25950us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:12.770179 (+  2946us) log.cc:842] Serialized 14 byte log entry
0809 23:19:12.938591 (+168412us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:12.938659 (+    68us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:12.938726 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:16.075216 (+3136490us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b1800463180. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11
0809 23:19:16.595787 (+520571us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b1800463180. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:19:16.596469 (+   682us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:16.596505 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:16.596620 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5804,"prepare.queue_time_us":18943,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":5949,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":13924,"spinlock_wait_cycles":98372480,"thread_start_us":156016,"threads_started":1,"wal-append.queue_time_us":169056}
I20260809 23:19:16.677758 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.647676 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.648278 24534 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:16.575861 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:16.679603 24544 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:16.680912 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.682598 24543 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:16.661849 29023 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.637135 28923 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:16.691574 24496 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.226s (SENT) [suppressed 44 similar messages]
W20260809 23:19:16.694921 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119165us
I20260809 23:19:16.695290 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.695572 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:16.683817 28754 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.709353 28993 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:16.710041 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:16.694262 28798 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 4.174s	user 0.022s	sys 0.017s
I20260809 23:19:16.688859 28783 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.718536 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:16.710554 29029 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.705418 28804 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:16.693302 29031 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.705346 28979 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:16.694418 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.705838 29003 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:16.730762 28959 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:16.717638 28993 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.797955 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.719512 28416 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:16.763216 28979 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:16.763446 28804 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:16.719419 28754 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.798977 25371 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:16.803351 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:16.808426 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 239912us
I20260809 23:19:16.811281 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.785561 25365 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:16.799717 28993 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.820729 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:16.796561 29003 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.795694 29025 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.749143 28783 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.809814 28979 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.824731 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:16.829180 24544 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 5
I20260809 23:19:16.827646 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:16.811184 28804 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.832616 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:16.844117 24544 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:16.817422 29038 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.847774 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:16.852909 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:16.853747 29040 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260809 23:19:16.844761 28835 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.855793 24539 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:16.854728 28754 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.859020 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.846953 29025 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.846834 28983 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:16.865398 24534 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:16.875633 29051 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.851231 29052 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.810427 29029 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.867889 29023 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:16.868281 28929 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:16.879226 28983 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:16.907562 29046 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:19:16.925601 24540 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 7. Ops: []
I20260809 23:19:16.932293 24534 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 23:19:16.891569 29003 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.846460 29031 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.879534 29038 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.941712 29038 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.889932 29055 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:16.962853 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:16.919039 28983 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:16.902660 29054 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:16.881034 29025 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.970341 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:16.976575 24534 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:16.924166 28929 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:16.977963 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:16.942790 28871 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260809 23:19:16.886360 29051 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.993916 28929 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.965727 29055 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:16.998894 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:16.999738 25365 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:16.961021 29038 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260809 23:19:16.982903 29054 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.030275 29055 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.040848 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 557) took 4107 ms. Trace:
W20260809 23:19:17.040007 29065 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260809 23:19:17.041400 24422 rpcz_store.cc:276] 0809 23:19:12.933865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.948439 (+ 14574us) service_pool.cc:227] Handling call
0809 23:19:12.948728 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:14.037405 (+1088677us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.040538 (+3003133us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54003eb7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:17.040762 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2484170112}
I20260809 23:19:17.030495 28255 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260809 23:19:17.045454 25365 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:17.058439 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:17.029791 28870 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:17.060839 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:17.054451 28416 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.073738 28416 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.075887 29047 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
I20260809 23:19:17.063225 29025 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.077533 29025 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.055051 28783 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.078678 28783 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:17.079170 28783 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:17.080459 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2395) took 4140 ms. Trace:
W20260809 23:19:17.064595 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 779) took 4524 ms (client timeout 1893 ms). Trace:
I20260809 23:19:17.080819 24317 rpcz_store.cc:276] 0809 23:19:12.939926 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.949279 (+  9353us) service_pool.cc:227] Handling call
0809 23:19:12.988427 (+ 39148us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:16.354586 (+3366159us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.080276 (+725690us) spinlock_profiling.cc:243] Waited 3.36 s on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:17.080408 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7735516416}
I20260809 23:19:17.064736 24309 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260809 23:19:17.063971 28900 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:17.079293 25373 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:17.078154 29025 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.092878 29025 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:16.890041 29052 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.070194 28993 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:16.998884 29051 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.082772 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 845) took 4098 ms. Trace:
I20260809 23:19:17.083073 24534 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 23:19:17.104960 25373 rpcz_store.cc:276] 0809 23:19:12.984841 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.985429 (+   588us) service_pool.cc:227] Handling call
0809 23:19:17.078764 (+4093335us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:17.079026 (+   262us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:17.082504 (+  3478us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.082709 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25046400}
I20260809 23:19:17.106890 28721 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.107000 29054 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.107686 28721 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:17.081431 24312 rpcz_store.cc:269] 0809 23:19:12.541139 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.573551 (+ 32412us) service_pool.cc:227] Handling call
0809 23:19:17.064355 (+4490804us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:17.064526 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":822328704}
I20260809 23:19:17.084820 28900 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:17.084964 28416 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.085758 25365 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:17.088210 28735 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:17.112413 28735 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:17.112879 25365 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:17.114428 29025 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.114988 25365 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 8. Ops: [6.7-6.7]
I20260809 23:19:17.112377 28900 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.116016 28959 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:17.117120 29054 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.114450 28416 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260809 23:19:17.088508 24540 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260809 23:19:17.090562 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:17.100111 29052 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.101226 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.121904 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:17.079798 28783 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.119594 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:17.094659 28870 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:17.114001 28735 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.134914 25365 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.136714 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.120400 28681 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.120993 28993 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.141289 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.141942 24317 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.142266 28979 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.146271 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.146505 24313 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:17.126554 28416 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260809 23:19:17.149612 28754 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.149866 24314 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260809 23:19:17.151403 28681 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.152724 28979 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.119937 29025 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.155771 28681 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.143193 28870 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.126652 29051 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.158471 28735 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.163861 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.165488 24314 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:17.125176 29054 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.150315 28754 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.192096 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.193140 24314 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:17.195079 24313 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260809 23:19:17.198530 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 230888us
I20260809 23:19:17.209064 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:17.210292 28914 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:17.210776 28914 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:17.211318 28914 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.211714 28754 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:17.209195 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.528s (SENT)
I20260809 23:19:17.213447 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.213663 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:17.215880 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.530s (SENT)
I20260809 23:19:17.230675 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.239144 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:17.240185 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.559s (SENT)
I20260809 23:19:17.238613 28738 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.227873 28681 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:17.174860 29051 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.247735 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.256369 24313 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:17.259312 28721 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.261185 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.263278 29063 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.268040 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.269703 24312 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.275137 28721 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.276902 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.278267 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.279860 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.281302 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.278914 24314 raft_consensus.cc:2362] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:17.285596 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.286288 24314 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:17.282629 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.287770 24313 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:17.288686 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.299546 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.299610 29025 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.302129 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.303139 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.304153 24314 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:17.306375 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.307746 28391 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:17.307803 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.213212 29047 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260809 23:19:17.335379 29051 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.250677 28983 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.350792 29054 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.352325 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.353263 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:17.355110 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.356523 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.357369 28983 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.357867 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.264452 29063 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.360741 28894 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:17.361367 28894 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:17.366135 24317 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:17.373745 24312 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:17.375990 24317 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:17.376144 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.358566 24313 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:17.382157 24313 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260809 23:19:17.378340 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 400446us
I20260809 23:19:17.393913 29063 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.395166 28873 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:17.395814 28873 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:17.397370 28873 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.399466 28873 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:17.404503 28914 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.409296 28914 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.410123 28914 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:17.411324 28914 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:17.411765 28914 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:17.412076 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 965) took 1084 ms. Trace:
I20260809 23:19:17.412199 28914 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.419509 28871 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:17.420053 28871 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:17.421128 28871 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.422513 28871 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:17.412679 25372 rpcz_store.cc:276] 0809 23:19:16.327280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:16.334086 (+  6806us) service_pool.cc:227] Handling call
0809 23:19:17.411865 (+1077779us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:17.412006 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26473216}
W20260809 23:19:17.438459 24269 connection.cc:232] RPC call timeout handler was delayed by -2.2391032287850976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260809 23:19:17.439348 24269 connection.cc:232] RPC call timeout handler was delayed by -2.1519912397488952s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
W20260809 23:19:17.439882 24269 connection.cc:232] RPC call timeout handler was delayed by -2.1464032288640738s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
W20260809 23:19:17.440271 24269 connection.cc:232] RPC call timeout handler was delayed by -1.8179032288026065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
W20260809 23:19:17.440639 24269 connection.cc:232] RPC call timeout handler was delayed by -1.1477345009334385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2747s for a response
I20260809 23:19:17.441609 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.449486 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.461360 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.463042 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:17.460337 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 781) took 4409 ms. Trace:
I20260809 23:19:17.466750 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.467825 24313 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:17.468967 24320 rpcz_store.cc:276] 0809 23:19:13.048930 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.228504 (+179574us) service_pool.cc:227] Handling call
0809 23:19:13.253203 (+ 24699us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:13.254207 (+  1004us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:13.254235 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:16.586863 (+3332628us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:16.589295 (+  2432us) log.cc:842] Serialized 52 byte log entry
0809 23:19:16.967411 (+378116us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:17.442772 (+475361us) log.cc:842] Serialized 14 byte log entry
0809 23:19:17.443424 (+   652us) log.cc:842] Serialized 14 byte log entry
0809 23:19:17.455464 (+ 12040us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.455564 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:17.457310 (+  1746us) raft_consensus.cc:1671] finished
0809 23:19:17.457335 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:17.457522 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11602,"prepare.queue_time_us":303421,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":87,"raft.queue_time_us":3006,"raft.run_cpu_time_us":552,"raft.run_wall_time_us":9677,"spinlock_wait_cycles":51253760,"thread_start_us":835795,"threads_started":2,"wal-append.queue_time_us":608297}
I20260809 23:19:17.456283 29054 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:17.476478 29067 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:17.491412 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.491492 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.464730 29063 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:17.476995 29025 raft_consensus.cc:2741] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 23:19:17.492619 29067 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:17.494736 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.476101 29051 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:17.496994 24320 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:17.493875 24314 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:17.497223 24313 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:17.498248 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.498682 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.506333 24313 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:17.483907 29052 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260809 23:19:17.514982 24320 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:17.517277 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.521625 24314 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:17.529858 28914 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.525903 29067 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.536262 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.538801 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.546008 24314 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:17.546926 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.547631 24314 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:17.540858 29067 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.547547 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.548818 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.550784 24314 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:17.552582 28900 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:17.551716 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.553141 28900 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.553527 24314 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:17.555256 28900 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.556497 28914 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.559656 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.560396 24313 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:17.561012 24314 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:17.561118 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.562112 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.562307 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.562883 24313 raft_consensus.cc:2362] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5. Re-sending same reply.
I20260809 23:19:17.545825 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.564604 24312 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:17.564890 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.566601 24313 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260809 23:19:17.544723 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.569304 24320 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:17.569711 28983 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.567718 24312 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:17.568775 24317 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:17.570247 28983 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.572271 28983 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.568776 28744 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:17.584868 28977 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.585382 28977 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.562906 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
W20260809 23:19:17.629853 29024 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.248s	user 0.004s	sys 0.004s
W20260809 23:19:17.662887 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 858) took 3368 ms (client timeout 1611 ms). Trace:
W20260809 23:19:17.663419 25369 rpcz_store.cc:269] 0809 23:19:14.294885 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:14.325907 (+ 31022us) service_pool.cc:227] Handling call
0809 23:19:17.662738 (+3336831us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540073e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:17.662824 (+    86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":309805952}
I20260809 23:19:17.707859 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.708963 28983 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260809 23:19:17.709110 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 468911us
I20260809 23:19:17.710839 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.711099 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.711647 24313 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:17.713073 28983 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.715054 28983 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.717443 28983 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:17.719174 28983 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:17.719947 28983 consensus_queue.cc:1225] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:17.720053 28937 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:19:17.720821 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.721609 28937 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:17.721669 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:17.722081 28937 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.722440 28983 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:17.722493 28937 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:17.722579 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.723091 28983 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:17.723515 28983 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:17.723598 28539 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260809 23:19:17.724578 29069 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.228s	user 0.003s	sys 0.000s
I20260809 23:19:17.724740 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:17.726164 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:17.728950 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2286) took 1124 ms. Trace:
I20260809 23:19:17.728995 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.732129 28983 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.733024 28937 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.735137 28937 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.729627 24539 rpcz_store.cc:276] 0809 23:19:16.604142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:16.609750 (+  5608us) service_pool.cc:227] Handling call
0809 23:19:16.610140 (+   390us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:16.874733 (+264593us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.728709 (+853976us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:17.728892 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":564371712}
I20260809 23:19:17.735661 28937 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:17.736092 28937 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:17.736575 28937 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:17.739291 29100 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:17.739769 29100 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:17.741822 29100 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:17.746107 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 959) took 1766 ms. Trace:
I20260809 23:19:17.746948 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2208) took 4495 ms. Trace:
I20260809 23:19:17.746541 25375 rpcz_store.cc:276] 0809 23:19:15.979425 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:16.163547 (+184122us) service_pool.cc:227] Handling call
0809 23:19:16.167964 (+  4417us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:16.168381 (+   417us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:16.168410 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:16.169114 (+   704us) log.cc:842] Serialized 26 byte log entry
0809 23:19:16.250103 (+ 80989us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:16.250159 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:16.250255 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:17.629084 (+1378829us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:17.745693 (+116609us) raft_consensus.cc:1671] finished
0809 23:19:17.745727 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:17.746019 (+   292us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1457,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"spinlock_wait_cycles":13622656,"thread_start_us":80472,"threads_started":1,"wal-append.queue_time_us":119470}
I20260809 23:19:17.747340 24546 rpcz_store.cc:276] 0809 23:19:13.251508 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.251707 (+   199us) service_pool.cc:227] Handling call
0809 23:19:13.252688 (+   981us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:13.418158 (+165470us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:13.887970 (+469812us) log.cc:842] Serialized 14 byte log entry
0809 23:19:14.037941 (+149971us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:17.734297 (+3696356us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:17.734691 (+   394us) log.cc:842] Serialized 26 byte log entry
0809 23:19:17.744228 (+  9537us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:17.744522 (+   294us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.744629 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:17.746586 (+  1957us) raft_consensus.cc:1671] finished
0809 23:19:17.746699 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:17.746842 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51356,"prepare.queue_time_us":349844,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":203426,"raft.run_cpu_time_us":437,"raft.run_wall_time_us":29121,"spinlock_wait_cycles":23190272,"thread_start_us":712890,"threads_started":4,"wal-append.queue_time_us":215737,"wal-append.run_cpu_time_us":1246,"wal-append.run_wall_time_us":1058833}
I20260809 23:19:17.749814 28983 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.747181 28937 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.751183 28983 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:17.751644 29103 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:17.755250 28937 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:17.756404 24323 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260809 23:19:17.758234 29100 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.760623 29100 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:17.771226 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.771840 24320 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:17.772238 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.772835 24323 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:17.773614 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.774129 24323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:17.775305 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.775899 24314 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:17.776621 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.777129 24314 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:17.778411 28590 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.778900 28590 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.779661 24320 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:17.780778 28590 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.780982 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.781600 24312 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:17.784914 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.785698 24546 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:17.782624 28590 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20260809 23:19:17.788118 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.774410 29105 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260809 23:19:17.793809 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.794732 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:17.795441 24541 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.795603 24540 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:17.796940 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.798436 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.799603 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.799585 29105 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:17.800717 24540 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:17.802659 24539 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:17.804898 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.805612 24539 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:17.809511 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.810285 24539 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:17.813275 29051 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.813482 29022 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.813829 29051 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.814105 29022 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.815194 29025 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.815769 28478 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.815881 29051 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.815708 29025 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.816171 29022 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.816545 29051 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.816448 28478 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.817059 29051 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.817754 29025 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.818302 28410 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.818607 28478 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.818832 28410 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.836617 28410 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.819679 28844 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:17.820732 29051 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.822418 29029 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:19:17.837598 28844 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:17.829492 28988 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:17.838250 28844 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:17.838528 28988 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.839887 28844 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.827719 28862 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.837756 29029 consensus_queue.cc:1225] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:17.841320 28862 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.831825 28873 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:17.842213 28873 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.843624 28862 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:17.844074 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.870s	user 0.017s	sys 0.035s
W20260809 23:19:17.844486 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.018s	sys 0.035s
I20260809 23:19:17.845276 28539 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:17.840678 28988 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.847012 28988 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:17.847302 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.847563 28988 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.848021 24430 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:17.843992 28873 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:17.850229 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.006s	sys 0.055s
W20260809 23:19:17.850623 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.008s	sys 0.056s
I20260809 23:19:17.851056 28844 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.851235 28988 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.851411 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 879) took 1085 ms. Trace:
I20260809 23:19:17.851925 25373 rpcz_store.cc:276] 0809 23:19:16.766106 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.107335 (+341229us) service_pool.cc:227] Handling call
0809 23:19:17.107870 (+   535us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:17.108343 (+   473us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:17.108370 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:17.109016 (+   646us) log.cc:842] Serialized 52 byte log entry
0809 23:19:17.851059 (+742043us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:17.851115 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.851195 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:17.851239 (+    44us) raft_consensus.cc:1671] finished
0809 23:19:17.851250 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:17.851351 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1460,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":741452,"threads_started":1,"wal-append.queue_time_us":376256}
I20260809 23:19:17.855507 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.860620 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.862025 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.864718 24422 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:17.847056 28841 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:17.866088 28841 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.870283 24430 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:17.870524 24539 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260809 23:19:17.867388 28783 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.744s	user 0.002s	sys 0.013s
I20260809 23:19:17.868067 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.872581 24422 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260809 23:19:17.873992 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.874408 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2211) took 3932 ms. Trace:
I20260809 23:19:17.874787 24424 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:17.875871 24543 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:17.874773 24535 rpcz_store.cc:276] 0809 23:19:13.941540 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.942551 (+  1011us) service_pool.cc:227] Handling call
0809 23:19:13.980616 (+ 38065us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:14.183516 (+202900us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.874199 (+3690683us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:17.874350 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457472256}
I20260809 23:19:17.875090 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2400) took 4892 ms. Trace:
I20260809 23:19:17.868080 28894 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:17.877283 24309 rpcz_store.cc:276] 0809 23:19:12.982521 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.186271 (+203750us) service_pool.cc:227] Handling call
0809 23:19:17.064516 (+3878245us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:17.083215 (+ 18699us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:17.874887 (+791672us) spinlock_profiling.cc:243] Waited 3.86 s on lock 0x7b5400225e50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:17.875031 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8884465024}
I20260809 23:19:17.878055 28894 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.878568 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.879748 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.880297 24309 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:17.880326 28841 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:17.881531 29055 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:17.881742 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.881953 29055 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:17.884435 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.885622 24424 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:17.887985 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.888741 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.871587 28783 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.891335 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.892020 24309 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.892107 24316 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:17.893417 28923 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:17.893849 28923 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:17.894213 28923 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:17.894693 28783 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:17.895112 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.890882 24323 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:17.896585 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.896713 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.897205 24422 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:17.899070 28859 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:17.899478 28859 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:17.899853 28859 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.900005 24422 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:17.902367 24422 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:17.904907 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.905642 24313 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:17.907248 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.908445 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.889420 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.905256 24312 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:17.884644 29055 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:17.910688 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.912061 24314 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:17.845386 24541 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:17.913311 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.916680 24323 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:17.935575 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.924345 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.934916 24424 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260809 23:19:17.933357 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.937260 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.938333 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.940335 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.941018 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.942476 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:17.914491 29105 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:17.940586 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.942093 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.946223 24424 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 23:19:17.941641 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.939913 29055 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:17.943079 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:17.944351 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.941694 28983 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:17.951197 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:17.948535 24314 raft_consensus.cc:2362] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6. Re-sending same reply.
W20260809 23:19:17.943439 28742 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 4.831s	user 0.024s	sys 0.006s
I20260809 23:19:17.954958 28590 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:17.955729 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.089933 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:17.955722 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.970542 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.090083 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.996474 24309 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:18.089843 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:17.972905 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:17.980063 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.047192 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:17.948310 29105 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.098537 24541 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 4: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:18.100023 24424 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:18.102329 24535 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:17.961691 28742 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.855s	user 0.024s	sys 0.006s
I20260809 23:19:18.102828 24423 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260809 23:19:18.103737 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:17.951853 28983 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:18.045959 28923 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.101624 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.159785 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.147987 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.160884 24424 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:18.157887 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.162511 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.103497 29068 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260809 23:19:18.163345 28783 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:18.163920 29068 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:18.107869 28742 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:18.046497 28859 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.161778 29077 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.003s
I20260809 23:19:18.159469 28983 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:18.164460 29068 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.167647 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.168087 29105 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.168628 28923 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:18.169106 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.169481 28783 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:18.198544 24314 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:18.163846 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.198503 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.185235 28894 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:18.187652 24543 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:18.188460 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.189345 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.191857 28742 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.204771 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.206111 24314 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:18.197731 28783 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.195858 24313 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260809 23:19:18.196077 24309 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:18.207772 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.194021 28859 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:18.202559 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.211033 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.210646 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:18.200816 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.214125 24314 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:18.214581 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.213726 24312 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6. Re-sending same reply.
I20260809 23:19:18.215816 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.216135 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.216945 24309 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:18.217108 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.217851 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.214953 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.219559 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.220992 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.212679 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:18.222617 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.223927 24423 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 23:19:18.223723 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.224504 24320 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:18.224797 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.225661 24424 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:18.227686 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.228366 24423 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:18.226627 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.228883 24424 raft_consensus.cc:2362] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6. Re-sending same reply.
I20260809 23:19:18.231146 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.232267 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.234489 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.234543 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.235564 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.236905 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.238054 24320 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:18.237603 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:18.238466 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.227118 28859 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:18.239161 24423 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:18.239460 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:18.245174 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 155442us
I20260809 23:19:18.246816 24309 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:18.247633 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.247880 24424 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:18.248075 24309 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.251082 24423 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:19:18.204090 28894 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.213212 28478 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260809 23:19:18.231470 29055 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:18.259730 24309 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:18.265030 24314 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:18.266584 28478 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:18.269872 28478 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.270349 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.267519 28894 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:18.271981 24309 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:18.272239 24314 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.227916 29068 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.265856 29055 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:18.283587 24314 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.287089 24314 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.292611 24309 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
W20260809 23:19:18.290589 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111350us
I20260809 23:19:18.297951 24314 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:18.303057 24309 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:18.245307 28783 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.283512 28894 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:18.308346 24309 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:17.257454 28804 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.281198 28478 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.316232 28478 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:18.316612 28804 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:18.317268 24422 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:18.314627 24535 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:18.290441 29055 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.300769 28590 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.310559 24314 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260809 23:19:18.325273 24309 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260809 23:19:18.321473 29068 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:18.332360 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:18.333802 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:18.333884 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:18.332545 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.340147 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:18.334425 29077 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:18.323261 28753 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.014s
I20260809 23:19:18.345259 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:18.346215 24496 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:18.346557 24496 leader_election.cc:401] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260809 23:19:18.348453 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:18.349817 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:18.325321 28859 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.351493 24496 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:18.354007 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.354156 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.354641 24435 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:18.323380 28894 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.340631 29100 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:18.363828 29135 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:18.364027 29100 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.364501 29135 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:18.365098 29135 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:18.365613 29135 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:18.366052 29100 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:18.345185 29077 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:18.352135 29122 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:18.361649 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.351737 29124 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:18.366904 29120 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:18.368573 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:18.359467 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.363386 28894 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:18.371038 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:18.366647 29100 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.004s
I20260809 23:19:18.359108 28753 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 23:19:18.366108 24352 debug-util.cc:398] Leaking SignalData structure 0x7b080005bfc0 after lost signal to thread 22536
I20260809 23:19:18.366026 29135 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.377858 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.381417 24435 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 23:19:18.369571 29077 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:18.397053 28840 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.206s	user 0.006s	sys 0.023s
W20260809 23:19:18.397477 28840 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.212s	user 0.006s	sys 0.024s
I20260809 23:19:18.373878 28859 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.399664 29055 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.401953 29135 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.404219 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 1976) took 1269 ms. Trace:
I20260809 23:19:18.403436 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.404583 24311 rpcz_store.cc:276] 0809 23:19:17.134803 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.135430 (+   627us) service_pool.cc:227] Handling call
0809 23:19:17.144335 (+  8905us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:17.289999 (+145664us) raft_consensus.cc:1520] Early marking committed up to index 1
0809 23:19:17.290041 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:17.290550 (+   509us) log.cc:842] Serialized 26 byte log entry
0809 23:19:17.515067 (+224517us) raft_consensus.cc:1631] Marking committed up to 1
0809 23:19:17.515109 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:18.403784 (+888675us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002535d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:18.403837 (+    53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:18.403892 (+    55us) raft_consensus.cc:1671] finished
0809 23:19:18.403910 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:18.404138 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1317,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":333872384,"thread_start_us":199069,"threads_started":1,"wal-append.queue_time_us":236710}
I20260809 23:19:18.375105 29122 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.406621 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.406877 29122 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.408205 24314 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:18.408633 24314 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.409402 24314 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.410331 24314 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.411868 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.412431 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:18.413376 24422 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:18.413867 29055 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:18.414721 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.415359 24435 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:18.415952 28877 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 4.902s	user 0.000s	sys 0.014s
I20260809 23:19:18.416860 24435 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:18.419286 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.420266 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:18.372239 29124 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:18.421547 29124 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:18.422209 28589 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:18.422598 28589 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:18.423396 28589 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.424423 28589 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:18.423949 28983 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.373370 29120 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:18.427150 29120 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:18.428053 28892 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:18.428143 28983 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:18.428548 28892 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.429414 28892 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.430447 28892 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:18.426770 25368 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:18.388305 28753 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260809 23:19:18.392449 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 1975) took 1258 ms (client timeout 1530 ms). Trace:
W20260809 23:19:18.432301 24316 rpcz_store.cc:269] 0809 23:19:17.133982 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.134121 (+   139us) service_pool.cc:227] Handling call
0809 23:19:18.392199 (+1258078us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:18.392372 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1713101312}
I20260809 23:19:18.394125 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.434973 24423 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 8. Ops: []
I20260809 23:19:18.436020 24423 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:18.403378 28754 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.144s	user 0.013s	sys 0.015s
W20260809 23:19:18.436733 28754 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.178s	user 0.013s	sys 0.015s
I20260809 23:19:18.345978 28590 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:18.374219 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 807) took 4482 ms (client timeout 1760 ms). Trace:
W20260809 23:19:18.440620 25366 rpcz_store.cc:269] 0809 23:19:13.892021 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.894322 (+  2301us) service_pool.cc:227] Handling call
0809 23:19:18.370893 (+4476571us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900bce 000000000034aea9 00007fac2872ca30 00007fac287f27b4 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:19:18.374118 (+  3225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5002112}
W20260809 23:19:18.383068 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.398s	user 0.035s	sys 0.020s
W20260809 23:19:18.442543 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.457s	user 0.035s	sys 0.020s
W20260809 23:19:18.444940 25323 connection.cc:232] RPC call timeout handler was delayed by -1.281718126963824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
W20260809 23:19:18.445390 25323 connection.cc:232] RPC call timeout handler was delayed by -1.2770309038460255s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.168s for a response
W20260809 23:19:18.445742 25323 connection.cc:232] RPC call timeout handler was delayed by -1.2508156858384609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260809 23:19:18.446136 25323 connection.cc:232] RPC call timeout handler was delayed by -1.1969671719707549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
W20260809 23:19:18.446417 25323 connection.cc:232] RPC call timeout handler was delayed by -1.1904436899349093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1774s for a response
I20260809 23:19:18.447778 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.448642 24311 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:18.449683 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:18.449841 24316 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:18.451223 24316 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:18.451943 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.453833 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:18.455794 24428 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 6. Ops: []
W20260809 23:19:18.446728 25323 connection.cc:232] RPC call timeout handler was delayed by -1.169430241920054s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260809 23:19:18.457448 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2206) took 5320 ms. Trace:
W20260809 23:19:18.457676 25323 connection.cc:232] RPC call timeout handler was delayed by -1.0229670538101345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
I20260809 23:19:18.458380 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:18.463173 29119 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:18.465579 29119 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:18.466271 29119 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 23:19:18.540509 29075 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:18.541162 29075 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:18.457826 24542 rpcz_store.cc:276] 0809 23:19:13.137002 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.137204 (+   202us) service_pool.cc:227] Handling call
0809 23:19:13.137587 (+   383us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:13.174518 (+ 36931us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:13.174553 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:13.191078 (+ 16525us) log.cc:842] Serialized 26 byte log entry
0809 23:19:13.252705 (+ 61627us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:13.252743 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:15.652930 (+2400187us) spinlock_profiling.cc:243] Waited 31.6 ms on lock 0x7b5400147654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:15.652976 (+    46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:18.415458 (+2762482us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:18.457227 (+ 41769us) raft_consensus.cc:1671] finished
0809 23:19:18.457247 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:18.457395 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8332,"spinlock_wait_cycles":77725824,"thread_start_us":44241,"threads_started":1,"wal-append.queue_time_us":236565}
I20260809 23:19:18.543963 28971 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:18.544499 28971 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:17.308449 28391 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:18.545140 24311 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:18.545389 28971 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.547498 28391 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.551716 29124 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.568827 28391 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.569940 29124 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:18.555409 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:18.571480 24541 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.573897 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.570010 29075 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.570039 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:18.576552 24540 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:18.576942 28517 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:18.577327 28517 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:18.577980 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:18.577661 28517 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:18.578877 29075 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:18.580322 24544 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:18.581660 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.582370 24541 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:18.583786 28804 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:18.584600 28804 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:18.590065 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.592238 24539 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:18.592916 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.585963 28804 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:18.601781 24539 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:18.444988 29120 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:18.564141 24164 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:18.642522 29034 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:18.441393 29077 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:18.681167 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 329735us
I20260809 23:19:19.050443 28171 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260809 23:19:19.052160 28754 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.015s	sys 0.027s
W20260809 23:19:19.052595 28754 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.016s	sys 0.028s
I20260809 23:19:19.052320 28171 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:19.052951 28171 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260809 23:19:18.664875 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 312085us
I20260809 23:19:18.609578 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:19.056928 24535 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:19.057843 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.058625 24535 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:18.610284 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:18.602746 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:18.467435 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:18.623955 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:19.252416 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 785001us
I20260809 23:19:19.253226 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.253679 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.254416 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:18.605336 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.257601 29075 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:18.610005 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 847) took 5616 ms. Trace:
I20260809 23:19:18.438671 24422 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:19.258139 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 816) took 1474 ms (client timeout 1614 ms). Trace:
I20260809 23:19:18.608153 24534 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:19.264074 24534 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:19.264557 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:18.609076 24545 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:19.270303 28569 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:19.271005 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.271242 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.271713 24545 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:19.272233 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.272804 24425 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:19.272945 24545 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:19:18.621599 24542 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:19.274173 24422 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:19.276156 24542 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 23:19:19.276665 24545 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:18.592293 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.277935 24540 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:18.616816 28410 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:18.586540 28517 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.283936 28517 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.284536 28517 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:18.560855 28738 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:18.561448 28870 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.311s	user 0.014s	sys 0.023s
I20260809 23:19:18.606940 28742 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:19.295233 28742 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.297896 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 975) took 2043 ms. Trace:
I20260809 23:19:19.298417 25363 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:18.564699 28882 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:18.675757 28940 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:19.303644 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:18.546726 28971 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:18.634646 28977 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.314288 24422 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260809 23:19:18.653030 28735 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.489s	user 0.013s	sys 0.002s
I20260809 23:19:19.316018 28777 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:19.317731 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.318521 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:18.671667 24314 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:19.319949 28569 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:19.320888 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2445) took 1152 ms. Trace:
I20260809 23:19:19.320601 28569 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.321360 24314 rpcz_store.cc:276] 0809 23:19:18.168225 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.326050 (+157825us) service_pool.cc:227] Handling call
0809 23:19:18.326465 (+   415us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.320046 (+993581us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.320627 (+   581us) spinlock_profiling.cc:243] Waited 81.6 ms on lock 0x7b540043add4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:19.320821 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":187591424}
I20260809 23:19:18.715592 29120 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.323113 28569 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.323313 29120 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.324054 29120 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.324589 28777 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:19.325554 28777 consensus_queue.cc:1225] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260809 23:19:19.326187 28940 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:19.326540 28940 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:18.708617 29034 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:18.710139 29077 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:18.922640 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.843s	user 0.017s	sys 0.019s
W20260809 23:19:18.991986 24352 debug-util.cc:398] Leaking SignalData structure 0x7b080005bf60 after lost signal to thread 24220
I20260809 23:19:19.014834 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 886) took 1775 ms. Trace:
I20260809 23:19:19.016219 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1970) took 1169 ms. Trace:
W20260809 23:19:19.018584 28859 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.010s	sys 0.019s
I20260809 23:19:17.238222 25365 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:19.061602 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:19.062083 24535 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260809 23:19:19.066360 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.015s	sys 0.047s
I20260809 23:19:19.065214 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:19.066965 28918 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:19.088538 28804 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.089015 29003 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.117539 28171 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:19.141743 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 872) took 2582 ms (client timeout 2623 ms). Trace:
I20260809 23:19:11.799533 24321 rpcz_store.cc:276] 0809 23:19:09.984793 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.870155 (+885362us) service_pool.cc:227] Handling call
0809 23:19:10.919845 (+ 49690us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:10.939857 (+ 20012us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:11.460054 (+520197us) log.cc:842] Serialized 14 byte log entry
0809 23:19:11.771446 (+311392us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:11.771523 (+    77us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:11.771593 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:11.771686 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:11.771829 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55887,"prepare.queue_time_us":209294,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":5741,"raft.queue_time_us":53594,"raft.run_wall_time_us":13639,"thread_start_us":676609,"threads_started":2}
W20260809 23:19:19.146342 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.012s	sys 0.013s
I20260809 23:19:19.160073 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:19.160948 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:19.163683 28822 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:19.163857 28855 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:19.163986 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0816452500876039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1948s for a response
I20260809 23:19:19.164170 28745 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:19.167835 24311 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:19:19.170020 24316 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:19.183907 29146 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.185230 24323 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:19.192739 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.196385 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2236) took 1246 ms. Trace:
I20260809 23:19:19.203305 28783 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:19.207808 24431 raft_consensus.cc:1203] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Deduplicated request from leader. Original: 3.3->[5.4-5.4]   Dedup: 5.4->[]
I20260809 23:19:19.211490 28889 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:19.214918 24539 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:19.223153 24164 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 4 to 5, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:19:19.225256 24168 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 339) took 27328 ms (client timeout 15000 ms). Trace:
I20260809 23:19:19.230396 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2022) took 1407 ms. Trace:
I20260809 23:19:19.237488 24544 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.326805 29120 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.354825 28742 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.329097 28517 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
W20260809 23:19:19.330214 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.250s	user 0.018s	sys 0.019s
I20260809 23:19:19.329596 28869 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
I20260809 23:19:19.357947 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 972) took 2698 ms. Trace:
I20260809 23:19:19.258129 29075 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.358551 29075 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:19.330479 25375 rpcz_store.cc:276] 0809 23:19:17.239188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.014176 (+1774988us) service_pool.cc:227] Handling call
0809 23:19:19.014469 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.014609 (+   140us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:19.014623 (+    14us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:19.014650 (+    27us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:19.014668 (+    18us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.014699 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.014782 (+    83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:19.330603 24430 rpcz_store.cc:276] 0809 23:19:17.855008 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.855149 (+   141us) service_pool.cc:227] Handling call
0809 23:19:19.016066 (+1160917us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:19.016149 (+    83us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32400640}
I20260809 23:19:19.364863 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.365309 24430 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 5. Ops: []
I20260809 23:19:19.365558 25375 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:19.258546 28711 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:19.260187 25370 rpcz_store.cc:276] 0809 23:19:12.993322 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:12.995327 (+  2005us) service_pool.cc:227] Handling call
0809 23:19:12.995711 (+   384us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:13.554410 (+558699us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:18.609738 (+5055328us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:18.609915 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1239543680}
I20260809 23:19:19.331041 28977 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:19.367627 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.368268 25370 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:19.331086 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 885) took 2214 ms (client timeout 2668 ms). Trace:
W20260809 23:19:19.368884 25365 rpcz_store.cc:269] 0809 23:19:17.117133 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.119297 (+  2164us) service_pool.cc:227] Handling call
0809 23:19:19.331021 (+2211724us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:19.369665 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:19.332192 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 267037us
W20260809 23:19:19.261639 25369 rpcz_store.cc:269] 0809 23:19:17.783761 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.605071 (+821310us) service_pool.cc:227] Handling call
0809 23:19:19.258071 (+653000us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:19.331601 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1965) took 1486 ms (client timeout 1951 ms). Trace:
I20260809 23:19:19.371459 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.332608 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:19.372076 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:19.262446 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.372608 24535 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:19.333619 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77255us
I20260809 23:19:19.373000 25366 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:19.374151 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.266988 28836 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:19.375195 28836 consensus_queue.cc:1225] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:19.334048 28804 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.374785 25366 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:19.375775 28836 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:19.376181 28836 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:19.376484 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.273073 28897 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:19.334631 28171 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.377703 28783 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.377907 28897 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:19.378355 28897 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:19.378244 28783 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.379230 28897 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.380115 28912 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:19.380409 28897 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.380699 28783 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.380604 28912 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:19.334646 24352 debug-util.cc:398] Leaking SignalData structure 0x7b0800153960 after lost signal to thread 24355
W20260809 23:19:19.334769 25368 rpcz_store.cc:269] 0809 23:19:16.559854 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:16.650249 (+ 90395us) service_pool.cc:227] Handling call
0809 23:19:19.141511 (+2491262us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b540055f854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:19.141662 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4020838784}
W20260809 23:19:19.383474 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08001e9ca0 after lost signal to thread 24469
I20260809 23:19:19.384007 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:19.384683 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08002e76a0 after lost signal to thread 24579
I20260809 23:19:19.385062 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.377206 25366 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
W20260809 23:19:19.385958 24352 debug-util.cc:398] Leaking SignalData structure 0x7b0800412620 after lost signal to thread 25410
W20260809 23:19:19.386296 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08004125e0 after lost signal to thread 25411
I20260809 23:19:19.280860 28410 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:19.279844 24545 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5. Ops: []
I20260809 23:19:19.386603 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.387118 28410 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.387802 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.388170 28410 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.388411 25366 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:19.335250 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.960s	user 0.012s	sys 0.013s
W20260809 23:19:19.389974 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.02141e+06us
I20260809 23:19:19.390599 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.337077 28855 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.391506 28783 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:19.392805 25370 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:19.292951 28870 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.043s	user 0.016s	sys 0.023s
I20260809 23:19:19.298311 25373 rpcz_store.cc:276] 0809 23:19:17.254892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.207915 (+1953023us) service_pool.cc:227] Handling call
0809 23:19:19.208308 (+   393us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.296586 (+ 88278us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:19.296630 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:19.296682 (+    52us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:19.296735 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.297610 (+   875us) spinlock_profiling.cc:243] Waited 88 ms on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:19.297663 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.297825 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202486528}
I20260809 23:19:19.393957 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.394199 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.342832 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.394690 25370 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.394896 25372 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:19.299778 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 810) took 5400 ms. Trace:
I20260809 23:19:19.300146 28565 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.395846 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.348855 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.396456 28565 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.397060 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.301923 28882 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.332459 28918 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.319027 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:19.347199 24431 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260809 23:19:19.400610 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 1984) took 1455 ms. Trace:
I20260809 23:19:19.240257 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:19.401034 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:19.401402 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 161247us
I20260809 23:19:19.402148 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:18.624511 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.372529 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.353685 24317 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260809 23:19:19.404749 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 1982) took 1467 ms (client timeout 1591 ms). Trace:
W20260809 23:19:19.405092 24427 rpcz_store.cc:269] 0809 23:19:17.937297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.624190 (+686893us) service_pool.cc:227] Handling call
0809 23:19:19.404694 (+780504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:19.265305 24534 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:19.328334 29034 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:19.406915 29034 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.403067 28569 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:19.332336 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.016s	sys 0.048s
I20260809 23:19:19.409382 28569 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:19.330798 28859 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.010s	sys 0.019s
I20260809 23:19:19.334324 29003 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.337482 28745 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.336776 28822 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.405680 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.337280 28984 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:19.338414 24269 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.760s (SENT)
I20260809 23:19:19.347411 29146 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:19.371341 24429 rpcz_store.cc:269] 0809 23:19:17.845339 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.061331 (+1215992us) service_pool.cc:227] Handling call
0809 23:19:19.331520 (+270189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:19.336410 24314 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260809 23:19:19.415123 24314 pending_rounds.cc:76] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 3
W20260809 23:19:19.337229 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0519452500157058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
I20260809 23:19:19.395514 25363 rpcz_store.cc:276] 0809 23:19:13.899806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:13.904144 (+  4338us) service_pool.cc:227] Handling call
0809 23:19:13.941136 (+ 36992us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.298896 (+5357760us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.299543 (+   647us) spinlock_profiling.cc:243] Waited 5.36 s on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:19.299719 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12319664896}
W20260809 23:19:19.416100 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0346836468670517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20260809 23:19:19.343004 24543 rpcz_store.cc:276] 0809 23:19:17.950236 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.950391 (+   155us) service_pool.cc:227] Handling call
0809 23:19:19.196183 (+1245792us) spinlock_profiling.cc:243] Waited 86.9 ms on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:19.196323 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199806848}
W20260809 23:19:19.416580 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0132452500984073s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
I20260809 23:19:19.416775 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.417397 25363 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.327589 28940 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.397073 24428 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:19.413051 25369 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:19.415326 24314 pending_rounds.cc:97] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260809 23:19:19.349021 24539 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:19.421515 24314 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:19.421792 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.420341 28940 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.422421 29077 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.422899 28745 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.422981 29077 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.425634 28873 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.425879 29077 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.426225 28873 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.349519 24312 rpcz_store.cc:276] 0809 23:19:17.822952 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.239855 (+416903us) service_pool.cc:227] Handling call
0809 23:19:18.264435 (+ 24580us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:18.269546 (+  5111us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:18.286389 (+ 16843us) log.cc:842] Serialized 14 byte log entry
0809 23:19:19.126225 (+839836us) spinlock_profiling.cc:243] Waited 9.74 ms on lock 0x7b18000e6a40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:19.208993 (+ 82768us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:19.209490 (+   497us) log.cc:842] Serialized 26 byte log entry
0809 23:19:19.209636 (+   146us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:19.209674 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.209737 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:19.230001 (+ 20264us) raft_consensus.cc:1671] finished
0809 23:19:19.230148 (+   147us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.230322 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6484,"prepare.queue_time_us":16712,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":7504,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":7744,"spinlock_wait_cycles":23478912,"thread_start_us":81074,"threads_started":1,"wal-append.queue_time_us":872201}
I20260809 23:19:19.413938 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:19.428489 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90125.2us
I20260809 23:19:19.428870 28873 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.429386 28873 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.413172 28984 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:19.429757 28873 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.430181 28984 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:19.430595 28889 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:19.431068 28859 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.431641 28889 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:19.431560 28859 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.432037 28889 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:19.432888 28889 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.433734 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.434070 28889 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:19.434531 24543 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:19.436528 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:19.438134 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66132.5us
I20260809 23:19:19.439339 29125 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:19.440029 29052 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:19.440584 29052 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.440660 29003 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.441073 29052 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.393005 28855 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:19.401073 24431 rpcz_store.cc:276] 0809 23:19:17.945172 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.046504 (+1101332us) service_pool.cc:227] Handling call
0809 23:19:19.046886 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:19.399907 (+353021us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.400415 (+   508us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:19.400558 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369821952}
W20260809 23:19:19.443063 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52541.6us
I20260809 23:19:19.443814 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:19.386111 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.444626 28739 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:19.444674 28881 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.445116 28739 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.445165 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:19.445710 28739 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:19.446557 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.450126 28882 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:19.445322 28881 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.402793 25372 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:19.480809 28881 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.480859 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:19.349431 24168 rpcz_store.cc:269] 0809 23:18:51.896750 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:18:51.896998 (+   248us) service_pool.cc:227] Handling call
0809 23:19:19.225149 (+27328151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:19:11.734978 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:19:11.870309 (+135331us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:19:11.870370 (+    61us) write_op.cc:435] Acquired schema lock
0809 23:19:11.870385 (+    15us) tablet.cc:660] Decoding operations
0809 23:19:11.871360 (+   975us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:19:11.871457 (+    97us) write_op.cc:641] Partition lock acquired for write op
0809 23:19:11.871487 (+    30us) tablet.cc:683] Acquiring locks for 6 operations
0809 23:19:11.872157 (+   670us) tablet.cc:699] Row locks acquired
0809 23:19:11.872188 (+    31us) write_op.cc:260] PREPARE: finished
0809 23:19:11.872391 (+   203us) write_op.cc:270] Start()
0809 23:19:11.872540 (+   149us) write_op.cc:276] Timestamp: P: 1786317551872304 usec, L: 0
0809 23:19:11.872591 (+    51us) op_driver.cc:331] REPLICATION: starting
0809 23:19:11.929238 (+ 56647us) log.cc:842] Serialized 2506 byte log entry
0809 23:19:12.645705 (+716467us) op_driver.cc:444] REPLICATION: finished
0809 23:19:12.770082 (+124377us) write_op.cc:301] APPLY: starting
0809 23:19:12.770185 (+   103us) tablet.cc:1365] starting BulkCheckPresence
0809 23:19:12.770399 (+   214us) tablet.cc:1368] finished BulkCheckPresence
0809 23:19:12.770540 (+   141us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:19:12.796521 (+ 25981us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:19:12.797037 (+   516us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0809 23:19:12.797085 (+    48us) write_op.cc:312] APPLY: finished
0809 23:19:12.797414 (+   329us) log.cc:842] Serialized 53 byte log entry
0809 23:19:12.797606 (+   192us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:12.797717 (+   111us) write_op.cc:454] Released schema lock
0809 23:19:12.797777 (+    60us) write_op.cc:341] FINISH: Updating metrics
0809 23:19:12.798121 (+   344us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:12.798143 (+    22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":920507,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":124029,"apply.run_cpu_time_us":4159,"apply.run_wall_time_us":28410,"num_ops":6,"prepare.queue_time_us":556817,"prepare.run_cpu_time_us":42431,"prepare.run_wall_time_us":405059,"raft.queue_time_us":358942,"raft.run_cpu_time_us":7475,"raft.run_wall_time_us":75533,"replication_time_us":773007,"thread_start_us":779626,"threads_started":4,"wal-append.queue_time_us":260237,"wal-append.run_cpu_time_us":11307,"wal-append.run_wall_time_us":1616092}]]}
I20260809 23:19:19.358278 25371 rpcz_store.cc:276] 0809 23:19:16.659995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:17.077816 (+417821us) service_pool.cc:227] Handling call
0809 23:19:17.078297 (+   481us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:17.078774 (+   477us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:17.078805 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:17.079444 (+   639us) log.cc:842] Serialized 52 byte log entry
0809 23:19:19.357604 (+2278160us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:19.357644 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.357709 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:19.357748 (+    39us) raft_consensus.cc:1671] finished
0809 23:19:19.357758 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.357892 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3540,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":15059,"thread_start_us":2277631,"threads_started":1,"wal-append.queue_time_us":438610,"wal-append.run_cpu_time_us":2241,"wal-append.run_wall_time_us":1005136}
I20260809 23:19:19.482564 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.436692 28822 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.452708 28859 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.452176 28918 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.454075 28565 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.349263 24164 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.329922 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:19.377095 28804 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.459218 28873 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.377059 28836 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.445667 28171 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.370716 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.410382 28569 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.462543 29146 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.463307 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.463663 28912 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:19.464478 29034 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:19.418537 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:19.444808 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.439020 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:19.443066 29052 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.366636 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.474670 28984 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:19.346251 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66370.5us
I20260809 23:19:19.467366 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:19.386602 24352 debug-util.cc:398] Leaking SignalData structure 0x7b0800412580 after lost signal to thread 28065
I20260809 23:19:19.471553 28739 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.475941 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.485411 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.593209 25368 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:19.594384 24164 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.345112 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:19.595625 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08002dc720 after lost signal to thread 28727
W20260809 23:19:19.596242 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08001b6ba0 after lost signal to thread 28766
W20260809 23:19:19.596803 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 178334us
W20260809 23:19:19.596905 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9c20 after lost signal to thread 28816
W20260809 23:19:19.597463 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08003f3dc0 after lost signal to thread 28833
W20260809 23:19:19.346272 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 81777.4us
W20260809 23:19:19.599623 24352 debug-util.cc:398] Leaking SignalData structure 0x7b08003bfd20 after lost signal to thread 28872
I20260809 23:19:19.600315 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:19.483400 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:19.601748 25372 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:19.509712 24544 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:19.593721 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2026) took 1753 ms. Trace:
I20260809 23:19:19.590699 25373 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:19.604099 25373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
W20260809 23:19:19.605031 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 275180us
I20260809 23:19:19.606503 24546 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 6. Ops: []
I20260809 23:19:19.607468 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 679) took 1001 ms. Trace:
I20260809 23:19:19.608043 24544 rpcz_store.cc:276] 0809 23:19:18.605576 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.610016 (+  4440us) service_pool.cc:227] Handling call
0809 23:19:19.607385 (+997369us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:19.609023 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164284us
I20260809 23:19:19.609225 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.609741 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:19.609858 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.602813 25370 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:19.611338 25375 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260809 23:19:19.347510 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64776.5us
I20260809 23:19:19.612461 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.614845 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:19.597452 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:19.620311 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.603190 24320 rpcz_store.cc:276] 0809 23:19:17.840157 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:18.245636 (+405479us) service_pool.cc:227] Handling call
0809 23:19:18.245949 (+   313us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:18.246318 (+   369us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:19:18.269687 (+ 23369us) log.cc:842] Serialized 14 byte log entry
0809 23:19:19.409280 (+1139593us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:19.409712 (+   432us) log.cc:842] Serialized 26 byte log entry
0809 23:19:19.544729 (+135017us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:19.593179 (+ 48450us) log.cc:842] Serialized 14 byte log entry
0809 23:19:19.593445 (+   266us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.593512 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:19.593547 (+    35us) raft_consensus.cc:1671] finished
0809 23:19:19.593557 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.593669 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2566,"prepare.queue_time_us":24643,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":2180,"raft.queue_time_us":45407,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":3017,"thread_start_us":1287156,"threads_started":4,"wal-append.queue_time_us":228200,"wal-append.run_cpu_time_us":1432,"wal-append.run_wall_time_us":1008332}
I20260809 23:19:19.621125 25365 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:19.621767 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:19.623534 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2316) took 1011 ms. Trace:
I20260809 23:19:19.623438 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.623960 24546 rpcz_store.cc:276] 0809 23:19:18.612540 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.238368 (+625828us) service_pool.cc:227] Handling call
0809 23:19:19.606254 (+367886us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.606988 (+   734us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.623341 (+ 16353us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b540029d450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:19.623477 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":845308160}
I20260809 23:19:19.624454 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 979) took 1021 ms. Trace:
I20260809 23:19:19.624867 25371 rpcz_store.cc:276] 0809 23:19:18.603026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.623187 (+1020161us) service_pool.cc:227] Handling call
0809 23:19:19.624402 (+  1215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:19.625092 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 976) took 1023 ms. Trace:
I20260809 23:19:19.625828 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.625447 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.625487 25370 rpcz_store.cc:276] 0809 23:19:18.601288 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.624365 (+1023077us) service_pool.cc:227] Handling call
0809 23:19:19.624631 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.624799 (+   168us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:19.624820 (+    21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:19.624860 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:19.624898 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:19.624949 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:19.625042 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:19.628099 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 189163us
I20260809 23:19:19.628407 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.629868 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:19.633378 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.623477 24164 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.636631 24164 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.504499 25363 raft_consensus.cc:2396] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:19.640300 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:19.644752 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:19.650228 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.650853 25370 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:19.651144 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:19.661494 25371 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 8. Ops: [6.7-6.7]
I20260809 23:19:19.662854 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.656991 24164 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 3, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 3 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.633486 29052 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:19.667860 29052 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:19.669186 29052 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:19.652355 24498 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:19.671439 24498 leader_election.cc:401] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:19.591342 28783 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.672274 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.679636 25370 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:19.680815 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.681690 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:19.682411 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:19.682838 25370 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:19.683182 25370 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:19.672439 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:19.673347 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.673480 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:19.675266 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:19.668563 24164 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 5, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.598877 28889 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:19.686170 24164 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:19:19.689129 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59297us
I20260809 23:19:19.622740 28754 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:19.690320 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.690814 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:19.694981 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50292us
I20260809 23:19:19.690089 24164 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 5, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:19.649835 28918 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:19.660660 28410 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:19.660339 28711 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:19.649834 28822 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:19.666826 28873 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:19.698500 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46275.1us
I20260809 23:19:19.693950 25368 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:19.661164 29077 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.787652 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:19.788992 25368 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:19.680190 28836 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:19.677227 28870 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:19.688716 29003 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
I20260809 23:19:19.765560 28923 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:19.782635 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92418.8us
I20260809 23:19:19.765765 29125 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6.
I20260809 23:19:19.624264 28804 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:19.672775 28881 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:19.747547 25372 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:19.766719 29173 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:19.771128 29189 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:19.767501 29178 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260809 23:19:19.762594 28869 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:19.782727 25373 raft_consensus.cc:595] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:19.676939 28742 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.122s
I20260809 23:19:19.775286 28410 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.650616 28855 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.140s
I20260809 23:19:19.762138 29172 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:19.696045 29146 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:19.697590 28783 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.790695 28711 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.689496 28940 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.116s
I20260809 23:19:19.689003 28971 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260809 23:19:19.837417 28410 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:19.695797 28738 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.116s
I20260809 23:19:19.801852 28539 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:19.802239 28923 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.811322 28804 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:19.813398 28870 raft_consensus.cc:2741] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 23:19:19.777199 29075 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.796411 28882 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:19.775537 29175 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
I20260809 23:19:19.827628 28881 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:19.800408 29077 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.826046 28918 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260809 23:19:19.849087 29172 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:19.863260 28923 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:19.832022 29189 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.881433 28918 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260809 23:19:19.894343 28897 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.513s	user 0.011s	sys 0.000s
I20260809 23:19:19.770375 24164 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 3 to 5, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 5 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:20.108903 29077 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:19.724208 24217 debug-util.cc:398] Leaking SignalData structure 0x7b080008c5e0 after lost signal to thread 22524
W20260809 23:19:19.783551 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 173879us
W20260809 23:19:20.110294 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800034600 after lost signal to thread 22536
W20260809 23:19:20.112257 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800352da0 after lost signal to thread 24220
I20260809 23:19:20.113446 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:19.960772 24269 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.507s (SENT)
W20260809 23:19:20.114801 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08002f51e0 after lost signal to thread 24355
W20260809 23:19:20.115669 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 155241us
I20260809 23:19:19.784525 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:20.117511 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08002e5d40 after lost signal to thread 24469
I20260809 23:19:20.118366 28777 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:19.984184 24422 pending_rounds.cc:76] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 5
I20260809 23:19:20.119284 24422 pending_rounds.cc:97] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 6.6
W20260809 23:19:20.119680 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 335039us
I20260809 23:19:19.985982 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.989008 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:19.989020 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:19.991511 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:19.803949 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:20.127314 24425 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:20.003684 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 310346us
W20260809 23:19:20.003806 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 330244us
I20260809 23:19:20.006192 25369 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:19.830525 25363 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:19.846127 25368 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:20.140395 28569 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:20.141885 25363 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:19.828683 25373 raft_consensus.cc:603] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:20.008137 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.144774 24429 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:20.014170 28754 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:20.146891 28754 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:19.861106 28539 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:20.151386 28539 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.152212 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:19.848876 28711 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:20.159654 25368 raft_consensus.cc:1203] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.6->[6.7-6.7]   Dedup: 6.7->[]
I20260809 23:19:20.160421 25368 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260809 23:19:20.162184 25368 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 4. Current term is 6. Ops: []
I20260809 23:19:19.872406 28739 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.889897 28984 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.912945 29172 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.165277 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.166908 25368 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260809 23:19:19.913553 29189 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:19.917802 29195 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.091s	user 0.004s	sys 0.000s
I20260809 23:19:20.170063 28739 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.068555 28783 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.170956 28739 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:19.863688 28804 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:20.171414 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 331499us
I20260809 23:19:20.171618 28739 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.889431 28873 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:20.173144 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:20.174371 28739 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:19.904928 29075 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.082566 29186 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:20.176733 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800303340 after lost signal to thread 24579
I20260809 23:19:20.176379 29186 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.178447 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800416b20 after lost signal to thread 25410
I20260809 23:19:20.178614 25371 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:20.179960 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 990) took 1116 ms. Trace:
I20260809 23:19:20.180313 25371 rpcz_store.cc:276] 0809 23:19:19.063330 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.680956 (+617626us) service_pool.cc:227] Handling call
0809 23:19:20.178232 (+497276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:20.179081 (+   849us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.179741 (+   660us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b5400601650. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:20.179904 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1142334208}
I20260809 23:19:19.903877 29003 consensus_queue.cc:1225] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260809 23:19:20.166906 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.183053 28804 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:19.985318 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.184666 29189 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:20.185240 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:20.185882 24431 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:20.186339 24431 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:20.187584 24431 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.189234 24431 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.190269 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.136960 29077 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.137471 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.138510 24428 raft_consensus.cc:2354] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7.
I20260809 23:19:20.174717 28873 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.191453 29077 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.192914 24428 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 23:19:20.194396 29077 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.191578 24425 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:20.195572 29077 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:20.195899 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:19:20.197788 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.198711 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.196475 29077 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.199601 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.200420 24535 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:20.200712 29186 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.200958 29077 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.138377 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.141386 28569 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.201709 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.203852 28569 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:19.861433 28735 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20260809 23:19:19.874861 28738 consensus_queue.cc:1225] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:19:20.155359 28918 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:20.207521 28918 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:20.091239 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 258403us
I20260809 23:19:20.207980 28918 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:20.208546 28977 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:20.209018 28977 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:20.211449 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60571.1us
I20260809 23:19:20.191417 28859 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:20.212848 28859 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:20.213253 28859 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.213985 25370 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.119742 24422 raft_consensus.cc:2866] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:20.185426 29189 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.075889 28881 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.178771 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800416b40 after lost signal to thread 25411
I20260809 23:19:20.171414 29120 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Stepping down as leader of term 4
I20260809 23:19:20.116112 29211 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:20.221894 28918 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.109828 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 4 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:20.235538 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.224924 29120 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:20.256759 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.256570 29211 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.257733 24539 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:20.257297 24164 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 3 to 6, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:20.210373 28977 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.264729 28804 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:20.266292 29034 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.641s	user 0.008s	sys 0.008s
W20260809 23:19:20.264751 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08003edd20 after lost signal to thread 28903
I20260809 23:19:20.275471 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.276479 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:20.273016 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:20.280699 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64182.7us
I20260809 23:19:20.281046 29218 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:20.281519 29218 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:20.282030 29218 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.282474 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.277737 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:20.278575 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:20.258757 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46330.4us
I20260809 23:19:20.310316 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:20.244845 28754 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.327723 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:20.345162 24427 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:20.345553 25370 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.345914 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:20.352188 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.341930 28754 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:20.351583 29191 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:20.356758 28822 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.357290 28822 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:20.358584 25366 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.969s	user 0.015s	sys 0.015s
I20260809 23:19:20.359031 29212 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.359201 28912 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:20.359694 28971 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:20.360144 29189 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.360534 29219 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.361510 29219 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.355958 28984 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:20.364012 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85726.2us
I20260809 23:19:20.363823 28984 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.356114 28841 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:20.366514 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44005.6us
I20260809 23:19:20.367489 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.367812 25372 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 4. Current term is 6. Ops: []
I20260809 23:19:20.368319 24496 leader_election.cc:398] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7.
I20260809 23:19:20.368667 24496 leader_election.cc:401] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260809 23:19:20.369630 28923 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.355945 28754 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.371584 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.361239 29120 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.379020 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 266099us
I20260809 23:19:20.342841 29211 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.364099 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.387244 25366 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:20.379763 28754 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:20.393764 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 111534us
I20260809 23:19:20.394340 28565 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:20.397094 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 978) took 1312 ms. Trace:
I20260809 23:19:20.397532 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.399554 28984 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.405288 24424 rpcz_store.cc:276] 0809 23:19:19.085005 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.111653 (+ 26648us) service_pool.cc:227] Handling call
0809 23:19:20.396112 (+1284459us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:20.396866 (+   754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2708745984}
I20260809 23:19:20.409883 24534 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:20.415052 24543 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:20.417384 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:20.421164 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96621.4us
I20260809 23:19:20.422454 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.423087 28171 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7.
I20260809 23:19:20.424290 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.424551 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:20.425758 29215 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260809 23:19:20.421257 28855 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.435046 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:20.410341 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 827) took 1164 ms. Trace:
I20260809 23:19:20.404919 29211 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.416267 25363 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260809 23:19:20.428671 28918 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.459748 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.467036 29219 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.467660 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 891) took 1268 ms. Trace:
I20260809 23:19:20.468060 25372 rpcz_store.cc:276] 0809 23:19:19.199426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.810178 (+610752us) service_pool.cc:227] Handling call
0809 23:19:19.810591 (+   413us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:20.368402 (+557811us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.467419 (+ 99017us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:20.467576 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136149888}
I20260809 23:19:20.472892 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34762 (request call id 979) took 1247 ms. Trace:
I20260809 23:19:20.473232 24435 rpcz_store.cc:276] 0809 23:19:19.225575 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.225835 (+   260us) service_pool.cc:227] Handling call
0809 23:19:20.472430 (+1246595us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:19:20.472695 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2497623808}
I20260809 23:19:20.475497 29120 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:20.476730 25375 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:20.478257 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 897) took 1277 ms. Trace:
I20260809 23:19:20.479069 29211 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.479885 29191 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.445781 28739 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.456560 28171 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:20.507236 28171 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:20.507754 28171 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:20.445150 25366 rpcz_store.cc:276] 0809 23:19:19.245607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.387554 (+141947us) service_pool.cc:227] Handling call
0809 23:19:20.401841 (+1014287us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:20.478593 25375 rpcz_store.cc:276] 0809 23:19:19.201204 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.853538 (+652334us) service_pool.cc:227] Handling call
0809 23:19:20.404328 (+550790us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:20.477184 (+ 72856us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.478059 (+   875us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400634550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:20.478203 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":870522112}
I20260809 23:19:20.510007 29192 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:20.512002 28711 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.512346 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:20.514650 28822 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.518440 24496 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:20.519845 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97566us
I20260809 23:19:20.483870 25371 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:20.470767 25370 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20260809 23:19:20.521003 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101549us
I20260809 23:19:20.521646 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 991) took 1439 ms. Trace:
I20260809 23:19:20.503293 28882 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:20.462494 28410 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.515288 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91204.4us
W20260809 23:19:20.516752 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53053.4us
W20260809 23:19:20.514758 29232 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260809 23:19:20.523211 28923 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.502686 24540 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:20.524786 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 984) took 1920 ms. Trace:
I20260809 23:19:20.503458 29211 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:20.528256 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 109590us
I20260809 23:19:20.525753 25370 rpcz_store.cc:276] 0809 23:19:19.081845 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.682463 (+600618us) service_pool.cc:227] Handling call
0809 23:19:19.682720 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:20.213139 (+530419us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54007d12a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f6951 00007fac23857321 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612
0809 23:19:20.521272 (+308133us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.521580 (+   308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1200849792}
I20260809 23:19:20.527436 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.429908 28881 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20260809 23:19:20.524046 28912 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:20.584401 24431 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260809 23:19:20.523768 28711 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:20.589387 24535 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:20.466348 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 898) took 1264 ms. Trace:
W20260809 23:19:20.593988 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 132893us
I20260809 23:19:20.589185 24431 pending_rounds.cc:76] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 3
I20260809 23:19:20.585208 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:20.595239 24431 pending_rounds.cc:97] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260809 23:19:20.595592 24431 raft_consensus.cc:2866] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:20.525321 28889 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260809 23:19:20.597019 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:20.598201 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.599813 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.602108 28410 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.525636 28171 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.603160 29211 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.603590 28539 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.604045 29211 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.605144 28171 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:20.602005 24534 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:20.605875 28539 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.594437 25363 rpcz_store.cc:276] 0809 23:19:19.201513 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:20.140733 (+939220us) service_pool.cc:227] Handling call
0809 23:19:20.141434 (+   701us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:20.456730 (+315296us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.466277 (+  9547us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:20.602700 29052 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.886s	user 0.018s	sys 0.004s
I20260809 23:19:20.607017 28977 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:20.622164 29052 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.624682 29052 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.607554 28912 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:20.628695 28862 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260809 23:19:20.610198 29003 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.613765 28711 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:20.603920 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.615618 24541 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:20.615782 29231 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:19:20.558060 25365 rpcz_store.cc:276] 0809 23:19:18.604330 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.632371 (+1028041us) service_pool.cc:227] Handling call
0809 23:19:19.691763 (+ 59392us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:19.784609 (+ 92846us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:20.175133 (+390524us) log.cc:842] Serialized 14 byte log entry
0809 23:19:20.328992 (+153859us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:20.347701 (+ 18709us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:20.351315 (+  3614us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:20.524422 (+173107us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b540064c8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:20.524467 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:20.524604 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5998,"prepare.queue_time_us":67348,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":24146,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":18872,"spinlock_wait_cycles":212998400,"thread_start_us":389570,"threads_started":2,"wal-append.queue_time_us":214306}
I20260809 23:19:20.616494 29219 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:20.620050 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.618319 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:20.620656 28841 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:20.628201 28912 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.618698 28822 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.639341 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.639794 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.601886 28754 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:20.654947 24269 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.951s (SENT)
I20260809 23:19:20.652565 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.660986 29052 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:20.662667 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:20.647360 28410 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:20.663344 28410 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:20.663430 24320 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:20.663800 28410 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:20.664914 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:20.668318 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50126.3us
I20260809 23:19:20.670145 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:20.647706 28841 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:20.676492 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 91405.1us
I20260809 23:19:20.687736 24427 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:20.652206 28881 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.694391 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:20.644466 29211 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260809 23:19:20.707099 29219 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:20.707607 29219 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:20.708019 29219 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.711932 28754 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:20.717801 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:20.718513 24317 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:20.718561 28859 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.505s	user 0.001s	sys 0.014s
I20260809 23:19:20.719647 28859 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.722306 28859 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:20.723457 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 126631us
I20260809 23:19:20.725561 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.726339 29034 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:20.687536 29218 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.733762 29218 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.734449 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.736210 24543 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.740686 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:20.741529 24534 raft_consensus.cc:2354] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:20.743672 24270 leader_election.cc:398] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:20.744033 24270 leader_election.cc:401] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260809 23:19:20.744169 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:20.745820 29211 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:20.748308 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.750267 28739 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:20.750689 28739 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:20.752151 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:20.753082 28836 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:20.753556 28836 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:20.755118 24312 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:20.757550 28927 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:20.759429 28539 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:19:20.760855 29191 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:20.761252 29191 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:20.761745 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 75550.6us
I20260809 23:19:20.762454 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:20.763728 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:20.764322 28859 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.764984 28859 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:20.765317 28859 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:20.765650 28859 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:20.765816 28984 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:20.768301 29125 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:20.768664 29125 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:20.762225 29191 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.769801 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.769524 29125 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.770386 24424 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:20.770292 29191 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:20.771548 28862 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:20.772086 28862 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.773526 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:20.774077 28862 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.774449 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:20.776839 28889 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:20.777247 28889 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:20.777869 25363 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.778115 28889 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.779414 28889 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:20.751857 28739 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.784255 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:20.770653 29125 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:20.795974 24424 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:20.802663 25369 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:20.807417 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:20.794735 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 93633.8us
I20260809 23:19:20.809083 29211 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:20.809613 25370 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.812784 24543 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:20.819171 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:20.820381 28754 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:20.821012 28754 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:20.822091 28754 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.827598 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:20.815974 24427 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:20.831182 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:20.835968 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:20.738586 29219 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.844076 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:20.845834 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:20.847422 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:20.747593 28410 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.853659 24312 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:20.854491 28881 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:20.856712 28923 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:20.856961 28881 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:20.857245 28923 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:20.858108 28881 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:20.859755 24317 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:20.848815 29125 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:20.861464 29125 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:20.862407 29125 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:20.863286 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 136325us
I20260809 23:19:20.874028 25375 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260809 23:19:20.875448 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:20.827721 29212 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:20.858275 28923 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:20.898329 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67260.1us
I20260809 23:19:20.899698 24317 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:20.806639 28841 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.901674 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.876998 29211 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:20.905115 28862 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.014s
I20260809 23:19:20.920176 28862 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:20.923926 25363 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:20.924635 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:20.925436 29052 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:20.927275 29052 raft_consensus.cc:2741] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260809 23:19:20.881894 29218 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:20.852581 29249 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.076s
I20260809 23:19:20.932243 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.939221 29219 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.939554 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.901839 28739 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:20.946210 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.948019 24425 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:20.948951 25365 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:20.948868 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.951315 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.952282 24431 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:20.848645 28836 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:20.941249 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.954914 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.936394 28410 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:20.976248 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:20.975260 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.979413 24435 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:20.979758 24424 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 5: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:20.980876 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.981937 24423 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:20.982883 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.944101 28862 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260809 23:19:20.983939 24425 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:20.992424 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:20.993242 24545 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:21.005507 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:21.006834 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70052.5us
I20260809 23:19:21.007359 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:20.948438 29212 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260809 23:19:21.011684 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:21.010733 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.014118 24425 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260809 23:19:21.009549 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.016192 24545 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:21.021567 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.005888 29191 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:21.023089 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 98658.3us
I20260809 23:19:21.009567 29219 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.020077 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.024796 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.033427 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.025789 24431 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:21.035235 24430 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:21.037329 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:21.039726 25365 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:21.040284 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.041949 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.042814 25371 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 4.
I20260809 23:19:21.044616 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:21.050227 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.051167 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.056689 28517 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260809 23:19:21.058516 24323 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260809 23:19:21.060179 28841 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.032519 29191 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.034953 29218 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.066088 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.034672 28754 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.076508 25371 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:21.079221 28859 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:21.080780 29271 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.077s	user 0.003s	sys 0.000s
I20260809 23:19:21.075631 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:21.048897 29189 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.046765 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:21.116603 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84125.9us
I20260809 23:19:21.125556 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.126416 25371 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:21.115767 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.082783 28859 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:21.131184 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:21.132825 25375 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:21.134527 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.113083 29211 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.137454 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.103266 29218 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:21.140946 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.142598 29125 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:21.142900 28841 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.143173 25365 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:21.150900 24323 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:21.156873 24316 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:21.167654 24431 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:21.159003 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.162233 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.170660 25370 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:21.172035 24539 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:21.171967 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.172245 25372 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:21.114732 28754 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:21.115669 29219 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:21.192265 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57905.7us
I20260809 23:19:21.195032 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.195786 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.201297 24312 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260809 23:19:21.184473 29125 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:21.205716 29125 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:21.209179 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.201733 24312 pending_rounds.cc:76] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 4
I20260809 23:19:21.210366 24312 pending_rounds.cc:97] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260809 23:19:21.190459 29191 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:21.211436 25365 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:21.213927 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.215111 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.215862 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.210638 24312 raft_consensus.cc:2866] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:21.218304 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.184429 29274 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20260809 23:19:21.197613 29052 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.217952 25372 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:21.175450 28841 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.189221 29211 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.225220 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.224763 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.234618 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.232697 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.235286 25370 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:21.241400 25368 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:21.244638 24317 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 23:19:21.248116 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.248950 24313 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
W20260809 23:19:21.250746 29259 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.303s	user 0.000s	sys 0.002s
I20260809 23:19:21.254841 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.255056 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:21.255722 25373 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
W20260809 23:19:21.256100 29262 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.269s	user 0.000s	sys 0.002s
I20260809 23:19:21.256152 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.257002 25368 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:21.257849 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:21.263193 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:21.240128 24378 connection.cc:232] RPC call timeout handler was delayed by -1.7755160499364138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
I20260809 23:19:21.271976 24428 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:21.238224 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.241055 29189 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:21.279033 24545 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:21.272872 24323 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:21.276084 25365 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:19:21.280009 24543 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:21.285234 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.289685 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:21.291246 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:21.293670 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:21.295292 25371 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:21.301836 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.272807 25363 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:21.311523 24320 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 5: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:21.307806 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.313383 24425 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:21.306471 24431 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:21.319671 29261 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.332s	user 0.003s	sys 0.000s
I20260809 23:19:21.295148 29189 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:21.329838 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.333318 24312 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:21.306246 29218 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.332994 24316 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:21.340965 24429 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:21.340121 24323 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:21.290273 28918 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
W20260809 23:19:21.344136 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105123us
I20260809 23:19:21.289706 29172 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
I20260809 23:19:21.348398 25375 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:21.345117 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.352986 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.328472 29191 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.337100 28754 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.372700 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.369402 24320 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:21.375099 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.376220 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:21.353308 29146 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:21.391377 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94690.8us
I20260809 23:19:21.402495 24431 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:21.396188 24378 connection.cc:232] RPC call timeout handler was delayed by -1.4550979051273316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26680000000000004s for a response
I20260809 23:19:21.404807 24543 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260809 23:19:21.404927 24378 connection.cc:232] RPC call timeout handler was delayed by -1.4046052198391408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
I20260809 23:19:21.353883 29219 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:21.406246 24378 connection.cc:232] RPC call timeout handler was delayed by -1.1950825368985534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
I20260809 23:19:21.405218 24323 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:21.420882 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.423556 24429 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:21.368261 28977 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:21.431546 25373 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:21.407102 28711 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.412750 29211 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:21.435308 29280 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260809 23:19:21.471562 24312 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260809 23:19:21.371560 29191 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.427182 29219 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:21.427387 29146 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:21.477837 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.479687 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.501833 28841 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:21.511777 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:21.513523 24543 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:21.481882 24312 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:21.535959 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.536669 24320 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:21.450814 28977 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.470655 28912 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260809 23:19:21.477082 28711 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.537717 29297 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:21.552520 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:21.542709 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92324.7us
I20260809 23:19:21.554768 24320 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
W20260809 23:19:21.546617 29279 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260809 23:19:21.476217 29189 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.546321 28841 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:21.540795 29299 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:21.569303 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.569891 29299 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:21.570676 29285 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260809 23:19:21.571175 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.571674 24312 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260809 23:19:21.571825 24313 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:21.572729 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.573912 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.574505 24312 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 3 FOLLOWER]: Advancing to term 5
I20260809 23:19:21.513260 28738 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.167s
W20260809 23:19:21.575378 29282 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260809 23:19:21.575491 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.576022 28738 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:21.576391 28738 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:21.577409 28841 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:21.577879 29296 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260809 23:19:21.579286 29287 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260809 23:19:21.578629 28841 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.582211 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:21.582872 24311 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:21.564203 29283 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.193s	user 0.000s	sys 0.005s
I20260809 23:19:21.571774 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:21.586429 24321 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260809 23:19:21.589257 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77610.8us
I20260809 23:19:21.590251 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.591106 29003 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:21.592587 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:21.556613 29277 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260809 23:19:21.572798 29299 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:21.598016 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47822.8us
I20260809 23:19:21.599701 24311 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:21.554467 29297 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:21.543408 29172 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 23:19:21.610757 24321 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:21.612383 24320 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:21.620150 28711 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:21.622257 28977 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.605962 29146 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.636415 28738 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.638912 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.650225 24425 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:21.670996 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.596782 28918 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
W20260809 23:19:21.672260 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 324638us
I20260809 23:19:21.678059 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.673202 24312 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:21.680047 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.695637 24311 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:21.696144 24311 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:21.677472 29191 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.702275 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:21.705376 24313 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:21.685068 29299 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:21.709522 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 154665us
I20260809 23:19:21.702064 28923 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:21.682902 29304 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.713225 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:21.713764 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121252us
I20260809 23:19:21.715796 24311 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:21.707337 28912 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:21.740038 24543 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:21.746467 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:21.746637 25372 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:21.749179 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.746229 24311 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:21.753185 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.749084 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.715065 28923 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:21.756178 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55742.3us
I20260809 23:19:21.757602 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.757790 24424 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:21.758795 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:21.744498 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.020s (SENT) [suppressed 34 similar messages]
I20260809 23:19:21.758024 29304 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.744220 29311 raft_consensus.cc:2741] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260809 23:19:21.717969 29299 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.771059 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.771634 29299 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:21.772380 29299 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:21.772780 29299 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:21.773171 29299 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:21.747774 28738 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:21.755236 29297 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.775600 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:21.776598 29297 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:21.743772 29191 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.783254 24430 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:19:21.714818 28918 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260809 23:19:21.784795 24423 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260809 23:19:21.843899 29322 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260809 23:19:21.845129 28923 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:21.847215 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 160609us
I20260809 23:19:21.848094 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:21.850211 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:21.851258 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 65221.9us
W20260809 23:19:21.851372 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94502.2us
I20260809 23:19:21.854632 28882 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:19:21.855088 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:21.856083 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69820.1us
W20260809 23:19:21.856534 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75216.2us
I20260809 23:19:21.856801 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:21.857249 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:21.858074 29313 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:21.859555 28836 raft_consensus.cc:2741] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260809 23:19:21.846688 28923 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:21.862592 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.870589 24311 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:19:21.858418 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:21.860280 28836 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:21.892282 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:21.863790 29172 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:21.893304 29313 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:21.900429 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.901288 29120 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:21.901685 29120 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:21.903268 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.904072 28836 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:21.905129 28836 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.905679 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.906594 29003 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:21.907178 28971 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:21.907253 29311 raft_consensus.cc:2741] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260809 23:19:21.893216 29125 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.015s
I20260809 23:19:21.907699 29003 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:21.902635 29120 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:21.908370 28889 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:21.907894 28971 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:21.908731 28889 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:21.906734 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.909606 29311 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:21.909874 29314 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:21.910374 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.911249 29311 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.915417 24545 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260809 23:19:21.915517 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57196.3us
I20260809 23:19:21.916393 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:21.903169 28862 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:21.911662 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.920133 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.925374 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.922853 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.926538 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:21.957679 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46084.5us
I20260809 23:19:21.974722 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:21.922636 29218 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.924945 28862 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260809 23:19:21.976835 24430 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260809 23:19:21.977958 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60779.9us
W20260809 23:19:21.982671 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51523.6us
I20260809 23:19:21.931425 29172 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:21.936538 29311 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:21.974105 29313 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 23:19:21.995316 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88337.1us
W20260809 23:19:21.977609 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49078.6us
I20260809 23:19:22.016811 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:21.981740 29211 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:21.997459 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:21.997862 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97530.3us
I20260809 23:19:21.909754 28889 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:21.982501 29302 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:21.997473 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.022552 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:21.995903 29172 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:21.979573 28984 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260809 23:19:22.029893 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.026163 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.027647 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:22.001502 28841 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:22.031560 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.032687 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.032691 24309 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:22.033217 29309 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:22.033907 28862 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:22.032815 29311 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.078095 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.016834 29297 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:21.929980 29304 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.090s
I20260809 23:19:22.067879 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.087152 28984 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260809 23:19:22.039884 28711 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:22.042743 29313 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.073122 29302 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:22.000156 29218 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:22.027143 29003 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.030637 29172 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:21.998910 29191 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
W20260809 23:19:22.079650 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53627.9us
I20260809 23:19:22.081526 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.078001 29052 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:22.075399 29309 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:22.040688 28841 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:22.080624 29211 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.030364 29189 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:22.093447 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.112517 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:22.083784 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56200.9us
I20260809 23:19:22.075158 29249 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260809 23:19:22.128157 24312 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:22.074110 28912 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:22.131551 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.132892 24311 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:22.126788 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:22.151885 25372 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
W20260809 23:19:22.137534 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44178.2us
I20260809 23:19:22.093751 29311 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:22.102478 28977 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.155061 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.102272 28889 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:22.126075 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:22.159091 25372 pending_rounds.cc:76] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 6
I20260809 23:19:22.105152 29274 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:22.092147 29297 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:22.160315 29327 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260809 23:19:22.164391 29314 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:22.196242 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:22.168779 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:22.155409 28711 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.103281 29218 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:22.130285 28841 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.176821 24320 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 23:19:22.087198 28859 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260809 23:19:22.167891 28754 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.106998 29309 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:22.189231 25372 pending_rounds.cc:97] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260809 23:19:22.209160 25372 raft_consensus.cc:2866] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:22.188886 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 110878us
I20260809 23:19:22.131526 28738 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 23:19:22.200121 29328 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260809 23:19:22.192754 29313 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election won for term 5
W20260809 23:19:22.200603 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69128.1us
I20260809 23:19:22.201663 28912 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.210507 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:22.217921 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43663.1us
I20260809 23:19:22.219873 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:22.195222 29331 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260809 23:19:22.238189 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.209494 29218 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.246994 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:22.247205 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:22.247853 25365 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.248389 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:22.249281 25368 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:22.250070 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:22.188886 28977 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.253731 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:22.258028 25363 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260809 23:19:22.258440 25363 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 6
I20260809 23:19:22.258791 25363 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260809 23:19:22.259083 25363 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:22.259459 25363 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260809 23:19:22.259301 29306 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 23:19:22.221551 28971 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.260648 28754 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:22.262722 24320 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:22.263399 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:22.264602 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.267619 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.267923 25365 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:22.268972 24316 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:22.260352 28971 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.286953 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:22.287628 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 196726us
I20260809 23:19:22.212872 29189 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260809 23:19:22.272850 29211 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:22.295614 29334 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.058s
I20260809 23:19:22.262048 29146 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:22.301947 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:22.325743 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:22.222067 29302 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.327298 24311 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:22.329605 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.259775 25363 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:22.271986 29313 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:22.301908 29322 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260809 23:19:22.290076 28754 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:22.285382 28738 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:19:22.358127 24314 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:22.357923 24316 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:22.259091 28739 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:22.320591 29338 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260809 23:19:22.298703 29052 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:22.297223 28881 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.381662 28881 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.385768 28881 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:22.387504 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 95342.6us
I20260809 23:19:22.389402 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:22.390663 29336 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.394726 28971 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:22.398241 29299 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.003s
I20260809 23:19:22.399153 29305 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.399224 28841 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.399945 29305 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.402453 29305 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:22.404043 29313 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.408646 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2467) took 1077 ms. Trace:
I20260809 23:19:22.409149 24309 rpcz_store.cc:276] 0809 23:19:21.331658 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:21.331830 (+   172us) service_pool.cc:227] Handling call
0809 23:19:21.347646 (+ 15816us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:22.074841 (+727195us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:22.408108 (+333267us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:22.408437 (+   329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1535498112}
W20260809 23:19:22.414007 29320 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.436s	user 0.000s	sys 0.001s
I20260809 23:19:22.416697 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:22.420698 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 156156us
I20260809 23:19:22.335561 29304 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.339912 29189 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:22.429124 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.398797 29299 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.436131 29299 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:22.436781 29297 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:22.438735 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:22.425971 29336 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:22.443216 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:22.443570 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:22.369421 28711 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:22.449604 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:22.426801 29273 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 23:19:22.454733 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 236473us
I20260809 23:19:22.463135 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:22.470415 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.432677 29189 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:22.483412 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:22.493093 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:22.554838 24317 raft_consensus.cc:2362] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5. Re-sending same reply.
I20260809 23:19:22.486665 29297 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260809 23:19:22.498560 24309 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:22.421226 28739 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.499522 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.559410 24320 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:22.495003 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:22.455394 29336 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:22.458663 28754 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260809 23:19:22.431396 29304 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.472041 29311 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:22.518744 24314 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:22.527524 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.575752 24316 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:22.539861 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:22.541605 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 214745us
I20260809 23:19:22.578003 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:22.473191 29218 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:22.548909 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 211532us
I20260809 23:19:22.526708 28841 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:22.527292 29299 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.501253 28711 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260809 23:19:22.553232 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:22.497852 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2469) took 1073 ms. Trace:
I20260809 23:19:22.511025 29309 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.557018 29189 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:22.581629 24317 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:22.582645 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.595111 29309 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:22.595583 24541 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:22.590127 24323 rpcz_store.cc:276] 0809 23:19:21.427823 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:21.428034 (+   211us) service_pool.cc:227] Handling call
0809 23:19:21.447603 (+ 19569us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:21.480568 (+ 32965us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:21.480603 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:21.523289 (+ 42686us) log.cc:842] Serialized 26 byte log entry
0809 23:19:21.725187 (+201898us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:21.725253 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:21.725378 (+   125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:22.413178 (+687800us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:22.497432 (+ 84254us) raft_consensus.cc:1671] finished
0809 23:19:22.497461 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:22.497773 (+   312us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2483,"raft.queue_time_us":7969,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":3338,"spinlock_wait_cycles":37918592,"thread_start_us":159142,"threads_started":1,"wal-append.queue_time_us":350868}
I20260809 23:19:22.597877 28739 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:22.584357 29274 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:22.606154 28923 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.742s	user 0.011s	sys 0.000s
I20260809 23:19:22.603973 29172 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.607615 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.649328 24428 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 4.
W20260809 23:19:22.609405 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 150383us
I20260809 23:19:22.611857 24320 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:22.612619 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.650483 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:22.639226 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47779.8us
I20260809 23:19:22.637828 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.652498 24321 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:22.645284 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45002.8us
I20260809 23:19:22.651012 24314 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:22.655134 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.657215 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:22.604161 29218 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:22.645553 29350 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260809 23:19:22.663400 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.665723 24423 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:22.682474 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.680284 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:22.695431 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.696151 24435 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:22.657389 29212 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:22.664217 28171 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.608407 29299 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.666714 29191 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:22.652834 29306 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260809 23:19:22.652312 29274 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.650297 29305 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260809 23:19:22.615764 29311 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.702908 29333 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:22.741488 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79098.3us
I20260809 23:19:22.737232 29249 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election won for term 5
W20260809 23:19:22.819000 29379 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
W20260809 23:19:22.824244 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79642.6us
I20260809 23:19:22.840801 24545 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:22.842865 29304 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:22.843142 29357 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260809 23:19:22.698753 29336 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.843636 29172 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:22.709425 28171 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.821276 29299 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:22.853890 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:22.851574 29249 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:22.905061 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:22.838265 28859 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260809 23:19:22.936621 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 396792us
I20260809 23:19:22.753455 29333 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:22.942044 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 244031us
I20260809 23:19:22.910200 29172 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:22.928306 29297 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:22.928624 29299 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.954834 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:22.959476 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:22.964057 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:22.964802 25375 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:22.967463 24422 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:22.967538 25373 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:22.968596 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.969331 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:22.970129 24428 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.967972 24422 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:22.974514 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:22.974813 24422 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:22.975768 24422 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.953397 29172 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:22.993482 29315 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.214s	user 0.004s	sys 0.002s
I20260809 23:19:22.994297 29274 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:22.995332 29297 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:22.993396 29377 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
I20260809 23:19:22.959576 28912 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20260809 23:19:22.979574 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:23.012440 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 459248us
I20260809 23:19:23.014865 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:23.015519 25375 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:23.016453 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:23.017091 25369 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:23.015027 25373 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260809 23:19:22.981004 29211 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.685s	user 0.013s	sys 0.000s
I20260809 23:19:22.979012 29313 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:22.991255 29355 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:22.962357 29299 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:23.029011 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 150052us
I20260809 23:19:23.030025 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:23.043576 29366 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260809 23:19:23.044104 29374 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:23.044626 29374 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:23.045027 29374 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:23.046010 29386 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:23.046504 29386 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:23.046825 29386 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:23.027554 25366 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:23.050885 29369 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:23.039675 29003 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.935s	user 0.011s	sys 0.002s
I20260809 23:19:23.051709 29003 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:23.052069 29369 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:23.053244 24431 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:23.053818 29369 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.057129 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:23.030274 29211 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 23:19:23.059911 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 409484us
W20260809 23:19:23.068073 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93628.8us
I20260809 23:19:23.069264 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.075234 24312 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:23.080129 24311 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260809 23:19:23.061380 29274 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:23.089496 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.091917 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.074159 29191 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:23.031813 29313 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:23.003981 29249 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.105525 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.116194 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.119004 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.120440 29313 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.120735 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.087471 29274 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:23.125079 29392 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:23.125494 29352 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:23.074671 29189 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:23.128417 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59478.4us
I20260809 23:19:23.129222 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:23.104055 28923 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:23.130355 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:23.139292 24424 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:23.031630 29355 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.142306 25373 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:23.117457 29212 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260809 23:19:23.130081 29361 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260809 23:19:23.060786 29172 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.116312 28859 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:23.130430 29274 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:23.108047 29003 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260809 23:19:23.130484 29365 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260809 23:19:23.137136 29191 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:23.130970 29299 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.214833 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.143184 28923 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:23.215813 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67920.5us
I20260809 23:19:23.230058 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.230896 24424 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:23.235387 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 290986us
W20260809 23:19:23.235543 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94498.1us
I20260809 23:19:23.238003 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:23.234375 29191 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.240890 29311 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:23.241534 29382 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:23.242244 29402 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260809 23:19:23.245288 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.246316 24424 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:23.130632 29411 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.116s
I20260809 23:19:23.248270 29411 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260809 23:19:23.249277 29317 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.273s	user 0.000s	sys 0.013s
I20260809 23:19:23.252370 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 712) took 1156 ms. Trace:
I20260809 23:19:23.252815 24541 rpcz_store.cc:276] 0809 23:19:22.095736 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.096091 (+   355us) service_pool.cc:227] Handling call
0809 23:19:22.096496 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:22.644829 (+548333us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.252037 (+607208us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:23.252251 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1147339520}
W20260809 23:19:23.255932 29354 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.603s	user 0.002s	sys 0.000s
I20260809 23:19:23.256345 29355 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:23.142426 29334 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260809 23:19:23.209062 28889 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:23.262971 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:23.142064 29376 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260809 23:19:23.276116 24422 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260809 23:19:23.278568 24422 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 6
I20260809 23:19:23.278837 24422 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260809 23:19:23.279112 24422 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:23.276932 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:23.263554 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:23.219952 28977 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:23.232719 28923 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:23.249279 29404 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260809 23:19:23.231951 29408 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:23.291059 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53137.6us
I20260809 23:19:23.242756 29191 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:23.292598 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56305.1us
I20260809 23:19:23.293813 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:23.294462 25373 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:23.296636 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:23.277838 29361 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:19:23.300356 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2371) took 1170 ms. Trace:
I20260809 23:19:23.225443 28859 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:23.303876 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.297339 24422 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:23.310765 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:23.311442 29388 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260809 23:19:23.313057 29414 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:23.313659 29414 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:23.315485 25369 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 5
I20260809 23:19:23.315698 29189 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:23.281816 28889 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:23.316725 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:23.317514 24430 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:23.317368 28889 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:23.181398 29392 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:23.320890 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:23.321513 24316 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:23.315814 25369 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:23.324784 25369 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:23.292236 29402 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:23.300751 24543 rpcz_store.cc:276] 0809 23:19:22.129593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.129747 (+   154us) service_pool.cc:227] Handling call
0809 23:19:22.130096 (+   349us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:22.169026 (+ 38930us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.169057 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:22.187993 (+ 18936us) log.cc:842] Serialized 26 byte log entry
0809 23:19:22.472122 (+284129us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:22.472172 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:22.472265 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:23.255082 (+782817us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:23.296355 (+ 41273us) raft_consensus.cc:1671] finished
0809 23:19:23.296379 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:23.300285 (+  3906us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3485,"raft.queue_time_us":24373,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"spinlock_wait_cycles":5248256,"thread_start_us":264343,"threads_started":1,"wal-append.queue_time_us":365296}
I20260809 23:19:23.344439 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.295720 29371 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:23.341926 24430 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:23.348271 29371 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.349071 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:23.299782 28977 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:23.350368 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:23.350751 28977 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:23.351224 28977 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:23.351737 29311 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:23.352270 29311 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:23.352632 28859 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:23.361287 24316 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:23.361478 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 216043us
I20260809 23:19:23.362488 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.363466 24316 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:23.297967 29394 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:23.367134 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 130392us
I20260809 23:19:23.365726 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:23.406905 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:23.408588 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46226.4us
I20260809 23:19:23.281970 29355 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20260809 23:19:23.409287 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.346987 29189 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:23.355952 28859 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:23.350456 29371 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.434273 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.435168 24316 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:23.350504 25366 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:23.364329 29420 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:23.484388 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:23.489969 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 192263us
I20260809 23:19:23.490439 25375 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:23.494656 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.498528 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.500324 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:23.502866 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89456.6us
I20260809 23:19:23.505066 24544 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:23.505163 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:23.508992 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:23.501356 24317 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:23.444319 28859 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:23.514227 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:23.442409 29333 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.519402 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.526355 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
W20260809 23:19:23.523876 24428 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.547s	user 0.008s	sys 0.003s
W20260809 23:19:23.527278 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 87040.4us
I20260809 23:19:23.524186 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:23.528717 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.532116 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:23.497985 29404 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260809 23:19:23.297619 29408 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.527598 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:23.356827 29402 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.548803 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:23.544556 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:23.552320 25371 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:23.548828 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.353189 29311 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.604139 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.549409 29392 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.530591 28859 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:22.540486 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:23.438436 29189 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:23.606067 29412 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260809 23:19:23.612423 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:23.549085 29333 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:23.635828 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 100350us
I20260809 23:19:23.636310 25371 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:23.562278 24317 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:23.630748 24270 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:23.635316 24544 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
W20260809 23:19:23.545306 29354 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260809 23:19:23.539611 29371 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:23.669651 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34762 (request call id 984) took 1907 ms. Trace:
I20260809 23:19:23.671952 24427 rpcz_store.cc:276] 0809 23:19:21.762625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:21.763021 (+   396us) service_pool.cc:227] Handling call
0809 23:19:23.664524 (+1901503us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:19:23.669582 (+  5058us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":573495680}
I20260809 23:19:23.516041 29420 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:23.677997 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 172951us
W20260809 23:19:23.665663 29395 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260809 23:19:23.660713 29392 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:23.665660 29371 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:23.693723 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:23.657339 29414 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:23.631513 29342 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.376s
I20260809 23:19:23.700582 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2073) took 1172 ms. Trace:
W20260809 23:19:23.694075 29393 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
I20260809 23:19:23.676461 29301 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:23.674481 29204 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:23.709352 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 191250us
I20260809 23:19:23.707628 24320 rpcz_store.cc:276] 0809 23:19:22.553741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.655017 (+101276us) service_pool.cc:227] Handling call
0809 23:19:22.686969 (+ 31952us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:22.693931 (+  6962us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.693960 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:22.702083 (+  8123us) log.cc:842] Serialized 52 byte log entry
0809 23:19:23.036425 (+334342us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:23.042331 (+  5906us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.048616 (+  6285us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.048785 (+   169us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.048863 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:23.700283 (+651420us) raft_consensus.cc:1671] finished
0809 23:19:23.700314 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:23.700502 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6595,"prepare.queue_time_us":4412,"prepare.run_cpu_time_us":4398,"prepare.run_wall_time_us":8400,"raft.queue_time_us":80642,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":21802,"spinlock_wait_cycles":21599360,"thread_start_us":292214,"threads_started":1,"wal-append.queue_time_us":352940}
I20260809 23:19:23.656739 29422 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
W20260809 23:19:23.726172 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89365.1us
I20260809 23:19:23.661114 29333 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.694170 29400 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:23.704268 29371 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:23.702234 29426 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:23.761315 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:23.665686 29418 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.379s
I20260809 23:19:23.674888 29423 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260809 23:19:23.766014 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:23.718547 29304 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:23.716780 29414 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:23.800410 29369 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:23.800940 29369 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:23.806820 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:23.784179 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:23.792973 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2071) took 1265 ms. Trace:
I20260809 23:19:23.831094 24323 rpcz_store.cc:276] 0809 23:19:22.527558 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.645314 (+117756us) service_pool.cc:227] Handling call
0809 23:19:22.686605 (+ 41291us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:22.881482 (+194877us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.881524 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:22.890726 (+  9202us) log.cc:842] Serialized 26 byte log entry
0809 23:19:23.071206 (+180480us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:23.478581 (+407375us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.791327 (+312746us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.791925 (+   598us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:23.792005 (+    80us) raft_consensus.cc:1671] finished
0809 23:19:23.792767 (+   762us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:23.792904 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26668,"prepare.queue_time_us":12926,"prepare.run_cpu_time_us":327,"prepare.run_wall_time_us":1011,"raft.queue_time_us":205884,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":82991,"thread_start_us":1016159,"threads_started":4,"wal-append.queue_time_us":198502}
I20260809 23:19:23.791630 29356 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:23.726423 29204 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.908200 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2075) took 1345 ms. Trace:
I20260809 23:19:23.984193 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.791054 29414 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.927369 25325 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:23.907194 29374 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.862s	user 0.013s	sys 0.001s
W20260809 23:19:24.008510 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.000s	sys 0.009s
W20260809 23:19:24.012156 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.001s	sys 0.010s
I20260809 23:19:24.010460 29374 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.018730 25365 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:23.964183 24314 rpcz_store.cc:276] 0809 23:19:22.562575 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.654824 (+ 92249us) service_pool.cc:227] Handling call
0809 23:19:22.681394 (+ 26570us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:22.698495 (+ 17101us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.698527 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:22.710183 (+ 11656us) log.cc:842] Serialized 52 byte log entry
0809 23:19:22.934803 (+224620us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:22.934871 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:22.934981 (+   110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:23.903900 (+968919us) raft_consensus.cc:1671] finished
0809 23:19:23.907969 (+  4069us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:23.908139 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9638,"raft.queue_time_us":22761,"spinlock_wait_cycles":14417792,"thread_start_us":108561,"threads_started":1,"wal-append.queue_time_us":258114}
I20260809 23:19:23.837028 29392 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:23.778668 29426 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.913275 29400 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:23.988526 29249 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.027s	sys 0.036s
W20260809 23:19:24.077654 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 316403us
I20260809 23:19:23.920115 29189 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:23.987186 29445 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:23.801955 29369 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:23.966656 29418 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.208801 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:23.978361 29408 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:24.210474 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:24.285490 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:23.989574 29402 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.235343 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:24.028645 29342 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.042372 28836 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:24.038968 29299 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:24.242189 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:24.200959 29249 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.027s	sys 0.037s
W20260809 23:19:24.288220 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80095.7us
W20260809 23:19:24.217648 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 391374us
I20260809 23:19:24.311479 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:24.234771 29445 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:24.288853 24534 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:24.249399 24314 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:24.301276 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.024921 29423 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.225s
I20260809 23:19:24.201042 29374 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:23.902549 29356 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:24.245040 29501 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:24.348400 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:24.294507 29301 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.310098 29474 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:24.378249 29456 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:24.352535 29369 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:24.381255 29433 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.660s	user 0.000s	sys 0.001s
I20260809 23:19:24.371920 29469 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:24.438623 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 511296us
I20260809 23:19:24.369803 29333 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.349774 29445 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:24.433554 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122167us
I20260809 23:19:24.445977 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.387526 29420 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.498092 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.451305 29469 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:24.312803 29418 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:24.495829 29447 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.672s	user 0.000s	sys 0.002s
I20260809 23:19:24.323390 29342 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:24.627094 25371 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.027s	sys 0.037s
W20260809 23:19:24.627734 25371 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.028s	sys 0.037s
I20260809 23:19:24.629223 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:24.671507 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.017s	sys 0.049s
W20260809 23:19:24.672037 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.017s	sys 0.049s
I20260809 23:19:24.691748 29414 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.692451 29414 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.693020 29414 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:24.548069 29433 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260809 23:19:24.390825 29449 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.757s
I20260809 23:19:24.560204 29469 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:24.395251 29392 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.704923 29414 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.703768 29204 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.560587 29402 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.597505 29363 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:24.758299 24541 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.019s	sys 0.035s
W20260809 23:19:24.712291 29437 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.560s	user 0.000s	sys 0.001s
W20260809 23:19:24.768232 24541 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.020s	sys 0.036s
I20260809 23:19:24.437491 29301 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:24.737212 28923 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.413s	user 0.012s	sys 0.002s
I20260809 23:19:24.739511 29189 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:24.401049 29501 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.766198 29494 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.818643 29304 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:24.818459 29494 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.819643 29304 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.820101 29501 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.820706 29494 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.821659 29501 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.822026 29426 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.822175 29501 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.827772 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:24.796448 28977 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.445s	user 0.010s	sys 0.003s
I20260809 23:19:24.799456 28171 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.805279 29513 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.805398 29301 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.805836 28923 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:24.808169 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2063) took 2338 ms (client timeout 1901 ms). Trace:
I20260809 23:19:24.774196 29414 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:24.809998 29484 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260809 23:19:24.816025 29334 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260809 23:19:24.817394 29438 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.777s	user 0.001s	sys 0.000s
I20260809 23:19:24.817770 29052 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:24.820268 29304 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:24.822575 29426 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.822661 29003 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.902688 29003 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.905079 29003 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.823171 25371 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:24.824222 29418 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.451949 29400 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.413805 29474 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.825937 29189 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.826335 29342 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.829022 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.698839 29503 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.833945 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 912) took 2675 ms. Trace:
I20260809 23:19:24.839550 29501 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.848834 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 866) took 1555 ms. Trace:
I20260809 23:19:24.849560 25375 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:24.852145 29274 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.608s	user 0.013s	sys 0.000s
I20260809 23:19:24.774578 29363 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:24.857007 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 841) took 1784 ms. Trace:
W20260809 23:19:24.867967 24311 rpcz_store.cc:269] 0809 23:19:22.469924 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.470078 (+   154us) service_pool.cc:227] Handling call
0809 23:19:24.807924 (+2337846us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:24.808081 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1199406848}
I20260809 23:19:24.809528 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 619) took 1498 ms. Trace:
I20260809 23:19:24.874989 24544 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:24.874894 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.846665 29521 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:24.858560 28859 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.226s	user 0.008s	sys 0.004s
I20260809 23:19:24.893765 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:24.894021 29334 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:24.968137 29334 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:24.958004 29274 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:24.971009 29274 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:24.900892 28923 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.902081 29304 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.945492 29342 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:24.973361 29342 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:24.870642 29462 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.892s	user 0.007s	sys 0.000s
I20260809 23:19:24.912539 29426 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:24.975452 29342 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:24.976504 29426 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:24.976759 24311 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260809 23:19:24.930573 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1035) took 1200 ms (client timeout 1521 ms). Trace:
I20260809 23:19:24.977382 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2080) took 1254 ms. Trace:
W20260809 23:19:24.977520 25371 rpcz_store.cc:269] 0809 23:19:23.712780 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.628878 (+916098us) service_pool.cc:227] Handling call
0809 23:19:24.913024 (+284146us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:24.913152 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443954560}
I20260809 23:19:24.977730 24311 rpcz_store.cc:276] 0809 23:19:23.723020 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.976086 (+1253066us) service_pool.cc:227] Handling call
0809 23:19:24.976432 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:24.977119 (+   687us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:24.977321 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:24.774549 29402 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.920459 29474 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.930531 29052 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:24.920449 29189 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:24.940271 25372 rpcz_store.cc:276] 0809 23:19:22.154691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.155293 (+   602us) service_pool.cc:227] Handling call
0809 23:19:22.155631 (+   338us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:22.303129 (+147498us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.303174 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:22.328965 (+ 25791us) log.cc:842] Serialized 52 byte log entry
0809 23:19:22.829924 (+500959us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:22.840549 (+ 10625us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.037514 (+196965us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.039594 (+  2080us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.039677 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:24.824465 (+1784788us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b1800366ca0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:24.829840 (+  5375us) raft_consensus.cc:1671] finished
0809 23:19:24.829873 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:24.830046 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103853,"prepare.queue_time_us":11944,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":8474,"raft.queue_time_us":192687,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":184,"spinlock_wait_cycles":17360000,"thread_start_us":550847,"threads_started":2,"wal-append.queue_time_us":611872}
I20260809 23:19:24.856726 28977 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:24.958654 24422 rpcz_store.cc:276] 0809 23:19:23.311202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.312784 (+  1582us) service_pool.cc:227] Handling call
0809 23:19:23.313121 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:23.313452 (+   331us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.313479 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:23.313937 (+   458us) log.cc:842] Serialized 26 byte log entry
0809 23:19:23.422245 (+108308us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:23.422286 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.422382 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:24.809169 (+1386787us) raft_consensus.cc:1671] finished
0809 23:19:24.809196 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:24.809430 (+   234us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":414,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"thread_start_us":107966,"threads_started":1,"wal-append.queue_time_us":299694}
I20260809 23:19:24.945560 25369 rpcz_store.cc:276] 0809 23:19:23.293758 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.293905 (+   147us) service_pool.cc:227] Handling call
0809 23:19:23.315262 (+ 21357us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:23.346852 (+ 31590us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.346886 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:23.353413 (+  6527us) log.cc:842] Serialized 52 byte log entry
0809 23:19:23.620249 (+266836us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:23.821732 (+201483us) log.cc:842] Serialized 14 byte log entry
0809 23:19:23.853169 (+ 31437us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.854230 (+  1061us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:24.848533 (+994303us) raft_consensus.cc:1671] finished
0809 23:19:24.848557 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:24.848741 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":692704,"raft.queue_time_us":473535,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":20469,"spinlock_wait_cycles":18169600,"thread_start_us":389254,"threads_started":3,"wal-append.queue_time_us":353589}
I20260809 23:19:24.962674 29363 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:24.985380 25371 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:24.958304 24316 rpcz_store.cc:276] 0809 23:19:23.072476 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.525559 (+453083us) service_pool.cc:227] Handling call
0809 23:19:23.552789 (+ 27230us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:23.721930 (+169141us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:23.767366 (+ 45436us) log.cc:842] Serialized 14 byte log entry
0809 23:19:24.332721 (+565355us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:24.332789 (+    68us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:24.332918 (+   129us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:24.842704 (+509786us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:24.842765 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:24.856928 (+ 14163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1761,"prepare.queue_time_us":47847,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":2947,"raft.queue_time_us":465725,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":34830,"spinlock_wait_cycles":388055808,"thread_start_us":564839,"threads_started":2,"wal-append.queue_time_us":600122}
I20260809 23:19:24.897238 29414 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.007718 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1036) took 1289 ms. Trace:
I20260809 23:19:25.012427 25371 rpcz_store.cc:276] 0809 23:19:23.718426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.978572 (+1260146us) service_pool.cc:227] Handling call
0809 23:19:24.978844 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.007440 (+ 28596us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.007657 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:25.013417 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.462s	user 0.079s	sys 0.182s
W20260809 23:19:25.013880 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.462s	user 0.079s	sys 0.182s
I20260809 23:19:25.017628 29340 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260809 23:19:25.024044 29340 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:25.024390 29340 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:25.025261 29340 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.026564 29323 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:25.002015 29052 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.028184 29402 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.029017 29052 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.029910 29340 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:25.030179 24316 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:25.030813 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2084) took 1306 ms. Trace:
I20260809 23:19:25.027062 29323 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:24.871814 29513 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.031122 24316 rpcz_store.cc:276] 0809 23:19:23.724393 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.008550 (+1284157us) service_pool.cc:227] Handling call
0809 23:19:25.029995 (+ 21445us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.030596 (+   601us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.030760 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:25.033733 25368 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:25.038668 29052 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:25.039254 29052 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.040109 29513 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.041175 29052 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.004172 29521 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.025195 29414 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.045338 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.045496 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:25.037915 29475 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.493s	user 0.000s	sys 0.001s
I20260809 23:19:25.047022 29518 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.043895 29323 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.070051 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.070338 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.070858 24544 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 2.
I20260809 23:19:24.987818 28859 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.071991 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.069255 24546 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.079118 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:24.953644 29503 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.085824 29342 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:25.086580 29334 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.086539 29342 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:25.088093 29524 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.088196 29342 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:25.088912 29524 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.082845 29513 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:19:25.091439 29513 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:25.091852 29513 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:19:25.092185 29445 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.593s	user 0.006s	sys 0.003s
I20260809 23:19:25.092666 29445 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.095469 29334 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.087807 29323 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:25.096141 29521 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.080015 29527 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.097193 29527 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.072569 29414 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:25.099826 29414 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.100584 29527 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.102911 29356 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.104707 29445 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.080587 29518 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.108273 29518 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:25.108665 29469 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:25.071224 29480 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.361s	user 0.001s	sys 0.000s
W20260809 23:19:25.060115 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.005s	sys 0.002s
W20260809 23:19:25.115212 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.006s	sys 0.002s
I20260809 23:19:25.116148 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2483) took 1627 ms. Trace:
I20260809 23:19:25.139587 24314 rpcz_store.cc:276] 0809 23:19:23.488909 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.340662 (+851753us) service_pool.cc:227] Handling call
0809 23:19:24.341165 (+   503us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:24.373402 (+ 32237us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:24.373448 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:24.408435 (+ 34987us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.115721 (+707286us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:25.115772 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.115865 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.115925 (+    60us) raft_consensus.cc:1671] finished
0809 23:19:25.115942 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.116085 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14300,"raft.queue_time_us":476004,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":617881,"threads_started":1,"wal-append.queue_time_us":678153}
W20260809 23:19:25.163070 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50499.4us
I20260809 23:19:25.077239 29525 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:25.113920 29386 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 2.067s	user 0.015s	sys 0.002s
I20260809 23:19:25.147644 29356 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.199553 29356 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.200433 29363 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.122509 29219 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.202067 29356 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.229909 29469 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.232903 29503 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.234329 28977 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:25.235006 29524 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:25.184864 29371 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.401s	user 0.013s	sys 0.000s
I20260809 23:19:25.249691 28738 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.262580 28738 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.245160 29386 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.202661 29363 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:25.255303 29365 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.390s	user 0.022s	sys 0.002s
I20260809 23:19:25.267980 29508 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.203698 29219 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.267508 29371 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.286755 28859 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.186383 29525 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:25.283885 29365 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.453s	user 0.023s	sys 0.002s
W20260809 23:19:25.295097 29456 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.863s	user 0.009s	sys 0.004s
I20260809 23:19:25.291340 28738 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.308786 29525 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.316797 29513 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.292448 29508 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.319218 29456 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:25.282351 29386 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.321439 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 913) took 1907 ms. Trace:
W20260809 23:19:25.325732 29311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.953s	user 0.000s	sys 0.038s
I20260809 23:19:25.329532 25373 rpcz_store.cc:276] 0809 23:19:23.413676 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.413804 (+   128us) service_pool.cc:227] Handling call
0809 23:19:23.414141 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:23.414556 (+   415us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.414595 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:23.443949 (+ 29354us) log.cc:842] Serialized 26 byte log entry
0809 23:19:23.786782 (+342833us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:23.901380 (+114598us) log.cc:842] Serialized 14 byte log entry
0809 23:19:24.098610 (+197230us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:24.098957 (+   347us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.321124 (+1222167us) raft_consensus.cc:1671] finished
0809 23:19:25.321155 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.321362 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10841,"prepare.queue_time_us":1279216,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":423046,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":30381,"thread_start_us":551551,"threads_started":3,"wal-append.queue_time_us":1747167}
W20260809 23:19:25.335891 29311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.000s	sys 0.039s
I20260809 23:19:25.349516 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.350327 25373 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:25.355492 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.356319 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
W20260809 23:19:25.358989 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.024s	sys 0.012s
W20260809 23:19:25.359308 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.024s	sys 0.012s
W20260809 23:19:25.359793 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 504021us
I20260809 23:19:25.359342 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.353564 29513 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.361019 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:25.363389 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.353936 29219 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.372781 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 921) took 1061 ms. Trace:
I20260809 23:19:25.373209 25372 rpcz_store.cc:276] 0809 23:19:24.311377 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.017890 (+706513us) service_pool.cc:227] Handling call
0809 23:19:25.018222 (+   332us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:25.028235 (+ 10013us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:25.028273 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:25.029019 (+   746us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.293635 (+264616us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:25.293680 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.293753 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.372546 (+ 78793us) raft_consensus.cc:1671] finished
0809 23:19:25.372595 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.372722 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":643,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":704,"thread_start_us":264241,"threads_started":1,"wal-append.queue_time_us":315867}
I20260809 23:19:25.381973 29371 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:25.387177 29369 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.964s	user 0.012s	sys 0.003s
I20260809 23:19:25.357594 29342 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.395138 29342 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:25.397017 29541 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.053s	user 0.002s	sys 0.000s
W20260809 23:19:25.402325 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2514) took 2741 ms (client timeout 1825 ms). Trace:
I20260809 23:19:25.407222 25366 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:25.418607 24431 rpcz_store.cc:269] 0809 23:19:22.661047 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.661881 (+   834us) service_pool.cc:227] Handling call
0809 23:19:25.402020 (+2740139us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.402169 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":845996928}
I20260809 23:19:25.431599 29508 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.425714 29518 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.425658 29513 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.436719 29518 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.445523 25372 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:25.437301 29513 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:25.450060 29427 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.825s	user 0.000s	sys 0.014s
I20260809 23:19:25.457329 29513 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.518060 24534 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:25.523872 29309 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:25.528661 29309 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.533249 29309 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:25.533653 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.076s	user 0.004s	sys 0.006s
W20260809 23:19:25.534032 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.589s	user 0.006s	sys 0.007s
I20260809 23:19:25.536506 28889 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.537034 28889 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.539561 28889 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.540835 28971 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.540122 29204 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.542801 29532 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.548094 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2485) took 1345 ms. Trace:
I20260809 23:19:25.548117 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.548691 24314 rpcz_store.cc:276] 0809 23:19:24.202730 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.161528 (+958798us) service_pool.cc:227] Handling call
0809 23:19:25.186579 (+ 25051us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.186827 (+   248us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:25.530741 (+343914us) log.cc:842] Serialized 14 byte log entry
0809 23:19:25.547774 (+ 17033us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:25.547826 (+    52us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:25.547870 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.547934 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.548039 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":325440,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":35507,"threads_started":2}
I20260809 23:19:25.551297 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.552047 24431 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:25.545437 28971 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:25.556425 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 868) took 2212 ms (client timeout 2562 ms). Trace:
I20260809 23:19:25.556744 28971 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.546986 29204 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:25.556815 25368 rpcz_store.cc:269] 0809 23:19:23.343768 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.343915 (+   147us) service_pool.cc:227] Handling call
0809 23:19:25.556223 (+2212308us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.556342 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3868211584}
I20260809 23:19:25.550376 29532 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.558686 29204 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.560215 29532 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.561583 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:25.562052 24314 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:25.568529 29359 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.874s	user 0.000s	sys 0.004s
I20260809 23:19:25.570590 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 857) took 3331 ms. Trace:
I20260809 23:19:25.571345 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.572105 24431 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:25.571014 25370 rpcz_store.cc:276] 0809 23:19:22.238912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.251014 (+ 12102us) service_pool.cc:227] Handling call
0809 23:19:22.251401 (+   387us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:22.323163 (+ 71762us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.323194 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:22.323676 (+   482us) log.cc:842] Serialized 26 byte log entry
0809 23:19:22.547520 (+223844us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:22.561684 (+ 14164us) log.cc:842] Serialized 14 byte log entry
0809 23:19:22.664649 (+102965us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:22.664761 (+   112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.567803 (+2903042us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:25.570276 (+  2473us) raft_consensus.cc:1671] finished
0809 23:19:25.570367 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.570520 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65075,"prepare.queue_time_us":18430,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":401460,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":30121,"spinlock_wait_cycles":4693248,"thread_start_us":326092,"threads_started":2,"wal-append.queue_time_us":293645}
I20260809 23:19:25.573246 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.573530 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1030) took 2558 ms. Trace:
I20260809 23:19:25.574494 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.578258 24431 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:25.574033 25363 rpcz_store.cc:276] 0809 23:19:23.014828 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.015346 (+   518us) service_pool.cc:227] Handling call
0809 23:19:23.015700 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:23.137300 (+121600us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.140013 (+  2713us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:23.257571 (+117558us) log.cc:842] Serialized 26 byte log entry
0809 23:19:24.014014 (+756443us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:24.014220 (+   206us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.573099 (+1558879us) spinlock_profiling.cc:243] Waited 72.4 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:25.573150 (+    51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.573208 (+    58us) raft_consensus.cc:1671] finished
0809 23:19:25.573225 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.573461 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15261,"raft.queue_time_us":28265,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":19585,"spinlock_wait_cycles":166436608,"thread_start_us":690240,"threads_started":1,"wal-append.queue_time_us":1214365}
I20260809 23:19:25.579442 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.580143 24431 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:25.581035 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.581632 24431 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:25.586099 24313 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:25.589969 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2486) took 1354 ms. Trace:
I20260809 23:19:25.590428 24314 rpcz_store.cc:276] 0809 23:19:24.235458 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.583742 (+1348284us) service_pool.cc:227] Handling call
0809 23:19:25.589382 (+  5640us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.589617 (+   235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:25.589642 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:25.589692 (+    50us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:25.589738 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.589791 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.589912 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:25.615667 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1237) took 2539 ms. Trace:
I20260809 23:19:25.616145 24314 rpcz_store.cc:276] 0809 23:19:23.076109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.612791 (+2536682us) service_pool.cc:227] Handling call
0809 23:19:25.615604 (+  2813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:25.618358 29417 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.618867 29417 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.623020 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 915) took 2115 ms. Trace:
I20260809 23:19:25.623278 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 922) took 1311 ms. Trace:
I20260809 23:19:25.623572 29417 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.623418 25375 rpcz_store.cc:276] 0809 23:19:23.508112 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.508339 (+   227us) service_pool.cc:227] Handling call
0809 23:19:24.849319 (+1340980us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:24.951093 (+101774us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.622797 (+671704us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400717850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.622954 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3082967296}
I20260809 23:19:25.623872 25371 rpcz_store.cc:276] 0809 23:19:24.311827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.013234 (+701407us) service_pool.cc:227] Handling call
0809 23:19:25.023700 (+ 10466us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.040833 (+ 17133us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:25.040879 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:25.040933 (+    54us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:25.040964 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.623052 (+582088us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:25.623102 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.623218 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38915840}
W20260809 23:19:25.628510 24311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.015s	sys 0.019s
W20260809 23:19:25.628814 24311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.015s	sys 0.019s
I20260809 23:19:25.629374 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2081) took 1906 ms. Trace:
I20260809 23:19:25.629637 24311 rpcz_store.cc:276] 0809 23:19:23.723343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.978565 (+1255222us) service_pool.cc:227] Handling call
0809 23:19:24.978844 (+   279us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:24.982533 (+  3689us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:24.982576 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:25.009176 (+ 26600us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.629139 (+619963us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:25.629166 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.629216 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.629246 (+    30us) raft_consensus.cc:1671] finished
0809 23:19:25.629252 (+     6us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.629330 (+    78us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3316,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":10988,"thread_start_us":566695,"threads_started":1,"wal-append.queue_time_us":395622}
I20260809 23:19:25.641814 29532 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:25.642472 29532 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.643059 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 608) took 3092 ms. Trace:
I20260809 23:19:25.644526 29532 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.643424 24425 rpcz_store.cc:276] 0809 23:19:22.550653 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.552232 (+  1579us) service_pool.cc:227] Handling call
0809 23:19:22.564200 (+ 11968us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:22.637247 (+ 73047us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:22.638681 (+  1434us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:22.649200 (+ 10519us) log.cc:842] Serialized 26 byte log entry
0809 23:19:22.781410 (+132210us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:22.781471 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.642609 (+2861138us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:25.642674 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.642722 (+    48us) raft_consensus.cc:1671] finished
0809 23:19:25.642732 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.642978 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21390,"raft.queue_time_us":18234,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":29092,"spinlock_wait_cycles":63351168,"thread_start_us":50872,"threads_started":1,"wal-append.queue_time_us":239107,"wal-append.run_cpu_time_us":1989,"wal-append.run_wall_time_us":1087904}
I20260809 23:19:25.651937 28836 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.652436 28836 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.655149 28836 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:25.657531 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 839) took 2585 ms (client timeout 1828 ms). Trace:
W20260809 23:19:25.658043 24313 rpcz_store.cc:269] 0809 23:19:23.071843 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.086365 (+ 14522us) service_pool.cc:227] Handling call
0809 23:19:25.657308 (+2570943us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.657458 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5671082880}
I20260809 23:19:25.660214 24317 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:25.660770 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.662042 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2480) took 2214 ms. Trace:
I20260809 23:19:25.662830 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.663553 24428 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:25.662524 24317 rpcz_store.cc:276] 0809 23:19:23.447825 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.637503 (+189678us) service_pool.cc:227] Handling call
0809 23:19:25.659948 (+2022445us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.660753 (+   805us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.661809 (+  1056us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b5400386dd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.661972 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4577520384}
I20260809 23:19:25.668980 29394 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.670077 29394 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.674540 29394 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.684693 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:25.686124 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.686895 24543 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:25.687613 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.688668 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.689368 24543 raft_consensus.cc:2354] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:25.690209 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.693907 25325 leader_election.cc:398] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:25.694279 25325 leader_election.cc:401] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:25.695559 29394 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 7
I20260809 23:19:25.710084 29394 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:25.713400 29414 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.715176 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:25.720446 24535 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 2 FOLLOWER]: Advancing to term 3
W20260809 23:19:25.722586 29419 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.406s	user 0.006s	sys 0.007s
I20260809 23:19:25.733560 24535 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 3.
I20260809 23:19:25.735179 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.736656 29459 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:19:25.737910 29459 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:25.738744 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:25.739010 29459 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:25.743464 24309 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.077s	user 0.077s	sys 0.099s
W20260809 23:19:25.744091 24309 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.079s	user 0.077s	sys 0.099s
W20260809 23:19:25.758140 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.993s	user 0.018s	sys 0.013s
W20260809 23:19:25.758497 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.993s	user 0.018s	sys 0.013s
I20260809 23:19:25.758520 29377 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:25.758944 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.9935e+06us
I20260809 23:19:25.759208 29377 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.761334 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 838) took 2690 ms. Trace:
I20260809 23:19:25.761801 24309 rpcz_store.cc:276] 0809 23:19:23.070667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.070807 (+   140us) service_pool.cc:227] Handling call
0809 23:19:23.071248 (+   441us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:23.210023 (+138775us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.213487 (+  3464us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:23.221027 (+  7540us) log.cc:842] Serialized 26 byte log entry
0809 23:19:23.622098 (+401071us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:23.663408 (+ 41310us) log.cc:842] Serialized 14 byte log entry
0809 23:19:25.758305 (+2094897us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.761035 (+  2730us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.761093 (+    58us) raft_consensus.cc:1671] finished
0809 23:19:25.761109 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.761248 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24674,"prepare.queue_time_us":13500,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":393248,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":167329,"thread_start_us":2508343,"threads_started":3,"wal-append.queue_time_us":530277}
W20260809 23:19:25.773492 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.214s	user 0.007s	sys 0.023s
W20260809 23:19:25.773852 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.215s	user 0.008s	sys 0.024s
I20260809 23:19:25.774866 29532 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.775450 29532 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.777511 29532 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.777943 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.778563 25371 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:25.779094 28836 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.779590 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.780104 29532 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.780200 25375 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:25.780671 29532 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:25.779620 28836 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.783679 28836 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.785039 29299 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.785497 29299 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:25.786744 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:25.789868 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 716) took 2300 ms. Trace:
I20260809 23:19:25.792986 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.794476 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:25.797943 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 832) took 3572 ms (client timeout 3273 ms). Trace:
W20260809 23:19:25.798333 24312 rpcz_store.cc:269] 0809 23:19:22.226133 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.226270 (+   137us) service_pool.cc:227] Handling call
0809 23:19:25.797757 (+3571487us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.797873 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1810281216}
I20260809 23:19:25.790251 24541 rpcz_store.cc:276] 0809 23:19:23.489542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.490042 (+   500us) service_pool.cc:227] Handling call
0809 23:19:23.490434 (+   392us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:23.711593 (+221159us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:23.725238 (+ 13645us) log.cc:842] Serialized 14 byte log entry
0809 23:19:24.773755 (+1048517us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:24.773804 (+    49us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:24.773848 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.789601 (+1015753us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:25.789652 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.789774 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14249,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":5514,"raft.queue_time_us":482612,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":11607,"spinlock_wait_cycles":508062592,"thread_start_us":1008250,"threads_started":2,"wal-append.queue_time_us":791001}
I20260809 23:19:25.799821 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.801055 25375 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:25.802726 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.804209 24423 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260809 23:19:25.805220 29299 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.805922 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.806674 25375 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:25.807260 24316 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:25.808189 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:25.808398 28836 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.808866 28836 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.809386 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2484) took 1607 ms. Trace:
I20260809 23:19:25.809798 24316 rpcz_store.cc:276] 0809 23:19:24.202343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.037505 (+835162us) service_pool.cc:227] Handling call
0809 23:19:25.806400 (+768895us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:25.807849 (+  1449us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.809167 (+  1318us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54003ca0d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:25.809304 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1747325824}
I20260809 23:19:25.810261 29532 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.810832 28836 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.811834 29352 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.812140 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:25.812268 29352 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.812830 24428 raft_consensus.cc:2354] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5.
I20260809 23:19:25.813969 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.814169 29352 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.814544 24428 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:25.814648 29352 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.815328 29352 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.815934 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.816524 24428 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.818075 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.818722 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:25.818090 29352 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:25.821439 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.820s	user 0.031s	sys 0.081s
W20260809 23:19:25.821758 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.821s	user 0.031s	sys 0.081s
I20260809 23:19:25.822425 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2025) took 1913 ms. Trace:
I20260809 23:19:25.822800 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.822747 24427 rpcz_store.cc:276] 0809 23:19:23.908645 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.908813 (+   168us) service_pool.cc:227] Handling call
0809 23:19:23.987039 (+ 78226us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:24.000046 (+ 13007us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:24.000092 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:24.000646 (+   554us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.822101 (+1821455us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:25.822140 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.822224 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.822266 (+    42us) raft_consensus.cc:1671] finished
0809 23:19:25.822276 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.822378 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12461,"raft.queue_time_us":708545,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":43325,"thread_start_us":1821130,"threads_started":1,"wal-append.queue_time_us":1439183}
I20260809 23:19:25.824586 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.825392 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.826386 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.833647 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.834518 24422 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:25.834635 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:25.835206 24542 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:25.818207 28836 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:25.839349 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.839762 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:25.839955 24435 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:25.840328 24422 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:25.835566 29299 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:25.851992 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2275) took 2775 ms. Trace:
I20260809 23:19:25.852610 24540 rpcz_store.cc:276] 0809 23:19:23.076502 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.076664 (+   162us) service_pool.cc:227] Handling call
0809 23:19:23.077011 (+   347us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:23.094330 (+ 17319us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:23.094370 (+    40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:23.130270 (+ 35900us) log.cc:842] Serialized 52 byte log entry
0809 23:19:23.412093 (+281823us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:23.412146 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:23.414857 (+  2711us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.448804 (+2033947us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:25.851602 (+402798us) raft_consensus.cc:1671] finished
0809 23:19:25.851639 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:25.851908 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12617,"raft.queue_time_us":27225,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":21969,"spinlock_wait_cycles":4684672,"thread_start_us":175779,"threads_started":1,"wal-append.queue_time_us":319130}
I20260809 23:19:25.853690 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.854403 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:25.855650 29299 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:25.856082 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:25.856271 29299 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.858719 29299 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:25.859079 29481 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.859625 29481 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.861775 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:25.863576 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.864756 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.865039 24544 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:25.866489 29212 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:25.866882 29212 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.866842 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.867533 29212 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:25.883662 25375 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:25.884703 29353 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:25.885599 29353 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.886211 29353 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:25.888643 25321 leader_election.cc:398] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5.
I20260809 23:19:25.889010 25321 leader_election.cc:401] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260809 23:19:25.890990 28841 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:25.891705 28841 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:25.892282 28841 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:25.893293 29377 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:25.894059 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.894698 25371 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:25.896098 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.897262 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.884716 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.896203 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.899842 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:25.898918 24423 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.904846 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.906885 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:25.905366 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.908102 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:25.907034 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.909611 25375 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260809 23:19:25.907835 24542 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:25.912890 24544 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 2. Current term is 3. Ops: []
I20260809 23:19:25.913848 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:25.914798 24427 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260809 23:19:25.915621 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:25.908679 24422 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.917320 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.920055 25371 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:25.920892 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:25.921764 25375 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:25.838721 28836 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.906658 29397 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:25.917637 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.925448 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.928174 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.918040 29513 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:25.916620 29394 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:25.931432 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:25.922376 29481 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:25.924450 29336 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.309s	user 0.017s	sys 0.025s
I20260809 23:19:25.944100 24425 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:25.945643 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.933327 29513 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:25.947480 29336 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.339s	user 0.018s	sys 0.025s
I20260809 23:19:25.936919 29352 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:25.945148 29204 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:25.957119 24424 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:25.937284 29394 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:25.952888 29342 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:25.961921 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.948706 29333 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:25.963116 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:25.965543 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:25.936751 28862 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260809 23:19:25.971460 28862 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:25.971807 28862 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:25.972620 28862 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.973973 28862 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:25.958315 29513 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:25.980010 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.980222 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.980654 24544 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:25.980934 24540 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:25.981515 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.981623 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.982547 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.983290 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.985230 29342 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:25.986188 29342 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:25.987581 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.961050 29204 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:25.962859 29394 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:25.996006 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.000473 24316 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:26.002199 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.966593 29333 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.021183 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:25.998663 29352 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.033223 24544 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:26.038465 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.039371 24316 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:26.023195 29333 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:26.004261 29204 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:26.068220 29333 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.083253 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.193516 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:26.194425 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.195027 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:26.195787 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.196290 24309 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:26.083504 24316 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260809 23:19:26.197152 29353 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.198184 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.198952 24316 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:26.199748 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.200811 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.084235 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.202612 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:26.203274 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.800s	user 0.016s	sys 0.012s
I20260809 23:19:26.203697 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.204792 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.205343 24311 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:26.206622 24428 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:26.037560 29352 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.089051 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.092664 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.093068 28836 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.093770 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.095029 24313 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260809 23:19:26.100692 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 916) took 2592 ms (client timeout 1893 ms). Trace:
W20260809 23:19:26.212355 25365 rpcz_store.cc:269] 0809 23:19:23.508553 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.508689 (+   136us) service_pool.cc:227] Handling call
0809 23:19:26.100429 (+2591740us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54002ab7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:26.100597 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1169865984}
I20260809 23:19:26.101867 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:26.104260 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:26.111227 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2074) took 3549 ms. Trace:
I20260809 23:19:26.006240 29212 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.085947 28971 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 23:19:26.179541 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:26.208626 24314 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:26.209800 24311 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260809 23:19:26.218331 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:26.217667 24311 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
I20260809 23:19:26.214376 24321 rpcz_store.cc:276] 0809 23:19:22.562248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:22.653378 (+ 91130us) service_pool.cc:227] Handling call
0809 23:19:22.683446 (+ 30068us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:25.611803 (+2928357us) spinlock_profiling.cc:243] Waited 897 us on lock 0x7b18004c8340. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:25.731920 (+120117us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b18004c8340. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac23858f6b 00007fac2386378b
0809 23:19:25.732121 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:25.732153 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:25.732838 (+   685us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.837376 (+104538us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:25.849411 (+ 12035us) log.cc:842] Serialized 14 byte log entry
0809 23:19:25.849752 (+   341us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.849872 (+   120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:25.853215 (+  3343us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18004c8340. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:26.110894 (+257679us) raft_consensus.cc:1671] finished
0809 23:19:26.110948 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:26.111147 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2433,"prepare.queue_time_us":11816,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":40980,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":6638,"spinlock_wait_cycles":9385984,"thread_start_us":143895,"threads_started":2,"wal-append.queue_time_us":291649}
W20260809 23:19:26.246840 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 299189us
I20260809 23:19:26.247208 24311 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:26.249332 24311 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:26.254993 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 337430us
W20260809 23:19:26.254441 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 287442us
I20260809 23:19:26.246548 29481 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.193537 29204 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:26.462627 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 244358us
I20260809 23:19:26.463913 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:26.453478 29212 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:26.453910 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.063s	user 0.016s	sys 0.021s
W20260809 23:19:26.476614 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 2.12813e+06us
I20260809 23:19:26.182667 28889 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260809 23:19:26.460628 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:26.458782 25375 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.011s	sys 0.002s
W20260809 23:19:26.491068 25375 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.011s	sys 0.002s
I20260809 23:19:26.457005 24314 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:26.483356 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.484058 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:26.457320 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.001s	sys 0.017s
I20260809 23:19:26.491195 25365 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:26.493558 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.001s	sys 0.017s
W20260809 23:19:26.494302 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 578707us
I20260809 23:19:26.469563 24309 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:26.496253 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:26.483848 29436 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:26.499271 29436 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:26.501168 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.425s	user 0.026s	sys 0.071s
I20260809 23:19:26.501343 29436 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.502128 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.503269 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.503928 29204 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.456447 29353 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.505016 29353 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:26.463210 29481 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:26.491575 29513 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.517s	user 0.006s	sys 0.011s
I20260809 23:19:26.505712 29481 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.506072 29513 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.506690 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.509321 29513 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.505388 29353 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:26.512900 29353 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.513346 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.514122 29353 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:26.517441 29333 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.518302 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.519076 29518 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.484669 29218 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:26.520951 29218 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:26.521327 29218 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:26.475950 28841 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.008s
I20260809 23:19:26.498862 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:26.527897 28841 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.528344 25363 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:26.530066 28841 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.530421 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.531241 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 735) took 1004 ms. Trace:
I20260809 23:19:26.531539 24545 rpcz_store.cc:276] 0809 23:19:25.526832 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.529814 (+1002982us) service_pool.cc:227] Handling call
0809 23:19:26.531184 (+  1370us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:26.533540 29397 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.608s	user 0.008s	sys 0.009s
I20260809 23:19:26.503950 24544 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:26.538642 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:26.543748 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.547173 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.547439 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.558547 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.539325 29579 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.536597 29479 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:26.562268 29479 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.562871 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.563863 24542 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:26.564695 29479 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:26.565586 29579 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:26.566418 29579 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.539592 29531 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Stepping down as leader of term 2
W20260809 23:19:26.571338 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.765s	user 0.015s	sys 0.025s
W20260809 23:19:26.543814 29552 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260809 23:19:26.571297 29531 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:26.533991 29397 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 4. Current term is 5.
I20260809 23:19:26.544461 29299 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.580506 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.573215 29516 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:26.582999 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 503426us
I20260809 23:19:26.581604 25375 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260809 23:19:26.585668 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.510s	user 0.027s	sys 0.072s
I20260809 23:19:26.586220 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.592134 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:26.594092 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.595955 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.596688 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:26.599387 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.600713 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:26.594898 25375 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:26.603834 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.605553 24534 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:26.605472 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:26.606411 25365 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:26.612998 29397 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:26.613716 29397 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.614398 29599 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:26.589879 29516 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.625259 29516 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.627032 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:26.627460 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.821s	user 0.016s	sys 0.025s
I20260809 23:19:26.628482 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.631693 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.633683 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.634758 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.637845 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.638665 24430 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:26.631040 29599 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:26.643369 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.644186 24545 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:26.645291 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1033) took 2933 ms (client timeout 2101 ms). Trace:
W20260809 23:19:26.645833 25366 rpcz_store.cc:269] 0809 23:19:23.712149 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.719719 (+  7570us) service_pool.cc:227] Handling call
0809 23:19:26.645052 (+2925333us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:26.645198 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3561343232}
I20260809 23:19:26.650842 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:26.651302 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.652784 24545 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:26.657050 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.657636 24546 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:26.659891 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 196046us
I20260809 23:19:26.659694 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.664398 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.685179 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.687688 24546 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:26.689342 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.691118 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 733) took 1165 ms. Trace:
I20260809 23:19:26.694723 24545 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:26.693100 24546 rpcz_store.cc:276] 0809 23:19:25.526156 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.652258 (+1126102us) service_pool.cc:227] Handling call
0809 23:19:26.691044 (+ 38786us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:26.694820 24534 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:26.698254 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.700770 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.699179 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:26.703750 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69330us
W20260809 23:19:26.705602 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44856.5us
W20260809 23:19:26.706408 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.022s	sys 0.055s
W20260809 23:19:26.706784 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.022s	sys 0.056s
I20260809 23:19:26.714027 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2482) took 3225 ms. Trace:
I20260809 23:19:26.714540 24323 rpcz_store.cc:276] 0809 23:19:23.488584 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.330941 (+842357us) service_pool.cc:227] Handling call
0809 23:19:24.379301 (+ 48360us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:24.733173 (+353872us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:24.734166 (+   993us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:24.750133 (+ 15967us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.055258 (+305125us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:26.586079 (+1530821us) log.cc:842] Serialized 14 byte log entry
0809 23:19:26.713606 (+127527us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:26.713714 (+   108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:26.713778 (+    64us) raft_consensus.cc:1671] finished
0809 23:19:26.713795 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:26.713952 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25844,"prepare.queue_time_us":456693,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":491519,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":21655,"thread_start_us":2192150,"threads_started":5,"wal-append.queue_time_us":337074,"wal-append.run_cpu_time_us":2621,"wal-append.run_wall_time_us":1002646}
W20260809 23:19:26.719841 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69127.3us
I20260809 23:19:26.704989 29607 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:26.704731 29602 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:26.711050 29615 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.727267 29607 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:26.750612 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46019.7us
I20260809 23:19:26.762312 24157 maintenance_manager.cc:419] P e3357d0fe4ef46db9a4739044fa527d5: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033345
I20260809 23:19:26.733932 29615 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.784010 29599 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:26.784370 29620 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:26.791186 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84703.4us
I20260809 23:19:26.763366 29607 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:26.766498 29621 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.794206 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:26.805364 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:19:26.806565 29598 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.303s	user 0.002s	sys 0.003s
W20260809 23:19:26.814467 24270 connection.cc:232] RPC call timeout handler was delayed by -1.2039312680717558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
I20260809 23:19:26.814019 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:26.816187 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:26.790341 29615 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:26.819646 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55562.4us
I20260809 23:19:26.817109 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:26.820307 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2481) took 3332 ms. Trace:
I20260809 23:19:26.820726 24320 rpcz_store.cc:276] 0809 23:19:23.488187 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:23.758331 (+270144us) service_pool.cc:227] Handling call
0809 23:19:23.779232 (+ 20901us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:23.779604 (+   372us) raft_consensus.cc:1520] Early marking committed up to index 2
0809 23:19:23.803721 (+ 24117us) log.cc:842] Serialized 14 byte log entry
0809 23:19:26.113902 (+2310181us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b1800467fe0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:26.705334 (+591432us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:26.706831 (+  1497us) log.cc:842] Serialized 26 byte log entry
0809 23:19:26.707043 (+   212us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:26.727087 (+ 20044us) log.cc:842] Serialized 14 byte log entry
0809 23:19:26.811738 (+ 84651us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b1800467fe0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:19:26.819969 (+  8231us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:26.820056 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:26.820110 (+    54us) raft_consensus.cc:1671] finished
0809 23:19:26.820126 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:26.820247 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12206,"prepare.queue_time_us":105221,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":10349,"raft.queue_time_us":694483,"raft.run_cpu_time_us":453,"raft.run_wall_time_us":22945,"spinlock_wait_cycles":11507328,"thread_start_us":618772,"threads_started":4,"wal-append.queue_time_us":2781166}
I20260809 23:19:26.823487 29625 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:26.823827 29409 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:26.824126 29625 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.824625 29409 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.826462 29625 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.813418 29621 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.833477 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:26.838027 29615 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.853643 29615 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:26.854182 29615 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.834450 29621 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.859799 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 727) took 2034 ms. Trace:
W20260809 23:19:26.860209 29394 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.853s	user 0.015s	sys 0.004s
I20260809 23:19:26.818186 29616 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:26.871632 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48484.3us
I20260809 23:19:26.869963 24539 rpcz_store.cc:276] 0809 23:19:24.825595 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:24.825733 (+   138us) service_pool.cc:227] Handling call
0809 23:19:24.826102 (+   369us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:24.826329 (+   227us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:24.826353 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:24.826790 (+   437us) log.cc:842] Serialized 26 byte log entry
0809 23:19:25.053208 (+226418us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:25.053288 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:25.053393 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:26.859539 (+1806146us) raft_consensus.cc:1671] finished
0809 23:19:26.859565 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:26.859728 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":225915,"threads_started":1,"wal-append.queue_time_us":428691,"wal-append.run_cpu_time_us":12102,"wal-append.run_wall_time_us":1190464}
I20260809 23:19:26.875063 29394 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:26.886855 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 166141us
I20260809 23:19:26.871342 29627 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.875645 29634 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:26.876936 29616 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:26.883401 29625 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:26.886863 29602 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.886879 29409 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.904415 29394 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.915457 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:26.913497 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 93127.2us
I20260809 23:19:26.914425 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:26.912938 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.917246 24314 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 4: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:26.918624 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.901381 29627 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:26.917603 24309 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:26.911252 29616 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:26.921123 29579 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.923964 29579 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.917224 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:26.901767 29636 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:26.926280 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.947697 24321 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:26.928947 29409 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:26.933889 29616 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:26.935029 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.607s	user 0.012s	sys 0.036s
W20260809 23:19:26.950361 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.622s	user 0.013s	sys 0.036s
I20260809 23:19:26.950816 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:26.951620 29639 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.952030 29639 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:26.952412 29639 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.952471 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:26.953370 29634 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:26.957233 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.957492 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.957881 24435 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.959151 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.960137 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:26.960923 24424 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:26.961673 29639 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.962343 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:26.963930 29639 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:26.964906 24435 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:26.965953 24496 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.825s (SENT) [suppressed 9 similar messages]
I20260809 23:19:26.937621 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:26.967574 29448 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:26.967968 29448 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:26.968312 29448 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:26.972011 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:26.915567 29625 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.972169 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:26.925323 29627 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.980157 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:26.981981 24496 connection.cc:232] RPC call timeout handler was delayed by -1.2558342660777271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
I20260809 23:19:26.986394 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.984458 29638 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:26.989979 29638 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:26.990058 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:26.990372 29638 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:26.978562 29635 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:26.949625 29636 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.004988 29636 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:26.915493 29602 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:26.990224 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:26.998536 29638 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.001231 29607 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.997833 29616 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.033192 29615 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.036304 24323 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260809 23:19:27.037176 29634 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 23:19:27.033967 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62133.9us
I20260809 23:19:26.984474 29409 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:27.057574 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9682544951792806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
I20260809 23:19:27.034547 29448 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:27.062881 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65304.3us
W20260809 23:19:27.063120 24498 connection.cc:232] RPC call timeout handler was delayed by -1.4462544950656593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
I20260809 23:19:27.065220 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.065846 24541 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.058645 29610 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.068459 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:27.072934 29615 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.073490 29627 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.074311 29607 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.075297 29439 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:27.077072 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.077270 29439 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:27.077657 29452 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:27.078435 29452 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:27.050912 29409 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.079355 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.071735 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.080125 29409 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.080503 24431 raft_consensus.cc:2362] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8. Re-sending same reply.
I20260809 23:19:27.075626 29627 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.086578 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.087415 24544 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:27.086694 29602 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.089355 29452 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.092908 29448 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.112679 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.113243 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.113935 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.114598 24431 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:27.119189 29625 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.071470 29610 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.120106 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.120922 24539 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.085011 29601 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.122159 29601 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.124399 29601 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.124949 29638 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.126065 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.130756 29120 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:27.133307 29352 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.914s	user 0.015s	sys 0.000s
I20260809 23:19:27.133709 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.138063 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.133793 29352 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.141135 24430 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.143231 29352 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.146046 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.136147 29579 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:27.159596 29579 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.161724 29579 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:27.164505 29481 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.658s	user 0.012s	sys 0.001s
I20260809 23:19:27.165105 29481 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:26.628959 29531 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.168895 24314 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:27.192044 24539 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:27.195216 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.195953 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:27.200331 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.200981 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:27.201699 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:27.202373 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.203032 24320 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:27.203294 29616 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:27.204203 29616 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:27.204842 29172 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:27.207762 29636 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:27.210088 29601 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:27.212100 24313 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.214553 25369 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:27.200549 29627 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.216004 29627 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:27.217368 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 874) took 2198 ms (client timeout 2032 ms). Trace:
W20260809 23:19:27.217767 25369 rpcz_store.cc:269] 0809 23:19:25.018690 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.053788 (+ 35098us) service_pool.cc:227] Handling call
0809 23:19:27.217181 (+2163393us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:27.217297 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4929300736}
I20260809 23:19:27.219978 29627 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.223460 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:26.581573 29299 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.224265 24309 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:27.224323 29299 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.225461 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.226210 24317 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:27.227798 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.231772 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.232275 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.232973 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:27.205812 29616 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.235292 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.236246 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:27.237169 29397 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:27.238783 29353 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.724s	user 0.009s	sys 0.005s
I20260809 23:19:27.240948 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.241856 24313 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:27.243214 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.244204 24321 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:27.246536 24427 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:27.134433 29610 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.175055 29579 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:27.249193 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.250474 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.263494 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.254441 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.264635 24312 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:27.254865 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.265584 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:27.255383 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 130140us
I20260809 23:19:27.263476 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.265569 24541 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:27.270747 24313 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:27.270563 24320 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:27.276219 29622 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:27.277233 29622 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:27.277568 29622 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:27.278328 29622 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.281085 29622 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.284961 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.285869 25366 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:27.278246 29299 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.299502 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.299975 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.300649 25366 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:27.305362 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.311394 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.313489 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.314297 29601 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.314759 29601 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.315194 29601 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.286819 29627 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.315945 29627 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.326507 29299 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:27.328016 28517 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:27.328552 28517 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.328969 28517 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:27.318647 29627 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.332523 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.333629 25368 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
W20260809 23:19:27.335693 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 265038us
I20260809 23:19:27.341261 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.342832 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.343721 25369 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:27.342451 25366 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:27.336220 29481 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.352111 29204 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.360662 24535 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260809 23:19:27.363580 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96650us
I20260809 23:19:27.366148 28517 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.344574 29602 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.369946 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.370823 24541 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:27.341444 28841 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260809 23:19:27.367388 29204 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.368002 29618 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:27.372120 29618 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.372661 28841 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:27.373063 28841 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:27.373942 28841 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.376708 29302 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.371686 29204 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.377264 29302 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.383355 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.383504 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.384588 25373 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:27.384830 29618 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:27.386549 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.387315 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.391330 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.398279 29302 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.399449 28841 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.403904 24539 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:27.408849 29621 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.409581 29621 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.413878 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.414963 25373 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:27.417155 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.417460 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.417927 24534 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:27.420483 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77753.3us
I20260809 23:19:27.421510 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.428071 24316 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:27.423301 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.429340 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:27.432981 25373 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.412084 28517 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.434356 28517 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:27.434813 28517 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.436753 28517 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.437198 28517 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:27.443616 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.451071 29204 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.453933 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2408) took 1476 ms. Trace:
I20260809 23:19:27.455842 24314 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:27.411931 29481 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.454416 24535 rpcz_store.cc:276] 0809 23:19:25.977903 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.985010 (+  7107us) service_pool.cc:227] Handling call
0809 23:19:25.985300 (+   290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:27.367726 (+1382426us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:27.453571 (+ 85845us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:27.453844 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3162173312}
I20260809 23:19:27.458314 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.458331 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.461544 24541 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:27.463711 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:27.464298 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.465198 25373 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 23:19:27.466033 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.383131 29601 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.467810 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.473130 24316 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:27.473526 29662 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:27.474395 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.465991 24314 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:27.476974 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.480685 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.482735 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.483484 24323 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:27.486094 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.486452 25368 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:27.486565 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.487311 24431 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260809 23:19:27.487536 24312 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260809 23:19:27.487692 24431 pending_rounds.cc:76] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 5
I20260809 23:19:27.488464 24431 pending_rounds.cc:97] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260809 23:19:27.486140 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.489405 24424 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:27.488742 24431 raft_consensus.cc:2866] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:27.496001 29601 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.500587 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.507925 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.508842 25373 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:27.509967 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.510812 25365 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:27.511953 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.512847 25370 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:27.518669 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.519423 25370 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:27.520521 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.525894 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.526670 25365 raft_consensus.cc:2362] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8. Re-sending same reply.
I20260809 23:19:27.527838 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.528693 25373 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:27.529786 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:27.530567 25370 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:27.533484 25325 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.539449 29204 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.540459 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.542995 29302 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:27.543607 29302 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:27.544764 29302 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.552397 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:27.553478 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.459833 29621 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:27.554780 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 131157us
I20260809 23:19:27.556152 29369 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.556999 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.558238 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.559016 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.559834 24314 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:27.471316 29616 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:27.559087 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.566501 25370 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:27.569329 28517 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:27.569942 28517 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.570437 28517 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:27.573649 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:27.574628 29621 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.575870 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:27.575994 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.576982 29621 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:27.577477 29621 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.577942 29621 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:27.587929 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.588122 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.589135 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.592252 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:27.593262 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.589246 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.596067 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.501941 29662 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:27.467546 29481 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.624892 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.597436 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.627976 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.565694 29616 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.590348 29482 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:27.592586 29627 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:27.627302 24323 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:27.632033 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.633215 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.634052 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.630503 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.595023 29299 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:27.587203 29661 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:27.624887 29625 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:27.592732 29425 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.625219 29615 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.619328 29579 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.626180 28517 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:27.642251 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:27.645748 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80423.2us
I20260809 23:19:27.627813 29638 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.626261 29397 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.643858 29627 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.653992 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.679657 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:27.656077 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:27.680678 24424 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.647756 29436 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.636473 29616 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:27.628409 29662 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.636049 29482 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:27.664559 24422 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:27.668473 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.671841 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.664119 29425 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:27.673943 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800319860 after lost signal to thread 22536
I20260809 23:19:27.674979 28517 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.676319 29125 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:27.649621 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.696127 24431 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 23:19:27.682240 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:27.647416 29625 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.645884 29661 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.684834 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.665264 24429 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:27.677832 29397 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:27.683390 29627 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:27.697547 24535 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.657301 29299 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:27.671643 29638 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.702386 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.704198 24424 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:27.650770 29509 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260809 23:19:27.696070 29482 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:27.662886 29458 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260809 23:19:27.658766 29615 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.736274 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:27.729413 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140369us
I20260809 23:19:27.723958 29436 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.738761 28984 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.723294 29661 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:27.726555 29397 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.773898 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.790043 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:27.737773 29638 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.777974 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 643) took 1143 ms. Trace:
I20260809 23:19:27.793195 24429 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.793097 24423 rpcz_store.cc:276] 0809 23:19:26.634367 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.634495 (+   128us) service_pool.cc:227] Handling call
0809 23:19:27.777714 (+1143219us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:27.777865 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348844800}
I20260809 23:19:27.778201 29425 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:27.762970 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.776150 29306 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
W20260809 23:19:27.782150 29663 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260809 23:19:27.781415 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:27.818207 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 136049us
I20260809 23:19:27.773209 29146 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:27.785111 24424 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:27.736207 29675 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:27.787510 24430 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:27.761245 29615 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.795450 25370 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:27.720464 29625 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:27.799393 29353 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:27.825003 29452 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:27.825488 29353 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.802738 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 637) took 1917 ms. Trace:
I20260809 23:19:27.803334 24435 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:27.787081 28984 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.805797 29601 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:27.806499 29635 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:27.806212 29638 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:27.808923 29664 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260809 23:19:27.811264 29306 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:27.741206 29508 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20260809 23:19:27.818781 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:27.819897 29146 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:27.821345 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:27.826519 24428 rpcz_store.cc:276] 0809 23:19:25.885010 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:25.895859 (+ 10849us) service_pool.cc:227] Handling call
0809 23:19:27.802530 (+1906671us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:27.802659 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":712268032}
I20260809 23:19:27.816679 29450 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:27.828343 29679 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:27.823732 29675 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.832010 28984 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:27.848270 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 238) took 17769 ms (client timeout 15000 ms). Trace:
I20260809 23:19:27.848425 29125 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:27.849279 24424 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:27.848562 29306 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.832201 29601 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:27.865659 29672 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260809 23:19:27.866334 29353 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.791080 29299 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.864444 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:27.867491 29458 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:27.837088 29481 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.846205 29450 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.837045 29635 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:27.877285 29450 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:27.860672 29675 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:27.864521 29601 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:27.888137 24544 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260809 23:19:27.861670 24164 rpcz_store.cc:269] 0809 23:19:10.117427 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:10.118762 (+  1335us) service_pool.cc:227] Handling call
0809 23:19:27.836119 (+17717357us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:19:20.689554 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:19:20.707970 (+ 18416us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:19:20.708028 (+    58us) write_op.cc:435] Acquired schema lock
0809 23:19:20.708057 (+    29us) tablet.cc:660] Decoding operations
0809 23:19:20.710851 (+  2794us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:19:20.710976 (+   125us) write_op.cc:641] Partition lock acquired for write op
0809 23:19:20.711006 (+    30us) tablet.cc:683] Acquiring locks for 13 operations
0809 23:19:20.929194 (+218188us) tablet.cc:699] Row locks acquired
0809 23:19:20.929234 (+    40us) write_op.cc:260] PREPARE: finished
0809 23:19:20.932348 (+  3114us) write_op.cc:270] Start()
0809 23:19:20.937612 (+  5264us) write_op.cc:276] Timestamp: P: 1786317560930901 usec, L: 0
0809 23:19:20.937669 (+    57us) op_driver.cc:331] REPLICATION: starting
0809 23:19:20.938586 (+   917us) log.cc:842] Serialized 5094 byte log entry
0809 23:19:21.084692 (+146106us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac2091a5fe 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb
0809 23:19:21.858611 (+773919us) op_driver.cc:444] REPLICATION: finished
0809 23:19:22.034112 (+175501us) write_op.cc:301] APPLY: starting
0809 23:19:22.034256 (+   144us) tablet.cc:1365] starting BulkCheckPresence
0809 23:19:22.034579 (+   323us) tablet.cc:1368] finished BulkCheckPresence
0809 23:19:22.034605 (+    26us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:19:22.037601 (+  2996us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:19:22.038009 (+   408us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0809 23:19:22.038064 (+    55us) write_op.cc:312] APPLY: finished
0809 23:19:22.038459 (+   395us) log.cc:842] Serialized 95 byte log entry
0809 23:19:22.078764 (+ 40305us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:22.094908 (+ 16144us) write_op.cc:454] Released schema lock
0809 23:19:22.106077 (+ 11169us) write_op.cc:341] FINISH: Updating metrics
0809 23:19:22.189630 (+ 83553us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:22.192393 (+  2763us) write_op.cc:454] Released schema lock
Metrics: {"dns_us":203,"mutex_wait_us":1072623,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":175138,"apply.run_cpu_time_us":6670,"apply.run_wall_time_us":183228,"num_ops":13,"prepare.queue_time_us":228599,"prepare.run_cpu_time_us":64029,"prepare.run_wall_time_us":601525,"raft.queue_time_us":174332,"raft.run_cpu_time_us":17638,"raft.run_wall_time_us":237160,"replication_time_us":902126,"spinlock_wait_cycles":10575104,"thread_start_us":574756,"threads_started":4,"wal-append.queue_time_us":345113,"wal-append.run_cpu_time_us":20372,"wal-append.run_wall_time_us":1798657}]]}
I20260809 23:19:27.851024 29509 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260809 23:19:27.875761 29481 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.895531 29481 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:27.895934 29481 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:27.896351 29696 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:27.899513 29662 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:27.911342 29635 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.870025 29618 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260809 23:19:27.873925 29146 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:27.879360 29621 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:27.943228 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 181406us
I20260809 23:19:27.946343 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.950179 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.951574 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.951840 29682 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:27.953156 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:27.954512 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:27.955260 29697 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:27.957072 24312 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:27.958613 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.933897 29125 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260809 23:19:27.953164 29690 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:27.952610 29666 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:27.947791 29458 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:27.957674 29682 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:27.970435 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:27.971454 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66739.4us
I20260809 23:19:27.971171 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.972154 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:27.974921 25375 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:27.971457 29666 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:27.971136 29690 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:27.978425 29703 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:27.970014 29638 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:27.974515 29353 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:27.993481 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:28.003468 24321 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:27.976619 29458 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.009160 29682 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.009624 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.991576 29690 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:27.988396 29666 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.011272 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.011447 29621 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.028964 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:28.011610 29699 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260809 23:19:28.012495 24314 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260809 23:19:28.013732 25363 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.485s	user 0.016s	sys 0.000s
W20260809 23:19:28.018646 29218 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.497s	user 0.009s	sys 0.004s
I20260809 23:19:28.019917 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:28.035673 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:27.992177 29306 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:19:28.025303 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 850436us
I20260809 23:19:28.038883 29481 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.039397 29682 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.041854 29481 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.026857 29125 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:28.029616 24321 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:19:28.028035 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 209282us
I20260809 23:19:28.051203 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.059007 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.001716 29353 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.042124 28984 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.067070 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.067766 29397 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.064258 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.070181 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.072146 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.084733 24317 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8.
I20260809 23:19:28.081804 24541 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:28.081244 29712 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260809 23:19:28.084637 25373 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.095403 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.097548 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.100654 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.103749 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.104873 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.105250 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.033746 25363 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:28.037700 29306 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:28.066749 29353 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.173344 25368 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260809 23:19:28.131171 29720 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:28.182174 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.066751 28984 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.082059 29172 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.106891 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:27.995151 29703 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.047089 29218 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.123530 29696 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:28.121935 29638 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.121698 29677 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.212905 25373 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:28.193683 25375 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:28.124388 29508 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.124261 29693 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.174080 29579 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
W20260809 23:19:28.230947 29532 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.726s	user 0.008s	sys 0.006s
I20260809 23:19:28.175452 29302 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:28.069540 29125 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.104503 29621 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
W20260809 23:19:28.190275 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 940) took 1719 ms (client timeout 1914 ms). Trace:
I20260809 23:19:28.220101 29218 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.221282 24311 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:28.221194 29450 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:28.227926 29714 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260809 23:19:28.230264 29702 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260809 23:19:28.192358 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.187204 29720 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:28.233168 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 239745us
I20260809 23:19:28.235620 24423 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:28.274840 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.275488 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.276305 25372 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
W20260809 23:19:28.199047 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92233.5us
I20260809 23:19:28.203944 29698 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.281843 24431 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:28.198637 29710 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:28.355444 29710 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.355916 29666 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.357175 24424 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:28.358099 29666 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.358867 29725 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:28.359493 24423 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:28.359951 29458 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.360757 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.362460 29621 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:28.363603 29621 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.364822 28984 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:28.368717 29125 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.372573 29397 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:28.348850 29722 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:19:28.380800 29601 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:28.382437 29615 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.001s
W20260809 23:19:28.351543 29625 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.001s
I20260809 23:19:28.396203 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.397354 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.304593 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:28.352823 29627 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.616s	user 0.011s	sys 0.000s
I20260809 23:19:28.192824 29616 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.353302 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.175807 29688 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.155s
I20260809 23:19:28.365209 29621 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:28.350024 29409 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.410135 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.265991 29448 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:28.401193 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:28.401378 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.412660 24544 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:28.402019 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800302780 after lost signal to thread 24220
I20260809 23:19:28.401974 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:28.405215 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.414116 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:28.414451 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080031aa80 after lost signal to thread 24355
W20260809 23:19:28.280252 29532 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.773s	user 0.009s	sys 0.006s
I20260809 23:19:28.417744 24535 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:28.418277 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.411032 24423 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.419269 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:28.423121 29436 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:28.423204 29693 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.423679 29436 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:28.424026 29693 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:28.425163 29693 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.398170 29720 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.427582 29709 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:28.427987 29709 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:28.282243 29579 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:28.351894 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77640.8us
W20260809 23:19:28.378820 29482 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.005s
I20260809 23:19:28.282146 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.292073 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:28.414791 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08001710e0 after lost signal to thread 24356
I20260809 23:19:28.411074 25373 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:28.357205 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.408982 29677 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:28.412744 28971 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:28.409801 29621 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:28.436949 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080013f860 after lost signal to thread 24469
I20260809 23:19:28.475651 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.476573 29621 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.477078 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.436115 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.383169 29625 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.475050 29740 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:28.418972 29448 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.441847 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:28.476189 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080003e900 after lost signal to thread 24470
I20260809 23:19:28.477802 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.382838 29615 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.418852 29218 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.465493 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.418424 29409 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:28.429114 29698 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.479490 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:28.430384 29661 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.654s	user 0.012s	sys 0.000s
I20260809 23:19:28.509970 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.415026 29725 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260809 23:19:28.409437 29627 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.511442 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.511973 24314 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:28.500428 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:28.401934 25363 rpcz_store.cc:269] 0809 23:19:26.460170 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.463043 (+  2873us) service_pool.cc:227] Handling call
0809 23:19:28.175483 (+1712440us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:28.508966 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.436014 29696 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.517408 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:28.273556 29306 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.525584 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.525975 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.526585 24428 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:28.475689 29703 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.521126 24423 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:28.532373 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.512836 25373 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:28.523366 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.533759 24425 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:28.534224 24316 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:28.535108 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.535831 25369 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:28.481315 29638 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.523118 29745 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.532109 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.536213 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.428851 29709 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.504089 28971 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.543081 24428 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.519591 29218 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.520756 29621 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:28.517829 29677 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.537981 29718 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:28.524750 29601 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.551963 24425 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:28.562906 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.559613 29709 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:28.548167 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.566851 25370 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.495682 29482 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:28.557770 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 155862us
I20260809 23:19:28.539884 25373 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:28.581171 24428 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:28.521032 29661 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.550227 29696 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.594726 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:28.595710 25375 raft_consensus.cc:2354] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:28.595783 25371 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:28.597635 29638 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.603247 25363 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:28.579068 29740 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.598166 29601 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:28.597075 29718 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:28.623700 25370 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:28.611655 29638 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:28.597659 29450 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.548429 29409 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.582751 29677 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20260809 23:19:28.647491 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.619267 29601 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.626699 29661 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:28.628794 29739 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260809 23:19:28.615393 24422 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:28.635573 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.645928 29718 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.652287 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.653374 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.653638 29718 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.613420 28971 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.654748 29532 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.655565 29532 raft_consensus.cc:2741] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 23:19:28.656219 29532 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:28.656759 29482 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.662117 29467 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:28.550185 29679 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260809 23:19:28.620136 29574 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.101s	user 0.009s	sys 0.000s
I20260809 23:19:28.547603 29745 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.614591 29710 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.650506 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.665747 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.665971 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:28.669844 25323 connection.cc:232] RPC call timeout handler was delayed by -1.8320934900548309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260809 23:19:28.616967 29616 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.580674 29579 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:28.695751 25323 connection.cc:232] RPC call timeout handler was delayed by -1.8053676059935242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
I20260809 23:19:28.697117 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:28.700485 25323 connection.cc:232] RPC call timeout handler was delayed by -1.696531212888658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20260809 23:19:28.700826 25323 connection.cc:232] RPC call timeout handler was delayed by -1.5522877781186253s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260809 23:19:28.701121 25323 connection.cc:232] RPC call timeout handler was delayed by -1.4838877781294286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260809 23:19:28.701403 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.704464 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.705734 29616 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.706037 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:28.706288 29758 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.707446 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:28.701400 25323 connection.cc:232] RPC call timeout handler was delayed by -1.4094312130473554s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260809 23:19:28.707791 29758 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:19:28.708819 25323 connection.cc:232] RPC call timeout handler was delayed by -1.3176312129944563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
W20260809 23:19:28.709172 25323 connection.cc:232] RPC call timeout handler was delayed by -1.2654312129598111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260809 23:19:28.709537 25323 connection.cc:232] RPC call timeout handler was delayed by -1.1955604830291122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
I20260809 23:19:28.714603 29745 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:28.716240 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65818.3us
I20260809 23:19:28.716758 24544 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:28.717644 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.720762 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.718175 25363 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:28.723098 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.723932 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.723805 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.724375 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.726840 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.727624 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.690649 29696 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.694526 29218 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.730170 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.730608 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.683362 29615 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.683621 29627 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.717963 29352 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:28.729229 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.733525 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.741072 24314 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.733875 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.735745 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.735953 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.717097 29619 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:28.676515 29625 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.739671 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.732713 29299 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:28.733745 29532 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.663236 29467 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.733121 29666 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:28.738401 29765 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:28.758424 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.759145 24321 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:28.744117 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.732903 29448 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:28.732657 29661 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260809 23:19:28.744086 29352 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.752091 24317 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 9
I20260809 23:19:28.730885 29760 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.754805 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.755842 28711 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:28.760005 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.785425 24321 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.765552 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.757738 29397 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.743731 29751 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:28.684283 29638 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.777370 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2109) took 1206 ms. Trace:
I20260809 23:19:28.780503 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.753715 29299 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.753391 29627 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:28.810886 24312 rpcz_store.cc:276] 0809 23:19:27.571433 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.580339 (+  8906us) service_pool.cc:227] Handling call
0809 23:19:27.580638 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:27.958451 (+377813us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:28.777146 (+818695us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:28.777311 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":865266944}
W20260809 23:19:28.772779 29675 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.864s	user 0.007s	sys 0.005s
I20260809 23:19:28.764139 29765 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:28.821906 24314 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:28.756309 29619 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.823153 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.734099 29601 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.831627 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.781026 29532 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:28.764024 29482 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.832847 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:28.834817 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.679009 29740 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.812261 29745 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.810077 29508 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.819437 29299 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:28.836347 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet e4ed6a400f2643da95b8f744ebab6f4c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260809 23:19:28.813442 29760 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:28.830421 29516 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:28.832413 29765 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:28.844447 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:28.836709 29513 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:28.843060 29579 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.837842 29532 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:28.851287 29516 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:28.857813 29754 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:28.879199 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87412.9us
I20260809 23:19:28.843532 29758 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:28.836055 28971 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260809 23:19:28.855405 29508 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260809 23:19:28.880707 29508 consensus_queue.cc:1225] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:28.849916 29397 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.851533 29482 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.016s
I20260809 23:19:28.883251 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.869458 29760 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.884788 24321 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:28.833112 24317 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
W20260809 23:19:28.858968 29756 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260809 23:19:28.829242 29627 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.880535 29758 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.881999 24378 leader_election.cc:398] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8.
I20260809 23:19:28.829504 29675 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.897964 24423 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:19:28.835623 29619 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.879995 29532 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:28.899665 24321 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:28.879298 29516 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:28.901602 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:28.905236 29627 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.901996 29397 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:28.907286 29397 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:28.908231 29758 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.908183 29397 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.881803 29751 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:28.906479 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.918488 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:28.902572 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.879479 29754 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.916791 24430 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:28.910753 28711 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:28.931156 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.906426 29770 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.920810 29661 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:28.932852 29770 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:28.933626 29579 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:28.934185 29579 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.934821 29579 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:28.931169 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:28.941546 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1045) took 1435 ms. Trace:
I20260809 23:19:28.943253 29627 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.945134 29627 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:28.924219 29754 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:28.927223 29771 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:28.920814 29745 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:28.945861 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1244) took 1678 ms. Trace:
I20260809 23:19:28.951543 24313 rpcz_store.cc:276] 0809 23:19:27.267202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.284350 (+ 17148us) service_pool.cc:227] Handling call
0809 23:19:28.943387 (+1659037us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f2726 00007fac287f2900 00007fac287a5b50 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1
0809 23:19:28.945693 (+  2306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8072576}
I20260809 23:19:28.921512 29299 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:28.961483 29690 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.930s	user 0.011s	sys 0.000s
I20260809 23:19:28.964038 29352 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:28.967859 29627 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:28.968083 29720 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.540s	user 0.011s	sys 0.000s
I20260809 23:19:28.968017 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.968541 29677 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.968888 25372 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:28.968711 29720 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.969367 29677 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.970271 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.968920 29627 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:28.971706 29627 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.972440 29675 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:28.973431 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2422) took 1973 ms. Trace:
I20260809 23:19:28.973836 24542 rpcz_store.cc:276] 0809 23:19:26.999860 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.000046 (+   186us) service_pool.cc:227] Handling call
0809 23:19:28.972718 (+1972672us) spinlock_profiling.cc:243] Waited 7.69 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:28.972902 (+   184us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.973097 (+   195us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:28.973122 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:28.973156 (+    34us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:28.973186 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:28.973238 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:28.973369 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17687424}
I20260809 23:19:28.941948 25368 rpcz_store.cc:276] 0809 23:19:27.506093 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.513627 (+  7534us) service_pool.cc:227] Handling call
0809 23:19:28.173076 (+659449us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:28.173750 (+   674us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:28.941367 (+767617us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b540076c5d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:28.941491 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1515549440}
I20260809 23:19:28.975888 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2543) took 2001 ms. Trace:
I20260809 23:19:28.976284 24435 rpcz_store.cc:276] 0809 23:19:26.975061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.975344 (+   283us) service_pool.cc:227] Handling call
0809 23:19:27.803140 (+827796us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:27.827325 (+ 24185us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:28.975698 (+1148373us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54002f8ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:28.975828 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1903080192}
I20260809 23:19:28.924736 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.973205 29675 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:28.941927 29758 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:28.979640 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:28.979059 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54403.6us
I20260809 23:19:28.982537 24313 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:28.983554 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:28.977768 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:28.984719 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:28.985123 24435 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:28.991950 28711 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:28.992209 29771 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:28.992774 28711 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:28.995318 29621 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:28.926450 29751 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.995769 29621 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:29.012451 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:29.015657 29621 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.018159 29621 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:29.021126 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 870) took 2106 ms (client timeout 1723 ms). Trace:
I20260809 23:19:29.021277 24309 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
W20260809 23:19:29.021557 24311 rpcz_store.cc:269] 0809 23:19:26.914289 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.915173 (+   884us) service_pool.cc:227] Handling call
0809 23:19:29.020920 (+2105747us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:29.021058 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3000600832}
I20260809 23:19:29.023016 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2098) took 1554 ms. Trace:
I20260809 23:19:29.023429 24309 rpcz_store.cc:276] 0809 23:19:27.468769 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.469136 (+   367us) service_pool.cc:227] Handling call
0809 23:19:29.021077 (+1551941us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:29.021823 (+   746us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.022814 (+   991us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54003a3ed0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:29.022960 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3568265088}
I20260809 23:19:29.025763 29677 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.026669 29666 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:29.027281 29666 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:29.028024 29299 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:29.026188 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:29.028553 29666 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:28.999256 24428 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:28.994282 28711 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.029649 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.030853 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:29.032737 24425 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:28.982116 29675 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:29.033838 29675 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.034112 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.000141 29452 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:28.982314 29172 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:29.004130 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2536) took 2047 ms (client timeout 2246 ms). Trace:
I20260809 23:19:29.004717 24313 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
W20260809 23:19:29.057880 24427 rpcz_store.cc:269] 0809 23:19:26.956616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.958855 (+  2239us) service_pool.cc:227] Handling call
0809 23:19:29.003972 (+2045117us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:29.004076 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":643470208}
I20260809 23:19:29.058807 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:29.059628 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.060411 24425 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:29.007548 29751 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.008172 24435 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:29.062361 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:29.063977 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.064308 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.042421 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:29.060355 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.030810 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:28.998425 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:29.068043 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69722.6us
I20260809 23:19:29.045871 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.031580 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 876) took 2452 ms. Trace:
I20260809 23:19:29.069499 25371 rpcz_store.cc:276] 0809 23:19:26.578979 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:26.582669 (+  3690us) service_pool.cc:227] Handling call
0809 23:19:26.604064 (+ 21395us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.598779 (+1994715us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.031329 (+432550us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b5400717854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:29.031526 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4580014848}
I20260809 23:19:29.053009 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:28.962234 29690 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.029156 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:28.934334 29770 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.046584 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.073531 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.073945 24435 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:29.076954 29532 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.079169 29693 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.083042 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:29.083468 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.087225 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:29.096215 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:29.093942 24309 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:29.094339 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.098027 24435 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:29.103992 24425 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:29.103399 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:29.113523 24311 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: [5.5-7.6]
I20260809 23:19:29.114702 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:29.115718 24429 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:29.117933 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58968us
I20260809 23:19:29.058475 29452 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.078292 29765 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.121847 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:29.082536 29352 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:29.126716 24435 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:29.088907 29705 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:29.095194 29333 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:29.131017 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.089085 29467 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:29.132534 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.132655 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.133620 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.131275 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.134637 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.140959 24309 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:29.143322 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.108552 29679 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:29.145136 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.149724 24544 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.150807 24425 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:29.152462 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.153610 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.086648 29677 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260809 23:19:29.114835 29513 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:29.123629 29666 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:29.117638 29720 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.098137 29616 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 23:19:29.176339 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.127243 29760 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:29.169713 24534 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:29.164517 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.172556 24425 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:29.170614 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.179576 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:29.183162 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.118990 29747 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.072s
I20260809 23:19:29.152192 29352 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:29.190285 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2552) took 1519 ms. Trace:
I20260809 23:19:29.195003 24422 rpcz_store.cc:276] 0809 23:19:27.671231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.693699 (+ 22468us) service_pool.cc:227] Handling call
0809 23:19:27.694065 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.651164 (+957099us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.190088 (+538924us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:29.190238 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2118225280}
I20260809 23:19:29.185297 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.159487 29333 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:29.174381 29679 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:29.184151 29760 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:29.213243 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:29.213305 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.187753 29450 raft_consensus.cc:2741] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260809 23:19:29.211730 24544 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:29.147501 29709 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260809 23:19:29.182247 29481 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:29.158357 29742 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260809 23:19:29.148419 29622 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.050s
I20260809 23:19:29.216723 29760 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:29.209311 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:29.189553 29627 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.210584 29337 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.240324 24543 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:29.218804 29725 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:29.238718 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:29.241926 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58849.8us
I20260809 23:19:29.180100 29754 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.252398 29742 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:29.256110 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.257082 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.274746 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.274873 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.276212 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.261806 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.269791 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.144344 29705 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.278237 24546 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:29.279531 29638 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:29.280566 29705 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.207405 29771 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260809 23:19:29.281543 29638 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.282373 29638 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:29.284622 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.287194 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.287945 29638 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.289281 29677 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.290740 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.291730 29638 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.293761 29638 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:29.309197 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47502.3us
I20260809 23:19:29.328890 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:29.330754 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:29.335520 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 65770.2us
I20260809 23:19:29.337560 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.338654 29783 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:29.339658 29783 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.339850 29337 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:29.340368 29709 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.340490 29337 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.330552 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 121327us
I20260809 23:19:29.354447 24543 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260809 23:19:29.357684 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83014.4us
I20260809 23:19:29.328183 29448 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260809 23:19:29.271070 29616 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:29.197804 29452 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.381392 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.373905 29333 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.388551 29776 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.203s	user 0.000s	sys 0.002s
I20260809 23:19:29.299693 29299 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.104s
I20260809 23:19:29.361410 29628 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.358517 29532 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.362783 29690 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:29.393278 29452 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.391397 29765 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.377388 29705 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260809 23:19:29.423607 29337 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.345508 29742 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.436008 29794 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.435878 29800 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:29.437985 29800 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:19:29.441341 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 131058us
I20260809 23:19:29.411533 29709 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.414760 29751 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:29.454303 24425 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:29.415019 29532 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:29.384212 29616 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.353751 29352 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:29.466248 29777 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260809 23:19:29.341045 29783 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.469056 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.494500 24425 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:29.496456 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:29.471606 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49307.9us
I20260809 23:19:29.493500 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.499434 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:29.502863 29516 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.003s
I20260809 23:19:29.503556 29516 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.505275 25363 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:29.506847 29305 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.510126 29628 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:29.510831 29628 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:29.511338 29628 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:29.511741 29780 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260809 23:19:29.361814 29679 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.513561 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2094) took 2049 ms. Trace:
I20260809 23:19:29.514437 24425 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:29.496402 29785 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260809 23:19:29.519586 29785 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:29.520007 29785 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.520666 29305 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:29.521857 24435 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:29.523916 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2103) took 1961 ms. Trace:
I20260809 23:19:29.513960 24320 rpcz_store.cc:276] 0809 23:19:27.464280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.464402 (+   122us) service_pool.cc:227] Handling call
0809 23:19:27.464728 (+   326us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:27.465071 (+   343us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:27.465096 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:27.471981 (+  6885us) log.cc:842] Serialized 26 byte log entry
0809 23:19:29.007002 (+1535021us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b1800243160. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:29.181597 (+174595us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:29.181639 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.181716 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:29.513137 (+331421us) raft_consensus.cc:1671] finished
0809 23:19:29.513157 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:29.513485 (+   328us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18,"raft.queue_time_us":2812,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9930,"spinlock_wait_cycles":21641984,"thread_start_us":174305,"threads_started":1,"wal-append.queue_time_us":1787474}
I20260809 23:19:29.524574 24316 rpcz_store.cc:276] 0809 23:19:27.562224 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.562341 (+   117us) service_pool.cc:227] Handling call
0809 23:19:27.591801 (+ 29460us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.534757 (+942956us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.523684 (+988927us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:29.523856 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2166943488}
W20260809 23:19:29.475561 29782 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.258s	user 0.002s	sys 0.000s
I20260809 23:19:29.528007 24435 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:29.498621 24427 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:29.522404 29305 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:29.531481 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:29.535831 24425 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:29.538156 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.538968 24320 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:29.542624 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:29.531073 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 78337.4us
I20260809 23:19:29.518131 29675 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.551821 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.562587 24309 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:29.562798 24316 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260809 23:19:29.569435 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2320) took 1614 ms. Trace:
I20260809 23:19:29.459136 29765 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260809 23:19:29.467389 29705 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
I20260809 23:19:29.570029 24535 rpcz_store.cc:276] 0809 23:19:27.954982 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.955142 (+   160us) service_pool.cc:227] Handling call
0809 23:19:27.955467 (+   325us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.429591 (+474124us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.569033 (+1139442us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:29.569313 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1062671744}
I20260809 23:19:29.578614 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.565896 29337 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
W20260809 23:19:29.583564 29172 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.521s	user 0.007s	sys 0.007s
I20260809 23:19:29.584343 29172 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.592208 29616 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.594130 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:29.594694 29482 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:29.594910 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:29.595522 29482 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.592340 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 489000us
I20260809 23:19:29.576421 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.583499 29337 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:29.609038 24422 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 8. Ops: []
I20260809 23:19:29.562402 29513 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:29.573778 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:29.611786 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:29.534299 29305 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.551332 29725 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260809 23:19:29.598564 29482 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:29.543121 29638 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.075s
I20260809 23:19:29.540771 29758 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260809 23:19:29.657586 24312 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260809 23:19:29.656433 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.548960 29693 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 23:19:29.665187 24312 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 8
I20260809 23:19:29.609869 29337 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.667404 24312 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260809 23:19:29.672740 24312 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:29.607995 29810 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:29.664999 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:29.528214 29785 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.682404 24309 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:29.652711 29369 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:29.694278 24545 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:29.691753 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:29.657852 29352 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260809 23:19:29.695605 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 335748us
I20260809 23:19:29.710624 24543 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:29.711128 24543 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.680536 29810 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:29.673450 29622 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:29.712051 24543 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.729507 24543 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:29.688525 29369 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:29.732740 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:29.702828 29754 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:29.738029 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:29.820568 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 244232us
I20260809 23:19:29.823578 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:29.823962 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:29.824355 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2323) took 1781 ms. Trace:
I20260809 23:19:29.824729 24541 rpcz_store.cc:276] 0809 23:19:28.043242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.043402 (+   160us) service_pool.cc:227] Handling call
0809 23:19:28.043671 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.099539 (+ 55868us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.824074 (+1724535us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54002706d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:29.824285 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46229760}
I20260809 23:19:29.739257 29337 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.829330 29337 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:29.830013 29797 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.264s
I20260809 23:19:29.831568 29814 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:29.734736 29305 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.835774 29369 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.836282 29305 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:29.740335 29785 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:29.836864 29369 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.840224 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:29.842247 29824 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260809 23:19:29.845608 29760 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.516s	user 0.000s	sys 0.011s
I20260809 23:19:29.846765 29622 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.846457 29832 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
W20260809 23:19:29.848253 29666 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.000s
I20260809 23:19:29.848332 29628 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.850443 29579 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.915s	user 0.011s	sys 0.004s
I20260809 23:19:29.851037 29579 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.852412 29832 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:29.854009 29579 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:29.855283 25370 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:29.722254 29810 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:29.836475 29675 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.860209 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.861310 29693 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:29.817437 29803 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.130s
W20260809 23:19:29.867380 29621 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.849s	user 0.017s	sys 0.006s
I20260809 23:19:29.871003 29797 consensus_queue.cc:1225] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:29.871945 29797 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Stepping down as leader of term 7
W20260809 23:19:29.871215 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080034db00 after lost signal to thread 24579
W20260809 23:19:29.872552 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800418a00 after lost signal to thread 24580
I20260809 23:19:29.872370 29797 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:29.874276 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08003ecf00 after lost signal to thread 25410
W20260809 23:19:29.876210 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 898) took 1570 ms (client timeout 1730 ms). Trace:
W20260809 23:19:29.876624 25375 rpcz_store.cc:269] 0809 23:19:28.305798 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.465156 (+159358us) service_pool.cc:227] Handling call
0809 23:19:29.876011 (+1410855us) spinlock_profiling.cc:243] Waited 98.8 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:29.876155 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227240192}
W20260809 23:19:29.832137 29816 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
W20260809 23:19:29.885821 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08003a23c0 after lost signal to thread 28918
W20260809 23:19:29.886525 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080038f260 after lost signal to thread 29211
I20260809 23:19:29.872381 24309 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:29.888899 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:29.889902 24309 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 8 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260809 23:19:29.891652 29797 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.894054 24269 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.730s (SENT)
I20260809 23:19:29.899463 24543 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:19:29.899631 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.901448 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.901630 25375 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:29.892582 29479 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:29.905462 29616 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.905609 29479 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:29.908124 29628 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:29.846196 29760 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.910862 25371 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:29.898969 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080036d760 after lost signal to thread 29309
W20260809 23:19:29.913662 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800322ca0 after lost signal to thread 29325
I20260809 23:19:29.914230 25375 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:29.915436 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:29.915640 24312 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:29.916028 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.916306 24312 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:29.916536 24316 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:29.916594 24321 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.914391 24320 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:29.918594 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08001803c0 after lost signal to thread 29377
W20260809 23:19:29.919034 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08002ede40 after lost signal to thread 29405
W20260809 23:19:29.920336 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 187715us
I20260809 23:19:29.921036 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:29.923595 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08002e2f40 after lost signal to thread 29408
W20260809 23:19:29.924124 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08003d6ba0 after lost signal to thread 29414
W20260809 23:19:29.924962 24574 debug-util.cc:398] Leaking SignalData structure 0x7b08003546c0 after lost signal to thread 29488
I20260809 23:19:29.917112 24312 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:29.882799 29516 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:29.953426 25365 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:29.955842 29605 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:29.960192 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:29.960893 29661 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:29.961536 29661 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:29.963053 29605 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:29.964326 29532 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:29.964967 29532 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.946045 29333 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260809 23:19:29.965972 24316 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:29.966562 29333 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:29.967069 29333 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:29.964017 29605 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:29.968104 29333 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:29.960937 29673 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.118s	user 0.000s	sys 0.010s
I20260809 23:19:29.854701 29579 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260809 23:19:29.975646 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1053) took 2458 ms. Trace:
I20260809 23:19:29.943267 29467 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.033s
I20260809 23:19:29.976706 29467 consensus_queue.cc:1225] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:29.976115 25366 rpcz_store.cc:276] 0809 23:19:27.517133 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.521778 (+  4645us) service_pool.cc:227] Handling call
0809 23:19:27.522167 (+   389us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:27.522594 (+   427us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:27.522630 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:27.523150 (+   520us) log.cc:842] Serialized 26 byte log entry
0809 23:19:27.640189 (+117039us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:27.640214 (+    25us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:27.640258 (+    44us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:29.960287 (+2320029us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:29.975314 (+ 15027us) raft_consensus.cc:1671] finished
0809 23:19:29.975337 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:29.975581 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1567,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":12849920,"thread_start_us":116673,"threads_started":1,"wal-append.queue_time_us":267279}
I20260809 23:19:29.978403 29828 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:29.979168 25366 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:29.979876 25365 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:19:29.981436 25366 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:29.985409 24316 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:29.989539 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:29.990156 25370 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:29.992075 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:29.986323 24321 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:29.992905 24313 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:29.994658 25366 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260809 23:19:29.976711 29333 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:30.005923 29448 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:30.007431 29448 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:30.007829 29448 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:30.009079 29448 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.010341 29448 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:29.944852 25375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34970da619114abc942b839d0862a300"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:30.011677 25375 raft_consensus.cc:595] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:30.011831 29508 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:30.012153 25375 raft_consensus.cc:603] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:30.012329 29508 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:30.013324 29508 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.014789 25370 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:30.018850 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:29.983791 29532 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.022303 24321 pending_rounds.cc:76] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 5
W20260809 23:19:30.024435 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285535us
I20260809 23:19:30.024721 24321 pending_rounds.cc:97] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260809 23:19:29.950902 29725 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260809 23:19:30.025103 24321 raft_consensus.cc:2866] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:30.023680 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.014686 29508 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:30.034915 25370 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:29.947211 29628 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260809 23:19:30.059415 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:30.061388 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:30.061285 29212 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:30.063625 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:30.067893 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.078718 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:29.932318 29479 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.087736 29479 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:30.092016 24544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "904621b566f64064bc1a7411ab27c677"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:19:30.092275 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:30.092690 24544 raft_consensus.cc:595] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:30.093025 24544 raft_consensus.cc:603] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:30.032292 29299 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:30.098237 25365 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:30.108279 24313 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:30.094631 29803 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:30.035730 29703 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.073s
I20260809 23:19:30.130757 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:30.036502 29661 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260809 23:19:30.134500 24313 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:30.104997 28841 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:30.143692 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:30.101965 29808 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:30.121919 29299 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:30.168386 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 651) took 2060 ms. Trace:
I20260809 23:19:30.196372 24424 rpcz_store.cc:276] 0809 23:19:28.107642 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.170481 (+ 62839us) service_pool.cc:227] Handling call
0809 23:19:28.188070 (+ 17589us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:28.357582 (+169512us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.145842 (+1788260us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54003ebcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:30.168328 (+ 22486us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":388341632}
I20260809 23:19:30.211686 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.135560 29760 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:30.218184 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121622us
I20260809 23:19:30.218045 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:30.176779 29808 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:30.166659 29693 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.226114 29808 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:30.221004 24425 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:30.224844 24313 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260809 23:19:30.210600 29825 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260809 23:19:30.232038 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:30.236634 24545 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260809 23:19:30.145051 28517 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260809 23:19:30.236609 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:30.175681 29579 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260809 23:19:30.226342 29693 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.145820 29660 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.312s
I20260809 23:19:30.221355 29803 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:30.142222 29532 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
W20260809 23:19:30.267575 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 140691us
I20260809 23:19:30.200825 28841 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.235332 29829 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:30.239657 29352 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.250634 29513 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:30.273741 29532 consensus_queue.cc:1225] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:30.271263 29369 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:30.296170 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70520.4us
I20260809 23:19:30.250123 29448 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260809 23:19:30.276465 29594 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260809 23:19:30.294242 29172 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:30.281291 29650 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260809 23:19:30.305526 29815 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:30.333304 29838 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.080s	user 0.002s	sys 0.000s
I20260809 23:19:30.280646 29834 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260809 23:19:30.285718 29352 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:30.344636 29835 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260809 23:19:30.342221 29708 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
I20260809 23:19:30.351888 29708 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260809 23:19:30.352234 29844 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260809 23:19:30.348896 29810 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.360808 29810 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.326040 29803 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:30.356626 29845 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260809 23:19:30.361809 29808 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.369366 29808 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.410300 24422 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260809 23:19:30.413532 24422 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:30.434357 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:30.441223 24320 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:30.445009 28841 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.445860 28841 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.446007 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:30.447134 24320 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:30.324992 29172 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.448292 29204 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:30.357290 29693 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:30.469874 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61547.8us
I20260809 23:19:30.478271 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:30.478801 29868 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:30.487949 24311 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 23:19:30.291616 29299 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.446820 28841 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.503592 29467 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.505635 29467 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.505888 29760 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:30.507068 29607 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.507225 29579 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.508373 29579 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.508468 29607 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:30.506004 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131310us
I20260809 23:19:30.516516 29831 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:30.521195 29333 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:30.522109 29333 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260809 23:19:30.524853 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2525) took 1821 ms. Trace:
I20260809 23:19:30.525394 24314 rpcz_store.cc:276] 0809 23:19:28.703376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.843913 (+140537us) service_pool.cc:227] Handling call
0809 23:19:28.853280 (+  9367us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.902637 (+ 49357us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:28.902668 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:28.902707 (+    39us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:28.902739 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.524326 (+1621587us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b540032fad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:30.524407 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:30.524658 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":113008896}
I20260809 23:19:30.532166 29661 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:30.532531 29808 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:30.534258 29808 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:30.534528 28841 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.535871 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2564) took 2301 ms. Trace:
W20260809 23:19:30.539165 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69241.6us
I20260809 23:19:30.517884 29831 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.307446 29369 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.534787 29808 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:30.536280 24429 rpcz_store.cc:276] 0809 23:19:28.234519 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.236996 (+  2477us) service_pool.cc:227] Handling call
0809 23:19:28.237312 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:29.116179 (+878867us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.535372 (+1419193us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:30.535685 (+   313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2019660672}
I20260809 23:19:30.551213 29467 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.552271 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2446) took 1506 ms. Trace:
I20260809 23:19:30.552769 24546 rpcz_store.cc:276] 0809 23:19:29.045901 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.046181 (+   280us) service_pool.cc:227] Handling call
0809 23:19:30.551810 (+1505629us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:30.552047 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":425230208}
I20260809 23:19:30.532202 29352 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.559818 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.562664 24534 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:30.560758 24541 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:30.565649 29369 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:30.562191 29860 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.002s
I20260809 23:19:30.577390 29831 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:30.507462 29760 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.585901 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:30.587059 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.586227 25365 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:30.591038 25371 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:30.591847 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:30.556097 29305 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.713s	user 0.013s	sys 0.004s
I20260809 23:19:30.595626 29607 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.591888 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 960) took 1670 ms. Trace:
I20260809 23:19:30.596992 29579 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.598378 24546 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:30.599822 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.600126 25366 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:30.596544 25363 rpcz_store.cc:276] 0809 23:19:28.921499 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.921702 (+   203us) service_pool.cc:227] Handling call
0809 23:19:28.922106 (+   404us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:29.505669 (+583563us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.591310 (+1085641us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:30.591656 (+   346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1340587776}
I20260809 23:19:30.603096 29808 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.612345 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:30.615491 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:30.615876 29675 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:30.616505 29675 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:30.617080 29675 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:30.618551 28889 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.604295 29760 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:30.619423 28889 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.627204 29864 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:30.628262 29864 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:30.628860 29864 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:30.629741 28889 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.624449 24544 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:30.632941 29436 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.635548 24320 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:19:30.623396 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.001s	sys 0.023s
W20260809 23:19:30.640175 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.001s	sys 0.024s
I20260809 23:19:30.640355 29850 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.646487 29305 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.647118 24544 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:30.647523 29305 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.649616 29531 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:19:30.646456 29869 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260809 23:19:30.662150 28517 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:30.663062 28517 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.663655 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.664665 25370 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:30.666479 28517 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:30.671196 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55791.8us
W20260809 23:19:30.658672 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 114038us
I20260809 23:19:30.641362 29850 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.650785 29305 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:30.651448 29650 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:30.676849 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:30.678615 28517 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.682404 29869 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:30.680431 29871 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.074s	user 0.001s	sys 0.003s
I20260809 23:19:30.690750 24542 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:30.666380 29833 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260809 23:19:30.707795 29682 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:30.708366 29870 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.158s	user 0.002s	sys 0.000s
I20260809 23:19:30.714310 29705 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
W20260809 23:19:30.718550 29508 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.652s	user 0.010s	sys 0.009s
I20260809 23:19:30.701923 29146 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:30.706526 29688 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.690907 29621 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.727751 29850 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:30.748665 29621 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.751834 29628 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.751641 28517 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.737152 29705 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:30.755139 29650 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.756067 28517 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.758050 29682 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.764238 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:30.765395 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.775712 25370 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:30.748905 29688 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.769493 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.780622 29877 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:30.780550 29305 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260809 23:19:30.764338 29869 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.788968 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.782095 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.792868 29682 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.799319 29878 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:30.792979 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:30.794587 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.787596 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:30.752665 29628 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.797377 29508 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:30.796770 29869 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:30.803035 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66283.9us
I20260809 23:19:30.810101 24316 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:30.812131 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.813593 29397 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.813807 29797 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:30.815387 29705 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.781662 29877 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.828066 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.814258 29397 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.830451 29305 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:30.837181 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54621.8us
W20260809 23:19:30.819032 29479 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.731s	user 0.014s	sys 0.009s
I20260809 23:19:30.852525 29509 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:30.852703 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:30.853312 29509 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:30.854151 29305 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:30.854820 29305 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:30.856202 24316 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:30.860045 29745 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.860816 29745 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.867134 29299 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.867952 29745 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.867812 29299 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.871737 29448 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:30.877394 29827 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.994s	user 0.001s	sys 0.001s
I20260809 23:19:30.877828 25363 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:30.848675 24429 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:30.884449 29621 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.885545 29877 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.887156 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.888609 24435 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:30.891563 29628 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.896701 24431 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 3.
I20260809 23:19:30.897698 29353 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.898414 29353 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.879222 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2078) took 1295 ms. Trace:
I20260809 23:19:30.901958 24427 rpcz_store.cc:276] 0809 23:19:29.584009 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.598882 (+ 14873us) service_pool.cc:227] Handling call
0809 23:19:29.599412 (+   530us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:29.599795 (+   383us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:29.599840 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:29.600318 (+   478us) log.cc:842] Serialized 26 byte log entry
0809 23:19:29.699985 (+ 99667us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:29.700023 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:29.700085 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:30.836482 (+1136397us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:30.878721 (+ 42239us) raft_consensus.cc:1671] finished
0809 23:19:30.878745 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:30.879022 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1164,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":139297024,"thread_start_us":99326,"threads_started":1,"wal-append.queue_time_us":242668}
I20260809 23:19:30.898463 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33980 (request call id 4) took 1724 ms. Trace:
I20260809 23:19:30.906759 24316 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:30.905316 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:30.908043 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.905323 25368 rpcz_store.cc:276] 0809 23:19:29.174237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.174635 (+   398us) service_pool.cc:227] Handling call
0809 23:19:30.897762 (+1723127us) spinlock_profiling.cc:243] Waited 7.63 ms on lock 0x7b54006e08a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5dc0 00007fac23866bd2 00007fac23865890 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272
0809 23:19:30.898228 (+   466us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17547520}
I20260809 23:19:30.881048 29299 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.918043 29682 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:30.921232 29819 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.079s	user 0.000s	sys 0.005s
W20260809 23:19:30.925556 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76018.6us
I20260809 23:19:30.921298 29172 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:30.943473 29705 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:30.943987 29734 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:30.838233 29878 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.923645 24539 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:30.933157 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:30.939074 29767 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:30.945480 24317 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:30.947052 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.913080 29305 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.951517 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:30.954501 24316 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:30.961144 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:30.961885 24425 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:30.955564 29635 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.964160 29635 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.944705 29675 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.969820 29675 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:30.980144 29479 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.980075 29803 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:30.981276 29479 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.944422 29734 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:30.929445 29760 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:30.918083 29628 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260809 23:19:30.855139 29509 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:30.991664 29299 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.006793 29299 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:30.994464 29305 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:30.998201 29635 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.004657 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:31.004812 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2559) took 3177 ms. Trace:
I20260809 23:19:31.011574 24430 rpcz_store.cc:276] 0809 23:19:27.827642 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.827839 (+   197us) service_pool.cc:227] Handling call
0809 23:19:27.842245 (+ 14406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:28.924741 (+1082496us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.994747 (+2070006us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003bae54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:31.004640 (+  9893us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2470795136}
I20260809 23:19:31.012943 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:31.013813 29450 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260809 23:19:31.005550 25368 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:31.017985 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.020278 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.020526 29299 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.021533 24309 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:31.024047 24429 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:31.014185 29353 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.013841 24320 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:31.026577 29647 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:31.032096 29647 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:31.032282 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:31.041949 29889 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:30.990976 29760 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:31.026116 29850 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:31.049026 29635 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:31.050511 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.050989 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.051165 24425 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:31.052189 29353 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.054180 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.056149 29481 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:30.998965 29734 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.055897 29628 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:19:31.049726 29876 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260809 23:19:31.054641 29760 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:31.077699 29896 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
W20260809 23:19:31.093755 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 103218us
I20260809 23:19:31.054801 29531 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:31.099375 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.080965 29481 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.100947 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:31.098460 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:31.078679 29878 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.101468 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.052846 29353 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.090867 29479 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.103565 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.110272 29509 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:31.115093 29895 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:19:31.115800 24427 pending_rounds.cc:76] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 4
I20260809 23:19:31.097363 29621 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.076932 29594 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260809 23:19:31.116163 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.101828 29481 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:31.116168 24427 pending_rounds.cc:97] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260809 23:19:31.110188 24430 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:31.094853 29397 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.130275 24427 raft_consensus.cc:2866] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:31.101465 29688 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.116662 29878 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:31.140148 24425 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:31.137667 29479 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:31.088392 29889 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.142557 29479 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.146059 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:31.145282 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.155653 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.138473 29808 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:31.139724 29878 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.149344 29479 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:31.165663 28711 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260809 23:19:31.167341 29889 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.163311 29635 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:31.170917 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.183931 29877 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.184633 29877 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.186964 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.191823 29710 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:31.186870 29877 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.198741 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:31.161091 29531 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.200253 29850 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:31.201028 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2568) took 2796 ms (client timeout 1958 ms). Trace:
I20260809 23:19:31.200937 29850 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:31.201475 24431 rpcz_store.cc:269] 0809 23:19:28.404823 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.405987 (+  1164us) service_pool.cc:227] Handling call
0809 23:19:31.200766 (+2794779us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b540018c254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:31.200950 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5695633152}
I20260809 23:19:31.203368 24422 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260809 23:19:31.203840 29353 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.161851 29767 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.188025 24320 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:31.172516 29635 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.162536 29878 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:31.171468 29734 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:31.151134 29621 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.223282 24316 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:31.225636 24311 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:31.227582 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:31.224131 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:31.231907 24542 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:31.227692 29713 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.235316 29531 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.235183 29713 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.237408 29713 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.238297 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.239030 24313 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:31.219450 29635 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:31.219377 28841 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:31.251756 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.253670 29481 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.255394 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:31.257556 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 84219.9us
I20260809 23:19:31.259418 24309 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:31.251587 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:31.247025 29450 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:31.266294 24543 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:31.269721 24543 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:31.269582 29450 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.270416 29450 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:31.271047 24313 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:31.270874 24543 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.272440 24543 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:31.270875 29531 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:31.275847 29901 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260809 23:19:31.275743 29531 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.276798 29621 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.265740 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.255659 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.291435 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:31.259281 29864 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.009s
I20260809 23:19:31.312804 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.315382 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:31.276822 29817 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.460s	user 0.005s	sys 0.000s
I20260809 23:19:31.255139 29448 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260809 23:19:31.269583 29218 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:31.255395 29146 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260809 23:19:31.321100 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.386449 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 909) took 1384 ms. Trace:
I20260809 23:19:31.312080 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:31.345969 24314 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:31.393641 25372 rpcz_store.cc:276] 0809 23:19:29.821649 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.196367 (+1374718us) service_pool.cc:227] Handling call
0809 23:19:31.196981 (+   614us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.197250 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:31.197284 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:31.197341 (+    57us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:31.205422 (+  8081us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:31.205494 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:31.205599 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:31.345716 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:31.345884 24313 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260809 23:19:31.403589 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:31.405400 29710 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.406203 29710 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.412806 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.414036 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:31.339288 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2110) took 3744 ms. Trace:
I20260809 23:19:31.427271 29710 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.428365 29353 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:31.429315 29353 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:31.431159 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 181721us
I20260809 23:19:31.431272 29146 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:31.431365 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:31.432091 24425 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:31.431875 29146 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.432868 28517 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
W20260809 23:19:31.346352 29818 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.477s	user 0.004s	sys 0.000s
W20260809 23:19:31.323716 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43731.6us
I20260809 23:19:31.445122 24543 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:31.348006 29481 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:31.447700 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102178us
I20260809 23:19:31.348006 29397 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:31.451695 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:31.321432 29850 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.347100 29800 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.277974 28841 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.449127 24316 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:31.279050 29767 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.314651 29864 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.471612 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.471086 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2068) took 2566 ms. Trace:
I20260809 23:19:31.453823 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:31.454574 29172 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:31.483737 29172 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:31.430729 29450 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:31.496273 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92766.1us
W20260809 23:19:31.501760 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50170.9us
I20260809 23:19:31.502807 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:31.505067 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73181.6us
I20260809 23:19:31.507083 24431 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:31.422753 24323 rpcz_store.cc:276] 0809 23:19:27.571898 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:27.638915 (+ 67017us) service_pool.cc:227] Handling call
0809 23:19:27.639284 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:27.648157 (+  8873us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:27.773878 (+125721us) log.cc:842] Serialized 14 byte log entry
0809 23:19:28.004891 (+231013us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:28.004983 (+    92us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:28.005038 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:31.309794 (+3304756us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:31.316447 (+  6653us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:31.316672 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25746,"prepare.queue_time_us":27463,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":1562,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":9398,"spinlock_wait_cycles":19803008,"thread_start_us":328553,"threads_started":2,"wal-append.queue_time_us":287390,"wal-append.run_cpu_time_us":7206,"wal-append.run_wall_time_us":1352232}
I20260809 23:19:31.476567 24428 rpcz_store.cc:276] 0809 23:19:28.686343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.067704 (+381361us) service_pool.cc:227] Handling call
0809 23:19:29.068091 (+   387us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:29.092987 (+ 24896us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:29.093034 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:30.963765 (+1870731us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:30.964348 (+   583us) log.cc:842] Serialized 26 byte log entry
0809 23:19:31.008289 (+ 43941us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:31.159235 (+150946us) log.cc:842] Serialized 14 byte log entry
0809 23:19:31.169780 (+ 10545us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:31.216059 (+ 46279us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:31.223204 (+  7145us) raft_consensus.cc:1671] finished
0809 23:19:31.234605 (+ 11401us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:31.252444 (+ 17839us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37374,"prepare.queue_time_us":230369,"raft.queue_time_us":50379,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":48975,"spinlock_wait_cycles":17185536,"thread_start_us":152072,"threads_started":2,"wal-append.queue_time_us":101063}
I20260809 23:19:31.454269 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:31.470531 29120 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 8 is higher.
I20260809 23:19:31.457857 29397 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:31.466343 29705 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.522s	user 0.010s	sys 0.010s
I20260809 23:19:31.469520 29850 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.522739 28836 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:31.523154 29926 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:31.523607 29850 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.524294 29926 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:31.524772 29926 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:31.525309 28836 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:31.526106 29850 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.487783 29635 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.531101 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:31.532058 29635 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.532224 25372 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:31.499645 29146 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:31.344338 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:31.475860 29218 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:31.523142 29917 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260809 23:19:31.535324 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 191252us
I20260809 23:19:31.517949 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.544337 24543 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:31.475193 29481 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260809 23:19:31.520119 29305 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.558326 29353 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.559384 24428 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:31.561412 29306 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:31.563599 29924 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:31.584110 29822 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.760s	user 0.004s	sys 0.000s
I20260809 23:19:31.583200 29924 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.520500 29397 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:31.588014 24546 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 23:19:31.521456 29531 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.468377 29800 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.577680 25370 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:19:31.603134 29864 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.580677 29878 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.581125 29299 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:31.582206 29747 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:31.582424 24429 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:31.543604 29767 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.543052 29931 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:31.625291 29931 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.628165 29931 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.605015 25372 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 8. Ops: []
I20260809 23:19:31.628226 29647 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.634780 24425 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:31.619920 29450 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.604858 29481 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260809 23:19:31.610993 29765 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.630486 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:31.631211 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:31.634262 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:31.611732 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 673) took 1358 ms. Trace:
W20260809 23:19:31.642143 29873 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.055s	user 0.009s	sys 0.004s
I20260809 23:19:31.640301 29682 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.654985 29682 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:31.644711 24435 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:31.650010 24422 rpcz_store.cc:276] 0809 23:19:30.253788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.417441 (+163653us) service_pool.cc:227] Handling call
0809 23:19:31.202990 (+785549us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.203755 (+   765us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:31.611495 (+407740us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b540018c250. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:31.611656 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1805614848}
I20260809 23:19:31.664237 29878 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.631145 29601 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:31.668241 25370 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 23:19:31.671367 29450 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:31.671125 29765 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.674533 29450 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:31.679137 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 231130us
W20260809 23:19:31.685518 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140690us
I20260809 23:19:31.729525 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:31.733599 24535 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:19:31.727218 29760 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.615s	user 0.013s	sys 0.000s
I20260809 23:19:31.744270 29737 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.745024 29834 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.745080 29218 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.745568 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 972) took 1230 ms. Trace:
I20260809 23:19:31.671141 29397 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.696281 29481 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.732093 29647 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:31.748199 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.744935 29737 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.744278 29409 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.737160 28711 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:31.745361 29918 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260809 23:19:31.683023 29305 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.685377 29931 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.697082 28517 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:31.759997 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.745971 25373 rpcz_store.cc:276] 0809 23:19:30.442255 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.445306 (+  3051us) service_pool.cc:227] Handling call
0809 23:19:30.446121 (+   815us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:30.447093 (+   972us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:30.447127 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:30.448188 (+  1061us) log.cc:842] Serialized 26 byte log entry
0809 23:19:30.502142 (+ 53954us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:30.502221 (+    79us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:30.502357 (+   136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:31.617305 (+1114948us) spinlock_profiling.cc:243] Waited 5.97 ms on lock 0x7b1800341d40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:31.619474 (+  2169us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:31.672164 (+ 52690us) raft_consensus.cc:1671] finished
0809 23:19:31.672186 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:31.672314 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":48985,"raft.run_cpu_time_us":540,"raft.run_wall_time_us":50537,"spinlock_wait_cycles":96103040,"thread_start_us":48447,"threads_started":1,"wal-append.queue_time_us":101736}
I20260809 23:19:31.760669 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:31.761376 29734 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.530s	user 0.007s	sys 0.004s
W20260809 23:19:31.774888 29925 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260809 23:19:31.759851 29409 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.868433 29409 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.748534 29760 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.869740 25363 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:31.870750 29760 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:31.745749 29834 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.871059 29394 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:31.874341 29394 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:31.874727 29394 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:31.878340 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2328) took 2747 ms (client timeout 1898 ms). Trace:
I20260809 23:19:31.878103 29800 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.758816 28711 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.829217 29815 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:31.783728 25366 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:31.741117 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 287758us
I20260809 23:19:31.745702 29452 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
W20260809 23:19:31.842598 29919 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
W20260809 23:19:31.864163 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 134444us
W20260809 23:19:31.878836 24539 rpcz_store.cc:269] 0809 23:19:29.130879 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.131041 (+   162us) service_pool.cc:227] Handling call
0809 23:19:31.878039 (+2746998us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54000b6154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:31.878228 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4099188608}
I20260809 23:19:31.893443 25373 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:31.783659 29481 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.894277 29305 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.894735 24544 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260809 23:19:31.895192 24544 pending_rounds.cc:76] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 6
I20260809 23:19:31.895470 24544 pending_rounds.cc:97] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260809 23:19:31.895726 24544 raft_consensus.cc:2866] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:31.896575 29481 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:31.897773 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:31.888211 28517 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.899719 29800 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:31.900839 29800 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:31.901566 29800 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:31.901980 29800 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:31.903192 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:31.886830 29479 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.720s	user 0.017s	sys 0.001s
I20260809 23:19:31.908149 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:31.909060 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 410589us
I20260809 23:19:31.911481 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:31.870975 29926 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.914463 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.914698 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:31.912060 25370 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:31.888470 29172 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:31.886077 29815 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:31.918891 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:31.922470 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.923368 24317 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:31.926477 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:31.927096 24320 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:31.873797 29853 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:31.894304 28862 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.933576 25323 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:31.806459 29931 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:31.935353 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:31.940582 24314 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:31.942629 24317 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:31.943616 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:31.944020 29450 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.944473 24316 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 10
I20260809 23:19:31.955960 29720 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:31.955987 29938 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:31.953730 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.910530 29760 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260809 23:19:31.932456 29853 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:31.918649 29479 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.956763 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:31.963901 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:31.966441 24539 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260809 23:19:31.969583 29808 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:31.956652 29937 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260809 23:19:31.971724 29397 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:31.973374 25373 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:31.941138 29924 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:31.977830 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 2.64525e+06us
I20260809 23:19:31.981866 24542 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:31.973245 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:31.983531 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.956307 29928 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:31.985601 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:31.986841 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:31.988220 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:31.989800 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:31.956477 29720 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:31.994825 24314 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:31.994779 25366 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:31.940524 28862 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:31.973567 29938 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.019164 24313 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:32.050189 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.051076 25373 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:32.024245 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:32.026768 24541 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:31.996673 29922 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.369s	user 0.000s	sys 0.003s
I20260809 23:19:32.056344 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.043917 25365 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260809 23:19:31.986800 29765 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.054256 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.020108 29720 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:32.020772 29937 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:19:32.057152 24539 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:32.058817 25365 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 5
I20260809 23:19:32.095547 25365 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260809 23:19:32.096650 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:32.097970 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:32.099635 24309 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:32.095494 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.033166 29450 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.024657 29397 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.109241 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.096196 25365 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:32.027127 29737 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.107092 24320 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:32.106767 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.112932 24542 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:31.996641 29815 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.108618 29450 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.118485 25366 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:31.962095 29800 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.024087 29353 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
W20260809 23:19:32.096599 29783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.074s	user 0.024s	sys 0.041s
I20260809 23:19:32.053865 29938 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:31.997705 29808 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.119539 24312 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.120083 29924 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.136386 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 773) took 1470 ms. Trace:
I20260809 23:19:32.103747 28711 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:32.125304 29873 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260809 23:19:32.125597 29397 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.139807 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2479) took 1411 ms. Trace:
I20260809 23:19:32.154482 24544 rpcz_store.cc:276] 0809 23:19:30.666266 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.667003 (+   737us) service_pool.cc:227] Handling call
0809 23:19:31.879324 (+1212321us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.896313 (+ 16989us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.136158 (+239845us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000b6150. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:32.136291 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787054848}
I20260809 23:19:32.125836 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.168931 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.169106 24313 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:32.109846 29853 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.130236 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.131311 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.171036 25373 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:32.150861 24314 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:32.046101 29951 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:32.099282 29890 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.151540 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.127194 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.147367 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.131458 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.137315 29742 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.120822 29450 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:32.182520 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55758.9us
W20260809 23:19:32.146279 29783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.310s	user 0.024s	sys 0.042s
I20260809 23:19:32.149149 29926 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.168486 24546 rpcz_store.cc:276] 0809 23:19:30.728877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.817076 (+ 88199us) service_pool.cc:227] Handling call
0809 23:19:30.817520 (+   444us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.588468 (+770948us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.139586 (+551118us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:32.139744 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1766282496}
W20260809 23:19:32.131798 29956 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260809 23:19:32.132612 29948 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260809 23:19:32.176501 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:32.137267 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:32.130901 29931 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.139911 28862 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.147909 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:32.181206 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51060.3us
I20260809 23:19:32.187163 24545 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.188041 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:32.232517 25366 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:32.153563 29737 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.132196 29939 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.148989 29823 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.131645 28971 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.239363 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.240684 29934 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:19:32.180615 24270 connection.cc:232] RPC call timeout handler was delayed by -1.7405480032321066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20260809 23:19:32.243676 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.175210 29878 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:32.255189 24270 connection.cc:232] RPC call timeout handler was delayed by -1.712646228261292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
I20260809 23:19:32.258970 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.259727 24546 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:32.245957 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.243861 24309 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:32.208845 29682 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.008s
I20260809 23:19:32.249547 24544 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:32.246580 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1077) took 2238 ms. Trace:
I20260809 23:19:32.255399 24320 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:32.261319 24311 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:32.175429 29937 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260809 23:19:32.173453 29924 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.271275 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.277259 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.277593 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:32.271507 29890 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.274547 25373 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:32.277957 24546 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:32.249915 29939 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:32.258140 29728 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:32.297281 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.279103 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.276043 24316 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:32.308041 29960 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260809 23:19:32.308835 29690 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:32.309392 29690 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.309805 29690 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.310849 29436 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.265875 25369 rpcz_store.cc:276] 0809 23:19:29.944567 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.947058 (+  2491us) service_pool.cc:227] Handling call
0809 23:19:29.994236 (+ 47178us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:30.016557 (+ 22321us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:30.016588 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:30.017145 (+   557us) log.cc:842] Serialized 52 byte log entry
0809 23:19:30.217288 (+200143us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:31.869294 (+1652006us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b180032a640. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:19:31.869483 (+   189us) log.cc:842] Serialized 14 byte log entry
0809 23:19:32.181910 (+312427us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.182025 (+   115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:32.182104 (+    79us) raft_consensus.cc:1671] finished
0809 23:19:32.182196 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:32.182322 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37198,"prepare.queue_time_us":87452,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":40724,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":24955,"spinlock_wait_cycles":15622784,"thread_start_us":516671,"threads_started":4,"wal-append.queue_time_us":438148,"wal-append.run_cpu_time_us":2594,"wal-append.run_wall_time_us":1103177}
I20260809 23:19:32.314421 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:32.315776 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.316885 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.318174 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.320632 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.321313 24546 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:32.324223 24545 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:32.261932 29725 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:32.281345 29809 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:32.327267 24321 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:32.283450 29935 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:32.272964 29479 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.332592 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.333546 29660 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260809 23:19:32.238241 29951 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.333824 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.255569 29394 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.281052 29650 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.282351 29930 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:32.254532 29737 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:32.370976 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 183451us
I20260809 23:19:32.349440 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.349721 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.340543 29924 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.298823 29939 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.266674 29682 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.300699 29934 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:32.357782 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:32.357945 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:32.375075 24312 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:32.375358 24435 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:32.358155 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:32.359159 25368 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260809 23:19:32.359536 29481 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:32.363479 29808 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.355875 29935 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:32.364454 29397 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:32.271413 28971 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.348728 29889 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260809 23:19:32.354805 29724 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:32.389719 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:32.365687 29938 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:32.367149 29601 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.685s	user 0.001s	sys 0.011s
I20260809 23:19:32.367481 29951 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.369247 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.370332 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:32.334075 29953 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260809 23:19:32.372746 25373 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260809 23:19:32.289914 29823 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.405520 29758 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.279686 29878 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.328718 29725 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:32.372014 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.380169 25366 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:32.419167 24270 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:32.395658 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.424084 24546 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:32.355818 29809 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:32.401252 24427 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 8.
I20260809 23:19:32.402297 24545 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:32.375625 29868 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.401635 24430 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:32.423230 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51360.4us
I20260809 23:19:32.396937 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 889) took 2469 ms. Trace:
I20260809 23:19:32.406698 29823 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.408694 29878 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.444386 24312 rpcz_store.cc:276] 0809 23:19:29.914917 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.915256 (+   339us) service_pool.cc:227] Handling call
0809 23:19:29.915531 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.891835 (+1976304us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5800a547b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:19:32.375495 (+483660us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.384351 (+  8856us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3674289536}
I20260809 23:19:32.418272 29690 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.446460 29615 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:32.470311 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.417508 29800 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.465032 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.401503 29724 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:32.468395 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:32.442692 29924 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:32.405963 29758 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.513705 24546 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:32.519608 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51260.3us
I20260809 23:19:32.420608 29737 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.445731 29765 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.469780 29615 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.446072 29809 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:32.416498 28971 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.521519 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:32.471063 24544 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
W20260809 23:19:32.525071 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60123.5us
I20260809 23:19:32.519905 29758 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:32.521025 29647 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.765s	user 0.009s	sys 0.005s
I20260809 23:19:32.528620 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.518056 29924 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.534621 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.521483 29823 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:32.538225 24546 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:32.546731 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:32.547902 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.463676 29868 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.424434 29930 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.531288 29615 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.555802 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:32.531545 29479 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.533227 29890 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.575080 29968 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.575083 29926 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.575650 29647 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.575811 29926 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.576512 29647 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.578104 29926 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.579713 29926 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.580581 29926 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.582161 29647 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.585660 29647 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.587239 29926 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.559386 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.552800 29737 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:32.591533 29737 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:32.545128 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:32.569598 29735 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.592156 29690 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.548863 28971 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.572430 24311 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260809 23:19:32.544627 29889 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:32.586452 29647 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.412010 29725 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.556257 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:32.605783 24316 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:32.595549 24378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.054s (SENT) [suppressed 4 similar messages]
I20260809 23:19:32.573529 29938 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.546131 29924 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:32.526502 29765 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:32.596024 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51031.4us
I20260809 23:19:32.623248 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:32.529306 29720 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.593220 29601 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.593698 29479 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.600615 29890 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:32.602893 29889 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.630345 29647 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.515247 29935 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.592555 29735 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.633160 24546 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:32.625178 29765 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.607468 29394 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:32.612929 29718 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260809 23:19:32.575676 29968 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.613188 29650 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260809 23:19:32.636271 29677 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:32.637622 29720 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.630970 29601 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.662549 29601 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.631171 29479 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.667815 29968 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.592728 29737 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.669478 29935 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:32.673197 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2061) took 4149 ms. Trace:
I20260809 23:19:32.467628 29934 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.648280 24546 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:32.635419 29889 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:32.660494 29855 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.662073 29677 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:32.673619 24423 rpcz_store.cc:276] 0809 23:19:28.523534 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:28.534964 (+ 11430us) service_pool.cc:227] Handling call
0809 23:19:28.535344 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:28.901350 (+366006us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.657578 (+3756228us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:32.673017 (+ 15439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":833169152}
I20260809 23:19:32.673580 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2487) took 1465 ms. Trace:
I20260809 23:19:32.609561 28971 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.676970 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.679045 24541 rpcz_store.cc:276] 0809 23:19:31.206049 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.215876 (+  9827us) service_pool.cc:227] Handling call
0809 23:19:32.670734 (+1454858us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:32.670877 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1706073856}
I20260809 23:19:32.678063 24546 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.631608 29939 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.695034 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.696414 29601 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.696717 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.697597 24311 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:32.698565 29601 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.699088 24540 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:32.699658 29794 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.700160 29810 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.696460 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.707935 24546 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.643532 29724 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.704026 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:32.689227 29696 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.974s	user 0.026s	sys 0.017s
I20260809 23:19:32.684188 29677 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.710922 24541 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 5.
I20260809 23:19:32.719805 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:32.689477 29890 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:32.700661 29810 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:32.720223 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 124718us
I20260809 23:19:32.720381 24541 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:32.689975 29720 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:32.714789 24317 raft_consensus.cc:1203] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[8.9-8.9]   Dedup: 8.9->[]
I20260809 23:19:32.730626 24317 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260809 23:19:32.719064 29696 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.141s	user 0.027s	sys 0.017s
I20260809 23:19:32.728320 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.700413 29794 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.716830 29823 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.693413 29878 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260809 23:19:32.722414 29436 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.748136 29436 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.750252 29436 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.744946 29823 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.730523 29601 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:32.732542 29720 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.751248 29601 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.706792 29647 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260809 23:19:32.747550 28971 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.686771 29855 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:32.756554 29601 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.757128 29878 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:32.775110 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 763) took 3587 ms. Trace:
I20260809 23:19:32.765662 24546 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260809 23:19:32.755173 29720 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.787467 29939 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:32.797510 24534 rpcz_store.cc:276] 0809 23:19:29.172696 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.181293 (+  8597us) service_pool.cc:227] Handling call
0809 23:19:30.561976 (+1380683us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:30.563187 (+  1211us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.755733 (+2192546us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54003dd450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:32.760343 (+  4610us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3158560640}
I20260809 23:19:32.787611 29823 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:32.791203 29212 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:32.789414 29677 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.744405 29976 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:32.796584 29916 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.163s	user 0.000s	sys 0.007s
I20260809 23:19:32.797896 24546 pending_rounds.cc:76] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:19:32.775022 29770 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:32.806260 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2330) took 3674 ms (client timeout 1909 ms). Trace:
I20260809 23:19:32.811178 29855 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:32.812091 29450 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.003s
I20260809 23:19:32.816443 24546 pending_rounds.cc:97] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260809 23:19:32.821398 24546 raft_consensus.cc:2866] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:32.737197 29615 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:32.819554 24540 rpcz_store.cc:269] 0809 23:19:29.131492 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:29.132269 (+   777us) service_pool.cc:227] Handling call
0809 23:19:32.805912 (+3673643us) spinlock_profiling.cc:243] Waited 3.32 s on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:32.806166 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7646255360}
W20260809 23:19:32.776881 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101193us
I20260809 23:19:32.798138 29890 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:32.791000 29878 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:32.822904 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:32.837438 24317 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:32.837756 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.803328 29976 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.828593 29728 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:32.828687 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 971) took 2757 ms. Trace:
I20260809 23:19:32.821869 29770 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.829370 29758 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.831468 29479 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.822610 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:32.835035 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 810835us
I20260809 23:19:32.840756 29976 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:32.846773 29879 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:32.851229 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 674786us
I20260809 23:19:32.872757 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.823827 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.838706 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:19:32.874094 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50387.4us
I20260809 23:19:32.842530 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:32.873102 25323 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:32.874874 25323 leader_election.cc:401] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260809 23:19:32.874732 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:32.836362 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 790) took 1078 ms. Trace:
I20260809 23:19:32.875665 24546 rpcz_store.cc:276] 0809 23:19:31.758895 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.547278 (+788383us) service_pool.cc:227] Handling call
0809 23:19:32.608728 (+ 61450us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:32.836099 (+227371us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.836294 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:32.877161 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.877789 25366 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:32.872757 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.928402 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 685) took 1765 ms. Trace:
I20260809 23:19:32.928977 24430 rpcz_store.cc:276] 0809 23:19:31.111162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.136745 (+ 25583us) service_pool.cc:227] Handling call
0809 23:19:32.850837 (+1714092us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:32.876332 (+ 25495us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2854497664}
W20260809 23:19:32.845796 29877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.225s	user 0.006s	sys 0.012s
I20260809 23:19:32.842433 29760 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:32.963074 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:32.835407 29794 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:32.937260 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:32.803920 29212 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.983806 24546 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:32.981594 24540 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:32.973800 25372 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:32.839521 25375 rpcz_store.cc:276] 0809 23:19:30.071674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.071880 (+   206us) service_pool.cc:227] Handling call
0809 23:19:30.072267 (+   387us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:30.097859 (+ 25592us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:30.097892 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:32.522306 (+2424414us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:32.571354 (+ 49048us) log.cc:842] Serialized 26 byte log entry
0809 23:19:32.674551 (+103197us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:32.679374 (+  4823us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.685713 (+  6339us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:32.828271 (+142558us) raft_consensus.cc:1671] finished
0809 23:19:32.828293 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:32.828536 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20479,"raft.queue_time_us":24547,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":10392,"spinlock_wait_cycles":9741568,"thread_start_us":102515,"threads_started":1,"wal-append.queue_time_us":160153}
I20260809 23:19:32.985620 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:32.872821 29728 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:32.823796 29450 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.989176 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.873569 29823 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:32.994603 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 142593us
W20260809 23:19:32.963984 29877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.343s	user 0.006s	sys 0.012s
I20260809 23:19:32.837034 29765 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:32.936533 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 599125us
I20260809 23:19:32.993449 24541 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:32.998881 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.999434 24541 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:32.999492 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:33.000124 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 794) took 1104 ms. Trace:
I20260809 23:19:33.000497 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.000430 24541 rpcz_store.cc:276] 0809 23:19:31.895749 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.999028 (+1103279us) service_pool.cc:227] Handling call
0809 23:19:32.999294 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:32.999929 (+   635us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.000077 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:33.001257 25375 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:33.001999 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2496) took 1082 ms. Trace:
I20260809 23:19:33.002398 24541 rpcz_store.cc:276] 0809 23:19:31.919774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.001194 (+1081420us) service_pool.cc:227] Handling call
0809 23:19:33.001480 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.001662 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:33.001686 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:33.001724 (+    38us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:33.001767 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.001821 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:33.001928 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:33.002808 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.003300 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:33.003885 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.005604 29666 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:33.006376 29666 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:33.008574 29666 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.010439 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.011138 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:33.012775 29934 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:33.012928 29785 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260809 23:19:32.936237 29987 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:32.992245 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.013495 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.962795 29696 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.025390 25368 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:33.004841 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:32.997912 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 793) took 1102 ms. Trace:
W20260809 23:19:33.017117 29800 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.002s
I20260809 23:19:33.017306 24542 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:32.987435 29397 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260809 23:19:32.995322 29728 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.014079 29809 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:33.061453 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2089) took 1771 ms (client timeout 1809 ms). Trace:
W20260809 23:19:33.085536 24429 rpcz_store.cc:269] 0809 23:19:31.290021 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.290331 (+   310us) service_pool.cc:227] Handling call
0809 23:19:33.061230 (+1770899us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:33.061370 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":612252416}
I20260809 23:19:33.070016 25369 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:33.037773 25366 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:33.091208 24546 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:32.938301 29879 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:32.992200 29705 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.057344 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.209088 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:33.013626 28711 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7.
I20260809 23:19:33.023880 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.026422 24313 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:33.014460 25365 raft_consensus.cc:1203] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Deduplicated request from leader. Original: 7.8->[8.9-8.9]   Dedup: 8.9->[]
I20260809 23:19:33.017560 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.096400 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.027722 29697 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.269s
I20260809 23:19:33.047437 29467 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.159s
I20260809 23:19:33.027554 29532 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:33.208442 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:33.232744 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 212165us
I20260809 23:19:33.066823 29770 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.254082 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.253996 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.026811 24540 rpcz_store.cc:276] 0809 23:19:31.895232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.984969 (+1089737us) service_pool.cc:227] Handling call
0809 23:19:32.997281 (+ 12312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:32.997482 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:32.997509 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:32.997566 (+    57us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:32.997627 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.997695 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:32.997841 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:33.232678 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 216254us
I20260809 23:19:33.255815 25366 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:33.068846 29615 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.232898 25368 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:33.023324 29987 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.252579 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.036387 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:33.071115 29765 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.081061 29758 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.041344 29962 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:33.025888 29988 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260809 23:19:33.240830 25365 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:19:33.041566 29747 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:33.255163 24546 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:33.072433 29212 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.016727 29823 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.283622 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:33.258948 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 222665us
I20260809 23:19:33.079020 29696 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.263976 29877 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:33.265482 29753 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:33.296501 29877 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.296957 29877 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:33.266372 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.266703 29879 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.267511 29450 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.268141 29728 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.268059 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.270043 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.023761 29855 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.232s
I20260809 23:19:33.027454 29337 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.274602 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 991) took 1288 ms. Trace:
I20260809 23:19:33.243376 29305 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:33.250864 29800 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:33.253612 28862 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:33.283986 29765 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 23:19:33.258972 29924 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.626s	user 0.009s	sys 0.002s
I20260809 23:19:33.027582 29890 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.256495 29532 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:33.288920 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.282226 29747 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:33.241819 29705 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.273691 29615 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.211515 29976 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.291775 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2353) took 1038 ms. Trace:
I20260809 23:19:33.292042 29696 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:33.292132 25321 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 8.
I20260809 23:19:33.302917 29760 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260809 23:19:33.360657 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68639.3us
I20260809 23:19:33.360841 25373 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:33.361605 29765 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.362141 29765 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.292972 29809 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:33.293875 29832 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:33.295231 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.364001 29832 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:33.295960 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.365271 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:33.364876 29832 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.366428 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.366767 29993 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:33.369969 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.298988 24309 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.299002 24316 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:33.375504 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:33.376314 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:33.297947 24317 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:33.364028 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:33.377830 24429 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.301367 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.046779 29794 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.216s
I20260809 23:19:33.272322 29467 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:33.019013 29878 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.274688 29962 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.283222 29172 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:33.262774 29987 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:33.274361 29677 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.065832 29942 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.196s
I20260809 23:19:33.305907 25363 rpcz_store.cc:276] 0809 23:19:31.986623 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.986769 (+   146us) service_pool.cc:227] Handling call
0809 23:19:32.019894 (+ 33125us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:32.020148 (+   254us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:32.095000 (+ 74852us) log.cc:842] Serialized 14 byte log entry
0809 23:19:33.274273 (+1179273us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:33.274327 (+    54us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:33.274381 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.274451 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:33.274558 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":854822,"prepare.queue_time_us":112398,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":48,"raft.queue_time_us":747433,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":32049,"thread_start_us":513154,"threads_started":2}
I20260809 23:19:33.288858 29479 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.346489 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.347363 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.395970 24320 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:33.314895 29889 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.627s	user 0.010s	sys 0.000s
I20260809 23:19:33.353271 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.290864 29212 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:33.303930 29810 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.318005 29305 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.357336 24543 rpcz_store.cc:276] 0809 23:19:32.260972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.319829 (+ 58857us) service_pool.cc:227] Handling call
0809 23:19:33.291597 (+971768us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54000b6154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:33.291695 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1125773184}
I20260809 23:19:33.378499 24546 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8. Re-sending same reply.
I20260809 23:19:33.364877 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.377390 28862 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:33.382611 29753 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:33.380070 29615 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.348228 29532 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.414561 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:33.340998 29803 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:33.319335 29800 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.301543 29397 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.345623 29747 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.302575 29808 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.386294 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.387319 24541 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:33.339591 29337 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.399227 24316 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:33.395433 24311 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:33.349789 29924 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.405299 24309 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:33.410300 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.494410 24546 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.407051 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:33.416440 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.389987 29962 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.400936 29878 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.396819 29934 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.509735 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.471521 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.478838 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.513636 25369 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.480424 24435 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:33.402184 29696 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.374508 29765 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.402036 29212 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:33.409632 29305 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.415550 29753 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:33.507392 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 121194us
I20260809 23:19:33.520224 24311 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:33.520823 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.522260 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.506217 24320 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:33.508500 24424 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:33.509754 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.476455 29800 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.477824 29677 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:33.517647 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.405501 29889 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.534789 25368 raft_consensus.cc:2396] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:33.545614 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:33.536264 24309 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.532583 29991 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:33.532557 29666 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.538882 29615 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:33.539228 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 134155us
I20260809 23:19:33.543190 29677 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:33.543332 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.553292 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.516042 29760 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:33.566650 24546 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:33.544180 28862 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.572460 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.544512 29939 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.573668 29337 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.574244 29976 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.533420 29753 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:33.470916 29397 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.526435 29212 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:33.587481 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.495677 29808 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.589226 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.572919 29758 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.589491 25369 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:33.592644 29615 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:33.549304 29467 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:33.593022 24313 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260809 23:19:33.567703 29879 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:33.595728 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 791) took 1801 ms. Trace:
I20260809 23:19:33.476140 29803 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.594430 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.527580 29934 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.586208 29760 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.613448 24309 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:33.533165 29696 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.629796 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.615995 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.604550 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:33.639596 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.645471 24320 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:33.630879 24534 rpcz_store.cc:276] 0809 23:19:31.794331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.965230 (+1170899us) service_pool.cc:227] Handling call
0809 23:19:32.965691 (+   461us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:32.965990 (+   299us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:33.273827 (+307837us) log.cc:842] Serialized 14 byte log entry
0809 23:19:33.595280 (+321453us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:33.595349 (+    69us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:33.595435 (+    86us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.595519 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:33.595659 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":449789,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":18382,"raft.queue_time_us":111,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13732,"thread_start_us":594144,"threads_started":2}
I20260809 23:19:33.643388 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.651396 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.653112 24309 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:33.654330 24430 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.630198 29615 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.579391 29666 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.652216 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:33.658578 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:33.610639 29705 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.692490 24543 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:33.644232 29760 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:33.632654 29758 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.689464 29931 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:33.690114 29479 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:33.690136 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.689520 29616 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:33.691558 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:33.699417 25369 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:33.694450 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:33.700484 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:33.695669 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.696250 29890 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.653815 29993 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:33.702250 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.709925 24546 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:33.685642 29337 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:33.701532 29120 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:33.658730 29615 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.615746 29939 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.705458 24543 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:33.612771 29877 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.716794 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:33.721931 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.721000 24320 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:33.708364 29479 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:33.713011 29436 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:33.706166 29801 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:33.727941 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2498) took 1776 ms. Trace:
I20260809 23:19:33.728462 24546 rpcz_store.cc:276] 0809 23:19:31.951858 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.716418 (+1764560us) service_pool.cc:227] Handling call
0809 23:19:33.727250 (+ 10832us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.727519 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:33.727556 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:33.727613 (+    57us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:33.727673 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.727748 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:33.727873 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:33.711236 29616 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:33.734699 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:33.737623 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 320572us
I20260809 23:19:33.718884 29337 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.731321 29935 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:33.739179 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:33.739375 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:33.743618 24309 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:33.741401 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:33.738596 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:33.745652 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.743655 25363 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
W20260809 23:19:33.752224 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 230050us
I20260809 23:19:33.724596 29931 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.725124 29924 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.779481 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2501) took 1684 ms. Trace:
I20260809 23:19:33.717803 29926 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260809 23:19:33.780385 24429 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:33.790688 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.786158 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.792095 24543 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:33.737818 29765 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:33.789064 24546 rpcz_store.cc:276] 0809 23:19:32.095057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.742078 (+1647021us) service_pool.cc:227] Handling call
0809 23:19:33.774484 (+ 32406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.774705 (+   221us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:33.774736 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:33.774781 (+    45us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:33.774842 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.774903 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:33.779435 (+  4532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:33.719439 29532 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.738570 29436 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:33.751899 29743 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:33.742782 29369 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.751250 29337 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:33.858059 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76165.9us
I20260809 23:19:33.841331 25375 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:33.841773 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:33.714545 29677 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.743853 29809 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:33.747429 29889 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:33.791962 24309 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:33.721892 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:33.909574 25369 raft_consensus.cc:595] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:33.910002 25369 raft_consensus.cc:603] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:33.744042 29801 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:33.752424 29935 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:33.772130 29991 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:33.774266 29120 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:33.783303 29770 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.859400 24320 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:33.712280 29890 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.943974 29890 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:33.943810 29801 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:33.946051 29120 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:33.863822 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:33.865236 29765 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:33.866153 29516 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.521s	user 0.006s	sys 0.016s
W20260809 23:19:33.866833 30000 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260809 23:19:33.867525 29479 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.868026 29616 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:33.875209 29999 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260809 23:19:33.875851 29735 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:33.876937 29800 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.878187 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2090) took 2584 ms. Trace:
I20260809 23:19:33.737478 29998 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.164s
I20260809 23:19:33.940601 29436 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:33.941478 24496 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.958s (SENT)
I20260809 23:19:33.947677 24546 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:33.936161 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:33.857920 29397 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.906901 30009 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:33.957563 29516 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.639s	user 0.006s	sys 0.016s
I20260809 23:19:33.793772 29931 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.930037 29935 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:33.935465 29889 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.971591 24431 rpcz_store.cc:276] 0809 23:19:31.294199 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.294351 (+   152us) service_pool.cc:227] Handling call
0809 23:19:31.322702 (+ 28351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.512896 (+190194us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.877888 (+2364992us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:33.878106 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":423496064}
W20260809 23:19:33.981371 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45280.7us
I20260809 23:19:33.906899 29369 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.984560 29705 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:33.986476 29800 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.017633 24543 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:33.989329 29962 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.990504 29993 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:33.993961 29342 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:33.996497 29930 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.025058 24546 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:33.977430 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:33.978468 24430 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:34.007421 29939 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.009799 29879 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:33.962224 29934 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:33.974105 29808 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.021128 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.021287 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:33.992457 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 792) took 2098 ms (client timeout 2111 ms). Trace:
I20260809 23:19:33.956137 29770 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:33.960189 29765 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.037308 24543 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:34.058905 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:34.029487 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.007180 30009 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.014467 29696 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.008071 29803 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.043783 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.045595 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:34.035704 29342 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.047566 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:34.047765 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106249us
I20260809 23:19:33.964319 29991 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.030544 29877 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.031065 29926 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:19:34.078475 30016 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:34.062217 29616 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:34.090755 30016 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:34.091264 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.091995 24323 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:34.091835 30016 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:34.092619 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2563) took 2574 ms (client timeout 2879 ms). Trace:
I20260809 23:19:34.061913 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:34.092989 24323 rpcz_store.cc:269] 0809 23:19:31.518257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.071699 (+2553442us) service_pool.cc:227] Handling call
0809 23:19:34.092464 (+ 20765us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.093991 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.055835 29993 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.096304 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.071985 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.097769 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:34.057416 24542 rpcz_store.cc:269] 0809 23:19:31.894697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.121988 (+227291us) service_pool.cc:227] Handling call
0809 23:19:33.992230 (+1870242us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:33.992397 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1949199104}
I20260809 23:19:34.024192 29705 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:33.955627 29120 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.059922 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.031322 29758 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.019924 29889 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.050659 29682 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:34.091523 29878 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.000s
I20260809 23:19:34.023926 29666 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.098904 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:34.059238 30002 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
W20260809 23:19:34.075366 29305 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.535s	user 0.009s	sys 0.003s
I20260809 23:19:34.134572 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.152686 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.171600 24311 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:34.100013 29808 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:34.129150 29705 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:34.205449 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70984.9us
I20260809 23:19:34.201881 29369 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.099725 30021 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:34.132326 29616 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:34.229254 30021 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:34.230232 30021 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.188303 29808 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:34.231930 29808 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.207412 29305 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.236521 24540 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:34.237349 29305 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.238590 29877 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:34.239171 29877 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:34.227461 30017 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260809 23:19:34.244067 29770 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.244678 29666 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.247565 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.248344 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:34.249897 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:34.280889 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2502) took 2152 ms. Trace:
I20260809 23:19:34.282241 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.284870 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.281358 24543 rpcz_store.cc:276] 0809 23:19:32.128863 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.221898 (+2093035us) service_pool.cc:227] Handling call
0809 23:19:34.242039 (+ 20141us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.280236 (+ 38197us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:34.280264 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:34.280312 (+    48us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:34.280365 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.280437 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.280826 (+   389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.281898 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:34.256562 29615 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.522s	user 0.009s	sys 0.000s
W20260809 23:19:34.262207 29753 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.632s	user 0.009s	sys 0.001s
W20260809 23:19:34.283823 29747 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.773s	user 0.012s	sys 0.000s
I20260809 23:19:34.250175 29765 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:34.301259 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 278424us
I20260809 23:19:34.302829 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.280777 30016 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:34.306735 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.286630 29926 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:34.286475 29369 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:34.331460 29369 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:34.332038 29369 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:34.335314 29931 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.335731 29962 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.335440 29616 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.331202 29926 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:34.337667 29987 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.937s	user 0.006s	sys 0.004s
I20260809 23:19:34.338423 29926 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.338176 29987 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.341565 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.322006 24316 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:34.322177 24311 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.323343 29622 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:34.284687 29337 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.252193 29342 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.324744 29747 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.322990 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:34.328810 29991 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.485586 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:34.341956 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.342746 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.487941 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:34.494022 24542 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:34.331236 29889 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.391585 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1089) took 2817 ms. Trace:
I20260809 23:19:34.304585 29878 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:34.497511 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:34.342495 29760 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.623s	user 0.005s	sys 0.005s
W20260809 23:19:34.381273 29212 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.749s	user 0.008s	sys 0.001s
I20260809 23:19:34.263473 29808 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.441993 28841 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.463969 30009 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.525698 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.413533 29666 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:34.528764 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.439921 29877 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.477361 29120 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:34.499642 25371 rpcz_store.cc:276] 0809 23:19:31.574437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.574663 (+   226us) service_pool.cc:227] Handling call
0809 23:19:31.577160 (+  2497us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:31.629587 (+ 52427us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:31.629615 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:31.629706 (+    91us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:31.629748 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.391318 (+2761570us) spinlock_profiling.cc:243] Waited 52.2 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:34.391368 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.391510 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":120099200}
I20260809 23:19:34.527478 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.527370 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.496080 29991 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:34.303076 29615 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.321306 29753 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.414572 29705 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.409788 29935 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.296482 29879 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.527041 29212 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.494671 29622 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.563681 29212 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.564319 29532 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.564958 24424 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:34.567118 29705 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.521236 29760 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.569804 29625 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:34.572703 29625 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:34.546654 30007 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.754s	user 0.000s	sys 0.004s
I20260809 23:19:34.573442 29615 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.501323 29878 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:34.574411 29747 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.574496 29878 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.532148 29770 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:34.555418 29991 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.582470 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2503) took 2453 ms. Trace:
I20260809 23:19:34.584995 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.585263 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.585737 24311 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:34.586051 25369 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.574154 29625 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.414089 29369 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.570045 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 700) took 1134 ms. Trace:
W20260809 23:19:34.573489 30005 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.730s	user 0.000s	sys 0.002s
I20260809 23:19:34.583328 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.558274 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.571908 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1095) took 1726 ms. Trace:
I20260809 23:19:34.551354 29120 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.626793 24428 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260809 23:19:34.618284 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2360) took 1382 ms (client timeout 1586 ms). Trace:
I20260809 23:19:34.584000 24543 rpcz_store.cc:276] 0809 23:19:32.129260 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.296341 (+2167081us) service_pool.cc:227] Handling call
0809 23:19:34.339985 (+ 43644us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:34.340442 (+   457us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:34.340482 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:34.368212 (+ 27730us) log.cc:842] Serialized 26 byte log entry
0809 23:19:34.496537 (+128325us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:34.581536 (+ 84999us) log.cc:842] Serialized 14 byte log entry
0809 23:19:34.582080 (+   544us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.582181 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:34.582238 (+    57us) raft_consensus.cc:1671] finished
0809 23:19:34.582256 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.582401 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2078,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6092,"thread_start_us":191971,"threads_started":2,"wal-append.queue_time_us":153703}
W20260809 23:19:34.636432 24540 rpcz_store.cc:269] 0809 23:19:33.236045 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.256620 (+ 20575us) service_pool.cc:227] Handling call
0809 23:19:34.618091 (+1361471us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:34.618223 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2057285504}
I20260809 23:19:34.622992 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.639427 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 689) took 2585 ms. Trace:
I20260809 23:19:34.639299 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.639822 24311 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:34.639892 24428 rpcz_store.cc:276] 0809 23:19:32.053585 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.626147 (+2572562us) service_pool.cc:227] Handling call
0809 23:19:34.626454 (+   307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.636238 (+  9784us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.639363 (+  3125us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:34.640597 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1102) took 1642 ms (client timeout 1885 ms). Trace:
W20260809 23:19:34.641011 25372 rpcz_store.cc:269] 0809 23:19:32.998665 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.620461 (+1621796us) service_pool.cc:227] Handling call
0809 23:19:34.640543 (+ 20082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.572072 29337 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.588716 24429 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:34.620882 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.646904 25369 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:34.648597 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:34.639746 25373 rpcz_store.cc:276] 0809 23:19:32.845952 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.846372 (+   420us) service_pool.cc:227] Handling call
0809 23:19:34.571728 (+1725356us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:34.571825 (+    97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1119940864}
I20260809 23:19:34.625972 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 695) took 1254 ms. Trace:
I20260809 23:19:34.650245 24435 rpcz_store.cc:276] 0809 23:19:33.371583 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.376942 (+  5359us) service_pool.cc:227] Handling call
0809 23:19:33.377328 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.518454 (+141126us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.625707 (+1107253us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:34.625912 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225142144}
I20260809 23:19:34.635556 24424 rpcz_store.cc:276] 0809 23:19:33.436028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.564603 (+128575us) service_pool.cc:227] Handling call
0809 23:19:33.572919 (+  8316us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.565437 (+992518us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.569811 (+  4374us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:34.569984 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2280911232}
I20260809 23:19:34.657804 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.578668 29770 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:34.661705 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.659626 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.658061 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.665238 24543 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:34.659091 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.666603 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.643425 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:34.666611 25369 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.668277 25371 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.667595 24316 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:34.670221 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.612644 29808 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.624020 29987 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.620496 29753 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.667507 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:34.674031 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:34.674997 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.679764 24542 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:34.680116 25372 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.688714 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:34.680686 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.625793 24170 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 6, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 6 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:34.622991 29868 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.684973 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.646641 29625 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.682850 29751 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:34.691303 24540 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.657594 29760 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:34.661581 29823 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.643699 29834 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:34.695189 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.696110 29808 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:34.697001 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.705000 24543 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.674451 29770 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:34.707998 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.700143 25371 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:34.700074 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:34.709656 25369 raft_consensus.cc:595] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:34.710091 25369 raft_consensus.cc:603] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:34.710093 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:34.711026 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 931) took 1052 ms. Trace:
I20260809 23:19:34.711266 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.711436 25371 rpcz_store.cc:276] 0809 23:19:33.658563 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.710249 (+1051686us) service_pool.cc:227] Handling call
0809 23:19:34.710532 (+   283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.710710 (+   178us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:34.710733 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:34.710773 (+    40us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:34.710812 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.710860 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.710976 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.715247 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 922) took 1358 ms. Trace:
I20260809 23:19:34.715638 24312 rpcz_store.cc:276] 0809 23:19:33.356893 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.368111 (+ 11218us) service_pool.cc:227] Handling call
0809 23:19:33.368511 (+   400us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:33.368852 (+   341us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:33.368879 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:33.369318 (+   439us) log.cc:842] Serialized 26 byte log entry
0809 23:19:33.644413 (+275095us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:33.644453 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:33.644515 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:34.572931 (+928416us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:34.708671 (+135740us) raft_consensus.cc:1671] finished
0809 23:19:34.708692 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.715170 (+  6478us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":368,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"spinlock_wait_cycles":5784320,"thread_start_us":274711,"threads_started":1,"wal-append.queue_time_us":362165}
I20260809 23:19:34.717564 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.718287 24312 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.719413 29935 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.719709 29991 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.719535 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.706948 24170 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:34.727702 25369 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:34.728049 24170 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 6, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 6 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:34.729254 25369 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:34.730468 29974 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.007s	user 0.000s	sys 0.009s
I20260809 23:19:34.721662 29120 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.700933 29751 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:34.731732 29751 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.732331 24425 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:34.733053 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2094) took 3064 ms. Trace:
I20260809 23:19:34.733335 24425 rpcz_store.cc:276] 0809 23:19:31.669116 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.731910 (+3062794us) service_pool.cc:227] Handling call
0809 23:19:34.732169 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.732833 (+   664us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.732999 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.734365 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 787) took 3004 ms. Trace:
I20260809 23:19:34.734720 24545 rpcz_store.cc:276] 0809 23:19:31.729601 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.443125 (+713524us) service_pool.cc:227] Handling call
0809 23:19:32.443521 (+   396us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:32.472976 (+ 29455us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:32.472999 (+    23us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:32.519277 (+ 46278us) log.cc:842] Serialized 52 byte log entry
0809 23:19:32.624854 (+105577us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:32.624907 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.624989 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:34.729899 (+2104910us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:34.734049 (+  4150us) raft_consensus.cc:1671] finished
0809 23:19:34.734071 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.734302 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1293,"raft.queue_time_us":13480,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":3041,"spinlock_wait_cycles":9673472,"thread_start_us":76299,"threads_started":1,"wal-append.queue_time_us":115990}
I20260809 23:19:34.736545 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.737105 24545 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:34.737625 25370 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.722813 29369 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:34.746048 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1103) took 1747 ms. Trace:
I20260809 23:19:34.729923 24170 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 5, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 5 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:34.722273 29878 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.755651 25371 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:34.762841 29481 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:34.775928 25371 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:34.776216 29481 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:34.776787 25371 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.777297 29481 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.777187 24170 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 6 to 9, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:34.752455 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:34.778780 29481 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:34.779193 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.759722 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:34.780432 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.780987 29878 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:34.754022 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44009.6us
I20260809 23:19:34.768460 29751 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.756803 24428 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:34.775579 24312 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:34.782783 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:34.749037 29815 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:34.778015 25371 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.761695 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.784703 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.712073 24316 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:34.762162 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:34.750803 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2483) took 3957 ms (client timeout 2879 ms). Trace:
I20260809 23:19:34.696676 29666 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:34.751848 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:34.760537 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2580) took 2073 ms (client timeout 1608 ms). Trace:
I20260809 23:19:34.773984 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.787318 29935 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:34.799582 24423 rpcz_store.cc:269] 0809 23:19:32.686719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.760263 (+2073544us) service_pool.cc:214] Skipping call since client already timed out
0809 23:19:34.760427 (+   164us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:19:34.789983 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.765002 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 696) took 1381 ms. Trace:
I20260809 23:19:34.797883 24431 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:34.799253 24535 rpcz_store.cc:269] 0809 23:19:30.793566 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:30.794253 (+   687us) service_pool.cc:227] Handling call
0809 23:19:34.750605 (+3956352us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:34.750717 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2141332224}
I20260809 23:19:34.761229 24424 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:34.790944 29801 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:34.803041 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:34.779076 24170 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 5 to 6, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 6 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:34.791925 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:34.746452 25365 rpcz_store.cc:276] 0809 23:19:32.999088 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.002195 (+  3107us) service_pool.cc:227] Handling call
0809 23:19:33.002655 (+   460us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:33.275797 (+273142us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.745280 (+1469483us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b540071c3a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:34.745827 (+   547us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:34.745990 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":288817408}
I20260809 23:19:34.800731 24427 rpcz_store.cc:276] 0809 23:19:33.383456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.383624 (+   168us) service_pool.cc:227] Handling call
0809 23:19:33.387952 (+  4328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.388191 (+   239us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:33.614200 (+226009us) log.cc:842] Serialized 14 byte log entry
0809 23:19:34.764367 (+1150167us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b18004a54e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:19:34.764598 (+   231us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:34.764649 (+    51us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:34.764715 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.764799 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.764933 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61377,"prepare.queue_time_us":351496,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":837,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":3445504,"thread_start_us":352140,"threads_started":2,"wal-append.queue_time_us":252292}
I20260809 23:19:34.751142 29808 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:34.806990 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.807646 24312 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.789419 29751 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.800794 24540 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.781203 24543 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:34.804032 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.814484 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.812660 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.817028 24423 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:34.778635 29770 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.814548 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.818022 24545 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:34.818328 25371 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:19:34.826649 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.815795 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:34.818591 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:34.827478 24378 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.054s (SENT)
I20260809 23:19:34.827329 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.819303 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.828400 24431 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:34.819617 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:34.829079 25365 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.820353 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.830559 25369 raft_consensus.cc:595] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:34.831097 25369 raft_consensus.cc:603] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:34.832535 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.832686 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.836745 24170 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 6, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 6 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:34.837214 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.838968 24535 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:34.840358 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.808434 29991 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:34.868294 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.801738 29666 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:34.835850 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:34.901949 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.885193 24425 raft_consensus.cc:2354] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:34.893594 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.887869 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2340) took 2716 ms. Trace:
W20260809 23:19:34.885411 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52956us
I20260809 23:19:34.929600 24543 rpcz_store.cc:276] 0809 23:19:32.171253 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.882830 (+2711577us) service_pool.cc:227] Handling call
0809 23:19:34.883209 (+   379us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.887520 (+  4311us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:34.887549 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:34.887592 (+    43us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:34.887635 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.887699 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:34.887800 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.886227 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.895486 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.893826 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:34.926501 24312 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:34.931080 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:34.933156 24424 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.926805 24170 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 5, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 5 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:19:34.888067 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47804.3us
I20260809 23:19:34.935056 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:34.927662 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 766213us
I20260809 23:19:34.909840 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260809 23:19:34.924911 29467 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.249s	user 0.014s	sys 0.005s
I20260809 23:19:34.827419 29878 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:34.814719 29801 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:34.937134 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.069s (SENT)
I20260809 23:19:34.924578 24540 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:34.942181 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.894079 28841 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:34.943924 28841 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:34.913355 29610 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260809 23:19:34.944974 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2342) took 2773 ms. Trace:
I20260809 23:19:34.945379 24540 rpcz_store.cc:276] 0809 23:19:32.171825 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.924134 (+2752309us) service_pool.cc:227] Handling call
0809 23:19:34.924409 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.944729 (+ 20320us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.944920 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.946033 29878 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:34.955987 24430 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 693) took 1585 ms (client timeout 2058 ms). Trace:
W20260809 23:19:34.956532 24430 rpcz_store.cc:269] 0809 23:19:33.370819 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.375983 (+  5164us) service_pool.cc:227] Handling call
0809 23:19:34.955823 (+1579840us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:34.955931 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637605120}
I20260809 23:19:34.957655 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.958508 29878 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:34.959918 29878 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:34.960563 29878 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.938663 24543 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:34.887744 29791 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:34.869596 29751 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:34.964478 29791 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:34.826735 29808 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:34.946900 29987 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:19:34.966365 29987 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:34.966805 29987 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:34.965971 29791 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.938591 29204 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:34.948418 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 928) took 1553 ms (client timeout 1959 ms). Trace:
W20260809 23:19:34.971223 25363 rpcz_store.cc:269] 0809 23:19:33.396981 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.587898 (+190917us) service_pool.cc:227] Handling call
0809 23:19:34.948137 (+1360239us) spinlock_profiling.cc:243] Waited 93.6 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:34.948341 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":215322496}
I20260809 23:19:34.971019 24170 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 4, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 4 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:34.948228 29801 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:34.949312 29212 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:34.949913 29725 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:34.953457 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:34.975374 24535 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
W20260809 23:19:34.939393 29467 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.283s	user 0.014s	sys 0.006s
I20260809 23:19:34.948143 24540 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:34.905902 29935 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:34.968578 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2343) took 2783 ms. Trace:
I20260809 23:19:34.938443 24320 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:34.981604 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2504) took 2748 ms. Trace:
I20260809 23:19:34.981699 24311 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:34.972585 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.972775 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.983819 24540 rpcz_store.cc:276] 0809 23:19:32.233151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.947581 (+2714430us) service_pool.cc:227] Handling call
0809 23:19:34.947978 (+   397us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.981346 (+ 33368us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.981550 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.974674 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.954638 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.981560 24543 rpcz_store.cc:276] 0809 23:19:32.185729 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.930704 (+2744975us) service_pool.cc:227] Handling call
0809 23:19:34.931086 (+   382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.968241 (+ 37155us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:34.968517 (+   276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:34.975288 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.977878 24423 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:34.927227 29666 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:34.986747 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:34.987486 24543 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:34.978587 29204 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:34.978776 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:34.983682 24312 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:34.974277 25365 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:34.987767 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:34.940866 29991 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:34.988720 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.978317 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:34.993613 24535 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:34.994895 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:34.995795 24423 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8.
I20260809 23:19:34.995679 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:34.996840 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:34.999833 24170 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:34.982697 29467 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:35.007418 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.009272 24323 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.005349 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:34.999346 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:35.006016 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.010356 24312 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:35.011757 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.012697 24316 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:35.004751 29204 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.015748 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.995101 29625 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:35.017778 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:34.989239 28862 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:35.026504 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.025507 29352 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:35.024070 24316 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:35.029207 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.028966 24535 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:35.030074 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:35.054745 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 750167us
I20260809 23:19:35.019672 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:35.010686 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.030344 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:35.058962 24543 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:34.984552 29935 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.059983 24316 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:35.033591 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2507) took 2773 ms. Trace:
I20260809 23:19:34.994799 29878 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.035444 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.060266 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:35.036528 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.025715 29625 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:35.046248 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.029062 29868 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260809 23:19:35.048396 29467 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.008230 29610 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:35.028582 29212 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260809 23:19:35.050995 24170 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:19:35.052189 29970 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.415s	user 0.015s	sys 0.010s
I20260809 23:19:35.033344 28862 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:34.968048 29987 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.061568 24535 rpcz_store.cc:276] 0809 23:19:32.259748 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.996639 (+2736891us) service_pool.cc:227] Handling call
0809 23:19:35.028771 (+ 32132us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.033308 (+  4537us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.033534 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.028947 29770 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.062323 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.063997 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:35.070869 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.069444 24431 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:19:35.076699 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 753755us
I20260809 23:19:35.069990 24317 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 4
I20260809 23:19:35.089679 24317 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260809 23:19:35.089980 24317 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:35.078222 24312 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:19:35.078322 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 910) took 3111 ms (client timeout 1880 ms). Trace:
W20260809 23:19:35.083377 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63813.4us
W20260809 23:19:35.091212 24321 rpcz_store.cc:269] 0809 23:19:31.958594 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.958840 (+   246us) service_pool.cc:227] Handling call
0809 23:19:35.069477 (+3110637us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:35.069604 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":338564480}
I20260809 23:19:35.061950 25365 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.088442 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.095870 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.098348 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.103822 29987 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:35.107221 29878 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:35.107689 29878 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:35.108101 29878 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:35.090437 29342 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:35.110555 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.111879 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.113029 29868 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:35.113121 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.113490 29868 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:35.113778 24435 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:35.113880 29745 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.091236 29751 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:35.110743 29342 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:35.114665 29745 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.115589 29342 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:35.116356 24422 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:35.116089 24170 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 4 to 5, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 5 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:35.082854 29204 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:35.120818 29204 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:35.111927 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.114593 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.172241 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.128098 24424 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:35.122602 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.128736 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.180395 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.189133 24312 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:35.174679 24266 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.809s (SENT)
I20260809 23:19:35.124612 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.194111 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:35.157895 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:35.136155 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62615.9us
I20260809 23:19:35.114311 29868 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.192341 24431 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260809 23:19:35.190994 24435 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:35.077683 29742 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:35.158414 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.198963 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.192656 24320 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:35.228089 24321 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.223538 24424 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10. Re-sending same reply.
I20260809 23:19:35.232281 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 914) took 3047 ms. Trace:
I20260809 23:19:35.233798 29610 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:35.232764 24314 rpcz_store.cc:276] 0809 23:19:32.184777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.184914 (+   137us) service_pool.cc:227] Handling call
0809 23:19:32.240064 (+ 55150us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:32.256832 (+ 16768us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:32.256888 (+    56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:32.278378 (+ 21490us) log.cc:842] Serialized 26 byte log entry
0809 23:19:32.517160 (+238782us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:32.517208 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:32.517284 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:33.265307 (+748023us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:35.051573 (+1786266us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:35.232090 (+180517us) raft_consensus.cc:1671] finished
0809 23:19:35.232111 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:35.232229 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9058,"raft.queue_time_us":17616,"raft.run_cpu_time_us":3531,"raft.run_wall_time_us":998963,"spinlock_wait_cycles":24193536,"thread_start_us":184411,"threads_started":1,"wal-append.queue_time_us":320351}
I20260809 23:19:35.236675 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.235502 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.234961 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.237327 24427 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:35.121825 29204 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.070957 29808 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:35.240466 24266 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.567s (SENT)
I20260809 23:19:35.242755 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.240437 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:35.239430 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.191395 29815 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:35.239743 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.232367 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.200462 29305 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.174106 29615 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 23:19:35.243496 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:35.347333 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.348295 25371 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.304131 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.250504 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.341158 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.321980 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.349848 24431 raft_consensus.cc:2362] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11. Re-sending same reply.
I20260809 23:19:35.350014 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.246768 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.224478 29765 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:35.248111 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2508) took 2985 ms. Trace:
I20260809 23:19:35.243446 24435 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10. Re-sending same reply.
I20260809 23:19:35.352725 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.243539 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.322383 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.353914 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.352003 24543 rpcz_store.cc:276] 0809 23:19:32.262364 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.060675 (+2798311us) service_pool.cc:227] Handling call
0809 23:19:35.060969 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.061170 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:35.154700 (+ 93530us) log.cc:842] Serialized 14 byte log entry
0809 23:19:35.247445 (+ 92745us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:35.247497 (+    52us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:35.247555 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.247617 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:35.248055 (+   438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17844,"prepare.queue_time_us":58511,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":41,"raft.queue_time_us":8923,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8987,"thread_start_us":83512,"threads_started":1,"wal-append.queue_time_us":193132}
I20260809 23:19:35.231123 28841 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
W20260809 23:19:35.355773 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 291845us
I20260809 23:19:35.243638 24321 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:35.313946 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.354267 25370 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:35.358146 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 117714us
I20260809 23:19:35.359164 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:35.354054 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.363011 24312 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:35.396035 24543 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260809 23:19:35.396438 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.242399 29742 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:35.245625 29610 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.398694 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.239271 29725 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.399451 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.430771 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.399711 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.400435 25370 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:35.431463 24435 raft_consensus.cc:2362] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11. Re-sending same reply.
I20260809 23:19:35.241734 29625 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.432521 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.401230 29823 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:35.309723 29856 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:35.399451 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.403765 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.433215 24435 raft_consensus.cc:2362] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9. Re-sending same reply.
I20260809 23:19:35.405370 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.405745 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.439383 25369 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
I20260809 23:19:35.406580 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.173957 29991 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.407536 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2510) took 3035 ms. Trace:
I20260809 23:19:35.241679 29868 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:35.407557 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:35.401748 29615 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.409435 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:35.396348 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 2.34862e+06us
I20260809 23:19:35.470311 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.471520 29718 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:35.425915 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:35.435873 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:35.402163 29797 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.477567 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2580) took 1099 ms. Trace:
I20260809 23:19:35.440964 24427 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:35.409909 29688 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.457103 25363 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:35.457824 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.459210 24543 rpcz_store.cc:276] 0809 23:19:32.371817 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.394577 (+3022760us) service_pool.cc:227] Handling call
0809 23:19:35.395787 (+  1210us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:35.407196 (+ 11409us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.407471 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.461982 24320 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260809 23:19:35.462708 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.340874 29369 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:35.395495 30077 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:35.477022 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.481738 24269 leader_election.cc:398] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
I20260809 23:19:35.470167 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2571) took 1147 ms. Trace:
I20260809 23:19:35.483153 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.485606 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:35.477931 24311 rpcz_store.cc:276] 0809 23:19:34.377782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.698902 (+321120us) service_pool.cc:227] Handling call
0809 23:19:35.477411 (+778509us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:35.477513 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627827456}
W20260809 23:19:35.530282 25325 connection.cc:232] RPC call timeout handler was delayed by -1.9399312930181623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
I20260809 23:19:35.531947 25372 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260809 23:19:35.437749 29856 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:35.533336 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.500321 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.488612 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2586) took 1108 ms. Trace:
I20260809 23:19:35.534149 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:35.542793 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 995) took 2480 ms (client timeout 1933 ms). Trace:
I20260809 23:19:35.543967 25369 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.545030 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.546023 25369 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8.
I20260809 23:19:35.547107 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.548429 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.387821 29305 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.522850 24312 rpcz_store.cc:276] 0809 23:19:34.323022 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.409122 (+1086100us) service_pool.cc:227] Handling call
0809 23:19:35.470109 (+ 60987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.543725 24424 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:35.557811 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.558447 24435 raft_consensus.cc:2362] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9. Re-sending same reply.
I20260809 23:19:35.535238 24269 leader_election.cc:401] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260809 23:19:35.484838 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:35.559751 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2595) took 1177 ms. Trace:
I20260809 23:19:35.560221 24314 rpcz_store.cc:276] 0809 23:19:34.382228 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.410198 (+1027970us) service_pool.cc:227] Handling call
0809 23:19:35.559695 (+149497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.487562 24431 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:35.537531 24543 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:35.533576 29747 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:35.563203 29747 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:35.563571 29747 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:35.541061 29718 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.564528 29622 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.487015 29299 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:35.566341 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 141068us
I20260809 23:19:35.536469 29369 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.567796 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:35.543507 24320 rpcz_store.cc:276] 0809 23:19:34.379958 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.395597 (+1015639us) service_pool.cc:227] Handling call
0809 23:19:35.488534 (+ 92937us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.569561 29369 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:35.570482 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.571113 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.571353 24311 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:35.571981 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2601) took 1133 ms. Trace:
I20260809 23:19:35.572227 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2598) took 1135 ms. Trace:
I20260809 23:19:35.572340 24311 rpcz_store.cc:276] 0809 23:19:34.438220 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.523638 (+1085418us) service_pool.cc:227] Handling call
0809 23:19:35.571906 (+ 48268us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.572989 24320 rpcz_store.cc:276] 0809 23:19:34.436944 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.570728 (+1133784us) service_pool.cc:227] Handling call
0809 23:19:35.572140 (+  1412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.573843 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.574697 24311 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:35.576499 30075 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:35.577008 30075 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:35.577893 30075 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.535813 29800 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:35.578950 29800 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.579809 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 928) took 1253 ms. Trace:
I20260809 23:19:35.580180 24320 rpcz_store.cc:276] 0809 23:19:34.326231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.573852 (+1247621us) service_pool.cc:227] Handling call
0809 23:19:35.579298 (+  5446us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.579499 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:35.579526 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:35.579567 (+    41us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:35.579603 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.579654 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:35.579758 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:35.581012 29800 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:35.585491 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:35.585413 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.588052 25371 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:35.585678 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:35.543751 25366 rpcz_store.cc:269] 0809 23:19:33.050639 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.208103 (+157464us) service_pool.cc:227] Handling call
0809 23:19:35.530762 (+2322659us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:35.531019 (+   257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14545792}
W20260809 23:19:35.572708 30069 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
I20260809 23:19:35.595326 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.599013 28862 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.533169 29935 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:35.599911 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 798) took 2592 ms (client timeout 2982 ms). Trace:
I20260809 23:19:35.470136 30025 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.142s
W20260809 23:19:35.600301 24534 rpcz_store.cc:269] 0809 23:19:33.007610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.705212 (+697602us) service_pool.cc:227] Handling call
0809 23:19:35.599705 (+1894493us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:35.599846 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1309521152}
I20260809 23:19:35.600771 29720 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:35.601222 29720 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:35.601328 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:35.602097 29720 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.523603 29688 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.608839 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.609573 29688 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.610014 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:35.611119 24424 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:35.604521 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:35.611023 30076 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.122s	user 0.002s	sys 0.000s
I20260809 23:19:35.566526 29718 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:35.612269 30079 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20260809 23:19:35.605587 29666 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.003s
I20260809 23:19:35.613680 29745 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:35.613376 29666 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.484355 29760 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260809 23:19:35.614626 24428 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:35.615127 30060 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.933s	user 0.003s	sys 0.000s
W20260809 23:19:35.616688 30055 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.934s	user 0.002s	sys 0.000s
I20260809 23:19:35.617013 29666 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:35.606226 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.619153 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.610823 25371 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:35.479619 29615 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.549409 29856 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:35.612366 25366 raft_consensus.cc:2396] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260809 23:19:35.459743 29725 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.620371 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:35.488273 29797 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.624948 24545 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:35.625356 24321 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:35.625974 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:35.644461 24312 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:35.602097 24534 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:35.627152 25370 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:19:35.646492 25370 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:35.648464 24424 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:35.648947 24424 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:35.655354 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 561527us
I20260809 23:19:35.656527 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:35.629171 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:35.630851 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:35.631204 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63485us
I20260809 23:19:35.664220 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:35.589706 29299 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:35.640097 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:35.622898 29823 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
I20260809 23:19:35.617178 29742 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.641878 24311 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:35.597203 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 115617us
I20260809 23:19:35.873019 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:35.873503 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 931) took 1546 ms. Trace:
I20260809 23:19:35.873864 24311 rpcz_store.cc:276] 0809 23:19:34.327077 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.591884 (+1264807us) service_pool.cc:227] Handling call
0809 23:19:35.641671 (+ 49787us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.873239 (+231568us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.873450 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:35.642470 29987 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.538s	user 0.008s	sys 0.002s
I20260809 23:19:35.679621 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:35.883679 24543 raft_consensus.cc:2354] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9.
I20260809 23:19:35.884613 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:35.649878 24424 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:35.886186 24424 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:35.704245 24498 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.608s (SENT)
I20260809 23:19:35.888630 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:35.872718 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 224556us
I20260809 23:19:35.891157 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:35.611017 29720 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:35.643198 30009 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 23:19:35.920060 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 263615us
I20260809 23:19:35.893795 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:35.875489 24427 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:35.880605 24496 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.569s (SENT)
I20260809 23:19:35.893285 29728 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:35.920859 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 256723us
W20260809 23:19:35.980786 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.011s	sys 0.039s
W20260809 23:19:35.981999 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.012s	sys 0.039s
I20260809 23:19:35.923302 24323 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 23:19:35.923575 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:35.642813 29718 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:35.878724 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:35.929530 29995 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:35.944661 29467 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:35.935137 29783 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:35.942657 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 238472us
I20260809 23:19:35.949064 24311 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:35.581027 29747 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:35.922878 29878 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.814s	user 0.011s	sys 0.001s
W20260809 23:19:35.956032 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.032s	sys 0.058s
I20260809 23:19:35.978300 29987 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.920792 29688 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:36.082916 29743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.012s	sys 0.050s
I20260809 23:19:35.879477 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:35.977026 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:36.267210 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.031s	sys 0.082s
W20260809 23:19:36.268148 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.031s	sys 0.083s
I20260809 23:19:35.921829 30009 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:35.920063 29823 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:35.930900 29299 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:35.989219 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:35.988420 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.026595 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:36.036489 24424 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:35.948181 29120 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:36.284582 24543 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:36.286020 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.287253 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 937) took 1046 ms. Trace:
I20260809 23:19:36.287696 24312 rpcz_store.cc:276] 0809 23:19:35.241262 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.556357 (+315095us) service_pool.cc:227] Handling call
0809 23:19:36.287176 (+730819us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:36.089157 24498 leader_election.cc:334] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.523s (SENT)
W20260809 23:19:36.289489 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 200365us
I20260809 23:19:36.091909 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.290594 24431 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:36.095736 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2574) took 1719 ms. Trace:
W20260809 23:19:36.207226 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.032s	sys 0.058s
I20260809 23:19:36.291778 24311 rpcz_store.cc:276] 0809 23:19:34.376032 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.893532 (+1517500us) service_pool.cc:227] Handling call
0809 23:19:35.948807 (+ 55275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.095381 (+146574us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.095660 (+   279us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:35.957623 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2495) took 4044 ms (client timeout 1948 ms). Trace:
W20260809 23:19:36.292907 24539 rpcz_store.cc:269] 0809 23:19:31.913184 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.100783 (+187599us) service_pool.cc:227] Handling call
0809 23:19:35.957414 (+3856631us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:35.957539 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":843607424}
W20260809 23:19:36.207051 25365 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.039s	sys 0.064s
I20260809 23:19:36.210983 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:36.294179 25365 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.039s	sys 0.064s
I20260809 23:19:36.303099 24534 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:36.263080 25363 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.017s	sys 0.096s
W20260809 23:19:35.963910 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72849.4us
I20260809 23:19:36.306457 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:35.992300 29995 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:36.267937 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 387348us
W20260809 23:19:36.309571 24496 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.582s (SENT)
I20260809 23:19:36.310045 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:35.878278 25366 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:35.990863 29450 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
W20260809 23:19:36.292068 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 896336us
I20260809 23:19:36.207165 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.314903 24312 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:36.307071 25363 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.017s	sys 0.096s
I20260809 23:19:36.306707 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:36.266830 29743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.013s	sys 0.050s
I20260809 23:19:35.944926 30074 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.263s
I20260809 23:19:35.990585 29728 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.278112 29968 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:36.320801 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.338658 24431 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:36.285270 30079 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
W20260809 23:19:36.323748 24498 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.598s (SENT)
I20260809 23:19:36.308265 24424 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:36.344502 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:36.344779 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.345302 24539 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 3.
I20260809 23:19:36.346210 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.271539 30009 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:36.324643 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:35.993738 29718 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.309044 29791 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.329340 29409 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:36.343822 29991 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.028s	sys 0.097s
I20260809 23:19:36.361744 24314 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:36.351513 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:36.362102 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.320654 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2575) took 1944 ms. Trace:
I20260809 23:19:36.353569 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:36.362843 24312 rpcz_store.cc:276] 0809 23:19:34.376376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.309972 (+1933596us) service_pool.cc:227] Handling call
0809 23:19:36.310403 (+   431us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.320349 (+  9946us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.320584 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.312868 29995 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:36.208346 29783 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.263188 29878 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.312904 29450 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:36.266983 29987 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.270062 29688 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:36.376180 24435 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.066s	sys 0.060s
I20260809 23:19:36.372555 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.383435 24543 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:36.384171 24435 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.066s	sys 0.060s
I20260809 23:19:36.283602 29823 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.384351 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2354) took 3719 ms. Trace:
I20260809 23:19:36.385268 24543 rpcz_store.cc:276] 0809 23:19:32.665386 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.353891 (+3688505us) service_pool.cc:227] Handling call
0809 23:19:36.377311 (+ 23420us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.384119 (+  6808us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.384303 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.337870 29120 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.343305 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:36.361783 29991 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.028s	sys 0.098s
I20260809 23:19:36.398269 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:36.363840 29725 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:36.405009 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61877.2us
I20260809 23:19:36.406208 24312 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:36.409489 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2578) took 2032 ms. Trace:
I20260809 23:19:36.409745 24320 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:36.410014 24312 rpcz_store.cc:276] 0809 23:19:34.377209 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.369720 (+1992511us) service_pool.cc:227] Handling call
0809 23:19:36.406001 (+ 36281us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.409170 (+  3169us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.409423 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.410732 24534 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:36.366137 29797 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:36.373680 30074 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260809 23:19:36.413790 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2516) took 3662 ms. Trace:
I20260809 23:19:36.370221 30085 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:36.416003 24534 rpcz_store.cc:276] 0809 23:19:32.750983 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.379431 (+3628448us) service_pool.cc:227] Handling call
0809 23:19:36.405273 (+ 25842us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.413503 (+  8230us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.413723 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.414372 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:36.418726 24534 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:36.413793 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1002) took 3020 ms. Trace:
I20260809 23:19:36.393700 29615 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.511538 24311 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:36.520556 25373 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
W20260809 23:19:36.415508 29625 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.026s	sys 0.060s
I20260809 23:19:36.412705 29968 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election won for term 7
W20260809 23:19:36.517198 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 925) took 3125 ms (client timeout 2153 ms). Trace:
W20260809 23:19:36.549813 25368 rpcz_store.cc:269] 0809 23:19:33.391709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.631469 (+239760us) service_pool.cc:227] Handling call
0809 23:19:36.516986 (+2885517us) spinlock_profiling.cc:243] Waited 86.9 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.517113 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199955840}
I20260809 23:19:36.466320 25375 rpcz_store.cc:276] 0809 23:19:33.393204 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.931923 (+538719us) service_pool.cc:227] Handling call
0809 23:19:33.961939 (+ 30016us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:34.028538 (+ 66599us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:34.072996 (+ 44458us) log.cc:842] Serialized 14 byte log entry
0809 23:19:34.299313 (+226317us) log.cc:842] Serialized 14 byte log entry
0809 23:19:34.692785 (+393472us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b18000a67e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:35.042586 (+349801us) spinlock_profiling.cc:243] Waited 7.71 ms on lock 0x7b18000a67e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:35.045004 (+  2418us) spinlock_profiling.cc:243] Waited 2.91 ms on lock 0x7b18000a67e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:19:35.045212 (+   208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:35.045271 (+    59us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:35.045342 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.413559 (+1368217us) spinlock_profiling.cc:243] Waited 66.4 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:36.413607 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.413723 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":314453,"prepare.queue_time_us":48020,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":92,"raft.queue_time_us":747075,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":18091,"spinlock_wait_cycles":231031680,"thread_start_us":225295,"threads_started":1,"wal-append.queue_time_us":304413,"wal-append.run_cpu_time_us":1467,"wal-append.run_wall_time_us":1018495}
I20260809 23:19:36.407051 29172 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:36.525967 24312 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:19:36.559796 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2581) took 2181 ms. Trace:
I20260809 23:19:36.560185 24312 rpcz_store.cc:276] 0809 23:19:34.378093 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.411147 (+2033054us) service_pool.cc:227] Handling call
0809 23:19:36.525644 (+114497us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.559481 (+ 33837us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.559730 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.463730 30085 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:36.407603 29204 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
W20260809 23:19:36.530916 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 113011us
I20260809 23:19:36.579138 29204 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:36.579357 29783 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:36.534416 24543 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:36.412026 29666 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:36.513499 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2521) took 3173 ms. Trace:
I20260809 23:19:36.580700 24534 rpcz_store.cc:276] 0809 23:19:33.340105 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.417997 (+3077892us) service_pool.cc:227] Handling call
0809 23:19:36.418334 (+   337us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.465841 (+ 47507us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.513426 (+ 47585us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.580171 29204 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.412084 29409 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:36.432844 24155 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.015s	sys 0.044s
I20260809 23:19:36.587779 29409 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.530061 24423 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:36.589401 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.590065 29409 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.415047 30074 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:36.590907 30074 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:36.591420 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.592829 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.593613 24423 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:36.594462 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.598945 29718 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:36.599819 29718 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:36.512491 24498 leader_election.cc:334] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.757s (SENT)
I20260809 23:19:36.571429 24312 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:36.600418 29718 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.601495 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2605) took 2095 ms. Trace:
I20260809 23:19:36.601897 24312 rpcz_store.cc:276] 0809 23:19:34.506213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.561060 (+2054847us) service_pool.cc:227] Handling call
0809 23:19:36.571258 (+ 10198us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.601235 (+ 29977us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.601438 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.571997 30083 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:36.603138 30083 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:36.603224 29718 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.571269 29747 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:36.583204 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 801) took 3573 ms. Trace:
I20260809 23:19:36.370401 29791 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.606472 29791 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.607331 24424 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:36.607051 29450 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.587222 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2522) took 3115 ms. Trace:
I20260809 23:19:36.608250 24534 rpcz_store.cc:276] 0809 23:19:33.472336 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.581496 (+3109160us) service_pool.cc:227] Handling call
0809 23:19:36.581840 (+   344us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.582011 (+   171us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.582034 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.582101 (+    67us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:36.582144 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.582195 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.587158 (+  4963us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.609119 29878 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.550776 29743 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260809 23:19:36.609881 24534 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:19:36.610445 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2523) took 3041 ms. Trace:
W20260809 23:19:36.510649 29808 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s	user 0.025s	sys 0.044s
W20260809 23:19:36.611094 29808 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.025s	sys 0.044s
I20260809 23:19:36.603643 30083 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:36.532274 29625 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.028s	sys 0.060s
I20260809 23:19:36.610581 29728 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:36.616521 29688 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.405638 29120 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.584336 29968 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:36.622416 29120 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:36.623205 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 703) took 2579 ms (client timeout 1867 ms). Trace:
I20260809 23:19:36.623365 29968 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:36.623857 24429 rpcz_store.cc:269] 0809 23:19:34.043611 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.047203 (+  3592us) service_pool.cc:227] Handling call
0809 23:19:36.622852 (+2575649us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.623100 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1173505408}
I20260809 23:19:36.625209 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.628193 29987 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.582727 29666 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:36.629848 29666 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:36.630149 29853 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:36.632925 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2581) took 1877 ms (client timeout 1959 ms). Trace:
W20260809 23:19:36.633349 24425 rpcz_store.cc:269] 0809 23:19:34.755250 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.773678 (+ 18428us) service_pool.cc:227] Handling call
0809 23:19:36.632725 (+1859047us) spinlock_profiling.cc:243] Waited 85.2 ms on lock 0x7b54003eb7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.632861 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195967744}
I20260809 23:19:36.634346 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:36.635571 29868 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.162s	user 0.008s	sys 0.004s
W20260809 23:19:36.635890 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2592) took 1877 ms (client timeout 1582 ms). Trace:
W20260809 23:19:36.636276 24428 rpcz_store.cc:269] 0809 23:19:34.758779 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.784446 (+ 25667us) service_pool.cc:227] Handling call
0809 23:19:36.635680 (+1851234us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.635805 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1878750080}
I20260809 23:19:36.637216 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.638301 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.639451 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 934) took 1413 ms. Trace:
I20260809 23:19:36.628310 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2595) took 1837 ms. Trace:
I20260809 23:19:36.639808 24321 rpcz_store.cc:276] 0809 23:19:35.225667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.474678 (+249011us) service_pool.cc:227] Handling call
0809 23:19:36.639233 (+1164555us) spinlock_profiling.cc:243] Waited 89.8 ms on lock 0x7b54000dc5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.639360 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206439808}
I20260809 23:19:36.640106 24430 rpcz_store.cc:276] 0809 23:19:34.792020 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.957393 (+165373us) service_pool.cc:227] Handling call
0809 23:19:36.628127 (+1670734us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.628254 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":167800576}
I20260809 23:19:36.641201 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.641757 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2614) took 1944 ms. Trace:
I20260809 23:19:36.642114 24321 rpcz_store.cc:276] 0809 23:19:34.697546 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.640938 (+1943392us) service_pool.cc:227] Handling call
0809 23:19:36.641255 (+   317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.641464 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.641487 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.641524 (+    37us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.641566 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.641614 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.641707 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.628983 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.643242 24316 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:36.622485 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2369) took 2279 ms. Trace:
I20260809 23:19:36.643996 24545 rpcz_store.cc:276] 0809 23:19:34.342689 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.832459 (+489770us) service_pool.cc:227] Handling call
0809 23:19:36.622327 (+1789868us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b54000b4d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.622432 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1820190592}
I20260809 23:19:36.644949 24545 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
W20260809 23:19:36.588231 24155 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s	user 0.015s	sys 0.048s
I20260809 23:19:36.646144 24425 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
I20260809 23:19:36.611624 24534 rpcz_store.cc:276] 0809 23:19:33.568995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.609238 (+3040243us) service_pool.cc:227] Handling call
0809 23:19:36.609567 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.610219 (+   652us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.610397 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.647466 24543 rpcz_store.cc:276] 0809 23:19:33.010222 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.407624 (+3397402us) service_pool.cc:227] Handling call
0809 23:19:36.534187 (+126563us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.582892 (+ 48705us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.583147 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.555399 29615 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.648280 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2531) took 2154 ms. Trace:
W20260809 23:19:36.651572 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 152543us
W20260809 23:19:36.649307 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 235008us
I20260809 23:19:36.557782 29797 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:36.656908 24312 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:36.654440 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2617) took 1717 ms. Trace:
I20260809 23:19:36.673635 24321 rpcz_store.cc:276] 0809 23:19:34.936638 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.653284 (+1716646us) service_pool.cc:227] Handling call
0809 23:19:36.653946 (+   662us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.654126 (+   180us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.654147 (+    21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.654188 (+    41us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:36.654235 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.654287 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.654392 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.674530 24543 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 23:19:36.675134 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2371) took 2114 ms. Trace:
I20260809 23:19:36.675853 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2179) took 1706 ms. Trace:
I20260809 23:19:36.655285 24423 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:36.675565 24543 rpcz_store.cc:276] 0809 23:19:34.560865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.651333 (+2090468us) service_pool.cc:227] Handling call
0809 23:19:36.674196 (+ 22863us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.674894 (+   698us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.675084 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.678985 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.656236 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:36.655025 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.680697 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.680845 24321 rpcz_store.cc:276] 0809 23:19:34.969359 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.674784 (+1705425us) service_pool.cc:227] Handling call
0809 23:19:36.675198 (+   414us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.675475 (+   277us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.675568 (+    93us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.675612 (+    44us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.675653 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.675707 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.675805 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.608264 29450 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:36.682643 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2181) took 1712 ms. Trace:
I20260809 23:19:36.683900 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.652221 24545 rpcz_store.cc:276] 0809 23:19:34.494463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.644542 (+2150079us) service_pool.cc:227] Handling call
0809 23:19:36.644807 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.648079 (+  3272us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.648233 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.684566 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:36.685979 24545 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:19:36.686190 24534 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:19:36.686450 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2376) took 1770 ms. Trace:
I20260809 23:19:36.686842 24545 rpcz_store.cc:276] 0809 23:19:34.915892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.685596 (+1769704us) service_pool.cc:227] Handling call
0809 23:19:36.685782 (+   186us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.686264 (+   482us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.686412 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.688949 24545 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:36.690069 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2377) took 1768 ms. Trace:
I20260809 23:19:36.686961 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2370) took 2146 ms. Trace:
I20260809 23:19:36.690493 24545 rpcz_store.cc:276] 0809 23:19:34.921431 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.688260 (+1766829us) service_pool.cc:227] Handling call
0809 23:19:36.688588 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.689733 (+  1145us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.690007 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.691577 24534 rpcz_store.cc:276] 0809 23:19:34.540684 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.650679 (+2109995us) service_pool.cc:227] Handling call
0809 23:19:36.681843 (+ 31164us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.686680 (+  4837us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.686885 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.657310 28889 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:19:36.674293 30096 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.053s	user 0.001s	sys 0.001s
W20260809 23:19:36.694639 25321 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.020s	sys 0.043s
W20260809 23:19:36.652938 24155 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 1.242s	user 0.016s	sys 0.049s
W20260809 23:19:36.697816 25321 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.020s	sys 0.044s
W20260809 23:19:36.698194 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 722156us
W20260809 23:19:36.698315 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 709963us
I20260809 23:19:36.699260 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2535) took 2072 ms. Trace:
I20260809 23:19:36.699787 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.699613 24543 rpcz_store.cc:276] 0809 23:19:34.626650 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.678954 (+2052304us) service_pool.cc:227] Handling call
0809 23:19:36.698786 (+ 19832us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.698984 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.699010 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.699046 (+    36us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:36.699076 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.699123 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.699213 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:36.655783 24498 leader_election.cc:334] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.885s (SENT)
W20260809 23:19:36.701444 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45701.6us
W20260809 23:19:36.701715 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45558.2us
I20260809 23:19:36.684108 24321 rpcz_store.cc:276] 0809 23:19:34.970103 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.681920 (+1711817us) service_pool.cc:227] Handling call
0809 23:19:36.682161 (+   241us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.682319 (+   158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.682359 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.682402 (+    43us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.682441 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.682485 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.682598 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:36.708595 24266 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.772s (SENT)
I20260809 23:19:36.709124 24321 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:36.709488 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.730537 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.731009 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.653075 29204 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:36.740034 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.747469 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2380) took 1142 ms. Trace:
I20260809 23:19:36.747853 24543 rpcz_store.cc:276] 0809 23:19:35.605466 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.701073 (+1095607us) service_pool.cc:227] Handling call
0809 23:19:36.746956 (+ 45883us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.747142 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.747164 (+    22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.747205 (+    41us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:36.747234 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.747279 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.747415 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:36.748279 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48574.1us
I20260809 23:19:36.741060 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2182) took 1738 ms. Trace:
I20260809 23:19:36.751881 24321 rpcz_store.cc:276] 0809 23:19:34.971124 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.707793 (+1736669us) service_pool.cc:227] Handling call
0809 23:19:36.708764 (+   971us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.709552 (+   788us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.709904 (+   352us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:36.697731 29342 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.582s	user 0.010s	sys 0.003s
W20260809 23:19:36.755903 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.006s	sys 0.018s
I20260809 23:19:36.753521 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.697326 28889 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:36.659068 29615 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.768118 29615 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.767654 29666 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.762212 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2538) took 1105 ms. Trace:
I20260809 23:19:36.772430 24543 rpcz_store.cc:276] 0809 23:19:35.657048 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.749125 (+1092077us) service_pool.cc:227] Handling call
0809 23:19:36.761700 (+ 12575us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.761858 (+   158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.761888 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.761924 (+    36us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.761966 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.762032 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.762152 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.746909 29747 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.732817 29625 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:36.763545 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.612301 30085 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:36.760053 29342 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.747033 29791 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:36.761638 24321 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:36.820849 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.772625 28889 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:36.777675 29482 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:36.780311 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.000s	sys 0.023s
W20260809 23:19:36.829952 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.000s	sys 0.024s
I20260809 23:19:36.782539 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2497) took 4846 ms. Trace:
I20260809 23:19:36.830664 24541 rpcz_store.cc:276] 0809 23:19:31.935795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:33.005798 (+1070003us) service_pool.cc:227] Handling call
0809 23:19:33.006336 (+   538us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:33.483373 (+477037us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.782158 (+3298785us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:36.782436 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":870464896}
I20260809 23:19:36.779767 29666 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.784296 29808 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
W20260809 23:19:36.785511 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2368) took 2443 ms (client timeout 1507 ms). Trace:
W20260809 23:19:36.827219 29720 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.894s	user 0.010s	sys 0.001s
W20260809 23:19:36.758551 29610 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.003s	sys 0.025s
W20260809 23:19:36.762522 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.006s	sys 0.018s
I20260809 23:19:36.803824 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:36.817255 29305 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:36.760874 29172 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.822381 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.771610 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:36.829226 30099 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260809 23:19:36.830185 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2610) took 2252 ms (client timeout 2876 ms). Trace:
W20260809 23:19:36.834271 24427 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.858s	user 0.009s	sys 0.001s
I20260809 23:19:36.835680 24543 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260809 23:19:36.838078 24542 rpcz_store.cc:269] 0809 23:19:34.341811 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.498659 (+156848us) service_pool.cc:227] Handling call
0809 23:19:36.785282 (+2286623us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.785419 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334027648}
I20260809 23:19:36.793527 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2506) took 4546 ms. Trace:
I20260809 23:19:36.764256 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:36.837375 25365 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.019s	sys 0.015s
W20260809 23:19:36.908427 25365 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.025s	sys 0.017s
I20260809 23:19:36.909550 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1109) took 1377 ms. Trace:
I20260809 23:19:36.908262 24540 rpcz_store.cc:276] 0809 23:19:32.247456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.990452 (+2742996us) service_pool.cc:227] Handling call
0809 23:19:34.990763 (+   311us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:34.997422 (+  6659us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:35.440274 (+442852us) log.cc:842] Serialized 14 byte log entry
0809 23:19:35.982365 (+542091us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:36.090562 (+108197us) log.cc:842] Serialized 26 byte log entry
0809 23:19:36.090768 (+   206us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:36.090815 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.090923 (+   108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:36.793270 (+702347us) raft_consensus.cc:1671] finished
0809 23:19:36.793298 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.793455 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1583,"prepare.queue_time_us":71995,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":3860,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":103101,"thread_start_us":887738,"threads_started":2,"wal-append.queue_time_us":1198407}
I20260809 23:19:36.911960 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:36.906389 24314 rpcz_store.cc:269] 0809 23:19:34.578144 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.561483 (+983339us) service_pool.cc:227] Handling call
0809 23:19:36.829769 (+1268286us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54003c32d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.830090 (+   321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1618679552}
I20260809 23:19:36.914404 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:36.843462 24428 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:36.916673 24427 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260809 23:19:36.917212 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2640) took 1269 ms. Trace:
I20260809 23:19:36.917670 24427 rpcz_store.cc:276] 0809 23:19:35.647464 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.647665 (+   201us) service_pool.cc:227] Handling call
0809 23:19:36.917152 (+1269487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:36.910849 25365 rpcz_store.cc:276] 0809 23:19:35.532176 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.532286 (+   110us) service_pool.cc:227] Handling call
0809 23:19:35.563452 (+ 31166us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.563670 (+   218us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:36.294559 (+730889us) log.cc:842] Serialized 14 byte log entry
0809 23:19:36.909085 (+614526us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.909163 (+    78us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.909229 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.909337 (+   108us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.909486 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25683,"prepare.queue_time_us":1007099,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":117232,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":95345,"thread_start_us":1320507,"threads_started":2}
W20260809 23:19:36.918988 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.007s	sys 0.018s
I20260809 23:19:36.918867 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:36.918915 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.920148 24321 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
W20260809 23:19:36.919409 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.007s	sys 0.019s
W20260809 23:19:36.920543 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.004s	sys 0.021s
W20260809 23:19:36.926132 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.004s	sys 0.021s
I20260809 23:19:36.810035 29747 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:36.940162 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:36.941511 24540 raft_consensus.cc:2396] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:36.819609 29797 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:36.933323 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.009s	sys 0.036s
W20260809 23:19:36.943183 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.009s	sys 0.037s
W20260809 23:19:36.943559 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.06411e+06us
W20260809 23:19:36.927404 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 620982us
I20260809 23:19:36.946480 25325 leader_election.cc:398] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9.
I20260809 23:19:36.921321 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:36.921037 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.951639 25325 leader_election.cc:401] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260809 23:19:36.952744 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:36.955077 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:36.956919 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.961373 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.870168 29868 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:36.897732 30075 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:36.840190 29625 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:36.914839 29610 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.332s	user 0.009s	sys 0.025s
I20260809 23:19:36.963470 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:36.951639 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:36.837880 29808 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:36.963848 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.836380 29467 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:36.941375 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1108) took 1409 ms. Trace:
I20260809 23:19:36.964993 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:36.940282 29987 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:36.986904 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:36.870451 29212 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260809 23:19:36.968953 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.970398 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.917711 29369 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:36.972500 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:36.964278 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:36.973093 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:36.974157 30098 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260809 23:19:36.998737 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:36.990752 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63948.2us
I20260809 23:19:36.999665 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.000516 24321 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:37.001040 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:36.990506 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.003728 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:36.974558 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2180) took 2004 ms (client timeout 2045 ms). Trace:
I20260809 23:19:36.836804 29532 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260809 23:19:36.925586 28841 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:36.967150 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.016491 24542 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:36.984594 25363 rpcz_store.cc:276] 0809 23:19:35.531773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.531906 (+   133us) service_pool.cc:227] Handling call
0809 23:19:35.583276 (+ 51370us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.583488 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:36.315717 (+732229us) log.cc:842] Serialized 14 byte log entry
0809 23:19:36.925017 (+609300us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:36.941054 (+ 16037us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:36.941126 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:36.941194 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:36.941315 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":987642,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"thread_start_us":1340782,"threads_started":2}
I20260809 23:19:36.839787 29791 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.127s
W20260809 23:19:36.920421 30067 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.000s	sys 0.024s
I20260809 23:19:36.886327 30077 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260809 23:19:36.993870 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:36.994546 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:36.995757 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.026540 25375 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:36.969937 29815 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:37.011308 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.029673 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.030000 24541 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:37.030766 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.030386 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:37.011816 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.027s	sys 0.026s
I20260809 23:19:37.011933 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:36.917660 29172 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.036357 24430 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
W20260809 23:19:37.005321 24316 rpcz_store.cc:269] 0809 23:19:34.969789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.187364 (+217575us) service_pool.cc:227] Handling call
0809 23:19:36.974329 (+1786965us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:36.974485 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3232462720}
I20260809 23:19:36.970870 29868 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.013024 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.015245 24543 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 23:19:37.018322 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.041530 29868 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.042817 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.015906 29718 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.004s
W20260809 23:19:37.044678 30051 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.362s	user 0.023s	sys 0.055s
I20260809 23:19:37.045063 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:37.015996 29856 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.353s	user 0.008s	sys 0.003s
I20260809 23:19:37.016887 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:36.988504 29720 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.011606 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.048692 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:37.032717 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.028s	sys 0.026s
W20260809 23:19:37.024567 30067 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.000s	sys 0.025s
I20260809 23:19:36.966804 29797 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.020083 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.046803 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:36.972231 29991 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260809 23:19:37.054723 25365 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
W20260809 23:19:37.035054 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.014s	sys 0.038s
I20260809 23:19:37.049959 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.049088 24321 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:37.057309 24316 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:37.057780 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:37.058200 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45259.7us
I20260809 23:19:37.058212 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.058507 24321 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:37.059587 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.059762 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.060375 24316 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:37.060526 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.061149 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.061662 24316 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.062323 29991 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.062599 29718 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.063031 29991 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:37.063436 29991 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:37.063619 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46813.3us
I20260809 23:19:36.987824 29615 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260809 23:19:36.975015 29625 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.065366 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.029683 29815 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.067592 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.071929 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.087836 24430 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:19:37.069093 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50269.4us
I20260809 23:19:37.068197 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 708) took 1996 ms. Trace:
I20260809 23:19:37.069901 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:37.021617 29305 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
W20260809 23:19:37.063827 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.014s	sys 0.038s
I20260809 23:19:37.011514 29987 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:37.099668 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 790096us
W20260809 23:19:37.100432 24496 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.598s (SENT)
I20260809 23:19:37.100858 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.101337 25371 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:37.080004 24540 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.073529 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2645) took 1155 ms. Trace:
I20260809 23:19:37.102703 24429 rpcz_store.cc:276] 0809 23:19:35.918600 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.065064 (+1146464us) service_pool.cc:227] Handling call
0809 23:19:37.073462 (+  8398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.103870 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.070420 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.053442 30077 consensus_queue.cc:1225] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 23:19:37.109071 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.110917 24543 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:37.066861 29720 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Stepping down as leader of term 6
W20260809 23:19:37.107457 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.010s	sys 0.014s
W20260809 23:19:37.123780 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.011s	sys 0.014s
I20260809 23:19:37.123929 24544 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:19:37.112520 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.125104 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.109683 24423 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:19:37.087049 30009 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.720s	user 0.007s	sys 0.003s
I20260809 23:19:37.128844 25368 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260809 23:19:37.112974 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:37.129906 24542 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.108891 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2648) took 1153 ms. Trace:
I20260809 23:19:37.132141 24429 rpcz_store.cc:276] 0809 23:19:35.955207 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.103589 (+1148382us) service_pool.cc:227] Handling call
0809 23:19:37.108818 (+  5229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.133229 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.134122 24429 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.113668 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.117798 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:37.098011 30114 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9.
I20260809 23:19:37.120831 29481 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:37.098639 29212 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.047715 29808 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.093339 29815 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.135182 24541 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:37.095965 29532 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:37.124748 29810 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.031307 30074 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.121276 29482 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:37.122494 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1007) took 3506 ms. Trace:
I20260809 23:19:37.232673 25373 rpcz_store.cc:276] 0809 23:19:33.616244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.702947 (+1086703us) service_pool.cc:227] Handling call
0809 23:19:36.520217 (+1817270us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.521295 (+  1078us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.122291 (+600996us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b5400634550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.122435 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4062500096}
W20260809 23:19:37.160331 29968 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.024s	sys 0.017s
I20260809 23:19:37.145493 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.234804 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.158373 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.107972 29878 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20260809 23:19:37.112746 29860 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:37.136379 29742 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.020s	sys 0.008s
W20260809 23:19:37.249949 29742 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.020s	sys 0.009s
I20260809 23:19:37.250875 24535 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
W20260809 23:19:37.179898 24496 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.608s (SENT)
W20260809 23:19:37.252403 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 102274us
W20260809 23:19:37.252894 24496 connection.cc:232] RPC call timeout handler was delayed by -1.1991415580268949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
W20260809 23:19:37.253337 24496 connection.cc:232] RPC call timeout handler was delayed by -1.1474164822138846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1542s for a response
W20260809 23:19:37.253655 24496 connection.cc:232] RPC call timeout handler was delayed by -1.1442163530737162s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
W20260809 23:19:37.253964 24496 connection.cc:232] RPC call timeout handler was delayed by -1.1379415581468493s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18700000000000003s for a response
I20260809 23:19:37.158591 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:37.237957 29968 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.025s	sys 0.017s
I20260809 23:19:37.255455 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:37.179448 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63634.8us
I20260809 23:19:37.159682 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.256462 25369 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:37.256867 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:37.254252 24496 connection.cc:232] RPC call timeout handler was delayed by -1.1179804340936244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260809 23:19:37.136547 29745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.015s	sys 0.014s
W20260809 23:19:37.259148 29745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.015s	sys 0.015s
W20260809 23:19:37.179103 24498 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.544s (SENT)
I20260809 23:19:37.259749 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.159087 25363 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:37.260488 25363 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:37.261294 25363 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.262562 25363 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
W20260809 23:19:37.199548 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86497.1us
I20260809 23:19:37.263034 29172 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.264302 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.198585 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.266872 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.267720 24423 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 23:19:37.268044 29987 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.198493 24316 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:37.269182 29987 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.112990 30120 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.272245 30074 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.275131 24423 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:37.198880 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2176) took 3180 ms. Trace:
I20260809 23:19:37.201583 30114 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.279178 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.281371 25363 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:37.227828 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.144790 29991 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.123370 29342 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:37.123271 29720 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:37.227348 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.291019 25368 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.123337 30077 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:37.054023 29856 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.144874 29481 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:37.219864 29808 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:37.243839 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.013s	sys 0.033s
W20260809 23:19:37.299548 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.014s	sys 0.034s
I20260809 23:19:37.144946 30115 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:37.244171 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.201916 30109 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election won for term 6
I20260809 23:19:37.231531 29810 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.248924 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.339775 25373 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:19:37.259387 24496 connection.cc:232] RPC call timeout handler was delayed by -1.0911415580194443s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
I20260809 23:19:37.269765 29987 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:37.146373 30009 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.091934 29868 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
I20260809 23:19:37.283138 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.294274 24542 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:19:37.294697 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 115621us
I20260809 23:19:37.297472 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:37.352210 24498 leader_election.cc:334] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.595s (SENT)
W20260809 23:19:37.353255 24498 leader_election.cc:334] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.631s (SENT)
W20260809 23:19:37.300585 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 299663us
I20260809 23:19:37.276684 24313 rpcz_store.cc:276] 0809 23:19:34.018443 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.018576 (+   133us) service_pool.cc:227] Handling call
0809 23:19:34.030800 (+ 12224us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:34.135775 (+104975us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:34.135808 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:34.173151 (+ 37343us) log.cc:842] Serialized 26 byte log entry
0809 23:19:34.459648 (+286497us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:34.459701 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.157949 (+2698248us) spinlock_profiling.cc:243] Waited 94.8 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.158012 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:37.158067 (+    55us) raft_consensus.cc:1671] finished
0809 23:19:37.158078 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:37.198804 (+ 40726us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2166,"raft.queue_time_us":2322,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":21950,"spinlock_wait_cycles":218028416,"thread_start_us":256090,"threads_started":1,"wal-append.queue_time_us":324760,"wal-append.run_cpu_time_us":1892,"wal-append.run_wall_time_us":1031033}
W20260809 23:19:37.315500 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 325070us
I20260809 23:19:37.354743 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:37.357846 30137 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260809 23:19:37.299505 29853 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260809 23:19:37.340659 24496 connection.cc:232] RPC call timeout handler was delayed by -1.0578415580093861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19590000000000002s for a response
I20260809 23:19:37.108554 29467 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.203651 29615 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.096s
I20260809 23:19:37.342682 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:37.298333 30083 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.683s	user 0.003s	sys 0.007s
I20260809 23:19:37.287810 29718 raft_consensus.cc:2741] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260809 23:19:37.227527 29743 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260809 23:19:37.201308 29212 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.198093 30108 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.160s
I20260809 23:19:37.359833 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.360023 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:37.319473 24429 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.367100 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.368151 25370 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:37.361092 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96876.6us
I20260809 23:19:37.379617 24540 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 6.
I20260809 23:19:37.317765 29682 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260809 23:19:37.235291 29532 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.296607 29666 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260809 23:19:37.348850 28889 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.348379 30074 raft_consensus.cc:2741] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 23:19:37.374418 24427 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:37.348289 29987 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:37.361105 29991 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:37.352869 30109 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:37.371397 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114619us
I20260809 23:19:37.380625 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.412117 25363 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:37.493432 25363 raft_consensus.cc:595] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.493534 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.366448 30114 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:19:37.375383 29204 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.000s
I20260809 23:19:37.378094 29747 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.408185 29666 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:37.319234 30106 consensus_queue.cc:1159] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 9 status { last_received { term: 8 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:37.410785 30009 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.400257 30115 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:37.527011 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.371069 29481 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.527715 24540 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:37.426003 29797 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:37.492558 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119689us
I20260809 23:19:37.495829 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.498252 29615 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 23:19:37.496805 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 253916us
I20260809 23:19:37.300379 30077 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.540000 29868 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.540426 30077 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.498430 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2644) took 1580 ms. Trace:
I20260809 23:19:37.540985 24427 rpcz_store.cc:276] 0809 23:19:35.918248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.353105 (+1434857us) service_pool.cc:227] Handling call
0809 23:19:37.369793 (+ 16688us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.498084 (+128291us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.498361 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.527318 29666 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.543118 24427 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260809 23:19:37.543838 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2099) took 1217 ms. Trace:
I20260809 23:19:37.375912 29720 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.544108 29666 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.544627 29720 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:37.545514 29481 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:37.546762 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 701) took 3514 ms (client timeout 1715 ms). Trace:
W20260809 23:19:37.547421 24422 rpcz_store.cc:269] 0809 23:19:34.032758 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.032919 (+   161us) service_pool.cc:227] Handling call
0809 23:19:37.546490 (+3513571us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b540018cc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.546645 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2272302976}
I20260809 23:19:37.548141 30150 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:37.376073 30083 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.513229 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.544160 24427 rpcz_store.cc:276] 0809 23:19:36.326196 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.542362 (+1216166us) service_pool.cc:227] Handling call
0809 23:19:37.542833 (+   471us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:37.543537 (+   704us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.543774 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.550309 25368 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 10 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260809 23:19:37.527554 30115 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.551173 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.552264 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.552384 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.552996 24543 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:37.553375 25368 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.553982 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.554251 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.555296 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.555886 25368 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
W20260809 23:19:37.501279 30075 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.530s	user 0.005s	sys 0.005s
I20260809 23:19:37.556622 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.557369 24321 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:37.558097 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.559177 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.559983 30083 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.560316 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.558670 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.561313 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.529350 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:37.561676 24321 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.562103 24540 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.383584 29853 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.131s
I20260809 23:19:37.562405 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.563001 24321 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:37.533456 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.563719 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.564222 24321 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.564869 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:37.565464 29450 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.869s	user 0.009s	sys 0.001s
I20260809 23:19:37.565743 29204 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:37.566584 29718 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.567272 29718 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:37.380192 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.533596 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.466778 28889 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.568722 28889 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:37.569274 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:37.570130 29868 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:19:37.571142 30083 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.571907 29868 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:37.534297 29856 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:37.551282 30115 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:37.091065 24435 rpcz_store.cc:276] 0809 23:19:35.072070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.559237 (+487167us) service_pool.cc:227] Handling call
0809 23:19:35.559657 (+   420us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:35.559861 (+   204us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:36.384929 (+825068us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.067603 (+682674us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.067780 (+   177us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:37.067830 (+    50us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:37.067903 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.067981 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:37.068131 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17461,"prepare.queue_time_us":334675,"prepare.run_cpu_time_us":170,"prepare.run_wall_time_us":20158,"raft.queue_time_us":18358,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":33165,"spinlock_wait_cycles":28265344,"thread_start_us":1456621,"threads_started":2}
I20260809 23:19:37.573748 24540 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:19:37.574538 24435 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:37.554802 25370 raft_consensus.cc:2396] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260809 23:19:37.567857 29718 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.570281 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 715) took 1694 ms. Trace:
I20260809 23:19:37.571611 30083 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.649238 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.650750 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:37.651095 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 371981us
I20260809 23:19:37.433971 29625 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.434456 29745 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:37.493945 25363 raft_consensus.cc:603] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:37.538231 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.581810 29212 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.579111 24541 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:37.656312 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:37.657544 24540 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:37.658349 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.572634 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:37.632126 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 252004us
I20260809 23:19:37.659700 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:37.659978 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87431.8us
I20260809 23:19:37.549474 24422 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:37.584851 24313 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260809 23:19:37.663822 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2576) took 3287 ms. Trace:
I20260809 23:19:37.663995 24313 pending_rounds.cc:76] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 4
I20260809 23:19:37.664384 24313 pending_rounds.cc:97] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260809 23:19:37.664258 24311 rpcz_store.cc:276] 0809 23:19:34.376669 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.312175 (+1935506us) service_pool.cc:227] Handling call
0809 23:19:36.341186 (+ 29011us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.518487 (+177301us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.663562 (+1145075us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b54003859d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.663760 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164801408}
I20260809 23:19:37.664798 24313 raft_consensus.cc:2866] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:37.665691 29995 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.280s	user 0.000s	sys 0.013s
I20260809 23:19:37.665967 24311 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260809 23:19:37.666111 29995 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.643572 29720 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.027s
I20260809 23:19:37.576391 29204 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:37.597229 30072 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.075s	user 0.025s	sys 0.049s
I20260809 23:19:37.583930 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 789) took 5841 ms. Trace:
I20260809 23:19:37.591562 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2567) took 3538 ms. Trace:
W20260809 23:19:37.579589 29939 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260809 23:19:37.606395 29832 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:37.582667 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2582) took 3204 ms (client timeout 2550 ms). Trace:
I20260809 23:19:37.646417 29532 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.537684 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.641283 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2118) took 1027 ms. Trace:
I20260809 23:19:37.551246 24427 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260809 23:19:37.683658 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:37.688303 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 150809us
I20260809 23:19:37.687944 24435 rpcz_store.cc:276] 0809 23:19:36.613762 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.573965 (+960203us) service_pool.cc:227] Handling call
0809 23:19:37.574316 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.640943 (+ 66627us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.641205 (+   262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.642493 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.690336 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.608841 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 930) took 3282 ms. Trace:
I20260809 23:19:37.618342 30085 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.575932 29610 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:19:37.578630 30123 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260809 23:19:37.611943 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1115) took 2077 ms. Trace:
I20260809 23:19:37.503122 29987 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:37.613395 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 930) took 4019 ms. Trace:
I20260809 23:19:37.654426 29745 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:37.647370 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.700244 24543 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:37.557407 24431 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:37.649317 29718 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:37.680827 24544 rpcz_store.cc:276] 0809 23:19:31.742507 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:32.543492 (+800985us) service_pool.cc:227] Handling call
0809 23:19:32.556645 (+ 13153us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.124289 (+4567644us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.583668 (+459379us) spinlock_profiling.cc:243] Waited 4.57 s on lock 0x7b5400147654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.583840 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10502995200}
I20260809 23:19:37.669376 24314 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260809 23:19:37.682246 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2509) took 5313 ms. Trace:
I20260809 23:19:37.703099 24535 rpcz_store.cc:276] 0809 23:19:32.368709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.085538 (+2716829us) service_pool.cc:227] Handling call
0809 23:19:35.157839 (+ 72301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.251310 (+2093471us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.673649 (+422339us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b54001fd954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.682180 (+  8531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4812956032}
I20260809 23:19:37.671159 29625 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:37.687208 24323 rpcz_store.cc:269] 0809 23:19:34.378374 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.157562 (+779188us) service_pool.cc:227] Handling call
0809 23:19:37.582440 (+2424878us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.582592 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1672550656}
I20260809 23:19:37.690938 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.673363 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.692232 24320 rpcz_store.cc:276] 0809 23:19:34.326810 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.581081 (+1254271us) service_pool.cc:227] Handling call
0809 23:19:35.581393 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.412442 (+831049us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.608605 (+1196163us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.608783 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904510720}
I20260809 23:19:37.707862 24323 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:37.707965 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.691054 24541 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:37.690550 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.711099 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:37.711773 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:37.712345 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:37.695415 29610 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:37.696609 25372 rpcz_store.cc:276] 0809 23:19:33.594289 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.702061 (+1107772us) service_pool.cc:227] Handling call
0809 23:19:35.531607 (+829546us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:35.532441 (+   834us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.613196 (+2080755us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54006d4cd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.613336 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1906925952}
I20260809 23:19:37.715586 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.646096 24423 rpcz_store.cc:276] 0809 23:19:35.876211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.268210 (+1391999us) service_pool.cc:227] Handling call
0809 23:19:37.268523 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.275596 (+  7073us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.569970 (+294374us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b540018c254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.570201 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14736128}
I20260809 23:19:37.696376 25371 rpcz_store.cc:276] 0809 23:19:35.534816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:35.620625 (+ 85809us) service_pool.cc:227] Handling call
0809 23:19:35.620922 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.108062 (+1487140us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.611659 (+503597us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b540073f5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.611873 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3404078336}
I20260809 23:19:37.698568 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2104) took 1356 ms. Trace:
I20260809 23:19:37.716521 25363 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.701359 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:37.716769 24427 rpcz_store.cc:276] 0809 23:19:36.341748 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.550636 (+1208888us) service_pool.cc:227] Handling call
0809 23:19:37.551004 (+   368us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:37.693570 (+142566us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.698493 (+  4923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.717411 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.669075 30083 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.663843 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2100) took 1337 ms. Trace:
I20260809 23:19:37.671159 29815 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.664124 29823 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.721593 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.721424 24309 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:37.671564 30067 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:37.732582 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:37.690347 24435 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 23:19:37.668677 29810 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.713649 24266 leader_election.cc:398] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10.
I20260809 23:19:37.701231 25370 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.717767 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.693393 29369 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.687263 29204 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.715569 30106 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.746258 29204 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:37.747157 24320 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:37.748961 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:37.749688 24320 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:37.752166 29369 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:37.753019 29369 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:37.753314 29369 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:37.754081 29369 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.755326 29610 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.766638 24535 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 23:19:37.689921 30109 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.769910 29369 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.722836 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:37.642683 25368 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.720664 24422 rpcz_store.cc:276] 0809 23:19:36.326564 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.548501 (+1221937us) service_pool.cc:227] Handling call
0809 23:19:37.549198 (+   697us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.663544 (+114346us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.663786 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.734200 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2122) took 1119 ms. Trace:
I20260809 23:19:37.777513 24435 rpcz_store.cc:276] 0809 23:19:36.614919 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.689730 (+1074811us) service_pool.cc:227] Handling call
0809 23:19:37.690053 (+   323us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.733974 (+ 43921us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.734148 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.778717 24313 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260809 23:19:37.779234 24313 pending_rounds.cc:76] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 7
I20260809 23:19:37.722626 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:37.777217 24542 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:37.779455 24313 pending_rounds.cc:97] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260809 23:19:37.780342 24313 raft_consensus.cc:2866] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:37.734596 25363 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
W20260809 23:19:37.778717 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48062.2us
I20260809 23:19:37.781020 25363 raft_consensus.cc:595] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:37.718614 30153 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:37.732036 30114 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260809 23:19:37.719810 29879 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:37.737455 29815 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.738458 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.732192 29965 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:37.739209 24472 maintenance_manager.cc:445] Time spent finding best maintenance operation: real 15.899s	user 0.074s	sys 0.093s
I20260809 23:19:37.794673 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2607) took 3223 ms. Trace:
I20260809 23:19:37.739570 24423 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:37.732462 30142 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.009s
I20260809 23:19:37.795591 24312 rpcz_store.cc:276] 0809 23:19:34.570769 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.602603 (+2031834us) service_pool.cc:227] Handling call
0809 23:19:36.604009 (+  1406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.692199 (+ 88190us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.793807 (+1101608us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b54002535d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.794516 (+   709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121207552}
I20260809 23:19:37.741195 29995 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:37.696348 30158 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260809 23:19:37.742866 24311 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:37.727475 29516 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.016s
I20260809 23:19:37.744493 25371 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260809 23:19:37.721452 29939 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:37.782459 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59905.9us
I20260809 23:19:37.680600 24317 rpcz_store.cc:276] 0809 23:19:34.052868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.071618 (+ 18750us) service_pool.cc:227] Handling call
0809 23:19:35.069788 (+998170us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:35.131833 (+ 62045us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:35.131880 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:35.192480 (+ 60600us) log.cc:842] Serialized 52 byte log entry
0809 23:19:35.474512 (+282032us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:35.987753 (+513241us) log.cc:842] Serialized 14 byte log entry
0809 23:19:35.988021 (+   268us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:35.988094 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:35.988136 (+    42us) raft_consensus.cc:1671] finished
0809 23:19:35.988151 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:37.591364 (+1603213us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54003c3cd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.591495 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18723,"prepare.queue_time_us":1080639,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":438462,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":115303,"spinlock_wait_cycles":2170463104,"thread_start_us":692601,"threads_started":2,"wal-append.queue_time_us":236140,"wal-append.run_cpu_time_us":4355,"wal-append.run_wall_time_us":1897917}
I20260809 23:19:37.732007 29352 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260809 23:19:37.721776 29532 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:37.786856 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2190) took 1003 ms. Trace:
I20260809 23:19:37.788196 25369 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:37.789646 30115 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:37.791954 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53569.7us
I20260809 23:19:37.854456 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.855664 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.802430 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2127) took 1055 ms. Trace:
I20260809 23:19:37.856372 24423 rpcz_store.cc:276] 0809 23:19:36.746592 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.716500 (+969908us) service_pool.cc:227] Handling call
0809 23:19:37.739334 (+ 22834us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.796722 (+ 57388us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.802370 (+  5648us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:37.849740 29995 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260809 23:19:37.858505 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 812) took 2292 ms. Trace:
I20260809 23:19:37.853024 29720 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260809 23:19:37.850703 29879 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.859936 29879 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.861954 29879 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:37.864027 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2643) took 1946 ms (client timeout 1958 ms). Trace:
W20260809 23:19:37.864451 24424 rpcz_store.cc:269] 0809 23:19:35.917797 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.345947 (+428150us) service_pool.cc:227] Handling call
0809 23:19:37.863875 (+1517928us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.863971 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":598801664}
I20260809 23:19:37.865008 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.849999 24316 rpcz_store.cc:276] 0809 23:19:36.783093 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.269020 (+485927us) service_pool.cc:227] Handling call
0809 23:19:37.269317 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.269513 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:37.524489 (+254976us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.786591 (+262102us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:37.786645 (+    54us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:37.786683 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.786726 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:37.786803 (+    77us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":372415,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":1264,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5818,"thread_start_us":516344,"threads_started":2,"wal-append.queue_time_us":322483}
I20260809 23:19:37.866699 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:37.787431 29467 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260809 23:19:37.834419 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52081.4us
I20260809 23:19:37.876969 24316 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:37.877012 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.785079 29666 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.797206 29481 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:19:37.839098 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63941.2us
I20260809 23:19:37.842540 24317 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:37.839772 25371 pending_rounds.cc:76] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 6
I20260809 23:19:37.881155 25371 pending_rounds.cc:97] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260809 23:19:37.867313 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:37.858911 24545 rpcz_store.cc:276] 0809 23:19:35.565891 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.692755 (+1126864us) service_pool.cc:227] Handling call
0809 23:19:36.785996 (+ 93241us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:36.786454 (+   458us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:36.786492 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:36.800929 (+ 14437us) log.cc:842] Serialized 26 byte log entry
0809 23:19:37.147575 (+346646us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:37.496065 (+348490us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.496335 (+   270us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.496412 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:37.499508 (+  3096us) raft_consensus.cc:1671] finished
0809 23:19:37.499564 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:37.858338 (+358774us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b54001fcf50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.858448 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":233266,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":2214,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":15176,"spinlock_wait_cycles":213506944,"thread_start_us":572728,"threads_started":2,"wal-append.queue_time_us":443562}
I20260809 23:19:37.881495 25371 raft_consensus.cc:2866] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:37.876240 30153 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.883811 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2102) took 1556 ms. Trace:
I20260809 23:19:37.884218 24428 rpcz_store.cc:276] 0809 23:19:36.327203 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.638064 (+310861us) service_pool.cc:227] Handling call
0809 23:19:37.883637 (+1245573us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b5400260cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.883751 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470391552}
I20260809 23:19:37.885390 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.886153 24428 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.876055 30106 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:37.887647 30106 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:37.888108 30106 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:37.889545 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2627) took 1043 ms. Trace:
I20260809 23:19:37.889896 24313 rpcz_store.cc:276] 0809 23:19:36.846254 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.689134 (+842880us) service_pool.cc:227] Handling call
0809 23:19:37.703020 (+ 13886us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.780739 (+ 77719us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.889263 (+108524us) spinlock_profiling.cc:243] Waited 71.4 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.889476 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":164334464}
I20260809 23:19:37.882206 29666 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.890985 29991 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:37.891593 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2646) took 1972 ms (client timeout 2022 ms). Trace:
I20260809 23:19:37.890506 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.849850 29532 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.895787 25372 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:37.785431 30141 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:37.903461 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2189) took 1120 ms. Trace:
I20260809 23:19:37.903836 24314 rpcz_store.cc:276] 0809 23:19:36.782773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.136493 (+353720us) service_pool.cc:227] Handling call
0809 23:19:37.668993 (+532500us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.703046 (+ 34053us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.903279 (+200233us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54003859d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.903403 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1121752320}
I20260809 23:19:37.905576 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:37.892001 24431 rpcz_store.cc:269] 0809 23:19:35.918966 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.352631 (+433665us) service_pool.cc:227] Handling call
0809 23:19:37.891441 (+1538810us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:37.891540 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2698263424}
I20260809 23:19:37.892602 24424 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:37.841545 29939 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.892877 24428 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:37.896381 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.878788 24316 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:37.879913 29467 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:37.904701 29808 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:37.900580 28841 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:37.900580 30153 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:37.915791 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.901026 30115 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:37.902024 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.913224 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.924067 24428 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:37.926369 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:37.914901 24321 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:37.918179 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2565) took 3866 ms. Trace:
I20260809 23:19:37.927275 24309 rpcz_store.cc:276] 0809 23:19:34.051908 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.072796 (+ 20888us) service_pool.cc:227] Handling call
0809 23:19:34.284889 (+212093us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.730238 (+3445349us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.917960 (+187722us) spinlock_profiling.cc:243] Waited 3.44 s on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:37.918123 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7902868864}
I20260809 23:19:37.901850 29532 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:37.923725 30153 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:37.929373 30153 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:37.915603 29467 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:37.914987 29939 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:37.932003 28889 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:37.935123 29518 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:37.935359 29718 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.936604 24314 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:37.936726 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.939193 29718 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.939133 30083 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:37.940327 29718 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:19:37.941495 30083 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:37.918678 29808 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:37.943002 30083 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:37.943992 30083 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.944172 29808 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.944999 30083 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:37.941599 29745 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.947481 29745 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:37.922344 28841 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:37.949538 28841 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.949785 29625 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:37.950680 28841 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:37.951565 29481 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:37.952028 29481 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:37.953166 29481 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:37.955673 29204 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.966136 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:37.954671 29481 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:37.972368 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:37.916886 30021 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:37.876785 29987 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:37.917637 29666 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.975653 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:37.997117 25366 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:38.000775 24425 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:37.969242 29823 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:37.969007 29688 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:38.005115 25363 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:38.009312 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.006855 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:38.064870 24316 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:38.067533 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.006811 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.070097 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.069687 29823 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.076591 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.080289 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:38.081653 30166 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260809 23:19:38.081885 30165 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260809 23:19:38.098598 29791 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.099146 29791 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.099363 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.100639 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.101109 29791 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.101751 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.102450 24431 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8.
I20260809 23:19:38.103101 29965 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:37.914515 24155 maintenance_manager.cc:643] P e3357d0fe4ef46db9a4739044fa527d5: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 11.129s	user 0.225s	sys 0.157s Metrics: {"bytes_written":29212,"cfile_init":1,"dirs.queue_time_us":203011,"dirs.run_cpu_time_us":463,"dirs.run_wall_time_us":1764,"drs_written":1,"lbm_read_time_us":511,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393349,"lbm_writes_1-10_ms":4,"lbm_writes_10-100_ms":10,"lbm_writes_lt_1ms":16,"peak_mem_usage":0,"raft.queue_time_us":621516,"raft.run_cpu_time_us":5516,"raft.run_wall_time_us":195176,"rows_written":63,"thread_start_us":1530249,"threads_started":3,"wal-append.queue_time_us":1004523}
I20260809 23:19:38.103535 29823 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:37.977244 30158 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260809 23:19:38.086417 29868 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260809 23:19:38.109071 29987 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.083772 24317 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:37.999019 29815 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
I20260809 23:19:38.110661 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:38.111482 24317 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:38.111963 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.112349 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.113540 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.078547 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.114102 29868 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:38.114574 25369 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:38.111372 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.116108 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:38.116333 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.116892 25369 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:38.117664 25370 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:19:38.120151 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.103549 29965 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.121016 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.121160 29868 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.121582 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:38.121843 24157 maintenance_manager.cc:419] P e3357d0fe4ef46db9a4739044fa527d5: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9770 bytes on disk
I20260809 23:19:38.122117 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.122397 29868 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:38.122969 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.123780 30141 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:38.124065 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.124091 24546 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260809 23:19:38.124758 24155 maintenance_manager.cc:643] P e3357d0fe4ef46db9a4739044fa527d5: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":339,"lbm_reads_lt_1ms":4}
I20260809 23:19:38.125263 29708 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.125828 29708 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.125149 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.125849 29868 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.127243 24431 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:38.127573 29868 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.128185 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.128794 24431 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:38.129547 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 954) took 1065 ms. Trace:
I20260809 23:19:38.129652 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.129961 25375 rpcz_store.cc:276] 0809 23:19:37.064253 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.071340 (+  7087us) service_pool.cc:227] Handling call
0809 23:19:37.100308 (+ 28968us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:37.299503 (+199195us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:37.357780 (+ 58277us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.738584 (+380804us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b180045bb00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:37.852256 (+113672us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:37.864534 (+ 12278us) log.cc:842] Serialized 26 byte log entry
0809 23:19:37.864686 (+   152us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:37.864717 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:37.864779 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.129261 (+264482us) raft_consensus.cc:1671] finished
0809 23:19:38.129372 (+   111us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.129497 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47537,"prepare.queue_time_us":35779,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":98863,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":33027,"spinlock_wait_cycles":8915968,"thread_start_us":299882,"threads_started":2,"wal-append.queue_time_us":715590}
I20260809 23:19:38.102159 29939 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.133159 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1025) took 1007 ms. Trace:
I20260809 23:19:38.133512 25366 rpcz_store.cc:276] 0809 23:19:37.125628 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.126942 (+  1314us) service_pool.cc:227] Handling call
0809 23:19:37.180956 (+ 54014us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.004079 (+823123us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.132944 (+128865us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b54002ab7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:38.133102 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1876503936}
I20260809 23:19:38.134713 29939 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.135620 30067 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Stepping down as leader of term 3
I20260809 23:19:38.136003 30067 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:38.136868 30067 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.138149 30067 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:38.140344 25363 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260809 23:19:38.142302 24266 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8.
I20260809 23:19:38.142654 24266 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:37.972193 30105 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:38.092909 25373 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:38.039278 29666 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:38.190877 24317 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:38.092161 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.191026 29666 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.107142 29532 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.192739 29610 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:37.974611 30115 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:38.144124 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2624) took 1298 ms. Trace:
I20260809 23:19:38.122799 24424 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:38.146807 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:37.969338 29615 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.092929 24320 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:37.974460 30127 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.171s
I20260809 23:19:38.156003 28841 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:19:38.157761 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:38.128859 30157 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.325s	user 0.000s	sys 0.001s
I20260809 23:19:38.161561 29708 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:38.140270 30164 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.247s	user 0.000s	sys 0.000s
I20260809 23:19:38.169801 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1003) took 1165 ms. Trace:
I20260809 23:19:38.171443 28889 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.071120 30075 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:38.172195 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34762 (request call id 1004) took 1167 ms. Trace:
I20260809 23:19:38.224893 24422 rpcz_store.cc:276] 0809 23:19:37.008095 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.777565 (+769470us) service_pool.cc:227] Handling call
0809 23:19:38.171663 (+394098us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:19:38.172061 (+   398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433181568}
W20260809 23:19:38.173025 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2365) took 4224 ms (client timeout 2100 ms). Trace:
W20260809 23:19:38.225947 24546 rpcz_store.cc:269] 0809 23:19:33.948575 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:34.093727 (+145152us) service_pool.cc:227] Handling call
0809 23:19:38.172772 (+4079045us) spinlock_profiling.cc:243] Waited 4.03 s on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:38.172939 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9266294656}
I20260809 23:19:38.173914 24539 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260809 23:19:38.227329 24539 pending_rounds.cc:76] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 5
I20260809 23:19:38.227622 24539 pending_rounds.cc:97] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260809 23:19:38.227888 24539 raft_consensus.cc:2866] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:38.179627 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2097) took 1902 ms. Trace:
W20260809 23:19:38.179713 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2116) took 1566 ms (client timeout 1933 ms). Trace:
W20260809 23:19:38.230540 24425 rpcz_store.cc:269] 0809 23:19:36.613110 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.650732 (+ 37622us) service_pool.cc:227] Handling call
0809 23:19:38.179499 (+1528767us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:38.179643 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3034152320}
I20260809 23:19:38.179943 24435 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260809 23:19:38.187086 30105 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.155010 29987 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8.
I20260809 23:19:38.196211 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.158212 29204 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.194058 24311 rpcz_store.cc:276] 0809 23:19:36.845396 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.666684 (+821288us) service_pool.cc:227] Handling call
0809 23:19:37.688275 (+ 21591us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.804508 (+116233us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.139780 (+335272us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:38.144041 (+  4261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124786560}
W20260809 23:19:38.201728 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44044.9us
I20260809 23:19:38.208122 29532 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.208659 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:38.208904 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 802) took 2735 ms. Trace:
I20260809 23:19:38.209497 29745 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:38.209901 29815 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.211850 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.259033 24534 rpcz_store.cc:276] 0809 23:19:35.473951 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.697461 (+1223510us) service_pool.cc:227] Handling call
0809 23:19:36.754534 (+ 57073us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:36.754790 (+   256us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:37.241587 (+486797us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.499383 (+257796us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.607919 (+108536us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18004d46a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:38.208198 (+600279us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b18004d46a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:19:38.208457 (+   259us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:38.208513 (+    56us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:38.208572 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.208634 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.208838 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":690609,"prepare.queue_time_us":602788,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":97,"raft.queue_time_us":718187,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":2142,"spinlock_wait_cycles":50107520,"thread_start_us":733383,"threads_started":2,"wal-append.queue_time_us":297123}
W20260809 23:19:38.213093 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 696412us
I20260809 23:19:38.193063 29666 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.213788 30114 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:38.202304 29615 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.205858 30115 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:38.216737 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70042.1us
I20260809 23:19:38.292557 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.288417 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:38.294031 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:38.218142 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 679941us
I20260809 23:19:38.295572 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.219581 29855 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.220350 29481 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.221711 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 944) took 1209 ms. Trace:
I20260809 23:19:38.301344 24320 rpcz_store.cc:276] 0809 23:19:37.012272 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.750286 (+738014us) service_pool.cc:227] Handling call
0809 23:19:37.750532 (+   246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.196531 (+445999us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.221500 (+ 24969us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:38.221656 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":786969472}
I20260809 23:19:38.302735 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.221499 24423 rpcz_store.cc:276] 0809 23:19:37.004033 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.886059 (+882026us) service_pool.cc:227] Handling call
0809 23:19:38.169082 (+283023us) spinlock_profiling.cc:243] Waited 70.1 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:38.169705 (+   623us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":161307392}
I20260809 23:19:38.228705 24429 rpcz_store.cc:276] 0809 23:19:36.276837 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.381313 (+1104476us) service_pool.cc:227] Handling call
0809 23:19:37.493252 (+111939us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:37.547176 (+ 53924us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:37.547542 (+   366us) log.cc:842] Serialized 14 byte log entry
0809 23:19:37.834355 (+286813us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:37.834411 (+    56us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:37.834470 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.179385 (+344915us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:38.179431 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.179570 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82473,"prepare.queue_time_us":193,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":56845,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":8825,"spinlock_wait_cycles":123326720,"thread_start_us":234189,"threads_started":1,"wal-append.queue_time_us":374897}
I20260809 23:19:38.305965 24429 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:38.234565 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2143) took 1202 ms. Trace:
I20260809 23:19:38.307466 24435 rpcz_store.cc:276] 0809 23:19:37.032520 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.778524 (+746004us) service_pool.cc:227] Handling call
0809 23:19:38.179634 (+401110us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.231610 (+ 51976us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.234378 (+  2768us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b540017f7d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:38.234508 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":921577344}
I20260809 23:19:38.309345 24435 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 3. Current term is 4. Ops: []
I20260809 23:19:38.195757 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:38.260840 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2512) took 5766 ms. Trace:
I20260809 23:19:38.310825 24539 rpcz_store.cc:276] 0809 23:19:32.494580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:36.358975 (+3864395us) service_pool.cc:227] Handling call
0809 23:19:38.173465 (+1814490us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.235583 (+ 62118us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.260281 (+ 24698us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54000b6650. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:38.260724 (+   443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4092307328}
I20260809 23:19:38.261966 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.312120 24546 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:38.264065 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:38.262750 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:38.313068 25365 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:38.268989 24314 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.262616 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:38.316075 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53519.2us
I20260809 23:19:38.316972 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.271178 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.278779 24424 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:38.277377 24317 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:38.266350 29745 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.283716 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.284272 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.327327 24425 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:38.283767 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.225097 29688 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:38.298360 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:38.299823 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.265770 30150 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.336719 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.280934 24321 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 12
I20260809 23:19:38.265612 29832 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.202337 29965 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:38.325727 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47709.7us
I20260809 23:19:38.339902 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:38.325806 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54762.4us
I20260809 23:19:38.355331 24321 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:19:38.360687 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62503.4us
I20260809 23:19:38.361621 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:38.361989 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:38.361460 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:38.364073 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:38.365887 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.366550 24539 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.285257 30114 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:38.235023 30075 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.318758 29987 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:38.234517 29467 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.373536 24317 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:38.234277 30153 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.373086 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.327456 29823 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:38.379892 25373 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:38.360054 30162 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260809 23:19:38.373822 29991 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:38.335088 29745 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:38.334520 30141 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:38.359815 29369 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.098s
I20260809 23:19:38.400379 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.368636 29832 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:38.394500 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 10 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:38.399284 29450 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20260809 23:19:38.360257 29688 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:38.387887 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.360306 30085 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:38.365728 29834 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.371115 30109 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260809 23:19:38.380337 29481 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260809 23:19:38.334637 29204 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:38.428236 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55227.2us
I20260809 23:19:38.371026 30150 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.370879 30021 raft_consensus.cc:2741] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260809 23:19:38.367367 30142 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.423657 24428 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:38.385548 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.380474 29742 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.371008 30124 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:38.415563 30125 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:38.456494 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56150.4us
I20260809 23:19:38.395583 30105 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.368040 30158 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260809 23:19:38.409885 29615 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:38.456779 24150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 178265us
I20260809 23:19:38.360286 29666 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260809 23:19:38.461127 24534 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.432890 29823 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.417186 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:38.450872 24546 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:38.504186 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.438894 29855 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:38.505616 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.470575 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.285887 29815 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:38.458135 30106 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.570s	user 0.006s	sys 0.006s
I20260809 23:19:38.410727 30115 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.512818 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.440500 29991 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:38.455049 30108 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.446127 29518 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
W20260809 23:19:38.483304 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 97833.7us
I20260809 23:19:38.550384 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.468389 29397 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:38.498100 24422 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:19:38.498358 30173 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260809 23:19:38.507810 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:38.575163 24539 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:38.511474 29823 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:38.510984 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:38.597719 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86839.8us
W20260809 23:19:38.474151 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.011s	sys 0.003s
I20260809 23:19:38.598860 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:38.599206 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.011s	sys 0.003s
I20260809 23:19:38.508764 29810 raft_consensus.cc:2741] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:19:38.495692 29808 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.488634 30125 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.519234 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.497466 29965 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.615892 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:38.532042 30169 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260809 23:19:38.523777 29991 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:38.535753 29467 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.536393 30181 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:38.540153 24534 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:19:38.543604 30178 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260809 23:19:38.544552 30175 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260809 23:19:38.546797 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.636624 24545 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:38.509609 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0512703240383416s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20260809 23:19:38.576783 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:38.588171 30150 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.616796 30125 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.588462 29815 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.571939 29397 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:38.651782 29397 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.653076 29397 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:38.653366 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:38.654601 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.655066 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 717) took 2778 ms. Trace:
I20260809 23:19:38.655206 24313 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:38.655628 30153 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.656448 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.655462 24430 rpcz_store.cc:276] 0809 23:19:35.876809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.313206 (+1436397us) service_pool.cc:227] Handling call
0809 23:19:37.352545 (+ 39339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:37.363049 (+ 10504us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:37.363087 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:37.381190 (+ 18103us) log.cc:842] Serialized 26 byte log entry
0809 23:19:37.917373 (+536183us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b18003249a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:38.231801 (+314428us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:38.231839 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.231921 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.654736 (+422815us) raft_consensus.cc:1671] finished
0809 23:19:38.654757 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.655002 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3429,"raft.queue_time_us":58942,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":15552,"spinlock_wait_cycles":6098688,"thread_start_us":314255,"threads_started":1,"wal-append.queue_time_us":1014269}
I20260809 23:19:38.657703 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.659021 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.659446 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2628) took 1572 ms. Trace:
I20260809 23:19:38.659777 24313 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:38.659900 24312 rpcz_store.cc:276] 0809 23:19:37.087367 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.796559 (+709192us) service_pool.cc:227] Handling call
0809 23:19:37.828149 (+ 31590us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:37.839124 (+ 10975us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:37.839158 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:37.867662 (+ 28504us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.235679 (+368017us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:38.235716 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.235771 (+    55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.658771 (+423000us) raft_consensus.cc:1671] finished
0809 23:19:38.658791 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.659380 (+   589us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":6068,"thread_start_us":324126,"threads_started":1,"wal-append.queue_time_us":501590}
I20260809 23:19:38.661911 24313 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:19:38.595785 30075 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.466769 30114 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.500051 30124 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.512684 29708 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.513442 24546 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:38.600951 24431 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260809 23:19:38.608635 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.611388 24435 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:38.613302 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.613479 25368 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:38.514874 29742 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:38.615797 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.617460 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:38.619870 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:38.575186 29204 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:38.626178 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.620766 29450 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260809 23:19:38.633375 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:38.633708 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.616377 29305 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:38.516588 30106 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:38.639426 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62770.3us
I20260809 23:19:38.640470 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.640813 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.637974 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.589788 29832 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:38.642313 29856 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.644069 29615 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:38.634773 30181 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:38.649308 30186 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.060s	user 0.003s	sys 0.000s
W20260809 23:19:38.650568 30183 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260809 23:19:38.679598 30109 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:38.679601 30153 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:38.673134 24539 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:38.695595 29305 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:38.673497 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60298.8us
I20260809 23:19:38.636001 29855 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:38.688642 29742 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:38.674292 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54516.3us
I20260809 23:19:38.674793 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:38.627107 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.680181 24534 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:38.627550 29834 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260809 23:19:38.721582 24422 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:38.685791 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52168.8us
W20260809 23:19:38.693372 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52999.3us
I20260809 23:19:38.694344 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:38.671960 29853 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:38.664635 24312 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:38.729496 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:38.730772 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:38.731495 24534 raft_consensus.cc:2354] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:38.720412 30109 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:38.732986 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2552) took 1104 ms. Trace:
I20260809 23:19:38.732792 30109 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.733352 24541 rpcz_store.cc:276] 0809 23:19:37.628332 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.710402 (+ 82070us) service_pool.cc:227] Handling call
0809 23:19:37.734262 (+ 23860us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.221132 (+486870us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.221179 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.221573 (+   394us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.483021 (+261448us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:38.483247 (+   226us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.483317 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.732793 (+249476us) raft_consensus.cc:1671] finished
0809 23:19:38.732814 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.732935 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":486479,"raft.queue_time_us":497544,"spinlock_wait_cycles":55900288,"thread_start_us":237685,"threads_started":1,"wal-append.queue_time_us":451062}
I20260809 23:19:38.735200 24539 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:38.715405 30153 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.735919 29965 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:38.736078 30153 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:38.736361 24539 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260809 23:19:38.736478 29965 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.736996 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2558) took 1083 ms. Trace:
I20260809 23:19:38.737430 24539 rpcz_store.cc:276] 0809 23:19:37.653258 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.735837 (+1082579us) service_pool.cc:227] Handling call
0809 23:19:38.736076 (+   239us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.736740 (+   664us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.736937 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:38.695899 29856 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.741950 29856 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:38.743000 29305 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.747879 29369 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:38.748180 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:38.748695 29369 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:38.749181 24312 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:38.749740 29369 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:38.752599 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2394) took 1125 ms. Trace:
I20260809 23:19:38.753163 24540 rpcz_store.cc:276] 0809 23:19:37.626774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.661293 (+ 34519us) service_pool.cc:227] Handling call
0809 23:19:37.661583 (+   290us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.218626 (+557043us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.218655 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.219072 (+   417us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.475322 (+256250us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.475372 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.479232 (+  3860us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.752315 (+273083us) raft_consensus.cc:1671] finished
0809 23:19:38.752346 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.752523 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":607086,"raft.queue_time_us":559030,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":9115,"thread_start_us":252864,"threads_started":1,"wal-append.queue_time_us":457673}
I20260809 23:19:38.755842 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:38.759601 24270 leader_election.cc:398] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:38.759948 24270 leader_election.cc:401] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:38.760710 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:38.704165 24546 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:38.666960 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:38.673295 24428 raft_consensus.cc:2354] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
W20260809 23:19:38.669641 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61062.5us
I20260809 23:19:38.767484 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.718688 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:38.680200 29518 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.034s
I20260809 23:19:38.689842 29204 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
W20260809 23:19:38.684551 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43805us
I20260809 23:19:38.779083 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.780756 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:38.676149 30075 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260809 23:19:38.707301 30085 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:38.692836 29615 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:38.696127 29760 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:38.774132 24266 leader_election.cc:398] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
W20260809 23:19:38.764178 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 92409.1us
W20260809 23:19:38.818499 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44549.4us
I20260809 23:19:38.796341 29965 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:38.728648 29666 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:38.782598 29481 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:38.786626 29832 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260809 23:19:38.799228 30141 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:38.785183 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.809805 24539 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260809 23:19:38.813846 29823 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:38.772622 29742 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:38.815137 29834 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:38.770727 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:38.762908 29369 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:38.938401 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 167791us
I20260809 23:19:38.944983 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.827761 24422 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:38.834138 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.014s	sys 0.000s
W20260809 23:19:38.947781 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.015s	sys 0.000s
W20260809 23:19:38.797165 30193 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260809 23:19:38.835916 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.001s	sys 0.014s
W20260809 23:19:38.952170 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.001s	sys 0.014s
W20260809 23:19:38.838774 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.000s	sys 0.015s
I20260809 23:19:38.836905 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2397) took 1192 ms. Trace:
I20260809 23:19:38.858135 30085 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:38.861868 24431 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:38.820372 29615 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:38.816726 29856 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:38.819375 29204 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:38.837446 29868 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.709s	user 0.010s	sys 0.001s
I20260809 23:19:38.819589 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:38.940060 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:38.838346 29666 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:38.950958 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:38.856089 29760 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
W20260809 23:19:38.906738 30182 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260809 23:19:38.952728 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.000s	sys 0.015s
I20260809 23:19:38.968343 30021 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:38.973927 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2555) took 1332 ms. Trace:
I20260809 23:19:38.964712 24535 rpcz_store.cc:276] 0809 23:19:37.644202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.768569 (+124367us) service_pool.cc:227] Handling call
0809 23:19:37.769102 (+   533us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.234544 (+465442us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.234594 (+    50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.235023 (+   429us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.622400 (+387377us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:38.622440 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.622506 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.633267 (+ 10761us) spinlock_profiling.cc:243] Waited 7.38 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:38.816637 (+183370us) raft_consensus.cc:1671] finished
0809 23:19:38.816662 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.836808 (+ 20146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":464948,"raft.queue_time_us":465685,"raft.run_cpu_time_us":1950,"raft.run_wall_time_us":402483,"spinlock_wait_cycles":16977280,"thread_start_us":336670,"threads_started":1,"wal-append.queue_time_us":506317}
I20260809 23:19:38.861510 30141 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:38.984939 24428 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:38.955470 29832 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:39.013633 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62765.9us
W20260809 23:19:39.022539 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203660us
I20260809 23:19:39.007685 24543 rpcz_store.cc:276] 0809 23:19:37.641238 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.736692 (+ 95454us) service_pool.cc:227] Handling call
0809 23:19:37.736990 (+   298us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.279980 (+542990us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.280010 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.300461 (+ 20451us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.632980 (+332519us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.633320 (+   340us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.633396 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:38.973694 (+340298us) raft_consensus.cc:1671] finished
0809 23:19:38.973716 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:38.973849 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":558549,"raft.queue_time_us":548770,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":4334,"thread_start_us":262052,"threads_started":1,"wal-append.queue_time_us":491281}
I20260809 23:19:38.833974 29745 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.027990 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2559) took 1372 ms. Trace:
I20260809 23:19:39.029090 24543 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:39.059998 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2401) took 1194 ms. Trace:
I20260809 23:19:39.060438 24543 rpcz_store.cc:276] 0809 23:19:37.865959 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.028325 (+1162366us) service_pool.cc:227] Handling call
0809 23:19:39.028647 (+   322us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.059713 (+ 31066us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.059943 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.029800 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:39.059649 24539 rpcz_store.cc:276] 0809 23:19:37.655950 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.738276 (+1082326us) service_pool.cc:227] Handling call
0809 23:19:38.801348 (+ 63072us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.861879 (+ 60531us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.027783 (+165904us) spinlock_profiling.cc:243] Waited 62 ms on lock 0x7b5400271fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:39.027932 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142637952}
I20260809 23:19:38.979409 29856 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:38.994303 29742 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:39.001207 29610 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.808s	user 0.012s	sys 0.001s
W20260809 23:19:39.177143 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.014s	sys 0.026s
W20260809 23:19:39.191987 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.015s	sys 0.026s
W20260809 23:19:39.164868 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.011s	sys 0.018s
I20260809 23:19:38.992869 29666 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:39.109349 24431 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:39.173236 24428 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:38.836064 30181 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.163836 24535 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:39.196237 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 822) took 1348 ms. Trace:
I20260809 23:19:39.196605 24535 rpcz_store.cc:276] 0809 23:19:37.847640 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.007676 (+1160036us) service_pool.cc:227] Handling call
0809 23:19:39.163383 (+155707us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.195953 (+ 32570us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.196152 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.197484 24427 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
W20260809 23:19:39.192731 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.011s	sys 0.018s
I20260809 23:19:39.197971 24427 pending_rounds.cc:76] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 7
I20260809 23:19:39.198273 24427 pending_rounds.cc:97] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 11.8
W20260809 23:19:39.187055 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.016s	sys 0.024s
I20260809 23:19:39.027225 30141 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:39.195137 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:39.028757 30106 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:39.198787 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.017s	sys 0.024s
I20260809 23:19:39.198565 24427 raft_consensus.cc:2866] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:19:39.193864 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.016s	sys 0.025s
I20260809 23:19:39.212438 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2563) took 1332 ms. Trace:
W20260809 23:19:39.220742 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.017s	sys 0.025s
I20260809 23:19:39.221029 24539 rpcz_store.cc:276] 0809 23:19:37.880350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.064725 (+1184375us) service_pool.cc:227] Handling call
0809 23:19:39.211817 (+147092us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.212043 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.212077 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.212132 (+    55us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:39.212182 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.212246 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.212372 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.177835 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2630) took 1171 ms. Trace:
I20260809 23:19:39.210166 24317 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:39.223228 29709 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:39.208665 24423 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 5. Current term is 7. Ops: []
I20260809 23:19:39.029966 29760 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:39.201520 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2639) took 1179 ms. Trace:
I20260809 23:19:39.108660 30075 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:38.981454 30085 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.193123 29924 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:39.108388 30124 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.058691 29832 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:39.167888 29481 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:39.174369 29688 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.703s	user 0.009s	sys 0.002s
I20260809 23:19:39.238390 24321 rpcz_store.cc:276] 0809 23:19:38.006400 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.006533 (+   133us) service_pool.cc:227] Handling call
0809 23:19:39.177408 (+1170875us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:39.177764 (+   356us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":498314624}
I20260809 23:19:39.227511 24320 rpcz_store.cc:276] 0809 23:19:38.021734 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.317114 (+295380us) service_pool.cc:227] Handling call
0809 23:19:38.317419 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.317638 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:38.320507 (+  2869us) log.cc:842] Serialized 14 byte log entry
0809 23:19:38.645555 (+325048us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b18004aec60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:39.201184 (+555629us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.201242 (+    58us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:39.201291 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.201352 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.201466 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":411,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":8681,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":14161,"spinlock_wait_cycles":22988544,"thread_start_us":546694,"threads_started":1}
I20260809 23:19:38.992173 29868 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.217073 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:39.227667 30105 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:19:39.275738 30153 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.001s
I20260809 23:19:39.267220 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2402) took 1348 ms. Trace:
I20260809 23:19:39.281764 24539 rpcz_store.cc:276] 0809 23:19:37.918767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.222540 (+1303773us) service_pool.cc:227] Handling call
0809 23:19:39.266562 (+ 44022us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.266784 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.266821 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.266887 (+    66us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:39.266954 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.267019 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.267148 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.176079 29856 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:39.187654 30158 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.092s
W20260809 23:19:39.275202 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.004s	sys 0.020s
W20260809 23:19:39.276628 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59639.4us
W20260809 23:19:39.304287 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.004s	sys 0.020s
W20260809 23:19:39.264302 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.011s	sys 0.026s
I20260809 23:19:39.167160 29212 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260809 23:19:39.306171 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:39.307015 24428 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:39.236874 29467 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.000s	sys 0.027s
I20260809 23:19:39.033465 30021 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:39.278050 30074 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.013s	sys 0.016s
I20260809 23:19:39.175195 29991 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:39.271103 24540 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.021s	sys 0.053s
W20260809 23:19:39.305423 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.012s	sys 0.026s
W20260809 23:19:39.276496 29305 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.008s	sys 0.019s
W20260809 23:19:39.309077 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.004s	sys 0.035s
I20260809 23:19:39.303738 30153 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.287971 29625 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:39.317874 24539 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:39.277567 29832 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:39.235749 30075 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:39.322273 24316 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
W20260809 23:19:39.345963 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.005s	sys 0.035s
I20260809 23:19:39.306313 30105 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:39.226729 29760 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:39.324283 30074 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.013s	sys 0.016s
W20260809 23:19:39.323968 29467 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.000s	sys 0.028s
I20260809 23:19:39.353252 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 825) took 1430 ms. Trace:
I20260809 23:19:39.306878 29853 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:39.346865 29305 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.010s	sys 0.021s
W20260809 23:19:39.347147 24540 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.025s	sys 0.058s
I20260809 23:19:39.322585 30108 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:39.359323 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.007s	sys 0.019s
I20260809 23:19:39.357539 30075 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:39.361654 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.012s	sys 0.035s
W20260809 23:19:39.388661 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.012s	sys 0.035s
I20260809 23:19:39.389343 29742 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:39.390008 29742 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:39.374176 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.012s	sys 0.034s
I20260809 23:19:39.377369 24312 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260809 23:19:39.355211 30125 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.708s	user 0.008s	sys 0.002s
I20260809 23:19:39.392486 29742 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.392184 30125 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:39.278743 29481 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.396662 29481 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.398350 24311 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:39.394688 30125 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.380075 24539 rpcz_store.cc:276] 0809 23:19:37.922517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.282995 (+1360478us) service_pool.cc:227] Handling call
0809 23:19:39.317640 (+ 34645us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.352937 (+ 35297us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.353181 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.348757 29745 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:39.383422 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.015s	sys 0.017s
W20260809 23:19:39.399969 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.015s	sys 0.017s
W20260809 23:19:39.400245 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 598929us
I20260809 23:19:39.400487 24539 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260809 23:19:39.401084 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2404) took 1444 ms. Trace:
I20260809 23:19:39.401374 24539 rpcz_store.cc:276] 0809 23:19:37.956607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.399836 (+1443229us) service_pool.cc:227] Handling call
0809 23:19:39.400185 (+   349us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.400837 (+   652us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.401035 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.401789 30124 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260809 23:19:39.403034 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 827) took 1193 ms. Trace:
I20260809 23:19:39.403369 24539 rpcz_store.cc:276] 0809 23:19:38.209309 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.402246 (+1192937us) service_pool.cc:227] Handling call
0809 23:19:39.402546 (+   300us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.402722 (+   176us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.402745 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.402783 (+    38us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:39.402819 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.402878 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.402988 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:39.404356 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.012s	sys 0.034s
W20260809 23:19:39.405052 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 644370us
W20260809 23:19:39.372218 30109 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.010s	sys 0.023s
W20260809 23:19:39.374985 29397 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.721s	user 0.007s	sys 0.005s
I20260809 23:19:39.408123 29666 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.409806 29397 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:39.279389 29204 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.411230 29397 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:39.278898 29688 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.411979 30153 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.412703 30158 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:39.365664 29625 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:39.418217 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:39.419103 24321 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:39.420279 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:39.354755 29823 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.413568 24317 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:39.345649 24428 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:39.420264 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:39.419365 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:39.412065 30021 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.426949 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:39.428711 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:39.385705 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.007s	sys 0.019s
W20260809 23:19:39.414023 30109 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.010s	sys 0.024s
I20260809 23:19:39.472723 24320 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260809 23:19:39.477293 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57118.9us
I20260809 23:19:39.478600 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:39.489841 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.026s	sys 0.018s
W20260809 23:19:39.490306 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.027s	sys 0.018s
I20260809 23:19:39.495396 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:39.498214 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2565) took 1261 ms. Trace:
I20260809 23:19:39.498701 24539 rpcz_store.cc:276] 0809 23:19:38.237228 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.404229 (+1167001us) service_pool.cc:227] Handling call
0809 23:19:39.497609 (+ 93380us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.497828 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.497859 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.497909 (+    50us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:39.497962 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.498024 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.498149 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.413213 30158 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:39.500727 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2566) took 1242 ms. Trace:
I20260809 23:19:39.506808 24539 rpcz_store.cc:276] 0809 23:19:38.258282 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.499797 (+1241515us) service_pool.cc:227] Handling call
0809 23:19:39.500131 (+   334us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.500345 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.500390 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.500441 (+    51us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:39.500491 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.500553 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.500670 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.509040 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2567) took 1232 ms. Trace:
I20260809 23:19:39.509482 24539 rpcz_store.cc:276] 0809 23:19:38.276707 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.507985 (+1231278us) service_pool.cc:227] Handling call
0809 23:19:39.508260 (+   275us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.508642 (+   382us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.508680 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:39.508734 (+    54us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:39.508789 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.508858 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.508981 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.512090 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:39.492810 30142 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.018s	sys 0.024s
W20260809 23:19:39.512811 30142 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.018s	sys 0.025s
I20260809 23:19:39.513666 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:39.492899 30198 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.484s	user 0.000s	sys 0.001s
I20260809 23:19:39.431236 29991 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.477171 29688 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.563428 30158 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.564133 29823 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:39.564765 29823 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.502501 30109 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:39.567503 30109 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.570174 30109 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:39.571498 30209 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260809 23:19:39.573014 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2136) took 2733 ms. Trace:
I20260809 23:19:39.573604 24427 rpcz_store.cc:276] 0809 23:19:36.840077 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.723207 (+883130us) service_pool.cc:227] Handling call
0809 23:19:37.778406 (+ 55199us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.219573 (+1441167us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.572689 (+353116us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003b9554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:39.572940 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3262493440}
W20260809 23:19:39.574504 30214 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260809 23:19:39.493454 29747 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:39.575278 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2200) took 1600 ms. Trace:
W20260809 23:19:39.576224 30211 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260809 23:19:39.577739 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1138) took 1084 ms. Trace:
I20260809 23:19:39.578161 25365 rpcz_store.cc:276] 0809 23:19:38.493664 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.494271 (+   607us) service_pool.cc:227] Handling call
0809 23:19:38.494722 (+   451us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.516547 (+ 21825us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.516575 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.517071 (+   496us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.950873 (+433802us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.950909 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.950998 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:39.577495 (+626497us) raft_consensus.cc:1671] finished
0809 23:19:39.577520 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.577670 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41178,"raft.queue_time_us":21874,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":6347,"thread_start_us":421016,"threads_started":1,"wal-append.queue_time_us":815020}
I20260809 23:19:39.583037 29666 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.553782 29625 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:39.413748 30108 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:39.586902 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:39.575806 24323 rpcz_store.cc:276] 0809 23:19:37.975277 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.975467 (+   190us) service_pool.cc:227] Handling call
0809 23:19:37.975741 (+   274us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:38.190647 (+214906us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:38.332369 (+141722us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.332398 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:38.370779 (+ 38381us) log.cc:842] Serialized 52 byte log entry
0809 23:19:38.816906 (+446127us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.855347 (+ 38441us) log.cc:842] Serialized 14 byte log entry
0809 23:19:38.880929 (+ 25582us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.881005 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:39.021927 (+140922us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b18003d1aa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:39.575011 (+553084us) raft_consensus.cc:1671] finished
0809 23:19:39.575040 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.575216 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69727,"prepare.queue_time_us":38269,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":113387,"raft.run_cpu_time_us":1266,"raft.run_wall_time_us":196191,"spinlock_wait_cycles":89755136,"thread_start_us":712686,"threads_started":2,"wal-append.queue_time_us":798928}
I20260809 23:19:39.596004 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:39.597332 24539 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260809 23:19:39.598251 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2569) took 1310 ms. Trace:
W20260809 23:19:39.599666 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 87609.6us
I20260809 23:19:39.601342 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:39.497162 29924 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:39.598718 24539 rpcz_store.cc:276] 0809 23:19:38.287808 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.510551 (+1222743us) service_pool.cc:227] Handling call
0809 23:19:39.510879 (+   328us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.597930 (+ 87051us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.598190 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:39.612367 24539 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260809 23:19:39.625433 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.016s	sys 0.027s
I20260809 23:19:39.591686 30085 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:39.632542 24314 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.017s	sys 0.026s
I20260809 23:19:39.592365 29747 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:39.604259 29853 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:39.544191 30124 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260809 23:19:39.616894 29924 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:39.638645 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.016s	sys 0.027s
I20260809 23:19:39.631155 29868 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.638645 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2573) took 1037 ms. Trace:
I20260809 23:19:39.494870 30105 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:39.621982 30109 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
W20260809 23:19:39.661124 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.008s	sys 0.030s
I20260809 23:19:39.662340 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1057) took 1399 ms. Trace:
I20260809 23:19:39.659778 24539 rpcz_store.cc:276] 0809 23:19:38.601743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.611571 (+1009828us) service_pool.cc:227] Handling call
0809 23:19:39.611942 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.638380 (+ 26438us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.638586 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:39.640334 24314 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.017s	sys 0.026s
W20260809 23:19:39.670564 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.009s	sys 0.031s
W20260809 23:19:39.665486 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 488655us
I20260809 23:19:39.637127 29467 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:39.646183 28889 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:39.668922 25372 rpcz_store.cc:276] 0809 23:19:38.263136 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.313416 (+ 50280us) service_pool.cc:227] Handling call
0809 23:19:38.313809 (+   393us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.314277 (+   468us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.314315 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.327118 (+ 12803us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.692283 (+365165us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:38.692339 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.693057 (+   718us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:39.662060 (+969003us) raft_consensus.cc:1671] finished
0809 23:19:39.662087 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:39.662267 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5996,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6293,"thread_start_us":364401,"threads_started":1,"wal-append.queue_time_us":602220}
I20260809 23:19:39.660617 29924 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:39.713225 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:39.714073 25365 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:39.676229 29853 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:39.733047 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:39.742220 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.009s	sys 0.024s
W20260809 23:19:39.742990 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.010s	sys 0.024s
W20260809 23:19:39.743348 30141 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.519s	user 0.005s	sys 0.006s
I20260809 23:19:39.743796 30141 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.744875 25375 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:39.679530 28889 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:39.677479 29823 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:39.700984 30158 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:39.771510 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.010s	sys 0.043s
I20260809 23:19:39.705787 30150 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:39.721223 30222 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:19:39.732406 30204 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.378s	user 0.000s	sys 0.001s
I20260809 23:19:39.722165 30124 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:39.681176 29467 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:39.863775 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:39.864521 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72957.4us
W20260809 23:19:39.806593 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.010s	sys 0.045s
W20260809 23:19:39.866024 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 836236us
I20260809 23:19:39.791136 29823 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:39.817029 29615 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.828s	user 0.008s	sys 0.003s
I20260809 23:19:39.792680 29853 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:39.894831 29204 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260809 23:19:39.895856 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.005s	sys 0.014s
W20260809 23:19:39.934145 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.005s	sys 0.014s
I20260809 23:19:39.936690 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:39.896428 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.016s	sys 0.034s
I20260809 23:19:39.816613 30222 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:39.926106 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s	user 0.003s	sys 0.029s
W20260809 23:19:39.925352 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.013s	sys 0.017s
W20260809 23:19:39.938979 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.016s	sys 0.034s
I20260809 23:19:39.860448 29688 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.003638 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:40.028260 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:40.033574 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:19:40.031013 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.006s	sys 0.008s
W20260809 23:19:40.034274 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.006s	sys 0.008s
W20260809 23:19:40.034569 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 643712us
W20260809 23:19:39.942524 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.003s	sys 0.029s
W20260809 23:19:40.034884 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.018s	sys 0.021s
W20260809 23:19:39.923760 29369 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.958s	user 0.013s	sys 0.000s
I20260809 23:19:39.805701 29991 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
I20260809 23:19:40.089221 29369 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:40.090898 30106 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:40.091549 30106 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:40.094107 30106 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:39.922387 30236 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.095388 30236 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.097913 30236 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:39.745671 29747 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.025954 25365 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:39.888587 29481 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:39.806258 30158 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:40.073546 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2633) took 2063 ms (client timeout 1549 ms). Trace:
I20260809 23:19:39.921041 30021 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:40.075603 30224 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.354s	user 0.000s	sys 0.001s
I20260809 23:19:40.076555 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:40.124229 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:40.102728 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 235854us
I20260809 23:19:40.126969 24496 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:40.103883 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2561) took 2368 ms. Trace:
I20260809 23:19:40.094630 30106 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.127892 24540 rpcz_store.cc:276] 0809 23:19:37.735748 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.755071 (+1019323us) service_pool.cc:227] Handling call
0809 23:19:38.755424 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.755654 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:39.384216 (+628562us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.103362 (+719146us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:40.103438 (+    76us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:40.103498 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.103669 (+   171us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.103805 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68043,"prepare.queue_time_us":553326,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":1278910,"threads_started":2}
W20260809 23:19:39.928696 29808 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.006s	sys 0.016s
W20260809 23:19:40.114442 24316 rpcz_store.cc:269] 0809 23:19:38.010085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.067257 (+ 57172us) service_pool.cc:227] Handling call
0809 23:19:40.073241 (+2005984us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:40.073450 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2826849536}
I20260809 23:19:39.933205 29823 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.115980 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 967) took 2004 ms. Trace:
W20260809 23:19:40.122390 30115 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.534s	user 0.022s	sys 0.016s
I20260809 23:19:40.160410 24311 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:39.922191 30228 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:39.861838 30150 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.168751 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:39.921108 30153 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.227s
W20260809 23:19:40.170070 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43568.1us
I20260809 23:19:39.885052 30124 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.069099 30108 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:40.223237 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.030s	sys 0.020s
I20260809 23:19:40.130810 30141 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:40.177438 30115 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.597s	user 0.022s	sys 0.016s
I20260809 23:19:40.190003 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:40.178241 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:40.238224 24311 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:40.168185 25366 rpcz_store.cc:276] 0809 23:19:38.111424 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.148410 (+ 36986us) service_pool.cc:227] Handling call
0809 23:19:38.206529 (+ 58119us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.225771 (+ 19242us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.225816 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.226311 (+   495us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.948291 (+721980us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:38.948348 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.948442 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.115728 (+1167286us) raft_consensus.cc:1671] finished
0809 23:19:40.115754 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.115911 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18386,"raft.queue_time_us":19627,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":3386,"thread_start_us":662325,"threads_started":1,"wal-append.queue_time_us":1004851}
I20260809 23:19:39.933665 29853 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:40.169349 29808 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.010s	sys 0.020s
W20260809 23:19:40.232540 30219 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.563s	user 0.001s	sys 0.001s
W20260809 23:19:40.235199 30181 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.014s	sys 0.017s
I20260809 23:19:40.123594 29481 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.242549 30142 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.013s	sys 0.039s
W20260809 23:19:40.219123 24539 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.014s	sys 0.029s
I20260809 23:19:40.115053 30240 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:39.930181 29615 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:40.191525 30074 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.015s	sys 0.017s
W20260809 23:19:40.223691 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.030s	sys 0.020s
I20260809 23:19:40.166977 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:40.168602 29810 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:40.070339 29688 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.233620 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.006s	sys 0.026s
W20260809 23:19:40.281590 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.007s	sys 0.026s
W20260809 23:19:40.193859 30221 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.014s	sys 0.014s
I20260809 23:19:40.177918 30242 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:39.921592 28889 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.184971 24316 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:19:40.258602 30142 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.013s	sys 0.040s
W20260809 23:19:40.243431 30223 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.565s	user 0.001s	sys 0.000s
W20260809 23:19:40.256258 24539 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.014s	sys 0.029s
I20260809 23:19:40.300415 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 821) took 2453 ms. Trace:
W20260809 23:19:40.246397 30125 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.018s	sys 0.035s
W20260809 23:19:40.239876 30127 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.933s	user 0.012s	sys 0.019s
W20260809 23:19:40.287472 30221 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.015s	sys 0.014s
I20260809 23:19:40.280822 30141 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:40.261109 24540 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
W20260809 23:19:40.252271 30181 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.025s	user 0.014s	sys 0.017s
W20260809 23:19:40.254717 29745 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.000s	sys 0.030s
I20260809 23:19:40.364619 29481 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:40.192607 30228 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.301054 29853 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:40.366318 29853 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:40.366917 30228 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:40.256903 29987 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.010s	sys 0.019s
I20260809 23:19:40.288916 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:40.367970 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2398) took 2720 ms. Trace:
I20260809 23:19:40.368944 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:40.171063 30021 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.334264 30242 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:40.370525 30232 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 23:19:40.260416 30074 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.015s	sys 0.017s
I20260809 23:19:40.300983 24546 rpcz_store.cc:276] 0809 23:19:37.847200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.810131 (+962931us) service_pool.cc:227] Handling call
0809 23:19:38.810465 (+   334us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.978787 (+168322us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.978833 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:38.978885 (+    52us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.978954 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.300115 (+1321161us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400146c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:40.300181 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.300345 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386549632}
W20260809 23:19:40.321895 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.034s	sys 0.018s
I20260809 23:19:40.368552 24534 rpcz_store.cc:276] 0809 23:19:37.647789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.731915 (+1084126us) service_pool.cc:227] Handling call
0809 23:19:38.732230 (+   315us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.735995 (+  3765us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.736023 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.736445 (+   422us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.304915 (+568470us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:39.304970 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.305066 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.367691 (+1062625us) raft_consensus.cc:1671] finished
0809 23:19:40.367714 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.367889 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3406,"raft.queue_time_us":9913,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":568079,"threads_started":1,"wal-append.queue_time_us":908810}
W20260809 23:19:40.406896 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.034s	sys 0.019s
W20260809 23:19:40.326112 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.025s	sys 0.035s
I20260809 23:19:40.174794 30106 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.213361 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.000s	sys 0.028s
W20260809 23:19:40.411695 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.000s	sys 0.028s
I20260809 23:19:40.320292 29467 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:40.339082 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 401006us
W20260809 23:19:40.333834 30127 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.022s	user 0.012s	sys 0.019s
I20260809 23:19:40.363034 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:40.424067 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:40.252532 29666 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:40.371528 30242 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:40.372151 29615 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:40.372377 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.007s	sys 0.028s
I20260809 23:19:40.215906 30222 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.434056 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.007s	sys 0.028s
I20260809 23:19:40.372577 29305 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:40.326130 28889 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:40.373040 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 965) took 1137 ms. Trace:
W20260809 23:19:40.338589 30233 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.433s	user 0.000s	sys 0.001s
I20260809 23:19:40.373663 30141 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.375864 30249 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.057s	user 0.002s	sys 0.000s
I20260809 23:19:40.431591 29467 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:40.420305 30228 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.475322 29467 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.475505 30228 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.477393 29467 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:40.459501 30150 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:40.378387 30243 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
W20260809 23:19:40.379266 30234 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
W20260809 23:19:40.379895 30230 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.310s	user 0.001s	sys 0.000s
W20260809 23:19:40.381933 30235 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
W20260809 23:19:40.325086 30125 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.018s	sys 0.035s
W20260809 23:19:40.382691 30248 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260809 23:19:40.383961 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2154) took 2412 ms. Trace:
W20260809 23:19:40.387342 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 197653us
I20260809 23:19:40.496176 24423 rpcz_store.cc:276] 0809 23:19:37.971912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.326006 (+354094us) service_pool.cc:227] Handling call
0809 23:19:38.381289 (+ 55283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.236693 (+855404us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.383674 (+1146981us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:40.383885 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1869431040}
I20260809 23:19:40.496641 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:40.497987 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:40.395382 30220 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.820s	user 0.011s	sys 0.004s
I20260809 23:19:40.404997 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2649) took 1791 ms. Trace:
W20260809 23:19:40.282881 30231 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.535s	user 0.000s	sys 0.001s
W20260809 23:19:40.409816 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.025s	sys 0.035s
W20260809 23:19:40.501189 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 756952us
I20260809 23:19:40.399370 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:40.408967 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 964) took 1613 ms. Trace:
I20260809 23:19:40.504011 24314 rpcz_store.cc:276] 0809 23:19:38.795979 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.836971 (+ 40992us) service_pool.cc:227] Handling call
0809 23:19:38.881611 (+ 44640us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:38.881880 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:39.671283 (+789403us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.408477 (+737194us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.408626 (+   149us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:40.408675 (+    49us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:40.408732 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.408800 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.408916 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5183,"prepare.queue_time_us":82306,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":4556,"raft.queue_time_us":7982,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5883,"spinlock_wait_cycles":8466176,"thread_start_us":1449463,"threads_started":2}
W20260809 23:19:40.225432 29212 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.030s	sys 0.019s
W20260809 23:19:40.334975 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.034s	sys 0.018s
W20260809 23:19:40.518733 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.034s	sys 0.018s
I20260809 23:19:40.519101 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:40.378387 30194 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.647s	user 0.013s	sys 0.044s
W20260809 23:19:40.424230 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61352.9us
W20260809 23:19:40.364837 29745 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.000s	sys 0.031s
I20260809 23:19:40.428244 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:40.542512 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:40.337354 30247 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:40.364753 30251 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:40.412330 30075 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.991s	user 0.004s	sys 0.008s
I20260809 23:19:40.433226 24316 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:40.407384 29939 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.458911 24317 rpcz_store.cc:276] 0809 23:19:39.235619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.236420 (+   801us) service_pool.cc:227] Handling call
0809 23:19:39.277006 (+ 40586us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.425847 (+148841us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.372800 (+946953us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:40.372988 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313631488}
I20260809 23:19:40.432895 29666 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:40.470240 29688 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:40.470873 30202 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.162s	user 0.005s	sys 0.014s
W20260809 23:19:40.472354 30253 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:19:40.458143 30158 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:40.477658 30228 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:40.377400 30021 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:40.432717 29987 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:40.487697 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.001s	sys 0.009s
I20260809 23:19:40.441656 29823 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:40.393431 30213 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.833s	user 0.006s	sys 0.008s
I20260809 23:19:40.388026 30240 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:19:40.510202 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:40.519107 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 712501us
I20260809 23:19:40.499583 24313 rpcz_store.cc:276] 0809 23:19:38.613219 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.675647 (+ 62428us) service_pool.cc:227] Handling call
0809 23:19:38.678442 (+  2795us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.680426 (+  1984us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.680456 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.682757 (+  2301us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.345492 (+662735us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:39.345552 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.345656 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.339170 (+993514us) raft_consensus.cc:1671] finished
0809 23:19:40.339191 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.404914 (+ 65723us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1531,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5993,"thread_start_us":662265,"threads_started":1,"wal-append.queue_time_us":949673}
I20260809 23:19:40.549984 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:40.488543 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.003s	sys 0.007s
I20260809 23:19:40.484779 30221 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:40.518154 29212 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.031s	sys 0.020s
I20260809 23:19:40.736708 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2646) took 2468 ms. Trace:
I20260809 23:19:40.740887 24309 rpcz_store.cc:276] 0809 23:19:38.268051 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.662849 (+394798us) service_pool.cc:227] Handling call
0809 23:19:38.681063 (+ 18214us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.681426 (+   363us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.681450 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.689084 (+  7634us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.490791 (+801707us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:39.490860 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.490967 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.736404 (+1245437us) raft_consensus.cc:1671] finished
0809 23:19:40.736433 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.736627 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":179,"raft.queue_time_us":1580,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3080,"thread_start_us":731549,"threads_started":1,"wal-append.queue_time_us":1068045}
W20260809 23:19:40.744158 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 234038us
W20260809 23:19:40.755445 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.000s	sys 0.024s
I20260809 23:19:40.526518 29855 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:40.518953 30124 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:40.611241 24311 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:40.834857 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2652) took 1155 ms. Trace:
I20260809 23:19:40.835306 24311 rpcz_store.cc:276] 0809 23:19:39.679007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.178003 (+498996us) service_pool.cc:227] Handling call
0809 23:19:40.834777 (+656774us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:40.558846 29834 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.312s	user 0.034s	sys 0.034s
W20260809 23:19:40.683012 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79286.5us
I20260809 23:19:40.557072 30108 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:40.683045 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:40.541504 30150 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.093s
I20260809 23:19:40.560364 30247 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:40.695493 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2159) took 2276 ms. Trace:
W20260809 23:19:40.694638 30105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.950s	user 0.016s	sys 0.037s
W20260809 23:19:40.754513 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.011s	sys 0.013s
I20260809 23:19:40.756909 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2399) took 3108 ms. Trace:
I20260809 23:19:40.603300 29939 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.527172 29305 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:40.814086 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.022s	user 0.007s	sys 0.024s
I20260809 23:19:40.815001 30276 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:40.689538 30243 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:19:40.836903 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2562) took 3062 ms. Trace:
I20260809 23:19:40.585191 30075 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:40.700660 29987 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:40.911679 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2395) took 3284 ms. Trace:
I20260809 23:19:40.917200 30127 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:40.756516 29709 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:40.697454 29204 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:40.912725 24534 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:40.565145 30251 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.924211 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.012s	sys 0.019s
W20260809 23:19:40.897192 29834 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.668s	user 0.034s	sys 0.034s
W20260809 23:19:40.901037 29853 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.001s
I20260809 23:19:40.814020 30240 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:40.700618 30228 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.220s
W20260809 23:19:40.922788 29856 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.599s	user 0.014s	sys 0.011s
I20260809 23:19:40.915849 30221 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.932293 30105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.179s	user 0.024s	sys 0.039s
I20260809 23:19:40.914988 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1048) took 2830 ms. Trace:
I20260809 23:19:40.916092 30154 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.920351 30247 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:40.921224 30106 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:40.924280 29855 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:40.924482 30124 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:40.959079 29747 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.009s	sys 0.014s
I20260809 23:19:40.969630 29987 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:40.922004 30108 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:40.993212 29832 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.616s	user 0.012s	sys 0.003s
I20260809 23:19:40.970643 30289 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.007761 29853 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:41.022490 30246 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260809 23:19:41.002043 30141 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:40.995563 30085 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.016707 30228 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:40.993000 30127 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:40.998859 30284 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.046427 30127 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.046950 30284 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:41.048502 30212 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.232s	user 0.003s	sys 0.000s
I20260809 23:19:41.042593 29832 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.049446 30284 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.050412 29832 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.050534 30221 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.051851 30114 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.052095 30251 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.053418 30075 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.044028 30085 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:41.033282 30300 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260809 23:19:41.031894 29747 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.013s	sys 0.014s
I20260809 23:19:40.965131 30276 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.017860 30150 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.031589 30232 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.008052 29856 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.054940 30228 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:41.017719 30154 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.007807 30240 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.055909 30289 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.065115 30232 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.072397 30232 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.072685 30289 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.073902 30276 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.074062 30228 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.076026 30228 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.076570 30154 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.085114 30240 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.089864 30247 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.096153 30247 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:40.695314 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.001s	sys 0.009s
I20260809 23:19:41.143314 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40854 (request call id 1256) took 1341 ms. Trace:
I20260809 23:19:41.144326 24311 rpcz_store.cc:276] 0809 23:19:39.801424 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.836214 (+1034790us) service_pool.cc:227] Handling call
0809 23:19:41.142856 (+306642us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:40.832602 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:41.146952 24309 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:40.800040 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.005s	sys 0.012s
I20260809 23:19:41.151829 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2213) took 1727 ms. Trace:
I20260809 23:19:41.152364 24312 rpcz_store.cc:276] 0809 23:19:39.424157 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.583858 (+159701us) service_pool.cc:227] Handling call
0809 23:19:39.620652 (+ 36794us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.620940 (+   288us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.620978 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:39.621531 (+   553us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.412223 (+790692us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:40.412262 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.412339 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.151232 (+738893us) raft_consensus.cc:1671] finished
0809 23:19:41.151291 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.151610 (+   319us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":769183,"threads_started":1,"wal-append.queue_time_us":1384896}
W20260809 23:19:41.155824 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 8 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:40.743189 24314 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:40.920670 24429 rpcz_store.cc:276] 0809 23:19:38.419142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.621236 (+202094us) service_pool.cc:227] Handling call
0809 23:19:38.621530 (+   294us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.621735 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.621758 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.631212 (+  9454us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.220005 (+588793us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:39.220060 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.220388 (+   328us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.394577 (+1174189us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:40.683119 (+288542us) raft_consensus.cc:1671] finished
0809 23:19:40.683150 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.695424 (+ 12274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39878528,"thread_start_us":536777,"threads_started":1,"wal-append.queue_time_us":758282}
I20260809 23:19:41.162827 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 969) took 1514 ms. Trace:
I20260809 23:19:41.163251 24323 rpcz_store.cc:276] 0809 23:19:39.648253 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.648616 (+   363us) service_pool.cc:227] Handling call
0809 23:19:39.649103 (+   487us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:19:39.668507 (+ 19404us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.668552 (+    45us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:19:39.669350 (+   798us) log.cc:842] Serialized 78 byte log entry
0809 23:19:40.183592 (+514242us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:40.558191 (+374599us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.558523 (+   332us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.558654 (+   131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.162074 (+603420us) raft_consensus.cc:1671] finished
0809 23:19:41.162124 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.162608 (+   484us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":62596,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":56531,"raft.run_cpu_time_us":3927,"raft.run_wall_time_us":83644,"thread_start_us":806373,"threads_started":2,"wal-append.queue_time_us":1326181}
I20260809 23:19:41.161160 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2209) took 1738 ms. Trace:
I20260809 23:19:41.167297 24321 rpcz_store.cc:276] 0809 23:19:39.422993 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.430362 (+  7369us) service_pool.cc:227] Handling call
0809 23:19:39.473690 (+ 43328us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:39.474268 (+   578us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.474311 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:39.475242 (+   931us) log.cc:842] Serialized 52 byte log entry
0809 23:19:40.282126 (+806884us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:40.282182 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.282272 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.160503 (+878231us) raft_consensus.cc:1671] finished
0809 23:19:41.160558 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.160939 (+   381us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2183,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5327,"thread_start_us":806464,"threads_started":1,"wal-append.queue_time_us":1575195}
I20260809 23:19:41.169600 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 734) took 3042 ms. Trace:
I20260809 23:19:41.169984 24424 rpcz_store.cc:276] 0809 23:19:38.126601 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.327920 (+201319us) service_pool.cc:227] Handling call
0809 23:19:38.328209 (+   289us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.479924 (+151715us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.479970 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.489550 (+  9580us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.007975 (+518425us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:39.008015 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.394016 (+1386001us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:40.394081 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.047923 (+653842us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:41.169030 (+121107us) raft_consensus.cc:1671] finished
0809 23:19:41.169064 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.169345 (+   281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3911,"raft.queue_time_us":4132,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":16406,"spinlock_wait_cycles":350199552,"thread_start_us":483698,"threads_started":1,"wal-append.queue_time_us":1170445}
I20260809 23:19:40.920732 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2155) took 2827 ms. Trace:
I20260809 23:19:41.172410 24425 rpcz_store.cc:276] 0809 23:19:38.093073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.327911 (+234838us) service_pool.cc:227] Handling call
0809 23:19:38.375757 (+ 47846us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.396276 (+ 20519us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.396308 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.396786 (+   478us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.631469 (+234683us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b1800370060. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:39.221491 (+590022us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:39.221547 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.221635 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.392617 (+1170982us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:40.920524 (+527907us) raft_consensus.cc:1671] finished
0809 23:19:40.920545 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.920672 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13410,"raft.queue_time_us":13629,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":17689,"spinlock_wait_cycles":104821760,"thread_start_us":589807,"threads_started":1,"wal-append.queue_time_us":1018758}
I20260809 23:19:40.695622 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:40.924305 24541 rpcz_store.cc:276] 0809 23:19:37.648094 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.734832 (+1086738us) service_pool.cc:227] Handling call
0809 23:19:38.735541 (+   709us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:38.762633 (+ 27092us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.762675 (+    42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:38.763322 (+   647us) log.cc:842] Serialized 52 byte log entry
0809 23:19:39.431136 (+667814us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:39.431189 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.431286 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.756673 (+1325387us) raft_consensus.cc:1671] finished
0809 23:19:40.756695 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.756831 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26826,"raft.queue_time_us":27164,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":15401,"thread_start_us":614159,"threads_started":1,"wal-append.queue_time_us":1405260}
I20260809 23:19:41.178680 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.180480 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:41.180708 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.181562 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 830) took 1436 ms. Trace:
I20260809 23:19:41.181905 24541 rpcz_store.cc:276] 0809 23:19:39.745678 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.180307 (+1434629us) service_pool.cc:227] Handling call
0809 23:19:41.181496 (+  1189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:41.182588 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2574) took 2551 ms. Trace:
I20260809 23:19:41.183511 24539 rpcz_store.cc:276] 0809 23:19:38.631509 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.672696 (+1041187us) service_pool.cc:227] Handling call
0809 23:19:39.673100 (+   404us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:40.300533 (+627433us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:40.300567 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:40.301029 (+   462us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.599357 (+298328us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:40.599413 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.599514 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.182039 (+582525us) raft_consensus.cc:1671] finished
0809 23:19:41.182073 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.182351 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30035,"raft.queue_time_us":220501,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":889105,"threads_started":2,"wal-append.queue_time_us":752442}
I20260809 23:19:41.184628 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:41.186074 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.191708 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 823) took 3315 ms. Trace:
I20260809 23:19:41.192150 24543 rpcz_store.cc:276] 0809 23:19:37.876499 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.061292 (+1184793us) service_pool.cc:227] Handling call
0809 23:19:39.061721 (+   429us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.062110 (+   389us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.062148 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:39.162776 (+100628us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.038012 (+875236us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:40.166416 (+128404us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.367214 (+200798us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b18004d38c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:19:40.367457 (+   243us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.367558 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.191144 (+823586us) raft_consensus.cc:1671] finished
0809 23:19:41.191276 (+   132us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.191541 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4001,"prepare.queue_time_us":130010,"prepare.run_cpu_time_us":1305,"prepare.run_wall_time_us":22718,"raft.queue_time_us":344791,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"spinlock_wait_cycles":21131520,"thread_start_us":810359,"threads_started":1,"wal-append.queue_time_us":1337897}
I20260809 23:19:40.963377 24544 rpcz_store.cc:276] 0809 23:19:37.627132 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.702643 (+ 75511us) service_pool.cc:227] Handling call
0809 23:19:37.720162 (+ 17519us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.220099 (+499937us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.220130 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.220533 (+   403us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.994143 (+773610us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:38.994205 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.994294 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.398353 (+1404059us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:40.899613 (+501260us) raft_consensus.cc:1671] finished
0809 23:19:40.899646 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.911600 (+ 11954us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":531930,"raft.queue_time_us":504168,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4432,"spinlock_wait_cycles":25799040,"thread_start_us":773231,"threads_started":1,"wal-append.queue_time_us":1081615}
I20260809 23:19:41.196038 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.196871 24544 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:41.197311 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2577) took 1094 ms. Trace:
I20260809 23:19:41.197619 24544 rpcz_store.cc:276] 0809 23:19:40.102692 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.195598 (+1092906us) service_pool.cc:227] Handling call
0809 23:19:41.197270 (+  1672us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:40.951509 24545 rpcz_store.cc:276] 0809 23:19:37.774726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.783427 (+1008701us) service_pool.cc:227] Handling call
0809 23:19:38.783821 (+   394us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:38.791012 (+  7191us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.791043 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:38.820879 (+ 29836us) log.cc:842] Serialized 52 byte log entry
0809 23:19:39.672318 (+851439us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:39.672369 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.672459 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.836600 (+1164141us) raft_consensus.cc:1671] finished
0809 23:19:40.836633 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.836815 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6723,"raft.queue_time_us":16212,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":789980,"threads_started":1,"wal-append.queue_time_us":1286009}
I20260809 23:19:41.188235 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.201347 24541 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:41.201994 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2415) took 1451 ms. Trace:
I20260809 23:19:41.202349 24541 rpcz_store.cc:276] 0809 23:19:39.751032 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.187582 (+1436550us) service_pool.cc:227] Handling call
0809 23:19:41.201914 (+ 14332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:41.181681 24534 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 23:19:41.190001 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2556) took 3544 ms. Trace:
I20260809 23:19:41.200186 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:40.983541 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.008144 25369 rpcz_store.cc:276] 0809 23:19:38.084839 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.119882 (+ 35043us) service_pool.cc:227] Handling call
0809 23:19:38.120171 (+   289us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.120369 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:38.120390 (+    21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:38.202776 (+ 82386us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:38.221995 (+ 19219us) log.cc:842] Serialized 26 byte log entry
0809 23:19:38.527427 (+305432us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:38.527933 (+   506us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:38.528043 (+   110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:40.378082 (+1850039us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b240036d418. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:19:40.914747 (+536665us) raft_consensus.cc:1671] finished
0809 23:19:40.914769 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:40.914903 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":138166912,"thread_start_us":261567,"threads_started":1,"wal-append.queue_time_us":462256}
I20260809 23:19:40.985261 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:41.297514 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.004s	sys 0.008s
I20260809 23:19:41.297793 25366 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
W20260809 23:19:41.018816 28889 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.551s	user 0.005s	sys 0.005s
I20260809 23:19:41.203869 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1142) took 1764 ms. Trace:
I20260809 23:19:40.995220 29204 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.198625 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.305445 24544 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:41.194584 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:41.307978 30181 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.186661 24321 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:41.206881 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.203893 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1071) took 1471 ms. Trace:
I20260809 23:19:41.311283 25363 rpcz_store.cc:276] 0809 23:19:39.732406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.067665 (+335259us) service_pool.cc:227] Handling call
0809 23:19:40.129131 (+ 61466us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:40.129560 (+   429us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:40.129597 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:40.130107 (+   510us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.990516 (+860409us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:40.990566 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.990637 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.203188 (+212551us) raft_consensus.cc:1671] finished
0809 23:19:41.203243 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.203603 (+   360us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1389,"raft.queue_time_us":103199,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7649,"thread_start_us":757134,"threads_started":1,"wal-append.queue_time_us":850683}
I20260809 23:19:41.210875 29305 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.213064 29369 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.210533 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.219421 29615 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:41.221504 29745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.002s	sys 0.024s
I20260809 23:19:41.221221 29666 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.324132 30153 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:41.221937 29625 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:41.325513 29625 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.223507 29823 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:41.226043 24546 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:41.023897 29855 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.227692 29856 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:41.064431 29924 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.343s	user 0.012s	sys 0.002s
I20260809 23:19:41.228600 29868 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.229256 30105 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.229749 29939 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.042330 30124 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.334213 29868 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.337934 29868 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.063934 30150 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.338899 29855 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.340327 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:41.340633 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 972) took 1040 ms. Trace:
I20260809 23:19:41.233060 30108 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.235316 30021 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.341190 24316 rpcz_store.cc:276] 0809 23:19:40.300353 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.368633 (+ 68280us) service_pool.cc:227] Handling call
0809 23:19:41.340249 (+971616us) spinlock_profiling.cc:243] Waited 63.4 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.340436 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145835904}
I20260809 23:19:41.238185 30085 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.228173 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:41.343120 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2653) took 1663 ms. Trace:
I20260809 23:19:41.343225 24323 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:41.298087 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.004s	sys 0.008s
I20260809 23:19:41.188882 24317 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:41.353377 30105 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.354327 30141 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:41.354583 29369 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.309298 30181 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.353457 29823 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.322969 29615 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:41.324411 29745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.002s	sys 0.024s
I20260809 23:19:41.314654 29305 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.325423 30153 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.368957 29305 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.355000 30141 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.054482 29853 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.370339 30141 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:41.304649 25368 rpcz_store.cc:276] 0809 23:19:39.438833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.480741 (+ 41908us) service_pool.cc:227] Handling call
0809 23:19:39.504700 (+ 23959us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.505109 (+   409us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.505145 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:39.505688 (+   543us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.261689 (+756001us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:40.261745 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.261824 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.203216 (+941392us) raft_consensus.cc:1671] finished
0809 23:19:41.203259 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.203604 (+   345us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65710,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":654970,"threads_started":1,"wal-append.queue_time_us":1506791}
I20260809 23:19:41.327423 24544 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:38878
I20260809 23:19:41.371721 24544 raft_consensus.cc:595] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:41.372243 24544 raft_consensus.cc:603] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:41.230259 30106 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.375540 29625 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.375380 30106 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.377533 30124 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.377899 30106 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.280752 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2410) took 1520 ms. Trace:
I20260809 23:19:41.378396 29823 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.378526 24534 rpcz_store.cc:276] 0809 23:19:39.748480 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.984697 (+1236217us) service_pool.cc:227] Handling call
0809 23:19:41.265403 (+280706us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:41.253499 24542 rpcz_store.cc:276] 0809 23:19:37.645021 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:37.780758 (+135737us) service_pool.cc:227] Handling call
0809 23:19:37.790830 (+ 10072us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:38.235405 (+444575us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:38.952535 (+717130us) log.cc:842] Serialized 14 byte log entry
0809 23:19:40.108849 (+1156314us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:40.127903 (+ 19054us) log.cc:842] Serialized 26 byte log entry
0809 23:19:40.166986 (+ 39083us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:40.167049 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:40.167160 (+   111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.189197 (+1022037us) raft_consensus.cc:1671] finished
0809 23:19:41.189244 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.189536 (+   292us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":544778,"prepare.queue_time_us":1020452,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":778378,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":43885,"thread_start_us":1711318,"threads_started":2,"wal-append.queue_time_us":1133394}
I20260809 23:19:41.383291 25368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:41.383251 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2575) took 1995 ms. Trace:
I20260809 23:19:41.343463 24309 rpcz_store.cc:276] 0809 23:19:39.679559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.742747 (+1063188us) service_pool.cc:227] Handling call
0809 23:19:41.343038 (+600291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:41.354010 30251 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:41.316817 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:41.357023 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 605604us
W20260809 23:19:41.358574 30217 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.805s	user 0.007s	sys 0.014s
I20260809 23:19:41.359560 29204 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.360075 29688 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.340859 30150 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:41.360705 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 375383us
I20260809 23:19:41.361033 30105 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.358675 29369 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:41.331516 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 125122us
W20260809 23:19:41.363184 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.853s (SENT) [suppressed 92 similar messages]
I20260809 23:19:41.341981 30021 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.364869 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2564) took 3484 ms. Trace:
I20260809 23:19:41.364671 29924 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.221975 25370 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:41.407927 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81095.2us
I20260809 23:19:41.409034 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:41.410251 30105 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:41.358762 30181 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.404196 29688 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.386665 24543 rpcz_store.cc:276] 0809 23:19:39.356872 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.307689 (+1950817us) service_pool.cc:227] Handling call
0809 23:19:41.308352 (+   663us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.308937 (+   585us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:41.309526 (+   589us) log.cc:842] Serialized 14 byte log entry
0809 23:19:41.351719 (+ 42193us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:41.351779 (+    60us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:41.351900 (+   121us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.351974 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.352087 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1888,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":6129,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":16087,"thread_start_us":21856,"threads_started":1}
I20260809 23:19:41.392714 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2416) took 1548 ms. Trace:
I20260809 23:19:41.395907 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2576) took 1460 ms. Trace:
I20260809 23:19:41.411355 30106 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260809 23:19:41.422654 29853 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:41.419045 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 88143.4us
W20260809 23:19:41.428741 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1045) took 3397 ms (client timeout 2077 ms). Trace:
I20260809 23:19:41.430456 24540 raft_consensus.cc:2354] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:41.391597 30251 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:41.431386 30167 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.970s	user 0.009s	sys 0.001s
I20260809 23:19:41.431715 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:41.431717 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.868s	user 0.008s	sys 0.024s
I20260809 23:19:41.432857 29924 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.432940 30153 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:41.434537 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2412) took 1685 ms (client timeout 1950 ms). Trace:
I20260809 23:19:41.477397 24544 rpcz_store.cc:276] 0809 23:19:39.839444 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.373458 (+1534014us) service_pool.cc:227] Handling call
0809 23:19:41.373960 (+   502us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.374609 (+   649us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:41.374638 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:41.374698 (+    60us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:41.374740 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.374797 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.388049 (+ 13252us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3688,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173}
I20260809 23:19:41.434758 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.410908 30105 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:41.434530 30255 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.096s	user 0.004s	sys 0.000s
I20260809 23:19:41.412269 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.416554 30021 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.436404 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.438462 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2654) took 1757 ms. Trace:
I20260809 23:19:41.418649 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1140) took 1994 ms. Trace:
I20260809 23:19:41.443257 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:41.443781 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:41.445572 29615 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.402418 24535 rpcz_store.cc:276] 0809 23:19:37.880716 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.220887 (+1340171us) service_pool.cc:227] Handling call
0809 23:19:39.269353 (+ 48466us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.269706 (+   353us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:39.269737 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:39.270369 (+   632us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.644458 (+374089us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:39.644498 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.644569 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.362430 (+1717861us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b24003c0158. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:19:41.364305 (+  1875us) raft_consensus.cc:1671] finished
0809 23:19:41.364330 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.364667 (+   337us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":884,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":11573,"spinlock_wait_cycles":8223616,"thread_start_us":325223,"threads_started":1,"wal-append.queue_time_us":639253}
W20260809 23:19:41.451571 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43661.1us
I20260809 23:19:41.452621 30222 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:41.454998 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.001s	sys 0.009s
I20260809 23:19:41.461709 30141 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.465202 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.505394 25369 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:41.466291 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.467276 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:41.468379 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:41.466728 29987 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:41.507831 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.002s	sys 0.009s
I20260809 23:19:41.491643 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:41.493999 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.065s	user 0.008s	sys 0.024s
I20260809 23:19:41.494083 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1068) took 1783 ms. Trace:
I20260809 23:19:41.501238 25375 rpcz_store.cc:276] 0809 23:19:39.423662 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.428398 (+  4736us) service_pool.cc:227] Handling call
0809 23:19:39.473780 (+ 45382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:39.745540 (+271760us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.417989 (+1672449us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:41.418407 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":623045888}
W20260809 23:19:41.494959 24546 rpcz_store.cc:269] 0809 23:19:39.749721 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.406309 (+656588us) service_pool.cc:227] Handling call
0809 23:19:41.434295 (+1027986us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b5400080a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.434466 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1841201536}
W20260809 23:19:41.487082 25371 rpcz_store.cc:269] 0809 23:19:38.031346 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:38.080016 (+ 48670us) service_pool.cc:227] Handling call
0809 23:19:41.428274 (+3348258us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.428518 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4954167040}
W20260809 23:19:41.489686 30167 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.109s	user 0.010s	sys 0.001s
I20260809 23:19:41.477799 24541 rpcz_store.cc:276] 0809 23:19:39.935249 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.263308 (+1328059us) service_pool.cc:227] Handling call
0809 23:19:41.395836 (+132528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:41.489892 30251 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.508590 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.508915 25370 rpcz_store.cc:276] 0809 23:19:39.711059 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.711228 (+   169us) service_pool.cc:227] Handling call
0809 23:19:39.787123 (+ 75895us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.407141 (+1620018us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.486965 (+ 79824us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54004a5bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:41.494001 (+  7036us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3298067712}
I20260809 23:19:41.516949 24323 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:41.517369 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:41.517968 29760 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:41.502959 30067 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.515187 24543 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:41.518909 29760 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.521878 30115 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.555112 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.556123 25375 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:41.555673 30115 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.557060 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.557850 25375 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:41.499938 24313 rpcz_store.cc:276] 0809 23:19:39.680726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.754828 (+1074102us) service_pool.cc:227] Handling call
0809 23:19:40.812370 (+ 57542us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:40.812775 (+   405us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:40.812815 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:40.835221 (+ 22406us) log.cc:842] Serialized 26 byte log entry
0809 23:19:41.318544 (+483323us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:41.350526 (+ 31982us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.361464 (+ 10938us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.437934 (+ 76470us) raft_consensus.cc:1671] finished
0809 23:19:41.437960 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.438220 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8571,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":76300,"thread_start_us":120176,"threads_started":1,"wal-append.queue_time_us":273900}
I20260809 23:19:41.522110 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.560801 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.561229 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.562403 29760 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.558807 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:41.522306 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1144) took 2040 ms (client timeout 2085 ms). Trace:
I20260809 23:19:41.571861 25375 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260809 23:19:41.571805 25373 rpcz_store.cc:269] 0809 23:19:39.481709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.481845 (+   136us) service_pool.cc:227] Handling call
0809 23:19:41.521991 (+2040146us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400631354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.522183 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4247641472}
I20260809 23:19:41.573108 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.522617 24317 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:41.523432 30228 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.575891 30228 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.562693 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.524696 30109 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.579752 25372 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:41.581051 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.580204 30109 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.582167 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.567709 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2420) took 1039 ms. Trace:
I20260809 23:19:41.519848 30067 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.524937 29708 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.526795 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:41.527607 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2414) took 1777 ms (client timeout 2181 ms). Trace:
I20260809 23:19:41.528632 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.530565 30106 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.530897 30141 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.531153 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.586337 30109 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:41.532526 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 120444us
I20260809 23:19:41.532128 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:41.533572 30105 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.535039 30331 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:41.536108 29709 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.545s	user 0.004s	sys 0.019s
I20260809 23:19:41.536491 29688 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.539307 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.516438 24544 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:41.545470 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.539765 29987 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.545809 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:41.591609 29815 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.271s	user 0.063s	sys 0.070s
I20260809 23:19:41.540645 24541 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:41.546795 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.547353 30339 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.547530 30150 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.547236 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.632612 25370 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:41.593364 30115 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.542479 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2212) took 2118 ms. Trace:
I20260809 23:19:41.547812 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.529556 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:41.549753 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.633611 24316 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.543586 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.549017 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.629869 25366 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:41.595008 30228 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.596691 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.582788 30251 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.599781 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.600689 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.601045 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.602348 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.573441 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:41.611608 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 255409us
I20260809 23:19:41.611835 24542 rpcz_store.cc:276] 0809 23:19:40.528662 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.383512 (+854850us) service_pool.cc:227] Handling call
0809 23:19:41.384016 (+   504us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:41.384697 (+   681us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:41.384724 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:41.385334 (+   610us) log.cc:842] Serialized 26 byte log entry
0809 23:19:41.494742 (+109408us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:41.494795 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.494881 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.567484 (+ 72603us) raft_consensus.cc:1671] finished
0809 23:19:41.567509 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.567635 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33646,"raft.queue_time_us":5612,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":26776,"thread_start_us":108085,"threads_started":1,"wal-append.queue_time_us":168285}
I20260809 23:19:41.637547 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.635496 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.630450 24541 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:41.638624 25372 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 23:19:41.629510 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:41.524878 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.09627e+06us
W20260809 23:19:41.623270 24540 rpcz_store.cc:269] 0809 23:19:39.750456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:40.541898 (+791442us) service_pool.cc:227] Handling call
0809 23:19:41.527297 (+985399us) spinlock_profiling.cc:243] Waited 881 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.527526 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2026023168}
W20260809 23:19:41.644155 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112126us
I20260809 23:19:41.641073 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2409) took 1890 ms. Trace:
I20260809 23:19:41.644150 24323 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:41.631950 29815 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.325s	user 0.064s	sys 0.070s
I20260809 23:19:41.627614 30331 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:41.645017 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.633463 24320 rpcz_store.cc:276] 0809 23:19:39.423869 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:39.474894 (+ 51025us) service_pool.cc:227] Handling call
0809 23:19:39.475284 (+   390us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:39.475530 (+   246us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:39.475557 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:39.475982 (+   425us) log.cc:842] Serialized 26 byte log entry
0809 23:19:39.923243 (+447261us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:39.923277 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:39.923350 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:41.386810 (+1463460us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:41.541991 (+155181us) raft_consensus.cc:1671] finished
0809 23:19:41.542013 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.542310 (+   297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47037312,"thread_start_us":446968,"threads_started":1,"wal-append.queue_time_us":783160}
W20260809 23:19:41.649272 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 119870us
I20260809 23:19:41.647938 24431 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:41.652063 24423 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 10
I20260809 23:19:41.649513 24425 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:41.644891 24539 rpcz_store.cc:276] 0809 23:19:39.746403 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.342323 (+1595920us) service_pool.cc:227] Handling call
0809 23:19:41.344958 (+  2635us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.386245 (+ 41287us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:41.386811 (+   566us) log.cc:842] Serialized 14 byte log entry
0809 23:19:41.628202 (+241391us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:41.628255 (+    53us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:41.633116 (+  4861us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.633189 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.636988 (+  3799us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4709,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":101,"raft.queue_time_us":6331,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":181,"thread_start_us":235581,"threads_started":1}
I20260809 23:19:41.654543 25368 raft_consensus.cc:595] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:41.655004 25368 raft_consensus.cc:603] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:41.655997 25375 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:41.657960 25375 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9.
I20260809 23:19:41.649755 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.634332 29709 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260809 23:19:41.650437 24428 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:41.648936 25365 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:41.636435 30109 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.649799 24430 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:41.645447 29760 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:41.665521 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.652300 30331 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.667042 24545 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:41.663053 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:41.672053 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.673452 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.674348 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.671743 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.676940 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.678117 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.650218 30067 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.645927 30106 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.677528 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.680473 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.682005 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.683383 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.684232 25369 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:41.686549 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.686645 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.688735 29808 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:41.690447 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:41.691497 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.691670 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2419) took 1572 ms. Trace:
I20260809 23:19:41.692243 24427 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 6.
I20260809 23:19:41.694141 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:19:41.694996 29987 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:41.695376 29987 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.695782 29987 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:41.696080 24422 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260809 23:19:41.698856 24542 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:41.699337 24542 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:41.700512 24542 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.701395 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.701942 24542 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:41.704783 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:41.706311 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59671.9us
I20260809 23:19:41.708153 29204 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:41.708729 29204 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:41.709733 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:41.709947 29204 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.714322 30286 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:41.715461 25363 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:41.715759 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:41.716392 25368 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:41.716746 30106 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.649032 29708 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.692065 24545 rpcz_store.cc:276] 0809 23:19:40.119291 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.199610 (+1080319us) service_pool.cc:227] Handling call
0809 23:19:41.691493 (+491883us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400147b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:41.691608 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":949697792}
I20260809 23:19:41.723083 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.725631 24543 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:41.649520 30150 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.727466 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.704795 30293 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:41.731088 24431 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:41.676168 24541 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.726468 24423 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:41.732777 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:41.734544 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47777.3us
W20260809 23:19:41.748842 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100701us
W20260809 23:19:41.747380 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 222832us
I20260809 23:19:41.744248 24425 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:41.751075 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 88474.1us
I20260809 23:19:41.751986 24541 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:41.754308 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.752170 24546 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:41.755918 24424 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:41.754920 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45329us
W20260809 23:19:41.761364 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56718.5us
I20260809 23:19:41.737258 30293 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.762483 30293 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.762590 29397 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:41.763173 29397 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.763926 29467 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:41.763662 30150 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:41.765277 30342 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260809 23:19:41.766011 29397 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.726748 30109 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:41.676249 30339 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.768420 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:41.768599 25371 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:41.769910 30150 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:41.761445 25372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:41.776093 25368 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:41.777580 25372 raft_consensus.cc:595] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:41.777982 25372 raft_consensus.cc:603] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:41.782402 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:19:41.772066 30141 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:41.784075 30141 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.784516 30141 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.788285 24270 leader_election.cc:398] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:41.788645 24270 leader_election.cc:401] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260809 23:19:41.789701 30339 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:41.777222 30150 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:41.768429 30109 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:41.808270 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46394.5us
I20260809 23:19:41.822098 25370 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:41.815502 25375 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:41.809585 30251 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:41.827256 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.827967 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.815276 30150 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:41.838340 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.841570 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.877978 24423 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:41.879495 24165 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 7, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:41.881062 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:41.881999 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:41.884086 30109 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:41.884639 30109 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.885129 30109 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.885512 30106 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:41.885926 30106 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.886264 30106 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:41.887574 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.838102 30251 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:41.894450 30251 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:41.894825 30251 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:41.898401 30349 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260809 23:19:41.902851 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.903765 24435 raft_consensus.cc:2362] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10. Re-sending same reply.
I20260809 23:19:41.907083 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:41.907418 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 65819us
I20260809 23:19:41.907933 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:41.910873 29305 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 23:19:41.912501 29305 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:41.913036 29305 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:41.919395 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:41.880476 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.926275 24542 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260809 23:19:41.930641 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:41.930893 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55534.7us
I20260809 23:19:41.913232 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.882040 24165 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 6, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 6 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:41.930485 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.936272 24429 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:41.910740 30337 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:41.941210 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:41.947391 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2422) took 1339 ms. Trace:
I20260809 23:19:41.952677 24541 rpcz_store.cc:276] 0809 23:19:40.608461 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.510471 (+902010us) service_pool.cc:227] Handling call
0809 23:19:41.540348 (+ 29877us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.938644 (+398296us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:41.938678 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:41.938718 (+    40us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:41.947140 (+  8422us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.947216 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:41.947336 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2007}
I20260809 23:19:41.955446 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:41.956015 24423 raft_consensus.cc:2362] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8. Re-sending same reply.
W20260809 23:19:41.955978 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80022.8us
I20260809 23:19:41.944207 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:41.960714 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:41.947259 24313 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
W20260809 23:19:41.967341 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48117us
I20260809 23:19:41.972671 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:41.973433 25370 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:19:41.975206 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 43667.2us
I20260809 23:19:41.975952 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:41.968430 24312 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:41.974079 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:41.951167 30337 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:41.914463 29305 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:41.992715 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.009574 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.011030 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.011340 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.012077 30331 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.012805 24422 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260809 23:19:42.013365 24428 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:42.014379 30331 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:41.992873 29467 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:42.012034 24427 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260809 23:19:42.016934 24323 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 23:19:41.948729 29987 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.015093 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.018440 24316 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:41.998601 29397 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:41.989626 30337 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.097859 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:42.099633 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 123756us
I20260809 23:19:42.100991 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.104224 24422 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:42.104645 24427 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260809 23:19:42.109349 30291 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:42.102980 24435 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:42.023998 29924 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:42.125977 24313 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:42.075213 29615 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9.
I20260809 23:19:42.077278 29397 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:42.143613 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.156925 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.110169 29305 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.157954 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.172820 24429 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:42.174499 24314 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.178073 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.179267 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.180075 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:42.180325 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.180891 25369 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:42.181409 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.182117 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:42.183322 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:42.183338 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.186036 24323 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:42.181737 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:42.186595 25369 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:42.182210 30106 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.164853 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.189108 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.194541 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.195607 24423 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:42.196491 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:42.189997 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195572us
I20260809 23:19:42.201996 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.135841 29924 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:42.198153 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2424) took 1020 ms. Trace:
I20260809 23:19:42.207227 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:42.157166 30124 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:42.166531 30153 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:42.166337 29397 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:42.212069 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:42.212177 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:42.206689 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.214964 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.213351 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:42.217350 24427 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:42.166054 29708 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:42.235070 24316 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:42.208981 24539 rpcz_store.cc:276] 0809 23:19:41.172760 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.658290 (+485530us) service_pool.cc:227] Handling call
0809 23:19:41.669014 (+ 10724us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:41.750537 (+ 81523us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:19:41.778639 (+ 28102us) log.cc:842] Serialized 14 byte log entry
0809 23:19:42.192341 (+413702us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:42.192401 (+    60us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:42.192449 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:42.192512 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:42.192615 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17731,"prepare.queue_time_us":27752,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1149,"raft.queue_time_us":66474,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":350731,"threads_started":2,"wal-append.queue_time_us":400595}
W20260809 23:19:42.243170 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 232844us
I20260809 23:19:42.243722 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:42.246102 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.231429 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.249276 24431 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260809 23:19:42.215293 29924 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:42.263638 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.266901 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.222628 30153 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:42.282701 24542 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.284305 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.285583 24540 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:42.287102 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.283725 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.291327 24317 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.295639 24545 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:42.286279 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:42.299804 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 325801us
I20260809 23:19:42.302189 24543 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:42.303557 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.304462 24544 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:42.265098 29708 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:42.226742 30251 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.305673 24541 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:42.317853 30375 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:42.323951 30106 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:42.337340 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:42.324535 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.281385 30153 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.330649 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.248216 30124 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.331424 24539 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:42.243723 29688 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20260809 23:19:42.352437 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.352869 30124 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:42.335899 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 365244us
I20260809 23:19:42.359100 29987 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.296861 29467 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:42.359516 30375 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.359887 29467 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.342628 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:42.362208 24323 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:42.383363 25375 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:42.384534 30251 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.320199 29708 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.386916 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:42.390612 24312 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:42.391978 29467 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.395488 24428 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.396803 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.397629 24431 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:42.398876 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:42.399968 30141 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.615s	user 0.013s	sys 0.004s
I20260809 23:19:42.400478 30141 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.401649 24424 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260809 23:19:42.402804 25371 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.361667 30375 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:42.408886 24423 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
W20260809 23:19:42.419579 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 157232us
W20260809 23:19:42.468400 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114433us
I20260809 23:19:42.469762 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:42.471081 29987 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:42.471588 29987 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:42.472044 29987 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:42.494122 24323 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:42.497902 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:42.499325 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:42.500347 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:42.501303 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80339.2us
W20260809 23:19:42.503122 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 189684us
I20260809 23:19:42.518210 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:42.525637 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.526657 24539 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:42.462033 30105 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:42.532681 24541 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:42.466558 30021 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:42.534662 24423 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260809 23:19:42.540328 24323 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:42.545751 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.515278 30141 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:42.549137 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 346295us
W20260809 23:19:42.538921 30293 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.776s	user 0.010s	sys 0.009s
I20260809 23:19:42.552179 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.497313 30106 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260809 23:19:42.562281 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.563162 24543 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:42.566844 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:42.570603 30264 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:42.573328 29823 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:19:42.577062 30109 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.000s
I20260809 23:19:42.577672 30109 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.542229 29708 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:42.584116 29708 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:42.537446 30021 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:42.589075 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:42.589915 24546 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.591459 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.592423 24545 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:42.608875 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.623093 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:42.626390 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.532881 30251 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260809 23:19:42.627138 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:42.628341 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.632826 25372 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:42.633023 30021 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.634193 30021 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:42.626757 29397 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.636986 29397 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.638164 29397 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:42.638506 25373 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
W20260809 23:19:42.623413 29709 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.032s	sys 0.036s
I20260809 23:19:42.643160 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:42.647488 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 379424us
I20260809 23:19:42.621492 30264 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:42.652122 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.651458 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:42.650553 29709 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.032s	sys 0.036s
I20260809 23:19:42.656194 24545 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:42.561359 30293 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.638113 30377 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260809 23:19:42.660178 30354 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.890s	user 0.006s	sys 0.005s
I20260809 23:19:42.661201 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.638710 29397 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.674484 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:42.644999 30373 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:42.676419 24544 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:42.679371 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:42.680110 24545 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.681998 24320 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:42.660629 30109 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.665050 30377 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:42.665270 30337 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.004s
I20260809 23:19:42.722636 30377 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 3 FOLLOWER]: Advancing to term 4
I20260809 23:19:42.722887 30337 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:42.724427 30383 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260809 23:19:42.725106 30293 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:42.725679 30337 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:42.695492 30351 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.885s	user 0.008s	sys 0.001s
I20260809 23:19:42.728058 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2418) took 2615 ms. Trace:
I20260809 23:19:42.728952 29987 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.728495 24534 rpcz_store.cc:276] 0809 23:19:40.112412 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.381357 (+1268945us) service_pool.cc:227] Handling call
0809 23:19:41.381947 (+   590us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:41.382605 (+   658us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:41.382636 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:41.422817 (+ 40181us) log.cc:842] Serialized 26 byte log entry
0809 23:19:41.439312 (+ 16495us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:41.613771 (+174459us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:41.613811 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:41.613905 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:42.616985 (+1003080us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:42.727784 (+110799us) raft_consensus.cc:1671] finished
0809 23:19:42.727814 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:42.727989 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53074,"raft.queue_time_us":7611,"raft.run_cpu_time_us":2771,"raft.run_wall_time_us":46483,"spinlock_wait_cycles":114111488,"thread_start_us":190493,"threads_started":1,"wal-append.queue_time_us":304340}
I20260809 23:19:42.696780 29924 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.700160 29397 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:19:42.702735 30366 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.520s	user 0.000s	sys 0.002s
I20260809 23:19:42.707239 30291 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:42.708367 30359 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.574s	user 0.000s	sys 0.004s
W20260809 23:19:42.715988 30355 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.606s	user 0.000s	sys 0.002s
I20260809 23:19:42.682140 30373 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:42.718426 29709 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:19:42.718696 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63166.3us
I20260809 23:19:42.719106 29708 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:42.719498 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.719264 24165 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 7, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:19:42.721714 30364 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.613s	user 0.004s	sys 0.000s
W20260809 23:19:42.722100 24170 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 370) took 23510 ms (client timeout 15000 ms). Trace:
I20260809 23:19:42.748477 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:42.735481 30150 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.799s	user 0.007s	sys 0.007s
I20260809 23:19:42.757856 24544 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260809 23:19:42.760303 24534 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260809 23:19:42.762686 24165 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 5, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 5 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:42.762771 30150 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.769385 30150 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.770146 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.770975 29708 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.771698 24430 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:42.773092 29708 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:42.750517 30291 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:42.776226 24545 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:42.781221 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:42.669713 30264 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:42.756453 24170 rpcz_store.cc:269] 0809 23:19:19.209533 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:19.209715 (+   182us) service_pool.cc:227] Handling call
0809 23:19:42.719555 (+23509840us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:19:36.084642 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:19:36.335479 (+250837us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:19:36.335550 (+    71us) write_op.cc:435] Acquired schema lock
0809 23:19:36.335591 (+    41us) tablet.cc:660] Decoding operations
0809 23:19:36.617384 (+281793us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:19:36.617502 (+   118us) write_op.cc:641] Partition lock acquired for write op
0809 23:19:36.617531 (+    29us) tablet.cc:683] Acquiring locks for 12 operations
0809 23:19:36.618620 (+  1089us) tablet.cc:699] Row locks acquired
0809 23:19:36.618665 (+    45us) write_op.cc:260] PREPARE: finished
0809 23:19:36.618942 (+   277us) write_op.cc:270] Start()
0809 23:19:36.619090 (+   148us) write_op.cc:276] Timestamp: P: 1786317576618813 usec, L: 0
0809 23:19:36.619131 (+    41us) op_driver.cc:331] REPLICATION: starting
0809 23:19:36.619916 (+   785us) log.cc:842] Serialized 4749 byte log entry
0809 23:19:37.587477 (+967561us) op_driver.cc:444] REPLICATION: finished
0809 23:19:37.970532 (+383055us) write_op.cc:301] APPLY: starting
0809 23:19:37.977769 (+  7237us) tablet.cc:1365] starting BulkCheckPresence
0809 23:19:38.547423 (+569654us) tablet.cc:1368] finished BulkCheckPresence
0809 23:19:38.547563 (+   140us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:19:40.270685 (+1723122us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:19:40.380354 (+109669us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=0,mrs_lookups=0
0809 23:19:40.380420 (+    66us) write_op.cc:312] APPLY: finished
0809 23:19:40.380918 (+   498us) log.cc:842] Serialized 113 byte log entry
0809 23:19:41.337265 (+956347us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac2091a5fe 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb
0809 23:19:41.529770 (+192505us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:41.530131 (+   361us) write_op.cc:454] Released schema lock
0809 23:19:41.530259 (+   128us) write_op.cc:341] FINISH: Updating metrics
0809 23:19:41.530966 (+   707us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:41.530994 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18183,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":306143,"apply.run_cpu_time_us":43686,"apply.run_wall_time_us":3611869,"cfile_cache_hit":46,"cfile_cache_hit_bytes":48024,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6215,"cfile_init":1,"lbm_read_time_us":62204,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":7,"num_ops":12,"prepare.queue_time_us":525972,"prepare.run_cpu_time_us":6682,"prepare.run_wall_time_us":661880,"replication_time_us":968221,"spinlock_wait_cycles":56114560,"thread_start_us":1450252,"threads_started":3,"wal-append.queue_time_us":1155670,"wal-append.run_cpu_time_us":1610,"wal-append.run_wall_time_us":1004297}]]}
W20260809 23:19:42.774705 29305 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.001s
I20260809 23:19:42.753284 30373 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:42.783764 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:42.783823 24165 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:42.880467 24539 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260809 23:19:42.880734 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 99590.9us
I20260809 23:19:42.885905 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.887810 29823 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:42.896384 29823 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:42.897614 29823 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.899226 30106 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:42.900010 30106 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:42.902520 30106 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:42.904155 24165 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:42.912081 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.915227 24423 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:42.916283 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.917441 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.918190 24431 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:42.917183 24165 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 5, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 5 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:42.919006 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:42.919754 30153 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.527s	user 0.003s	sys 0.006s
I20260809 23:19:42.920606 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.919428 24165 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 6, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 6 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:42.922180 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:42.922860 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:42.922293 24165 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 6, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 6 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:42.924466 24165 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 5, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 5 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:42.784037 29987 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:42.929986 30264 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:42.931491 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45962.1us
I20260809 23:19:42.919658 24428 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:42.926448 24165 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:42.920229 30153 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.942955 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:42.945940 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:42.946743 24427 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:42.949991 24165 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:42.951642 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:42.961120 24165 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:42.958621 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:42.970894 24546 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:42.972308 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:42.968405 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 839) took 1552 ms. Trace:
I20260809 23:19:42.942071 30264 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:42.966822 24165 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 5, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 5 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:42.983598 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2191) took 1107 ms. Trace:
I20260809 23:19:42.974886 24535 rpcz_store.cc:276] 0809 23:19:41.416129 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.662819 (+246690us) service_pool.cc:227] Handling call
0809 23:19:41.836737 (+173918us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:41.879297 (+ 42560us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:41.879662 (+   365us) log.cc:842] Serialized 14 byte log entry
0809 23:19:41.954648 (+ 74986us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:41.997246 (+ 42598us) log.cc:842] Serialized 26 byte log entry
0809 23:19:41.997466 (+   220us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:41.997825 (+   359us) log.cc:842] Serialized 14 byte log entry
0809 23:19:42.016412 (+ 18587us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:42.016499 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:42.721092 (+704593us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:42.767369 (+ 46277us) raft_consensus.cc:1671] finished
0809 23:19:42.767397 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:42.968167 (+200770us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54000b5c50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:42.968331 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22048,"prepare.queue_time_us":5060,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":92,"raft.queue_time_us":79532,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":2020,"spinlock_wait_cycles":702457088,"thread_start_us":74595,"threads_started":1,"wal-append.queue_time_us":228840}
I20260809 23:19:42.974598 29615 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:42.990674 30337 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:42.990872 29615 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:42.990576 24165 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 8 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:42.992159 29615 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:42.992338 24165 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 6, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 6 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:42.984233 24430 rpcz_store.cc:276] 0809 23:19:41.872129 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:41.880164 (+  8035us) service_pool.cc:227] Handling call
0809 23:19:42.978958 (+1098794us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:42.979102 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1950686592}
I20260809 23:19:42.993723 24165 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 1 to 6, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 6 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:19:42.984575 29987 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:43.003151 24165 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 7, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:42.963981 30377 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.000229 29924 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:43.005345 29987 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:42.991431 30337 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.023108 29987 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:43.024606 24165 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:43.022719 24543 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 4.
I20260809 23:19:43.026576 24165 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 6, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:19:43.029088 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 510962us
I20260809 23:19:43.004263 30264 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:43.033453 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 483342us
W20260809 23:19:43.068430 24540 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.003s	sys 0.019s
I20260809 23:19:43.068542 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:43.069402 24540 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.004s	sys 0.019s
I20260809 23:19:43.069751 24546 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:43.059406 24165 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 7, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:19:43.088137 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50547.9us
I20260809 23:19:43.092248 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.085405 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:43.085852 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.007s	sys 0.018s
W20260809 23:19:43.095940 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.008s	sys 0.019s
I20260809 23:19:43.087042 24165 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:19:43.072568 29855 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.009s	sys 0.016s
I20260809 23:19:43.100240 24165 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 7, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:43.097162 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:43.108187 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:43.099073 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.004s	sys 0.023s
W20260809 23:19:43.109829 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.004s	sys 0.023s
W20260809 23:19:43.110415 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 561558us
I20260809 23:19:43.111941 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:43.114902 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.118285 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.085039 30106 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.124503 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:43.120558 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52228.6us
I20260809 23:19:43.126560 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.085680 29397 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:43.099656 30293 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:43.096712 30124 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.014s	sys 0.011s
W20260809 23:19:43.106021 29855 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.009s	sys 0.016s
I20260809 23:19:43.127458 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:43.136726 30124 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.015s	sys 0.011s
I20260809 23:19:43.110780 30153 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.134625 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.135263 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.109242 30251 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:43.168962 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:43.169492 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:43.138844 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.014s	sys 0.017s
W20260809 23:19:43.170835 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.014s	sys 0.017s
I20260809 23:19:43.131551 24165 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 9, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 9 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:19:43.141530 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.170409 25373 raft_consensus.cc:2362] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9. Re-sending same reply.
I20260809 23:19:43.173724 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:43.179791 25370 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:43.179849 24541 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:43.121649 30141 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:43.132727 29688 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:43.178067 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43517.3us
W20260809 23:19:43.161412 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.012s	sys 0.019s
W20260809 23:19:43.185281 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.014s	sys 0.020s
I20260809 23:19:43.185436 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:43.186375 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 661135us
I20260809 23:19:43.190366 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.169550 30251 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:43.216804 30251 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:43.219061 30343 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
I20260809 23:19:43.117722 30390 raft_consensus.cc:2741] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:19:43.221674 30390 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:43.222144 30390 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:43.223296 25371 raft_consensus.cc:2362] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10. Re-sending same reply.
I20260809 23:19:43.192497 30293 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:43.224686 30293 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260809 23:19:43.208770 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 641968us
I20260809 23:19:43.210562 29853 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:43.232913 29853 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.140959 30106 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.237263 30106 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.215085 30371 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.011s
W20260809 23:19:43.216324 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87185.1us
I20260809 23:19:43.241365 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.243108 29305 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:43.243667 30331 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:43.244102 30331 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.244467 30331 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.244271 30390 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.252115 29305 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:43.252704 29305 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.254722 29305 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.255709 29853 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:43.258337 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:43.259696 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.207886 30141 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:43.262229 30141 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:43.262494 30264 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.262661 30141 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.264518 30337 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.265488 29467 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:43.199086 29924 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:19:43.266880 30358 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.883s	user 0.004s	sys 0.001s
I20260809 23:19:43.261552 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.206537 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:43.270584 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64116.2us
I20260809 23:19:43.271822 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.272396 24546 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.272814 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:43.184260 30150 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:43.135993 29397 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.280592 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.279162 30337 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:43.283905 29708 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:19:43.286722 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81204.7us
I20260809 23:19:43.289916 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.204121 30153 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260809 23:19:43.292609 30391 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.421s	user 0.001s	sys 0.001s
I20260809 23:19:43.292701 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:43.289223 30377 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.293764 24320 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:43.295526 30141 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.297547 30141 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.288533 24541 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:43.181880 30109 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.103s
I20260809 23:19:43.293785 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:43.306895 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:43.191594 29688 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:43.279429 30021 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:43.290894 30371 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:43.316025 24541 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:19:43.316742 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55354.2us
I20260809 23:19:43.305532 30153 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:43.312084 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:43.284584 30150 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.310286 30106 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:43.313334 29688 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.289964 30373 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.320690 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.287830 29987 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.330319 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.319295 30153 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:43.330454 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:43.285050 30291 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.325270 30150 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:43.331941 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.333432 30293 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:43.325433 30021 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:43.343308 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:43.325610 30106 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:43.359247 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:43.328120 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.006s	sys 0.083s
W20260809 23:19:43.369567 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.006s	sys 0.083s
W20260809 23:19:43.370128 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 825747us
W20260809 23:19:43.368779 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.022s	sys 0.023s
W20260809 23:19:43.371515 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.022s	sys 0.023s
I20260809 23:19:43.327603 24546 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:43.411886 29987 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.354429 30021 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:43.422369 24546 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:43.427759 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53947.9us
I20260809 23:19:43.431058 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.369572 30337 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:43.374145 30106 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:43.441767 30021 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:43.447731 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.449213 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.449985 24542 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:43.387305 30377 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.458352 29987 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:43.463809 30373 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.471773 30411 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:43.472298 30291 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.472398 30411 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:43.473448 30411 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.474895 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.475261 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.476159 30153 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.479204 30408 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:43.480026 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.491086 24542 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:19:43.480329 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:43.481501 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:43.503449 24428 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 4: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:43.482220 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.482300 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.504268 24427 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:43.504192 24435 raft_consensus.cc:2354] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:43.483004 29987 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:43.483239 30377 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.419732 29708 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:19:43.493628 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 362377us
I20260809 23:19:43.503362 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:43.504136 24429 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:43.506410 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58816.6us
I20260809 23:19:43.457921 30337 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.428457 30394 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:43.513552 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:43.511148 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 320867us
I20260809 23:19:43.515825 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.516094 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.513614 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.516812 24542 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:43.518137 24496 leader_election.cc:398] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:43.518965 24496 leader_election.cc:401] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260809 23:19:43.525264 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.524716 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.506050 29855 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.528568 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.526860 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:43.439882 29924 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260809 23:19:43.509367 29708 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.538432 24498 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9.
I20260809 23:19:43.542807 24498 leader_election.cc:401] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
W20260809 23:19:43.542096 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 309729us
I20260809 23:19:43.539328 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.540643 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.514904 30394 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.515067 30373 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:19:43.517338 29397 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.527002 30392 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:43.504540 30331 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.532928 29853 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:43.541265 29709 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:43.563853 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.507195 30150 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:43.562635 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 444154us
I20260809 23:19:43.544543 29305 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:43.548018 29708 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:43.577458 24539 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:43.545784 30396 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:43.557911 30271 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.463s	user 0.020s	sys 0.037s
I20260809 23:19:43.556874 30394 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:43.549528 30375 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:43.562060 30109 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:43.589075 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:43.559859 30392 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.546226 30251 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.506376 30408 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.573774 30373 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:43.671675 30373 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.533989 30337 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.623571 30392 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.590153 29853 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.630136 30331 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.700084 30331 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.700723 30331 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.630764 29467 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.631175 30396 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:43.703210 29467 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.703766 30396 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.631431 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2581) took 1299 ms. Trace:
I20260809 23:19:43.704939 24540 rpcz_store.cc:276] 0809 23:19:42.332517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:42.338340 (+  5823us) service_pool.cc:227] Handling call
0809 23:19:42.389241 (+ 50901us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:43.069960 (+680719us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:43.070007 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:43.105948 (+ 35941us) log.cc:842] Serialized 26 byte log entry
0809 23:19:43.511791 (+405843us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:43.511829 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.518073 (+  6244us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:43.631214 (+113141us) raft_consensus.cc:1671] finished
0809 23:19:43.631236 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:43.631371 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107167,"raft.queue_time_us":226373,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":4765,"thread_start_us":947162,"threads_started":2,"wal-append.queue_time_us":497819}
I20260809 23:19:43.633452 30106 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.577107 29397 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:43.708134 29397 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.635587 30153 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.625430 30398 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:43.625041 30375 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:43.729377 30331 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.729631 30375 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:43.730082 30375 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:43.731313 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2596) took 1026 ms. Trace:
I20260809 23:19:43.733028 24541 rpcz_store.cc:276] 0809 23:19:42.704585 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.374294 (+669709us) service_pool.cc:227] Handling call
0809 23:19:43.410434 (+ 36140us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:43.410713 (+   279us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:43.411176 (+   463us) log.cc:842] Serialized 14 byte log entry
0809 23:19:43.730330 (+319154us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:43.730993 (+   663us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:43.731051 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.731118 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:43.731239 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2329,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"thread_start_us":318730,"threads_started":1}
I20260809 23:19:43.626085 30424 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.735360 30424 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.587150 29987 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:43.626981 30410 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260809 23:19:43.576296 29709 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:43.743423 29709 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:43.585237 30271 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.700s	user 0.021s	sys 0.038s
I20260809 23:19:43.628453 30150 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.654893 24542 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:43.738008 30424 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:43.666177 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 307211us
I20260809 23:19:43.675192 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2444) took 1026 ms. Trace:
W20260809 23:19:43.674733 30415 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
W20260809 23:19:43.599149 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 457483us
I20260809 23:19:43.751281 24545 rpcz_store.cc:276] 0809 23:19:42.649066 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:42.777771 (+128705us) service_pool.cc:227] Handling call
0809 23:19:42.778131 (+   360us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:42.778415 (+   284us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:42.778451 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:42.778980 (+   529us) log.cc:842] Serialized 26 byte log entry
0809 23:19:43.374518 (+595538us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:43.374557 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.374628 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:43.674993 (+300365us) raft_consensus.cc:1671] finished
0809 23:19:43.675013 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:43.675139 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":490381,"threads_started":1,"wal-append.queue_time_us":766452}
I20260809 23:19:43.756381 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:43.780977 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:43.779017 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 227075us
I20260809 23:19:43.791383 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2591) took 1105 ms. Trace:
I20260809 23:19:43.793931 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:43.760809 29397 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.801330 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:19:43.812846 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 250981us
I20260809 23:19:43.813318 24545 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:43.783025 30431 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260809 23:19:43.819064 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:43.792126 30377 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:43.790714 30293 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:43.788828 30398 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:43.828486 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2594) took 1124 ms. Trace:
W20260809 23:19:43.832818 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 292254us
I20260809 23:19:43.831390 24543 rpcz_store.cc:276] 0809 23:19:42.703956 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.031456 (+327500us) service_pool.cc:227] Handling call
0809 23:19:43.059463 (+ 28007us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:43.087332 (+ 27869us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:19:43.505339 (+418007us) log.cc:842] Serialized 14 byte log entry
0809 23:19:43.827851 (+322512us) log.cc:842] Serialized 14 byte log entry
0809 23:19:43.828063 (+   212us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:43.828120 (+    57us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:43.828191 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.828276 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:43.828418 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14130,"prepare.queue_time_us":150792,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":36,"thread_start_us":655148,"threads_started":2}
I20260809 23:19:43.799889 30153 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:43.837438 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 256168us
I20260809 23:19:43.840603 30397 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260809 23:19:43.842275 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.809968 29467 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:43.920259 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:43.913599 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 157329us
W20260809 23:19:43.912886 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 373611us
I20260809 23:19:43.819965 30431 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:43.913722 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 375447us
I20260809 23:19:43.925104 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:43.919255 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 100274us
I20260809 23:19:43.938333 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:43.933208 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91016.3us
I20260809 23:19:43.947891 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:43.948781 24429 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:43.949597 30427 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:43.950317 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:43.950181 30427 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.940378 29397 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:43.953333 29397 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.953794 29397 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:43.876516 30106 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.957299 30106 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:43.957837 30106 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.959648 30251 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:43.960978 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:43.969509 30397 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:43.969939 30397 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:43.970275 30397 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:43.971513 24545 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:43.979745 30337 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:43.983399 30453 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:43.984169 30453 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:43.985281 30453 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.820480 30390 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.123s
I20260809 23:19:43.948813 29467 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:43.952867 30434 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260809 23:19:44.001439 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 204340us
W20260809 23:19:44.034307 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84070.9us
W20260809 23:19:44.037423 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 99276.3us
I20260809 23:19:43.836920 30021 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:43.932248 30431 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:44.058140 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.058882 25375 raft_consensus.cc:2354] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:44.060266 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.060266 30021 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:44.066032 30431 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:43.988493 30251 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260809 23:19:44.073717 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:44.075338 24540 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
W20260809 23:19:44.041772 29688 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.706s	user 0.019s	sys 0.011s
I20260809 23:19:44.079605 24543 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:44.079941 24541 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:44.082093 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 295545us
I20260809 23:19:44.082185 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:44.087342 24545 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:44.096071 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.052826 29924 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:44.140854 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.036841 30408 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:44.122829 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.156394 24498 leader_election.cc:398] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:44.156863 24498 leader_election.cc:401] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:44.158142 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:44.159044 24545 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 7: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:44.160280 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:44.163403 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.167280 30458 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:44.167725 30458 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:44.168076 30458 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:44.157218 30141 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.022s	sys 0.036s
W20260809 23:19:44.177618 30141 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.022s	sys 0.037s
W20260809 23:19:44.181586 29708 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.586s	user 0.008s	sys 0.000s
I20260809 23:19:44.182164 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 4.
I20260809 23:19:44.182029 29708 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.042696 30429 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.116s
I20260809 23:19:44.157011 29924 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:44.186154 30331 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:44.192049 30429 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:44.192741 30150 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
W20260809 23:19:44.195775 30436 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:19:44.196396 30427 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:44.197691 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:44.200670 30441 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260809 23:19:44.138353 30399 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.132s
I20260809 23:19:44.208688 30431 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:44.217269 30412 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.636s	user 0.005s	sys 0.000s
I20260809 23:19:44.138700 30398 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.221747 30264 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:43.792634 24535 rpcz_store.cc:276] 0809 23:19:42.686302 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:42.990215 (+303913us) service_pool.cc:227] Handling call
0809 23:19:43.009066 (+ 18851us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:43.027905 (+ 18839us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:43.511563 (+483658us) log.cc:842] Serialized 14 byte log entry
0809 23:19:43.790921 (+279358us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:43.791005 (+    84us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:43.791075 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.791157 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:43.791302 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43061,"prepare.queue_time_us":115274,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":18539,"thread_start_us":689189,"threads_started":2}
W20260809 23:19:44.269155 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56914.1us
I20260809 23:19:44.121141 29688 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.168998 30106 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:44.230237 30343 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.002s	sys 0.018s
W20260809 23:19:44.275182 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 351999us
W20260809 23:19:44.255340 30375 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.002s
W20260809 23:19:44.275085 30343 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.002s	sys 0.018s
I20260809 23:19:44.268947 30150 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:44.236449 29853 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.010s
I20260809 23:19:44.271654 30264 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:44.276623 30371 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.956s	user 0.012s	sys 0.016s
I20260809 23:19:44.248363 30429 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:44.320926 24323 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:44.275799 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:44.231359 29924 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:44.272027 30377 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:44.315181 30399 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:44.316123 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:44.316674 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:44.317205 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:44.318051 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:44.318571 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 198568us
I20260809 23:19:44.318871 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:44.319139 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:44.369571 24540 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:44.322109 24321 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260809 23:19:44.322829 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:44.362504 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 238309us
I20260809 23:19:44.233095 30331 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.324389 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.288815 29467 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:44.371330 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.373478 24535 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:44.370625 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 94873.5us
I20260809 23:19:44.312788 29853 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9.
I20260809 23:19:44.321120 30371 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:44.311118 30459 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:44.310811 30448 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:44.410686 24317 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:44.376012 24311 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:44.414656 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:44.413703 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:44.369779 29709 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.003s
I20260809 23:19:44.430182 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:44.430296 29709 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.433204 29709 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:44.319805 30431 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
W20260809 23:19:44.423418 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45874.8us
I20260809 23:19:44.424858 24320 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:44.425873 24323 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:44.435390 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:44.439718 30150 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.377700 29823 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:44.309319 30375 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.424021 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.453799 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.446781 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2589) took 1787 ms. Trace:
I20260809 23:19:44.473646 25371 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:44.474416 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60620us
I20260809 23:19:44.475484 24321 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:44.476682 24269 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.853s (SENT)
W20260809 23:19:44.477692 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63276.1us
I20260809 23:19:44.369760 30461 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260809 23:19:44.459693 24544 rpcz_store.cc:276] 0809 23:19:42.659298 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:42.798518 (+139220us) service_pool.cc:227] Handling call
0809 23:19:42.798914 (+   396us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:42.799454 (+   540us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:42.799494 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:42.800047 (+   553us) log.cc:842] Serialized 26 byte log entry
0809 23:19:43.372887 (+572840us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:43.372946 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:43.373033 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:44.216450 (+843417us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:44.417349 (+200899us) raft_consensus.cc:1671] finished
0809 23:19:44.417373 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:44.446688 (+ 29315us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1628,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":7930240,"thread_start_us":572375,"threads_started":1,"wal-append.queue_time_us":727812}
W20260809 23:19:44.484889 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 404826us
I20260809 23:19:44.493727 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:44.525527 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:19:44.376960 30399 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.546826 24545 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:44.551388 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98272.4us
I20260809 23:19:44.480554 29823 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:44.587175 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:44.497887 29688 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:44.597466 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 118285us
I20260809 23:19:44.599409 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2686) took 1107 ms. Trace:
I20260809 23:19:44.599617 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:44.617578 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:44.628919 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 172666us
I20260809 23:19:44.445546 30264 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:44.630570 24269 leader_election.cc:334] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.001s (SENT)
I20260809 23:19:44.633730 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:44.635218 25371 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:44.595539 30397 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.625s	user 0.013s	sys 0.005s
I20260809 23:19:44.647503 29688 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:19:44.648428 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56235.9us
I20260809 23:19:44.648826 29615 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:44.649121 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.649415 29615 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:44.610066 30468 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
I20260809 23:19:44.486610 30485 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260809 23:19:44.652721 24535 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:19:44.595084 30331 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:44.609562 30469 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.309s	user 0.000s	sys 0.001s
I20260809 23:19:44.451102 30459 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.457803 30448 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:44.611088 30471 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260809 23:19:44.658228 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:44.608474 30464 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
I20260809 23:19:44.660552 24540 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:44.652964 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 856) took 1103 ms. Trace:
W20260809 23:19:44.662115 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.010s	sys 0.037s
W20260809 23:19:44.681200 30392 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.983s	user 0.014s	sys 0.009s
W20260809 23:19:44.685019 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.010s	sys 0.037s
W20260809 23:19:44.685567 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 761263us
I20260809 23:19:44.685447 30392 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.686455 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:44.610451 30467 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260809 23:19:44.671506 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:44.687642 30392 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.688241 30450 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:19:44.688685 30450 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:44.683542 24546 rpcz_store.cc:276] 0809 23:19:43.549464 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.640479 (+ 91015us) service_pool.cc:227] Handling call
0809 23:19:43.658205 (+ 17726us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:43.711129 (+ 52924us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:43.711175 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:43.711831 (+   656us) log.cc:842] Serialized 52 byte log entry
0809 23:19:44.038616 (+326785us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:44.071580 (+ 32964us) log.cc:842] Serialized 14 byte log entry
0809 23:19:44.078125 (+  6545us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:44.078221 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:44.652744 (+574523us) raft_consensus.cc:1671] finished
0809 23:19:44.652767 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:44.652903 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32795,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":20178,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":31060,"thread_start_us":243658,"threads_started":1,"wal-append.queue_time_us":469322}
I20260809 23:19:44.689473 30450 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.678710 24314 raft_consensus.cc:595] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:44.605259 24431 rpcz_store.cc:276] 0809 23:19:43.487545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.487692 (+   147us) service_pool.cc:227] Handling call
0809 23:19:43.488054 (+   362us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:43.488293 (+   239us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:43.796278 (+307985us) log.cc:842] Serialized 14 byte log entry
0809 23:19:44.594732 (+798454us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:44.594796 (+    64us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:44.594878 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:44.594986 (+   108us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:44.595135 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2496,"prepare.queue_time_us":689293,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":116291,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6772,"thread_start_us":1025108,"threads_started":3,"wal-append.queue_time_us":491176}
I20260809 23:19:44.604362 30124 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:44.682471 29708 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:44.691037 29709 raft_consensus.cc:2741] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260809 23:19:44.690989 30124 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.693372 30124 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:44.690338 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:44.678764 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 206016us
W20260809 23:19:44.669473 30394 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.068s	user 0.017s	sys 0.000s
I20260809 23:19:44.670619 30465 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:44.500079 30429 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.688308 30392 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260809 23:19:44.680058 30498 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:44.708231 30431 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:44.753013 30264 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:44.755687 30431 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.721797 30450 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:44.759068 30291 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:44.754060 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.760319 25370 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:44.762257 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131699us
I20260809 23:19:44.763660 30331 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:44.764803 30331 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:44.765283 30331 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:44.749593 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.771850 25371 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:44.723268 29855 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.184s	user 0.026s	sys 0.005s
I20260809 23:19:44.719693 30472 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:44.774130 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:44.661275 30397 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:44.788880 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 139830us
I20260809 23:19:44.789685 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:44.775354 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85120.1us
I20260809 23:19:44.653513 29467 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.695938 30474 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.103s
I20260809 23:19:44.775849 29688 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:44.799373 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 113030us
I20260809 23:19:44.797791 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:44.729810 30390 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260809 23:19:44.783835 30465 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:44.784585 29855 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:44.710047 30394 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.815331 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:44.816289 24542 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:44.817480 30419 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:19:44.823423 30397 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.825261 30459 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.829033 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 992) took 1705 ms. Trace:
I20260809 23:19:44.811959 30124 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:44.829375 25372 rpcz_store.cc:276] 0809 23:19:43.123559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.124537 (+   978us) service_pool.cc:227] Handling call
0809 23:19:43.142896 (+ 18359us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:43.230666 (+ 87770us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:43.653612 (+422946us) log.cc:842] Serialized 14 byte log entry
0809 23:19:44.266066 (+612454us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:44.266125 (+    59us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:44.277284 (+ 11159us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:44.828779 (+551495us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:44.828835 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:44.828974 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":554009,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":251134,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":13932,"spinlock_wait_cycles":94071680,"thread_start_us":920125,"threads_started":3,"wal-append.queue_time_us":505252}
W20260809 23:19:44.838332 30477 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260809 23:19:44.708835 29615 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:44.832276 29397 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.878s	user 0.011s	sys 0.005s
W20260809 23:19:44.840716 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140647us
I20260809 23:19:44.777050 29823 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.836746 30375 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.852032 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.795298 30482 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.256s
I20260809 23:19:44.837584 30431 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:44.842101 30498 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:44.850662 30153 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.925s	user 0.005s	sys 0.009s
I20260809 23:19:44.841442 30124 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:44.861032 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:44.851537 30503 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260809 23:19:44.858433 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.828209 29688 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.858130 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.881469 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:44.855119 30462 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:44.900012 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.900454 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:44.879241 30124 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:44.894722 30480 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260809 23:19:44.893954 30153 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:44.852380 29397 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:44.870808 30399 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:44.911878 24535 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:44.925886 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.909238 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:44.926671 24323 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:44.926904 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.913565 30504 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.123s
I20260809 23:19:44.914265 30479 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:44.914604 30512 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:44.914920 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.915253 30487 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:44.915612 30520 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:44.869632 30498 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.995082 30498 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.995759 25365 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:44.915879 30511 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260809 23:19:44.996387 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.996634 30496 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:44.996892 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 858) took 1398 ms. Trace:
I20260809 23:19:44.916543 30291 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:44.997649 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.998323 25366 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 4 FOLLOWER]: Advancing to term 5
I20260809 23:19:44.998574 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:44.918551 25372 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:19:44.916484 30443 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.650s	user 0.000s	sys 0.002s
I20260809 23:19:44.922142 24321 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:45.000134 30509 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.155s
I20260809 23:19:45.000674 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.001466 30509 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:44.898900 30491 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.103s
I20260809 23:19:45.001828 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.002610 24321 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:44.997598 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.997232 24534 rpcz_store.cc:276] 0809 23:19:43.598136 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.657888 (+ 59752us) service_pool.cc:227] Handling call
0809 23:19:43.707577 (+ 49689us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:43.713418 (+  5841us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:43.761084 (+ 47666us) log.cc:842] Serialized 14 byte log entry
0809 23:19:44.289320 (+528236us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:44.309303 (+ 19983us) log.cc:842] Serialized 52 byte log entry
0809 23:19:44.309471 (+   168us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:44.309782 (+   311us) log.cc:842] Serialized 14 byte log entry
0809 23:19:44.309934 (+   152us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:44.310022 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:44.715659 (+405637us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:44.996621 (+280962us) raft_consensus.cc:1671] finished
0809 23:19:44.996648 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:44.996820 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21883,"prepare.queue_time_us":49824,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":43460,"raft.queue_time_us":68234,"raft.run_cpu_time_us":2916,"raft.run_wall_time_us":1024024,"spinlock_wait_cycles":29636864,"thread_start_us":451912,"threads_started":1,"wal-append.queue_time_us":612020}
I20260809 23:19:45.004683 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.006326 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:44.997246 30392 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:45.007588 30392 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:44.941617 24323 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:44.943559 24313 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:44.945711 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.010010 24309 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:44.906692 30448 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:44.946255 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:44.979645 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 181933us
I20260809 23:19:44.905778 30461 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:19:44.983089 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:44.983318 25368 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:44.984026 24311 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:44.912484 30264 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Leader election won for term 5
I20260809 23:19:44.991847 25370 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:44.984016 30479 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:45.014371 25371 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:44.992866 30531 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:44.996234 25373 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:44.996030 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:44.915632 30494 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:45.026028 24320 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.018002 24312 raft_consensus.cc:2354] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
I20260809 23:19:45.022513 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.020296 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:44.986541 24378 connection.cc:232] RPC call timeout handler was delayed by -1.6668205298483372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1589s for a response
I20260809 23:19:45.025043 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:45.027729 24378 connection.cc:232] RPC call timeout handler was delayed by -1.5786205297335982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16870000000000002s for a response
W20260809 23:19:45.029441 24378 connection.cc:232] RPC call timeout handler was delayed by -1.5399205298162997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
W20260809 23:19:45.030043 24378 connection.cc:232] RPC call timeout handler was delayed by -1.3320205297786742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260809 23:19:45.030383 24378 connection.cc:232] RPC call timeout handler was delayed by -1.3203205298632383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20260809 23:19:45.030715 24378 connection.cc:232] RPC call timeout handler was delayed by -1.238420529756695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260809 23:19:45.032013 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:44.936504 30525 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:45.009168 30520 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:45.029776 24316 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:45.044193 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.043124 24313 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:45.050251 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.057442 24431 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:45.037456 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.059297 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.059866 24427 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.062628 24309 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:45.063715 24311 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.024173 30479 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:44.935595 30431 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.103s
I20260809 23:19:44.988919 30485 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:45.018245 30424 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:45.067087 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:45.074473 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52053.4us
I20260809 23:19:45.074877 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.075277 24320 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.075766 24430 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:19:45.027387 30498 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.068778 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.069025 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.001856 30509 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.038825 30264 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.073689 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.075913 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:45.070714 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45904.7us
I20260809 23:19:45.079324 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:45.076658 24422 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.081459 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:45.029440 30153 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:45.061539 30520 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.039907 30525 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:45.088445 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56517.1us
I20260809 23:19:45.086411 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.065783 30394 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.077499 30524 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:45.101076 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.077679 30461 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:45.089052 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.091665 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.101394 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.077531 30487 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.115543 24435 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.090302 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.112071 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.068444 30495 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260809 23:19:45.116511 24320 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.002619 30512 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.118599 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.120450 30486 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.131s
I20260809 23:19:45.108752 30525 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.122354 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:45.122238 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.121851 24429 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:45.117127 30458 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.941s	user 0.005s	sys 0.008s
I20260809 23:19:45.124740 24423 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:19:45.127460 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.129037 30523 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:45.129490 30523 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.130309 30523 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.131630 30523 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.112133 30524 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:45.136004 30545 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260809 23:19:45.135761 30524 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.028792 30531 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.159701 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:45.126639 24427 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:45.159689 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.147434 24320 raft_consensus.cc:2354] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:45.121420 30487 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.080413 30498 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.121645 30392 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.083931 30424 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:45.168027 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77782.7us
I20260809 23:19:45.168788 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.126595 30462 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.175056 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.175634 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:45.173563 24422 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:45.176528 24424 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:45.177680 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.108840 30153 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.083319 30485 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.191165 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.185210 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.094820 30264 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.192395 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:45.177068 29688 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.191845 24541 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:45.211211 24534 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.182337 30461 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.177044 29397 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.159135 30458 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.215830 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:45.216046 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.184020 30504 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:45.186381 30496 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260809 23:19:45.233393 24431 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:19:45.227645 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.234560 25366 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 5.
I20260809 23:19:45.236946 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.237550 24313 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:45.246680 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61833.2us
I20260809 23:19:45.249477 24535 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:45.251228 30461 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:45.252605 30461 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:45.253510 30522 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:19:45.254598 30522 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:45.257122 30522 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.165868 30509 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.257958 24429 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:45.258298 24429 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.257974 30522 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.259120 24429 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.259851 30522 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:45.260215 24429 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.262439 30461 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:19:45.263558 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:45.264277 30485 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:45.264760 30461 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:45.265671 30536 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:45.266651 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.267403 24321 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.268098 30531 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.268220 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.268499 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.268841 24316 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:45.269553 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.270263 30153 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:45.270344 30460 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260809 23:19:45.271348 30460 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:45.271406 30392 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.256067 30331 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.256291 30527 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:45.265692 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.281721 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.270668 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.273006 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.255313 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.276795 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.275589 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.283438 24427 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:45.284267 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.284526 24321 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:45.285094 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.286765 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.235720 30291 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:45.281627 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.311585 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.283360 24316 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:45.320778 24429 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:19:45.320887 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.244074 30431 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.323463 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.322599 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.326196 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.320000 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:45.329380 24429 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.330781 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:45.329258 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.333740 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.335489 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.336645 24321 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:45.338557 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.339596 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:45.339896 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.340610 24431 raft_consensus.cc:2354] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:45.340931 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.341305 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.342031 24431 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:45.344918 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.346493 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.271768 30460 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:45.314072 30494 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:45.314487 30291 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:45.310829 30375 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.320369 30535 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:45.356652 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.354665 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.283336 30512 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.369683 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.345026 30106 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:45.383172 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 307301us
I20260809 23:19:45.346315 30431 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.350481 29823 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:45.311179 30392 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.353766 30494 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.389261 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.394134 24431 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:45.357234 30291 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.368500 30375 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:45.360507 30535 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:45.402472 24541 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260809 23:19:45.388433 29924 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.010s	user 0.013s	sys 0.000s
I20260809 23:19:45.382128 29397 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.408433 24545 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:19:45.359130 30474 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260809 23:19:45.408725 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:45.409475 24431 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:45.386119 30448 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:45.393939 24429 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:45.422639 24429 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260809 23:19:45.423293 24429 pending_rounds.cc:76] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:19:45.423429 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.403642 30106 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:45.428185 24431 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:45.401614 29467 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:45.462852 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 178642us
I20260809 23:19:45.423502 24429 pending_rounds.cc:97] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260809 23:19:45.477672 24429 raft_consensus.cc:2866] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:45.477846 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.368677 30479 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.406190 29823 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:45.484258 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164377us
I20260809 23:19:45.388288 30512 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260809 23:19:45.408759 30535 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.414844 30408 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.406054 30498 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.483832 24435 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:45.484994 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.421271 30424 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:45.510758 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 156170us
I20260809 23:19:45.590497 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.510372 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:45.422609 30487 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:45.596669 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:45.598194 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88983.8us
I20260809 23:19:45.605058 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:45.606343 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96060.8us
W20260809 23:19:45.607069 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 129747us
I20260809 23:19:45.606750 24498 leader_election.cc:398] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:45.607483 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:45.614967 24266 leader_election.cc:398] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:45.615314 24266 leader_election.cc:401] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260809 23:19:45.503846 30458 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.419497 29924 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.584326 30450 raft_consensus.cc:2741] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260809 23:19:45.620505 30485 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260809 23:19:45.657171 30461 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:45.657662 30461 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.591127 30109 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:45.659304 30109 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:45.660403 30109 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.661819 30109 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.419831 29397 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.504778 30549 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.664745 29397 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.666407 30392 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.596074 30424 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.667126 30424 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.669130 30424 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.405773 30460 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.695132 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.695497 30460 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:45.695794 24541 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:45.696830 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.698485 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:45.648371 30574 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:45.699502 30458 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:45.648759 30498 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.589542 30408 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.704281 29924 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:45.705015 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.705463 30408 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.484591 29467 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.705935 24543 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:45.590529 30555 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.059s
I20260809 23:19:45.707145 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2608) took 1594 ms. Trace:
I20260809 23:19:45.707391 29467 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.486565 30494 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.709964 30494 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:45.658634 30461 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.717183 30498 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.720315 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.721338 24428 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:45.721536 30461 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.722361 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.723258 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.724645 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.725315 24429 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:45.729192 30398 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:45.729676 30398 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:45.650599 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 861) took 1852 ms. Trace:
I20260809 23:19:45.653582 30141 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:45.480603 30525 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.731387 30141 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:45.732448 30141 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.733084 30525 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:45.655144 30331 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:45.734073 30577 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:45.635604 29823 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.676693 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:45.730669 30398 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.692294 30479 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:45.617332 30106 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
W20260809 23:19:45.693907 30557 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.212s	user 0.000s	sys 0.000s
I20260809 23:19:45.761425 30398 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:45.763334 30124 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.827s	user 0.011s	sys 0.001s
W20260809 23:19:45.737376 30558 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.371s	user 0.001s	sys 0.000s
W20260809 23:19:45.764004 30488 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.009s	user 0.002s	sys 0.015s
W20260809 23:19:45.755158 30520 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.649s	user 0.011s	sys 0.000s
I20260809 23:19:45.764756 30520 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.766438 25366 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:45.707528 24543 rpcz_store.cc:276] 0809 23:19:44.113285 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.142052 (+ 28767us) service_pool.cc:227] Handling call
0809 23:19:44.167049 (+ 24997us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.706376 (+1539327us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:45.706914 (+   538us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:45.707092 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3538630912}
I20260809 23:19:45.731273 24542 rpcz_store.cc:276] 0809 23:19:43.798101 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.798651 (+   550us) service_pool.cc:227] Handling call
0809 23:19:43.811260 (+ 12609us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:44.816700 (+1005440us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:45.650314 (+833614us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400146c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:45.650526 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2310877056}
W20260809 23:19:45.623512 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 138577us
I20260809 23:19:45.769614 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.678452 24534 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:45.745529 30502 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:45.772697 30587 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:45.748554 24313 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.510s	user 0.009s	sys 0.000s
W20260809 23:19:45.760790 30523 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.629s	user 0.009s	sys 0.003s
W20260809 23:19:45.746814 30505 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.984s	user 0.000s	sys 0.003s
I20260809 23:19:45.779525 24424 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260809 23:19:45.774250 24313 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:45.781957 24424 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:45.782366 24424 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.782478 29305 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:45.779295 30150 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:45.792202 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 31) took 1314 ms. Trace:
I20260809 23:19:45.795053 24314 rpcz_store.cc:276] 0809 23:19:44.478166 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.478961 (+   795us) service_pool.cc:227] Handling call
0809 23:19:45.792013 (+1313052us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54003c32d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:45.792106 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300443008}
I20260809 23:19:45.801745 24428 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:45.786758 30556 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
W20260809 23:19:45.806022 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52416.9us
I20260809 23:19:45.785678 30458 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:45.782915 29305 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:45.839581 24428 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:45.840173 24428 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:45.854199 30375 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:45.855430 24428 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.856881 24428 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:45.859015 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 160630us
I20260809 23:19:45.859349 30539 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:45.859894 30539 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:45.860314 30539 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:45.860416 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:45.862349 30424 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:45.862982 30424 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:45.864060 30424 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.866214 30502 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:45.867475 29305 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.807509 30574 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:45.872627 30574 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.871179 30535 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:45.880535 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 111013us
I20260809 23:19:45.764109 30124 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.889513 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:45.909569 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 299588us
I20260809 23:19:45.913113 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.913283 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.913743 25365 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:45.915998 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 307457us
I20260809 23:19:45.917758 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.917480 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.883894 30392 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:45.918524 25369 raft_consensus.cc:2362] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10. Re-sending same reply.
W20260809 23:19:45.923956 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.006s	sys 0.003s
W20260809 23:19:45.924424 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.006s	sys 0.003s
W20260809 23:19:45.924755 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 710992us
I20260809 23:19:45.925099 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:45.925388 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:45.922716 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:45.928802 25372 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:45.929792 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 314986us
W20260809 23:19:45.921020 24269 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.140s (SENT)
I20260809 23:19:45.922346 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.932897 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.933640 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:45.783390 24424 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:45.922793 24544 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:45.935250 24424 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:45.935851 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:45.940275 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.936905 24429 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:45.949126 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:45.950220 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:45.952296 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66275.6us
W20260809 23:19:45.917490 30264 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.061s	sys 0.072s
W20260809 23:19:45.953560 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.000s	sys 0.011s
W20260809 23:19:45.974501 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.000s	sys 0.012s
I20260809 23:19:45.976301 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:45.880540 30291 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:45.971455 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.979924 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:45.971560 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.802311 30587 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:45.981088 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47481.3us
I20260809 23:19:45.982029 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:45.970551 25363 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.007s	sys 0.005s
W20260809 23:19:45.990391 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.012s	sys 0.031s
W20260809 23:19:45.990746 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.012s	sys 0.031s
I20260809 23:19:45.991757 25369 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:45.992816 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:45.980320 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55033.6us
I20260809 23:19:45.997671 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:45.965463 30371 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:45.991225 30337 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:45.991405 30373 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
W20260809 23:19:45.974089 30264 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.062s	sys 0.072s
I20260809 23:19:46.023384 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.030906 24429 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:46.031538 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:46.032976 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:46.021592 24534 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:46.021840 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:45.949154 30585 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:46.039746 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57817.2us
I20260809 23:19:46.044775 24541 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:19:46.045367 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:46.046149 25369 raft_consensus.cc:2354] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
I20260809 23:19:46.046857 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:46.047883 24541 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:46.047843 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:46.048637 25369 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:46.049480 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:46.050455 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:46.051954 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.052615 24314 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:46.039224 30375 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.053802 30375 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.055801 30375 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:46.056444 30458 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.056941 30458 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.057504 30587 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.058092 30587 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:46.058543 30587 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:45.978433 30106 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:45.992256 30392 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:46.003961 30592 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:45.991597 29709 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:46.040009 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.010874 25365 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:46.059619 24321 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:46.060338 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.067564 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:46.069537 24316 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:46.115999 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:46.118651 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.021533 30337 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:46.126084 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:46.060487 30535 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:46.129456 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:46.130059 24311 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:46.131600 30337 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.135257 29709 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:46.135777 29709 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.137724 29709 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.059571 30377 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:46.140625 30291 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:46.058903 30458 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:46.059486 30435 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260809 23:19:46.148072 30472 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:46.069509 30375 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260809 23:19:46.149945 30375 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:46.150372 30375 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:46.150502 30504 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:46.156311 30504 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:46.156513 24311 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:46.157197 30504 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.158552 30504 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:46.142639 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2615) took 1283 ms. Trace:
I20260809 23:19:46.163594 24544 rpcz_store.cc:276] 0809 23:19:44.859677 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.861792 (+  2115us) service_pool.cc:227] Handling call
0809 23:19:44.862091 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.951417 (+1089326us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.142405 (+190988us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:46.142581 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2426725120}
I20260809 23:19:46.165263 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2473) took 1099 ms. Trace:
I20260809 23:19:46.165834 24535 rpcz_store.cc:276] 0809 23:19:45.065670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.065857 (+   187us) service_pool.cc:227] Handling call
0809 23:19:46.164922 (+1099065us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:46.165063 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384100224}
I20260809 23:19:46.172355 24435 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:46.180562 24423 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
W20260809 23:19:46.183674 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.016s	sys 0.059s
W20260809 23:19:46.183997 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.016s	sys 0.059s
W20260809 23:19:46.184440 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 577045us
I20260809 23:19:46.186093 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.187619 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:46.068651 30585 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:46.199590 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:46.200307 24545 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.018s	sys 0.039s
W20260809 23:19:46.200726 24545 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.018s	sys 0.039s
I20260809 23:19:46.188941 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.201835 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:46.195466 24313 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:46.203436 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:46.205026 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:46.051970 30124 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.110803 30523 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:46.197606 24309 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.009s	sys 0.006s
W20260809 23:19:46.213013 24309 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.010s	sys 0.006s
W20260809 23:19:46.211829 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.030s	sys 0.032s
W20260809 23:19:46.216713 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.030s	sys 0.032s
I20260809 23:19:46.218261 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:46.212628 30523 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.119741 30520 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:46.253111 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.067114 30522 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
I20260809 23:19:46.194293 30453 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:19:46.270953 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83615.8us
I20260809 23:19:46.190235 30535 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.060789 30106 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.283344 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.284551 24429 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:19:46.271245 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2196) took 1211 ms. Trace:
I20260809 23:19:46.286111 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:46.151440 30375 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:46.197527 30448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.003s	sys 0.013s
I20260809 23:19:46.197309 30458 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.209973 30585 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:46.268920 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2467) took 1574 ms. Trace:
I20260809 23:19:46.274303 30575 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:46.293390 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.293864 24546 rpcz_store.cc:276] 0809 23:19:44.694506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.694672 (+   166us) service_pool.cc:227] Handling call
0809 23:19:44.709891 (+ 15219us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:44.710739 (+   848us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:44.733047 (+ 22308us) log.cc:842] Serialized 14 byte log entry
0809 23:19:45.773182 (+1040135us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b18003e11c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:46.268389 (+495207us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:46.268467 (+    78us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:46.268540 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.268652 (+   112us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:46.268838 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63968,"prepare.queue_time_us":22124,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":76540,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":7813,"spinlock_wait_cycles":20521984,"thread_start_us":424000,"threads_started":1}
I20260809 23:19:46.292141 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1159) took 2216 ms. Trace:
I20260809 23:19:46.190331 30496 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.092s
I20260809 23:19:46.062737 30392 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.285949 24430 rpcz_store.cc:276] 0809 23:19:45.060211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.119402 (+ 59191us) service_pool.cc:227] Handling call
0809 23:19:45.143088 (+ 23686us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.165870 (+ 22782us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:45.385910 (+220040us) log.cc:842] Serialized 14 byte log entry
0809 23:19:46.270667 (+884757us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:46.270735 (+    68us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:46.270915 (+   180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.271038 (+   123us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:46.271172 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1506,"prepare.queue_time_us":199688,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":13031,"thread_start_us":1019114,"threads_started":2}
I20260809 23:19:46.279194 30512 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:46.275249 30271 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:19:46.293159 30448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.003s	sys 0.013s
I20260809 23:19:46.282682 30481 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:46.365594 24430 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:46.311064 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.305449 30124 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:46.312201 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:46.294070 30396 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.114s	user 0.024s	sys 0.038s
I20260809 23:19:46.309192 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:46.282534 29397 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:46.292577 30459 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:46.281901 30150 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:46.353163 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64237.1us
I20260809 23:19:46.365154 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:46.292900 30592 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:46.343482 30375 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:46.304047 25375 rpcz_store.cc:276] 0809 23:19:44.066299 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.066450 (+   151us) service_pool.cc:227] Handling call
0809 23:19:44.066762 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:44.072324 (+  5562us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:44.159942 (+ 87618us) log.cc:842] Serialized 14 byte log entry
0809 23:19:45.747414 (+1587472us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b1800500500. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:46.282446 (+535032us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:46.282500 (+    54us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:46.282554 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.282631 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:46.282747 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85966,"prepare.queue_time_us":82059,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":7784,"raft.queue_time_us":92700,"spinlock_wait_cycles":88064384,"thread_start_us":448807,"threads_started":1}
I20260809 23:19:46.342381 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:46.407656 24431 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:46.408257 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 427543us
I20260809 23:19:46.370921 30573 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.372236 24425 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:46.292804 30535 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.379312 24429 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:46.382278 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:46.366650 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55745.9us
I20260809 23:19:46.374153 30408 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.355772 30494 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:46.305990 30453 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260809 23:19:46.436551 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.437446 24323 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:46.413036 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.439757 24431 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [7.6-9.7]
I20260809 23:19:46.440855 24430 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:46.441226 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:46.385417 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86001.1us
W20260809 23:19:46.404906 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.017s	sys 0.026s
W20260809 23:19:46.449366 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.017s	sys 0.027s
I20260809 23:19:46.376319 30124 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.353926 30448 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:46.384872 30514 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.010s	sys 0.028s
I20260809 23:19:46.373072 30605 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.292418 30587 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.308688 30458 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.427376 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.384987 30589 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:46.371771 30390 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:46.473356 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.477016 24311 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:46.477710 30124 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:46.477876 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:46.478363 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.479156 24430 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260809 23:19:46.479364 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:46.530575 24316 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:46.480671 30598 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260809 23:19:46.530272 24312 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:46.481475 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:46.482033 30592 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.486879 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:46.501680 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:46.538033 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 155910us
W20260809 23:19:46.538219 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 486223us
W20260809 23:19:46.538525 24424 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.603s	user 0.012s	sys 0.001s
I20260809 23:19:46.541029 25375 raft_consensus.cc:2354] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
W20260809 23:19:46.376781 30396 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.197s	user 0.024s	sys 0.039s
I20260809 23:19:46.429692 30150 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:46.439515 30494 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:46.524384 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51123.3us
I20260809 23:19:46.526876 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:46.498381 30514 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.010s	sys 0.029s
W20260809 23:19:46.527887 24428 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.670s	user 0.005s	sys 0.006s
I20260809 23:19:46.581116 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:46.581460 24428 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:19:46.581691 24316 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:46.472797 29397 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:46.592221 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:46.423103 30473 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
W20260809 23:19:46.573706 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92324.3us
I20260809 23:19:46.641469 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:46.641803 24428 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:46.501681 30511 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260809 23:19:46.642638 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:46.502993 29709 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.577656 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:46.416311 30573 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.438735 30408 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.586943 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:46.587929 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:46.485407 30512 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:46.592602 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.679607 24430 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
W20260809 23:19:46.564282 30398 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.802s	user 0.012s	sys 0.000s
I20260809 23:19:46.593760 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:46.573675 30531 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.184s	user 0.011s	sys 0.041s
W20260809 23:19:46.644549 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66975.3us
I20260809 23:19:46.730019 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:46.577667 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:46.671438 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:46.732172 24535 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:46.582792 30450 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:46.611555 30424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.016s	sys 0.050s
I20260809 23:19:46.585006 30494 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:46.745029 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:46.592334 29397 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:46.751040 24428 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260809 23:19:46.524459 30484 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.149s
I20260809 23:19:46.731529 24546 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:46.643610 30586 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:46.668331 30251 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:46.680006 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 484693us
I20260809 23:19:46.584801 30390 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.176s
I20260809 23:19:46.777451 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1101) took 1379 ms. Trace:
I20260809 23:19:46.783980 25371 rpcz_store.cc:276] 0809 23:19:45.397676 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.397797 (+   121us) service_pool.cc:227] Handling call
0809 23:19:45.398159 (+   362us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:45.405723 (+  7564us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:45.405758 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:45.429382 (+ 23624us) log.cc:842] Serialized 26 byte log entry
0809 23:19:45.951978 (+522596us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:45.952032 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:45.952120 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:46.777216 (+825096us) raft_consensus.cc:1671] finished
0809 23:19:46.777254 (+    38us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:46.777394 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24523,"raft.queue_time_us":22075,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":53983,"thread_start_us":358854,"threads_started":1,"wal-append.queue_time_us":776166}
W20260809 23:19:46.787715 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.610s (SENT) [suppressed 20 similar messages]
W20260809 23:19:46.751538 30531 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.366s	user 0.011s	sys 0.042s
I20260809 23:19:46.733651 30150 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.788767 24316 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:46.646989 30396 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.137s
I20260809 23:19:46.472128 30585 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.735807 30106 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:46.766646 30424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.017s	sys 0.051s
I20260809 23:19:46.649228 30461 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20260809 23:19:46.770363 30479 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:46.815330 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.016s	sys 0.074s
W20260809 23:19:46.830525 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.016s	sys 0.074s
I20260809 23:19:46.507182 30605 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.773950 29397 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:46.788280 30536 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:46.806833 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.012s	sys 0.036s
I20260809 23:19:46.678977 29709 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.834695 25370 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
W20260809 23:19:46.840037 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.013s	sys 0.036s
I20260809 23:19:46.841897 24544 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:46.814359 30396 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260809 23:19:46.845402 30428 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260809 23:19:46.847170 30573 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.848990 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.805814 30251 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:46.772367 30450 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:46.859445 24428 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
W20260809 23:19:46.859694 24309 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.021s	sys 0.037s
W20260809 23:19:46.860697 24309 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.021s	sys 0.037s
I20260809 23:19:46.861598 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:46.836247 30106 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:46.799644 30373 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260809 23:19:46.899796 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:46.900628 24428 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:46.902527 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:46.890993 30574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.023s	sys 0.050s
I20260809 23:19:46.907909 30608 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:46.907826 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:46.805647 30150 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.856786 30535 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:46.913528 30574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.023s	sys 0.051s
I20260809 23:19:46.776753 30448 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.947012 30593 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.897720 29709 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.840801 30479 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.960793 24431 pending_rounds.cc:76] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:19:46.961228 24431 pending_rounds.cc:97] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:19:46.961529 24431 raft_consensus.cc:2866] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:46.971045 30390 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.971644 30390 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.972435 30458 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.973471 30458 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:46.973930 30458 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:46.974403 30458 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:46.975678 30426 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:46.976344 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 768) took 1306 ms. Trace:
I20260809 23:19:46.898706 30536 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
W20260809 23:19:46.978022 24425 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.004s
I20260809 23:19:46.976722 24423 rpcz_store.cc:276] 0809 23:19:45.670150 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.675170 (+  5020us) service_pool.cc:227] Handling call
0809 23:19:46.976164 (+1300994us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:46.976286 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":834182528}
I20260809 23:19:46.977349 24535 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:46.986455 24425 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:46.962359 30612 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.406s	user 0.000s	sys 0.001s
I20260809 23:19:46.984933 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:46.955650 30408 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:46.965677 30624 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:46.964316 29709 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:47.008265 30124 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:47.014855 30124 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:47.011518 30542 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 1.994s	user 0.007s	sys 0.010s
I20260809 23:19:46.976373 30426 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:47.018240 30426 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.958724 30565 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.337s
I20260809 23:19:46.998689 30548 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:46.929647 30608 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:46.947650 30593 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.009703 30450 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:46.989984 30461 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.992946 30574 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:46.991633 30622 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:19:47.075829 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 433228us
I20260809 23:19:47.077893 24269 leader_election.cc:398] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12.
W20260809 23:19:47.079464 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 492608us
I20260809 23:19:47.080507 24378 leader_election.cc:398] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
I20260809 23:19:47.082089 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:47.092818 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 500767us
W20260809 23:19:47.074100 30337 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.018s	sys 0.043s
I20260809 23:19:47.025250 29709 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:47.122859 24429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.008s	sys 0.031s
I20260809 23:19:47.066581 30523 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:47.137633 30493 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.234s	user 0.014s	sys 0.000s
I20260809 23:19:47.095979 30450 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:47.142904 24540 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:47.132891 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:46.793061 30586 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.145236 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 866) took 2902 ms. Trace:
I20260809 23:19:47.152420 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:47.124275 24429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.008s	sys 0.031s
W20260809 23:19:47.098091 30337 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.018s	sys 0.043s
I20260809 23:19:47.065944 30548 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:47.148221 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:47.121436 29709 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:47.152518 24539 rpcz_store.cc:276] 0809 23:19:44.242648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.242842 (+   194us) service_pool.cc:227] Handling call
0809 23:19:44.267814 (+ 24972us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:44.287311 (+ 19497us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:44.287367 (+    56us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:44.301234 (+ 13867us) log.cc:842] Serialized 26 byte log entry
0809 23:19:44.546702 (+245468us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:44.546744 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:44.546825 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:46.832411 (+2285586us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:47.139663 (+307252us) raft_consensus.cc:1671] finished
0809 23:19:47.139685 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.145151 (+  5466us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3315,"raft.queue_time_us":15392,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6570,"spinlock_wait_cycles":25955200,"thread_start_us":126294,"threads_started":1,"wal-append.queue_time_us":435447}
W20260809 23:19:47.165334 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83286.1us
I20260809 23:19:47.091104 30461 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:47.150574 30602 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.492s	user 0.001s	sys 0.000s
I20260809 23:19:47.144089 30523 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:47.092697 30574 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:47.150381 30124 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:47.066299 30251 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.184605 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:47.170758 30617 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.329s	user 0.000s	sys 0.001s
W20260809 23:19:47.170503 30623 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.358s	user 0.001s	sys 0.000s
I20260809 23:19:47.177690 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:47.249229 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.008s	sys 0.018s
W20260809 23:19:47.254427 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.126s	user 0.008s	sys 0.018s
W20260809 23:19:47.254848 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77233.9us
I20260809 23:19:47.255208 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1099) took 1858 ms. Trace:
I20260809 23:19:47.255563 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:47.255532 25363 rpcz_store.cc:276] 0809 23:19:45.397058 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.397201 (+   143us) service_pool.cc:227] Handling call
0809 23:19:45.415675 (+ 18474us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.415860 (+   185us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:45.985093 (+569233us) log.cc:842] Serialized 14 byte log entry
0809 23:19:47.254872 (+1269779us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:47.254949 (+    77us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:47.254993 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.255053 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.255158 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":233410,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"thread_start_us":1680627,"threads_started":2,"wal-append.queue_time_us":1213461}
I20260809 23:19:47.281915 25373 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:47.284710 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:47.286211 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:47.295321 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:47.296881 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:47.298856 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:47.180481 30427 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:47.256846 30504 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.074s	user 0.012s	sys 0.001s
I20260809 23:19:47.248189 30150 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.173727 30450 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.119s
I20260809 23:19:47.249425 30605 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:47.294524 30448 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:47.292290 30573 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:47.333204 24545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.023s	sys 0.010s
I20260809 23:19:47.182054 30523 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:47.337414 25375 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:47.320421 30607 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260809 23:19:47.283790 30559 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:47.282236 30514 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:47.369037 24545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.024s	sys 0.011s
W20260809 23:19:47.369752 29823 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.007s	sys 0.023s
I20260809 23:19:47.451686 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2481) took 1602 ms. Trace:
W20260809 23:19:47.320935 30473 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.042s	sys 0.073s
W20260809 23:19:47.332775 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.005s	sys 0.025s
W20260809 23:19:47.457767 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.005s	sys 0.025s
I20260809 23:19:47.459918 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:47.461041 24313 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 23:19:47.324931 30486 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:47.334343 30618 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.419s	user 0.000s	sys 0.001s
I20260809 23:19:47.248785 30124 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:47.321960 30390 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:47.373392 25363 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:19:47.327633 30608 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.384997 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 875) took 1900 ms. Trace:
I20260809 23:19:47.339916 30645 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:47.355445 30448 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:47.458288 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.072s	sys 0.090s
I20260809 23:19:47.478658 30652 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
W20260809 23:19:47.530194 29397 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.674s	user 0.009s	sys 0.000s
I20260809 23:19:47.331730 30427 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.508365 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:47.532523 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1012) took 1564 ms (client timeout 1838 ms). Trace:
W20260809 23:19:47.532876 25372 rpcz_store.cc:269] 0809 23:19:45.968293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.023484 (+ 55191us) service_pool.cc:227] Handling call
0809 23:19:47.508126 (+1484642us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:19:47.532453 (+ 24327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8925056}
W20260809 23:19:47.533417 30637 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.390s	user 0.000s	sys 0.001s
I20260809 23:19:47.534142 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.509567 30492 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:47.535492 30492 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:47.535849 30492 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:47.535925 30620 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:47.536468 30620 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.541992 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:47.543464 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:47.544139 24313 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:47.544304 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:47.547634 24428 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:47.462882 24323 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:47.368947 30573 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:47.452203 24543 rpcz_store.cc:276] 0809 23:19:45.849768 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.850367 (+   599us) service_pool.cc:227] Handling call
0809 23:19:45.850728 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:45.905655 (+ 54927us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:45.905694 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:45.906181 (+   487us) log.cc:842] Serialized 26 byte log entry
0809 23:19:46.373073 (+466892us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:46.373128 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.375003 (+  1875us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:47.451384 (+1076381us) raft_consensus.cc:1671] finished
0809 23:19:47.451414 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.451604 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86947,"raft.queue_time_us":56588,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":6071,"thread_start_us":349762,"threads_started":1,"wal-append.queue_time_us":716824}
I20260809 23:19:47.373975 30605 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:19:47.454901 30596 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.387s	user 0.001s	sys 0.000s
W20260809 23:19:47.457543 30473 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.046s	sys 0.076s
W20260809 23:19:47.464186 29823 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.007s	sys 0.023s
I20260809 23:19:47.577371 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:47.522404 24545 rpcz_store.cc:276] 0809 23:19:45.484732 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.485020 (+   288us) service_pool.cc:227] Handling call
0809 23:19:45.485356 (+   336us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.530763 (+ 45407us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:19:46.201121 (+670358us) log.cc:842] Serialized 14 byte log entry
0809 23:19:47.384619 (+1183498us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:47.384678 (+    59us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:19:47.384737 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.384813 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.384944 (+   131us) inbound_call.cc:173] Queueing success response
0809 23:19:47.532893 (+147949us) spinlock_profiling.cc:243] Waited 5.04 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
Metrics: {"mutex_wait_us":60948,"prepare.queue_time_us":1063553,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":123445,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":21169,"spinlock_wait_cycles":11597056,"thread_start_us":1792160,"threads_started":3,"wal-append.queue_time_us":864462}
W20260809 23:19:47.575392 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.016s	sys 0.011s
I20260809 23:19:47.450654 30644 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:47.629624 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.018s	sys 0.011s
I20260809 23:19:47.633152 25375 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260809 23:19:47.516918 30539 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.656s	user 0.015s	sys 0.001s
I20260809 23:19:47.377962 30651 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:47.449309 30600 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260809 23:19:47.586378 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.056s	sys 0.076s
W20260809 23:19:47.523000 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.073s	sys 0.090s
W20260809 23:19:47.510825 30633 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.356s	user 0.000s	sys 0.000s
W20260809 23:19:47.510094 30616 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.560s	user 0.000s	sys 0.001s
I20260809 23:19:47.533215 30427 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:47.586261 30495 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:19:47.726951 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.002s	sys 0.013s
W20260809 23:19:47.739768 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.002s	sys 0.013s
W20260809 23:19:47.740159 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 614198us
I20260809 23:19:47.530812 29397 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.645265 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:47.710378 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 807896us
I20260809 23:19:47.726114 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:47.516036 30461 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.626717 30620 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.731474 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:47.681108 24311 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.004s	sys 0.008s
W20260809 23:19:47.750366 24311 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.007s	sys 0.008s
I20260809 23:19:47.750499 25363 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:47.520870 30567 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Stepping down as leader of term 6
I20260809 23:19:47.522641 30645 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:47.735164 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.624210 30608 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:47.505340 30574 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:47.661288 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:47.744200 24317 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.010s	sys 0.006s
I20260809 23:19:47.581315 30573 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:47.355675 30548 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.511788 30459 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:47.710700 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.057s	sys 0.098s
W20260809 23:19:47.753324 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.000s	sys 0.017s
I20260809 23:19:47.786665 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:47.787103 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.000s	sys 0.017s
W20260809 23:19:47.759420 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.017s	sys 0.024s
I20260809 23:19:47.734119 30504 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:47.810602 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.811478 25363 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 5.
I20260809 23:19:47.787425 30459 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:47.815798 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:47.744522 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.004s	sys 0.012s
I20260809 23:19:47.757391 24428 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:47.819561 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:47.515244 30593 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.816442 24539 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 5.
I20260809 23:19:47.809239 30504 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:47.756222 25375 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:47.760074 30645 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:47.457587 30514 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:47.802799 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.020s	sys 0.026s
W20260809 23:19:47.792001 30426 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.014s	sys 0.034s
I20260809 23:19:47.804523 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.522819 30448 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:47.517383 30124 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:47.750967 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.005s	sys 0.019s
W20260809 23:19:47.635099 30392 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.007s	sys 0.023s
W20260809 23:19:47.823405 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.004s	sys 0.012s
W20260809 23:19:47.757774 24317 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.010s	sys 0.007s
I20260809 23:19:47.748180 30479 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:47.519485 30375 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.109s	user 0.013s	sys 0.000s
I20260809 23:19:47.753592 30567 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:47.517876 30486 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.663908 30539 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:47.845378 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.011s	sys 0.047s
I20260809 23:19:47.895434 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2225) took 1394 ms. Trace:
W20260809 23:19:47.872325 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 719970us
W20260809 23:19:47.901252 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.090s	user 0.011s	sys 0.047s
I20260809 23:19:47.872427 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 776) took 2162 ms. Trace:
I20260809 23:19:47.748698 30651 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:47.788295 30586 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:47.873370 25369 raft_consensus.cc:595] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:47.813460 30495 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:47.875910 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.877246 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:47.910156 24546 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:47.910498 24428 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:47.809643 29823 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:47.879714 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:47.878955 30592 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:47.818780 30608 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:47.665609 30644 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:47.887032 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 808100us
I20260809 23:19:47.839159 30504 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:47.876577 30426 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.014s	sys 0.038s
I20260809 23:19:47.901467 24424 rpcz_store.cc:276] 0809 23:19:46.501068 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.588571 (+ 87503us) service_pool.cc:227] Handling call
0809 23:19:46.588963 (+   392us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:46.648783 (+ 59820us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:46.648814 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:46.649259 (+   445us) log.cc:842] Serialized 26 byte log entry
0809 23:19:47.633534 (+984275us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:47.633587 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.633679 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:47.895047 (+261368us) raft_consensus.cc:1671] finished
0809 23:19:47.895082 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.895352 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80787,"raft.queue_time_us":80244,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":55803,"thread_start_us":876226,"threads_started":1,"wal-append.queue_time_us":1204490}
I20260809 23:19:47.904846 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:47.901269 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2227) took 2902 ms. Trace:
I20260809 23:19:47.902969 24422 rpcz_store.cc:276] 0809 23:19:45.710270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.792737 (+ 82467us) service_pool.cc:227] Handling call
0809 23:19:45.793066 (+   329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:45.793276 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:45.793300 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:45.793730 (+   430us) log.cc:842] Serialized 26 byte log entry
0809 23:19:46.472090 (+678360us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:46.472136 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.472219 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:47.832794 (+1360575us) raft_consensus.cc:1671] finished
0809 23:19:47.832815 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.872350 (+ 39535us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":655836,"threads_started":1,"wal-append.queue_time_us":1156147}
W20260809 23:19:47.889405 30635 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.585s	user 0.005s	sys 0.003s
W20260809 23:19:47.886332 30392 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.008s	sys 0.024s
W20260809 23:19:47.881978 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.006s	sys 0.019s
I20260809 23:19:47.889211 30603 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:47.897284 30641 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:48.040371 30660 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:48.097913 29823 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.098377 30660 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.899183 30124 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.098215 30641 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.906277 30605 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:48.103787 30592 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.104290 30605 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:47.903205 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2619) took 2093 ms. Trace:
I20260809 23:19:48.099809 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.036275 24546 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:19:48.059410 30682 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.100425 24431 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:19:48.105883 30124 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.107014 30495 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:48.107524 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:47.889317 30626 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.803s	user 0.001s	sys 0.001s
I20260809 23:19:48.069530 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2710) took 1901 ms. Trace:
I20260809 23:19:47.912467 30492 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.036455 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:48.007066 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:48.101464 30608 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.105959 24424 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:48.115921 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:48.055738 30567 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.103284 30426 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.103164 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.068616 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.120591 24431 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:47.894449 30523 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.105726 24542 rpcz_store.cc:276] 0809 23:19:45.809767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.809889 (+   122us) service_pool.cc:227] Handling call
0809 23:19:45.844086 (+ 34197us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.844299 (+   213us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:46.365920 (+521621us) log.cc:842] Serialized 14 byte log entry
0809 23:19:47.902764 (+1536844us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:47.902832 (+    68us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:47.902907 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.903011 (+   104us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.903144 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":827127,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":141064,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":6653,"thread_start_us":2018277,"threads_started":3,"wal-append.queue_time_us":727771}
W20260809 23:19:48.036384 30494 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.243s	user 0.013s	sys 0.000s
W20260809 23:19:48.096259 30570 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.006s	sys 0.008s
W20260809 23:19:48.156210 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.013s	sys 0.037s
W20260809 23:19:48.156586 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.013s	sys 0.037s
W20260809 23:19:48.156925 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 579632us
W20260809 23:19:48.027865 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 127211us
I20260809 23:19:48.100750 24317 rpcz_store.cc:276] 0809 23:19:44.985231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.023133 (+ 37902us) service_pool.cc:227] Handling call
0809 23:19:45.023410 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.023590 (+   180us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:45.066233 (+ 42643us) log.cc:842] Serialized 14 byte log entry
0809 23:19:46.143744 (+1077511us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b180045c460. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:47.158160 (+1014416us) log.cc:842] Serialized 14 byte log entry
0809 23:19:47.526177 (+368017us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b180045c460. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:47.887254 (+361077us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:47.887322 (+    68us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:47.887387 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.887467 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:47.887594 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10827,"prepare.queue_time_us":45785,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":546177,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":35028,"spinlock_wait_cycles":28578688,"thread_start_us":1662539,"threads_started":3,"wal-append.queue_time_us":1874372}
I20260809 23:19:48.097915 30603 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:48.069751 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 332978us
W20260809 23:19:47.913064 30460 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.005s	sys 0.011s
W20260809 23:19:48.184376 30460 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.208s	user 0.005s	sys 0.011s
I20260809 23:19:48.110575 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 879) took 2230 ms. Trace:
W20260809 23:19:48.185869 30506 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260809 23:19:48.158210 29397 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.186383 30506 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:48.105661 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:48.186749 29397 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:48.186965 30506 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.187378 30603 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:48.187673 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 82135.3us
I20260809 23:19:48.188100 30678 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260809 23:19:48.188912 30674 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260809 23:19:48.189709 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
W20260809 23:19:48.190047 30666 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.293s	user 0.000s	sys 0.000s
I20260809 23:19:48.171317 30577 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:48.191421 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 789) took 1186 ms. Trace:
W20260809 23:19:48.192147 30632 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.057s	user 0.003s	sys 0.000s
W20260809 23:19:48.192392 30677 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
I20260809 23:19:48.194571 25366 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:48.195190 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1008) took 2299 ms. Trace:
I20260809 23:19:48.195262 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2628) took 2108 ms. Trace:
I20260809 23:19:48.195557 25366 rpcz_store.cc:276] 0809 23:19:45.895549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.064012 (+168463us) service_pool.cc:227] Handling call
0809 23:19:48.194050 (+2130038us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:48.194256 (+   206us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.194916 (+   660us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.195143 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18035968}
I20260809 23:19:48.195616 24541 rpcz_store.cc:276] 0809 23:19:46.086538 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.113013 (+ 26475us) service_pool.cc:227] Handling call
0809 23:19:46.187172 (+ 74159us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:46.187475 (+   303us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:46.187513 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:46.195189 (+  7676us) log.cc:842] Serialized 26 byte log entry
0809 23:19:46.840690 (+645501us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:46.840744 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.840834 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:48.191469 (+1350635us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:48.195011 (+  3542us) raft_consensus.cc:1671] finished
0809 23:19:48.195030 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.195204 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25874688,"thread_start_us":563692,"threads_started":1,"wal-append.queue_time_us":810261}
I20260809 23:19:48.196913 30594 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:48.197377 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.197924 30271 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:48.198328 30271 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:48.198650 30658 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:48.199856 30658 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:19:48.200177 30658 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:48.200284 29305 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:48.201017 30658 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.201838 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:48.202560 24312 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:48.055637 30651 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.185245 24534 rpcz_store.cc:276] 0809 23:19:45.879987 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.064418 (+184431us) service_pool.cc:227] Handling call
0809 23:19:46.065144 (+   726us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:46.065915 (+   771us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:46.065965 (+    50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:46.066809 (+   844us) log.cc:842] Serialized 52 byte log entry
0809 23:19:46.611265 (+544456us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:46.611329 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:46.611438 (+   109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:48.110324 (+1498886us) raft_consensus.cc:1671] finished
0809 23:19:48.110375 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.110518 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1459,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":497186,"threads_started":1,"wal-append.queue_time_us":883936}
W20260809 23:19:48.164628 30570 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.008s	sys 0.008s
I20260809 23:19:48.109632 24429 rpcz_store.cc:276] 0809 23:19:46.168071 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.415316 (+247245us) service_pool.cc:227] Handling call
0809 23:19:46.441608 (+ 26292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:46.441876 (+   268us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:47.153770 (+711894us) log.cc:842] Serialized 14 byte log entry
0809 23:19:48.069093 (+915323us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:48.069164 (+    71us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:48.069236 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.069321 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.069460 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1068917,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":264691,"thread_start_us":1549198,"threads_started":3,"wal-append.queue_time_us":745917}
I20260809 23:19:48.093420 24545 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:48.149829 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.212615 24542 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:48.213804 24542 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:48.108009 30495 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.149737 30661 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:48.097187 30587 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.005s	sys 0.012s
W20260809 23:19:48.155519 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 252714us
I20260809 23:19:48.108395 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:47.888922 30508 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260809 23:19:48.233482 30504 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:48.112054 30625 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.767s	user 0.000s	sys 0.001s
I20260809 23:19:48.162477 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.235700 24428 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:48.165207 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2237) took 2928 ms. Trace:
I20260809 23:19:48.172958 24424 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260809 23:19:48.236472 24309 rpcz_store.cc:276] 0809 23:19:45.236388 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.236522 (+   134us) service_pool.cc:227] Handling call
0809 23:19:45.277005 (+ 40483us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:45.277244 (+   239us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:45.338828 (+ 61584us) log.cc:842] Serialized 14 byte log entry
0809 23:19:46.890578 (+1551750us) log.cc:842] Serialized 14 byte log entry
0809 23:19:48.164828 (+1274250us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:48.164886 (+    58us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:48.164939 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.165017 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.165144 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72785,"prepare.queue_time_us":936460,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":328642,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":7697,"thread_start_us":2653854,"threads_started":4,"wal-append.queue_time_us":1452654,"wal-append.run_cpu_time_us":8753,"wal-append.run_wall_time_us":1019640}
W20260809 23:19:48.240142 30685 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:19:48.242336 30649 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:48.242911 30649 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.245069 30649 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.245769 30371 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:19:48.246165 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:48.247622 30472 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:48.248026 30472 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:48.248376 30472 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:48.248965 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.251286 30660 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.007833 30652 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:48.150058 30494 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.256469 30506 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.257006 30506 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.257489 30494 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.257409 30506 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.153344 30150 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.153644 30375 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:48.258594 30150 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.259115 30375 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:48.259507 30375 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:48.260601 30150 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.108424 30124 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.187258 29397 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.191798 24425 rpcz_store.cc:276] 0809 23:19:47.005017 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.005186 (+   169us) service_pool.cc:227] Handling call
0809 23:19:47.005832 (+   646us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:47.006416 (+   584us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:47.006449 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:47.007630 (+  1181us) log.cc:842] Serialized 520 byte log entry
0809 23:19:47.355316 (+347686us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:47.355357 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.355428 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:48.191179 (+835751us) raft_consensus.cc:1671] finished
0809 23:19:48.191202 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.191360 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4872,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":346666,"threads_started":1,"wal-append.queue_time_us":818384}
I20260809 23:19:48.254619 24317 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
W20260809 23:19:48.112038 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.002s	sys 0.011s
W20260809 23:19:48.266839 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.003s	sys 0.012s
I20260809 23:19:48.103156 25375 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 11. Ops: []
I20260809 23:19:48.208282 30658 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:48.209236 30524 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:48.209292 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.209112 29615 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:48.166486 30567 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:48.214329 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:48.221347 30657 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.465s	user 0.001s	sys 0.000s
I20260809 23:19:48.221486 24534 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
W20260809 23:19:48.281484 30650 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.777s	user 0.001s	sys 0.001s
I20260809 23:19:48.282480 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2229) took 1775 ms. Trace:
I20260809 23:19:48.216895 30459 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.282838 24430 rpcz_store.cc:276] 0809 23:19:46.506627 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.680270 (+173643us) service_pool.cc:227] Handling call
0809 23:19:46.757887 (+ 77617us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:46.773204 (+ 15317us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:46.773237 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:46.815725 (+ 42488us) log.cc:842] Serialized 26 byte log entry
0809 23:19:47.249636 (+433911us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:47.249712 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.249808 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:48.282244 (+1032436us) raft_consensus.cc:1671] finished
0809 23:19:48.282272 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.282413 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20150,"raft.queue_time_us":25387,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8846,"thread_start_us":384715,"threads_started":1,"wal-append.queue_time_us":652896}
I20260809 23:19:48.221926 30497 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:48.222193 24312 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 6. Current term is 7. Ops: []
I20260809 23:19:48.285054 30459 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.285120 30497 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.216086 30586 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:48.227568 30458 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.253s	user 0.012s	sys 0.000s
I20260809 23:19:48.228121 30592 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:48.230625 30613 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.645s	user 0.001s	sys 0.001s
I20260809 23:19:48.230432 30661 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.231618 30522 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:48.233943 30508 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:48.236073 30264 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:48.109974 30644 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.238633 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:48.253243 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1010) took 2253 ms. Trace:
I20260809 23:19:48.100375 29823 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.249040 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 12) took 2338 ms. Trace:
I20260809 23:19:48.198727 30594 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:48.270963 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:48.215247 29709 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.042s	user 0.008s	sys 0.004s
I20260809 23:19:48.285807 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:48.376587 30656 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.376888 24312 raft_consensus.cc:2362] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11. Re-sending same reply.
I20260809 23:19:48.376976 25369 rpcz_store.cc:276] 0809 23:19:45.910308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.050310 (+140002us) service_pool.cc:227] Handling call
0809 23:19:48.248877 (+2198567us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:48.248953 (+    76us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4142447360}
I20260809 23:19:48.276417 30271 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.377843 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.379649 30271 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.277587 30514 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.278108 30603 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:48.377213 30656 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.381031 30603 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.274132 30435 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.384137 30435 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.386451 30435 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.160486 30426 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.389137 30426 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:48.397035 30573 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.592s	user 0.007s	sys 0.002s
I20260809 23:19:48.397511 30573 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.399801 29397 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.401587 30504 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.286682 25363 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:48.408128 24312 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:48.409174 24309 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:48.282857 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2257) took 1783 ms. Trace:
I20260809 23:19:48.303045 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.106743 30682 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:48.255548 30587 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.006s	sys 0.013s
W20260809 23:19:48.411286 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 197016us
I20260809 23:19:48.408985 24321 rpcz_store.cc:276] 0809 23:19:46.000197 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.000326 (+   129us) service_pool.cc:227] Handling call
0809 23:19:46.000625 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:46.109037 (+108412us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.238827 (+2129790us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:48.253170 (+ 14343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134080256}
I20260809 23:19:48.425345 30509 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:48.434998 24314 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:48.412657 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.279464 30675 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.443555 30675 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.446331 30675 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.342684 30645 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.417104 30594 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:48.418169 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 770) took 2741 ms (client timeout 2188 ms). Trace:
W20260809 23:19:48.447291 24435 rpcz_store.cc:269] 0809 23:19:45.676277 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.676424 (+   147us) service_pool.cc:227] Handling call
0809 23:19:48.417956 (+2741532us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:48.418073 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":861175552}
I20260809 23:19:48.448743 30594 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.383554 30603 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:48.419920 30628 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.469s	user 0.002s	sys 0.002s
I20260809 23:19:48.446650 30497 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.450093 30645 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.284973 30486 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.420485 30375 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.453150 30375 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.336115 30605 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.421110 30573 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.422029 25368 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260809 23:19:48.454406 30465 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.273021 30652 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.346659 30506 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.421618 30586 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.367690 30539 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:48.376904 29709 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:48.431005 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.408s	user 0.003s	sys 0.021s
W20260809 23:19:48.479066 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.456s	user 0.003s	sys 0.021s
I20260809 23:19:48.412289 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.410039 24311 rpcz_store.cc:276] 0809 23:19:46.499612 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.544171 (+ 44559us) service_pool.cc:227] Handling call
0809 23:19:46.583620 (+ 39449us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:46.583892 (+   272us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:46.583927 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:46.646452 (+ 62525us) log.cc:842] Serialized 26 byte log entry
0809 23:19:47.158375 (+511923us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:47.758527 (+600152us) log.cc:842] Serialized 14 byte log entry
0809 23:19:47.793015 (+ 34488us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:47.793139 (+   124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:48.269753 (+476614us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:48.282572 (+ 12819us) raft_consensus.cc:1671] finished
0809 23:19:48.282684 (+   112us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:48.282802 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34187,"prepare.queue_time_us":367829,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":45094,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":11904,"spinlock_wait_cycles":10938752,"thread_start_us":957466,"threads_started":2,"wal-append.queue_time_us":639358}
I20260809 23:19:48.430733 24320 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:48.409361 30264 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:48.415482 30522 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:48.340138 30679 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.426914 30508 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.465512 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:48.478233 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.487821 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.480669 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:48.425740 30504 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260809 23:19:48.436872 30448 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.002s
I20260809 23:19:48.301359 30492 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:48.531309 24541 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:48.532311 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.503836 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:48.533572 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.537669 24535 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:48.540572 24309 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:48.549769 24313 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:19:48.563553 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:48.565189 24534 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 10. Ops: []
I20260809 23:19:48.567611 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.591528 24539 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:48.599551 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61854.3us
I20260809 23:19:48.622138 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:48.590312 25369 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:48.480235 30474 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.633068 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:48.633845 24546 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:48.631996 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:48.652572 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.656524 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.658102 24425 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260809 23:19:48.656545 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.667263 24431 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
I20260809 23:19:48.416388 30458 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.679414 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.486297 30506 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:48.499235 30504 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:48.455107 30465 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.690168 24311 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:48.691569 24422 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:48.706599 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.706991 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.692329 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.697484 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1177) took 1080 ms. Trace:
I20260809 23:19:48.709314 25363 rpcz_store.cc:276] 0809 23:19:47.581854 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.822876 (+241022us) service_pool.cc:227] Handling call
0809 23:19:48.662167 (+839291us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:48.662342 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":865414912}
I20260809 23:19:48.698266 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.711236 24534 raft_consensus.cc:2354] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:48.712044 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:48.712204 30699 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:48.712724 24534 raft_consensus.cc:2354] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
I20260809 23:19:48.704403 24541 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:48.430876 30472 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:48.700749 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.703233 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.569418 30605 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:48.631251 30497 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:48.653359 30523 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:48.685101 24313 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:48.651875 30471 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260809 23:19:48.707432 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:48.720116 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:48.722815 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:48.659296 30431 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:48.661406 30474 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.512498 30679 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:48.725063 24321 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 11. Ops: []
I20260809 23:19:48.725507 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:48.725049 24544 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:48.804306 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.705760 30504 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:48.690984 30428 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:49.093815 24430 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260809 23:19:49.094189 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 493657us
W20260809 23:19:48.813926 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 96853.5us
I20260809 23:19:48.753129 24546 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:48.537792 30448 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.096787 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:48.776895 30675 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.117165 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:49.138239 24430 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:49.096701 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:48.679090 30514 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:48.782375 30659 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.274s	user 0.003s	sys 0.005s
W20260809 23:19:48.783071 30615 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.340s	user 0.000s	sys 0.008s
W20260809 23:19:48.787671 30707 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
W20260809 23:19:49.148408 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51809.5us
I20260809 23:19:49.153546 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:49.154639 24422 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 9
I20260809 23:19:48.783538 30472 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:49.156841 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 434043us
I20260809 23:19:48.780450 30508 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:49.179554 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:49.181011 24539 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:49.186751 24313 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:49.189363 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:49.193368 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:48.756965 30497 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.194557 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:49.195796 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:48.748759 30605 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:49.196380 30605 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:49.196389 25375 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:49.198612 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:49.199661 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:49.200201 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:49.195910 30497 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:49.203615 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:49.215848 24430 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:49.220358 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:49.227833 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:49.231443 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:48.791189 29709 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:49.247022 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:48.705260 30506 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.120550 30460 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:48.790046 30682 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:48.792426 30661 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:49.250041 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129641us
I20260809 23:19:49.258469 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:49.250193 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:49.147428 30496 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.256054 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:49.247998 24546 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:49.147574 30524 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.268694 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:49.282269 24311 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:49.173610 30461 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:49.204161 29688 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.292016 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:49.216647 30523 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:49.295770 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.434s	user 0.007s	sys 0.079s
I20260809 23:19:49.285939 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:48.718572 30264 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.329011 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:49.357679 25363 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:19:49.328382 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 147578us
W20260809 23:19:49.358870 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.498s	user 0.007s	sys 0.083s
I20260809 23:19:49.136197 30431 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.360071 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:48.701313 30522 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:49.373000 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1001) took 3484 ms (client timeout 1838 ms). Trace:
W20260809 23:19:49.468518 25370 rpcz_store.cc:269] 0809 23:19:45.892583 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.892967 (+   384us) service_pool.cc:227] Handling call
0809 23:19:49.372785 (+3479818us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.372920 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1874263808}
W20260809 23:19:49.371294 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1002) took 3477 ms (client timeout 2613 ms). Trace:
W20260809 23:19:49.469889 25368 rpcz_store.cc:269] 0809 23:19:45.893643 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.893796 (+   153us) service_pool.cc:227] Handling call
0809 23:19:49.371051 (+3477255us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.371203 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5753210624}
W20260809 23:19:49.369418 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 886) took 1629 ms (client timeout 1836 ms). Trace:
I20260809 23:19:49.471491 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:49.472222 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:49.329010 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:49.334354 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59311.4us
W20260809 23:19:49.473536 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.378s	user 0.035s	sys 0.065s
I20260809 23:19:49.346482 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:49.474442 24546 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:49.149494 30675 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.458113 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2613) took 4511 ms. Trace:
I20260809 23:19:49.483650 24540 rpcz_store.cc:276] 0809 23:19:44.858983 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:44.860180 (+  1197us) service_pool.cc:227] Handling call
0809 23:19:47.139980 (+2279800us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:47.143391 (+  3411us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:49.369816 (+2226425us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b5400012cd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.369992 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5242111744}
W20260809 23:19:49.458849 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2235) took 4193 ms (client timeout 1517 ms). Trace:
W20260809 23:19:49.485143 24320 rpcz_store.cc:269] 0809 23:19:45.177275 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.177463 (+   188us) service_pool.cc:227] Handling call
0809 23:19:49.370438 (+4192975us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b54002535d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.370569 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7480130432}
I20260809 23:19:49.286509 30661 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:49.473841 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.378s	user 0.035s	sys 0.065s
W20260809 23:19:49.284689 30109 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s	user 0.022s	sys 0.070s
W20260809 23:19:49.471958 24535 rpcz_store.cc:269] 0809 23:19:47.739905 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.740022 (+   117us) service_pool.cc:227] Handling call
0809 23:19:49.369149 (+1629127us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.369343 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1724878720}
I20260809 23:19:49.164220 30514 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.355373 30508 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20260809 23:19:49.499924 30514 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:49.356523 30684 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:49.490202 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:49.494989 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 883) took 1840 ms. Trace:
W20260809 23:19:49.498082 30109 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.446s	user 0.025s	sys 0.073s
I20260809 23:19:49.449312 30523 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:49.554859 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:49.488839 24422 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:49.518764 24543 rpcz_store.cc:276] 0809 23:19:47.654215 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.655266 (+  1051us) service_pool.cc:227] Handling call
0809 23:19:47.655639 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:47.655895 (+   256us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:48.095360 (+439465us) log.cc:842] Serialized 14 byte log entry
0809 23:19:49.494588 (+1399228us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:49.494650 (+    62us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:49.494710 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:49.494781 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:49.494914 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":784603,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":52,"thread_start_us":1837938,"threads_started":2}
I20260809 23:19:49.137058 30428 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:49.494467 30699 consensus_queue.cc:1159] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 10 status { last_received { term: 10 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 10 index: 10 } }
I20260809 23:19:49.494971 30695 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:49.569336 30695 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:49.569710 30695 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:49.570348 30428 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:49.573218 30506 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:49.573788 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 790) took 2553 ms. Trace:
I20260809 23:19:49.574848 30508 consensus_queue.cc:1225] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:49.489187 29709 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:49.577905 29709 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:49.551947 24430 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:49.287369 30460 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.558209 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:49.563617 30427 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:49.491792 30264 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:49.294901 30524 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.509064 30661 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:49.567137 30458 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:49.568004 30535 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:49.355039 30150 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.155s
I20260809 23:19:49.287729 30496 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.578830 29709 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.375049 30377 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.250s
I20260809 23:19:49.324514 29688 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:49.722108 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:49.715430 24539 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.534s	user 0.015s	sys 0.005s
I20260809 23:19:49.789260 24539 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:49.634128 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:49.793790 24317 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:49.492254 30486 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:49.785624 24422 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:49.655532 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:49.574229 24423 rpcz_store.cc:276] 0809 23:19:47.020789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.021451 (+   662us) service_pool.cc:227] Handling call
0809 23:19:47.021815 (+   364us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:47.022145 (+   330us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:47.022175 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:47.022587 (+   412us) log.cc:842] Serialized 26 byte log entry
0809 23:19:48.479452 (+1456865us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:48.479501 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.479564 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:49.573580 (+1094016us) raft_consensus.cc:1671] finished
0809 23:19:49.573603 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:49.573729 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3038,"raft.queue_time_us":10071,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":708,"thread_start_us":1456528,"threads_started":1,"wal-append.queue_time_us":2549226}
I20260809 23:19:49.801503 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:49.785553 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2724) took 1069 ms. Trace:
I20260809 23:19:49.861617 24430 rpcz_store.cc:276] 0809 23:19:48.716503 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:49.473095 (+756592us) service_pool.cc:227] Handling call
0809 23:19:49.551534 (+ 78439us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:49.785224 (+233690us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:49.785494 (+   270us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:49.737346 30504 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.000s
I20260809 23:19:49.708688 30141 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:49.510811 30684 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:49.864969 25369 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:49.579538 30523 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:49.704974 30509 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.248s	user 0.007s	sys 0.003s
I20260809 23:19:49.874042 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2643) took 1382 ms. Trace:
I20260809 23:19:49.978478 24541 rpcz_store.cc:276] 0809 23:19:48.491683 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.532059 (+ 40376us) service_pool.cc:227] Handling call
0809 23:19:49.873826 (+1341767us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.873967 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":393153920}
I20260809 23:19:49.708844 30448 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:49.878041 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2485) took 2124 ms. Trace:
I20260809 23:19:49.653501 30109 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.716562 30656 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:49.624680 30431 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:49.623576 30682 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:49.590808 30675 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:49.957507 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 882) took 2230 ms (client timeout 2101 ms). Trace:
W20260809 23:19:50.005015 24545 rpcz_store.cc:269] 0809 23:19:47.644780 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.644936 (+   156us) service_pool.cc:227] Handling call
0809 23:19:49.874772 (+2229836us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:49.874905 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":353004800}
I20260809 23:19:49.717146 30679 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:49.707028 30514 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:49.601879 30494 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.356s
I20260809 23:19:49.970216 30684 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:49.730944 30589 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:49.807911 30427 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.026760 30687 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.969s	user 0.018s	sys 0.020s
I20260809 23:19:50.027057 30460 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.016793 30337 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.027901 30337 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.029155 30495 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:49.972096 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:49.983168 24544 rpcz_store.cc:276] 0809 23:19:47.753709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.753847 (+   138us) service_pool.cc:227] Handling call
0809 23:19:47.792472 (+ 38625us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.807102 (+1014630us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:49.877738 (+1070636us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b5400147b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:49.877968 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2144155520}
I20260809 23:19:49.852995 30458 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:49.810277 30577 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:50.037112 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1018) took 3915 ms. Trace:
I20260809 23:19:50.037664 24314 rpcz_store.cc:276] 0809 23:19:46.122085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.125005 (+  2920us) service_pool.cc:227] Handling call
0809 23:19:46.194102 (+ 69097us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.458237 (+2264135us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.036864 (+1578627us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:50.037053 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5153007872}
W20260809 23:19:50.039535 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1179) took 2403 ms (client timeout 2999 ms). Trace:
W20260809 23:19:50.040114 25372 rpcz_store.cc:269] 0809 23:19:47.636549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.901939 (+265390us) service_pool.cc:227] Handling call
0809 23:19:50.039341 (+2137402us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:50.039471 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179870592}
I20260809 23:19:50.033039 24423 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:49.998986 24422 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:49.995918 24427 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:50.001565 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2725) took 1262 ms. Trace:
I20260809 23:19:50.033592 30337 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.034442 30458 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.035125 30496 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:49.873085 30504 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.003535 30109 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.010892 30682 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:50.100200 30458 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:50.047854 30593 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:50.128022 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.901s	user 0.000s	sys 0.062s
W20260809 23:19:50.128517 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.902s	user 0.000s	sys 0.062s
I20260809 23:19:50.031701 30589 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:50.140074 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1017) took 3660 ms. Trace:
I20260809 23:19:50.045116 30577 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:50.118564 24311 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.002s
I20260809 23:19:50.144554 24311 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:50.030547 30331 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:50.048112 24430 rpcz_store.cc:276] 0809 23:19:48.739674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:49.870872 (+1131198us) service_pool.cc:227] Handling call
0809 23:19:49.997297 (+126425us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:50.001215 (+  3918us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:50.001248 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:50.001289 (+    41us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:50.001323 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.001386 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.001505 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3574,"raft.queue_time_us":3664,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8453}
W20260809 23:19:50.137586 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.933s	user 0.053s	sys 0.109s
W20260809 23:19:50.178643 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.974s	user 0.053s	sys 0.109s
I20260809 23:19:50.141535 30481 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:50.161274 30450 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.128865 30593 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.162092 30109 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:50.162604 30667 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 2.001s	user 0.013s	sys 0.043s
I20260809 23:19:50.167575 30524 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:50.169184 30427 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.176216 30504 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.140583 25365 rpcz_store.cc:276] 0809 23:19:46.479934 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:46.483271 (+  3337us) service_pool.cc:227] Handling call
0809 23:19:46.483622 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:46.483938 (+   316us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:46.515524 (+ 31586us) log.cc:842] Serialized 14 byte log entry
0809 23:19:48.226636 (+1711112us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b180032ee40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:19:50.139591 (+1912955us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:50.139661 (+    70us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:50.139729 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.139889 (+   160us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.140019 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":39905,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3707,"spinlock_wait_cycles":11513216,"thread_start_us":1902045,"threads_started":1}
I20260809 23:19:50.132040 29688 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:50.187328 30667 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 2.148s	user 0.014s	sys 0.044s
W20260809 23:19:50.176921 30695 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.607s	user 0.004s	sys 0.004s
W20260809 23:19:50.191926 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.897s	user 0.008s	sys 0.036s
W20260809 23:19:50.197428 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.903s	user 0.008s	sys 0.036s
W20260809 23:19:50.193120 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.932s	user 0.017s	sys 0.025s
I20260809 23:19:50.145772 30641 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.015403 30514 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.177124 30331 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:50.205909 30331 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.207056 30514 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.207687 30331 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:50.199988 30651 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.990s	user 0.019s	sys 0.025s
W20260809 23:19:50.213732 30651 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.004s	user 0.019s	sys 0.025s
W20260809 23:19:50.198832 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.938s	user 0.017s	sys 0.025s
I20260809 23:19:50.199872 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:19:50.192126 30675 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:50.220414 24311 raft_consensus.cc:595] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:50.222100 24311 raft_consensus.cc:603] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:50.195425 30661 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.199173 29688 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:50.221061 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.96015e+06us
W20260809 23:19:50.204705 30605 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.008s	user 0.013s	sys 0.000s
I20260809 23:19:50.188189 30450 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.312021 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:50.183104 30682 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:50.229234 30512 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.932s	user 0.004s	sys 0.043s
I20260809 23:19:50.198897 30695 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.352340 25321 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.090s	user 0.017s	sys 0.033s
W20260809 23:19:50.370498 25321 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.109s	user 0.017s	sys 0.033s
W20260809 23:19:50.372634 24266 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.804s	user 0.045s	sys 0.053s
W20260809 23:19:50.373078 24266 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.805s	user 0.045s	sys 0.053s
W20260809 23:19:50.372879 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 2.11116e+06us
W20260809 23:19:50.373567 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.80556e+06us
W20260809 23:19:50.357142 30512 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.062s	user 0.004s	sys 0.044s
W20260809 23:19:50.406054 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.028s	sys 0.024s
W20260809 23:19:50.406471 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.028s	sys 0.024s
I20260809 23:19:50.290422 30605 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.290563 29688 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.417421 30661 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.418480 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:19:50.422564 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2691) took 2961 ms. Trace:
W20260809 23:19:50.423475 30689 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.365s	user 0.006s	sys 0.013s
I20260809 23:19:50.425073 30577 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:50.374361 30680 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.019s	user 0.041s	sys 0.081s
I20260809 23:19:50.228566 30675 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.374207 30251 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.193s	user 0.041s	sys 0.058s
I20260809 23:19:50.433290 24311 raft_consensus.cc:595] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:50.433243 24323 rpcz_store.cc:276] 0809 23:19:47.461142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.635332 (+174190us) service_pool.cc:227] Handling call
0809 23:19:47.739699 (+104367us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:47.745405 (+  5706us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:47.745446 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:47.755345 (+  9899us) log.cc:842] Serialized 26 byte log entry
0809 23:19:48.115127 (+359782us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:48.115165 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:48.115225 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:50.421305 (+2306080us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b2400056228. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:19:50.422234 (+   929us) raft_consensus.cc:1671] finished
0809 23:19:50.422253 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.422505 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46315,"raft.queue_time_us":10130,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":34267,"spinlock_wait_cycles":33765760,"thread_start_us":282329,"threads_started":1,"wal-append.queue_time_us":769039}
I20260809 23:19:50.442088 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:50.444741 24430 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:50.448913 24423 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260809 23:19:50.445500 30680 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.083s	user 0.041s	sys 0.081s
W20260809 23:19:50.445436 30251 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.268s	user 0.042s	sys 0.058s
I20260809 23:19:50.464344 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.439298 30458 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.439061 30460 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.469710 25366 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:50.471808 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:50.474296 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:50.475301 25365 raft_consensus.cc:595] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:50.471161 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.475723 25365 raft_consensus.cc:603] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:50.477712 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.437846 30661 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:50.479748 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s	user 0.039s	sys 0.050s
W20260809 23:19:50.474956 30724 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
W20260809 23:19:50.492110 30264 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.684s	user 0.012s	sys 0.000s
I20260809 23:19:50.493088 24425 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:50.493680 30509 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.494212 30509 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.495616 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:19:50.496892 30651 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.497462 30651 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.500808 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2222) took 5211 ms (client timeout 3201 ms). Trace:
W20260809 23:19:50.501191 24427 rpcz_store.cc:269] 0809 23:19:45.289429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.369242 (+ 79813us) service_pool.cc:227] Handling call
0809 23:19:50.500593 (+5131351us) spinlock_profiling.cc:243] Waited 4.62 s on lock 0x7b540023da54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:50.500715 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10634918656}
I20260809 23:19:50.501870 29688 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.502426 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:50.474264 30458 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.483829 30535 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:50.498054 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:50.527339 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:50.485455 30251 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
W20260809 23:19:50.488919 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.040s	sys 0.050s
I20260809 23:19:50.483959 30448 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:50.483807 30656 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260809 23:19:50.468398 30512 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:50.520016 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.480453 30377 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:50.539496 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.510865 30450 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.544430 24430 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:50.563273 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:50.558542 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.756s	user 0.011s	sys 0.047s
I20260809 23:19:50.557878 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 896) took 1331 ms. Trace:
W20260809 23:19:50.531625 30548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.871s	user 0.026s	sys 0.034s
W20260809 23:19:50.564430 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.762s	user 0.011s	sys 0.047s
I20260809 23:19:50.564538 24534 rpcz_store.cc:276] 0809 23:19:49.226821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:49.373429 (+146608us) service_pool.cc:227] Handling call
0809 23:19:49.495651 (+122222us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:49.495892 (+   241us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:19:49.555017 (+ 59125us) log.cc:842] Serialized 14 byte log entry
0809 23:19:50.546626 (+991609us) log.cc:842] Serialized 14 byte log entry
0809 23:19:50.557509 (+ 10883us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:50.557567 (+    58us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:50.557615 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.557697 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.557816 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14786,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":11420,"thread_start_us":985558,"threads_started":1}
I20260809 23:19:50.570892 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.571727 24427 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:50.481802 30460 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.576275 24435 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.885s	user 0.024s	sys 0.035s
W20260809 23:19:50.576833 24435 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.886s	user 0.024s	sys 0.035s
I20260809 23:19:50.577756 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.579006 24427 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260809 23:19:50.556370 30448 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:50.578562 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:50.557778 30512 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:50.568346 30548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.918s	user 0.027s	sys 0.034s
I20260809 23:19:50.557554 30647 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:50.563489 30337 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.559270 30680 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.584379 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.568367 30715 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.569644 30377 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:50.580317 30679 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:50.557881 30535 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:50.592415 30512 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:50.584666 29688 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:50.563974 30453 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.842s	user 0.016s	sys 0.032s
I20260809 23:19:50.588693 30448 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:50.615528 30153 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:50.654500 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.611074 30715 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.630264 30377 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.631815 30641 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:50.632125 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2258) took 3480 ms. Trace:
W20260809 23:19:50.652132 30141 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.689s	user 0.009s	sys 0.004s
W20260809 23:19:50.650339 30453 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.928s	user 0.017s	sys 0.033s
W20260809 23:19:50.644286 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2721) took 1965 ms (client timeout 2544 ms). Trace:
I20260809 23:19:50.646560 30509 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.618582 30531 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:50.613821 25373 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:50.641848 30593 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.648819 30577 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.655814 30645 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
I20260809 23:19:50.660710 30629 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:50.661039 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.661646 30458 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.661046 30535 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.628096 30605 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:50.662981 30693 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.421s	user 0.008s	sys 0.035s
I20260809 23:19:50.628232 30679 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.663281 30727 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.005s
I20260809 23:19:50.612474 30337 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.627833 30695 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:50.721546 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.400s	user 0.017s	sys 0.042s
I20260809 23:19:50.658952 24316 rpcz_store.cc:276] 0809 23:19:47.151452 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.151611 (+   159us) service_pool.cc:227] Handling call
0809 23:19:49.371697 (+2220086us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:49.372106 (+   409us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:49.372147 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:49.372203 (+    56us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:49.372270 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:49.372349 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.631940 (+1259591us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54002535d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:50.632061 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1311,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":21819,"spinlock_wait_cycles":5104694656}
W20260809 23:19:50.671132 24425 rpcz_store.cc:269] 0809 23:19:48.679022 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.679134 (+   112us) service_pool.cc:227] Handling call
0809 23:19:50.644051 (+1964917us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:50.644211 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4169112960}
W20260809 23:19:50.674162 24424 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.961s	user 0.036s	sys 0.034s
I20260809 23:19:50.612563 30647 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.657469 30153 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:50.619784 30680 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.738977 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:50.739750 24425 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:50.653368 29688 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:50.719936 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58971.8us
I20260809 23:19:50.668963 30398 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.652496 30651 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.673377 30531 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:50.725242 30577 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:19:50.735927 24424 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.023s	user 0.036s	sys 0.039s
W20260809 23:19:50.736333 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.185s	user 0.014s	sys 0.027s
I20260809 23:19:50.720077 30573 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:19:50.744324 30398 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.764071 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.250s	user 0.015s	sys 0.027s
I20260809 23:19:50.737643 30661 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:50.764561 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.40477e+06us
I20260809 23:19:50.767056 24380 leader_election.cc:398] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:50.767773 24380 leader_election.cc:401] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:50.768752 25371 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260809 23:19:50.816418 30651 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:50.816725 30426 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.817114 30651 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.817394 30426 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.752784 30658 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260809 23:19:50.820281 30474 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.820711 30426 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.822623 30337 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.823474 30651 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.834197 30398 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:50.838092 30523 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.826s	user 0.008s	sys 0.002s
I20260809 23:19:50.839145 30460 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.839706 30593 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:50.840457 30106 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.841039 30106 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.843081 30675 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:50.845921 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 79065.5us
I20260809 23:19:50.846755 24380 leader_election.cc:398] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
I20260809 23:19:50.763015 30577 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:50.847373 30577 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:50.847810 30586 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.852916 30679 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.855808 29688 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:50.752555 30695 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:50.757560 24425 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:50.856833 30695 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.857322 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.858110 30573 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:50.858412 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:50.859042 24425 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:50.859954 30573 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:19:50.862061 30732 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:19:50.862146 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1108) took 3524 ms. Trace:
W20260809 23:19:50.735873 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.480s	user 0.017s	sys 0.045s
I20260809 23:19:50.862887 25373 rpcz_store.cc:276] 0809 23:19:47.338063 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.338977 (+   914us) service_pool.cc:227] Handling call
0809 23:19:47.377752 (+ 38775us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:50.678776 (+3301024us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.861930 (+183154us) spinlock_profiling.cc:243] Waited 3.23 s on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:50.862089 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7428092672}
I20260809 23:19:50.757902 30652 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.739432 30153 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:50.736958 30629 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:50.877357 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.878029 24545 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:50.880774 24545 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:19:50.881394 24544 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:19:50.687057 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2493) took 2484 ms. Trace:
W20260809 23:19:50.856289 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1118) took 1626 ms (client timeout 1893 ms). Trace:
W20260809 23:19:50.883662 25369 rpcz_store.cc:269] 0809 23:19:49.230164 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:49.231123 (+   959us) service_pool.cc:227] Handling call
0809 23:19:50.856030 (+1624907us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:50.856216 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1454270208}
I20260809 23:19:50.847074 24380 leader_election.cc:401] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260809 23:19:50.678733 30567 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.482s	user 0.022s	sys 0.048s
W20260809 23:19:50.873255 30716 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:19:50.872929 30106 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.904022 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:50.888056 30567 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.691s	user 0.022s	sys 0.049s
I20260809 23:19:50.858839 30695 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:50.838630 30523 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:50.890328 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.409s	user 0.018s	sys 0.041s
W20260809 23:19:50.948143 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.475s	user 0.020s	sys 0.042s
W20260809 23:19:50.948978 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.4767e+06us
I20260809 23:19:50.949683 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:50.890767 25375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.529s	user 0.016s	sys 0.036s
I20260809 23:19:50.892076 24425 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:50.952306 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.953096 24425 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:50.878211 30522 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:50.896534 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49921.9us
W20260809 23:19:50.890434 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.430s	user 0.048s	sys 0.073s
W20260809 23:19:50.960119 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.500s	user 0.048s	sys 0.073s
W20260809 23:19:50.960721 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.60062e+06us
I20260809 23:19:50.960949 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.961431 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:50.962020 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.962541 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:50.880517 30645 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260809 23:19:50.914642 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.915112 30647 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.922081 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:50.978272 24534 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:50.903280 30150 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10.
I20260809 23:19:50.923890 24539 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:50.884791 24542 rpcz_store.cc:276] 0809 23:19:48.202410 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.216406 (+ 13996us) service_pool.cc:227] Handling call
0809 23:19:48.264651 (+ 48245us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.264863 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:50.197878 (+1933015us) log.cc:842] Serialized 14 byte log entry
0809 23:19:50.686617 (+488739us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:50.686675 (+    58us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:50.686742 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.686814 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:50.686986 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":175935,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":28533,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":2559,"thread_start_us":2391037,"threads_started":2}
I20260809 23:19:50.925289 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.926082 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1173) took 3383 ms. Trace:
I20260809 23:19:50.980484 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:50.980181 25366 rpcz_store.cc:276] 0809 23:19:47.542442 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.196807 (+654365us) service_pool.cc:227] Handling call
0809 23:19:48.197114 (+   307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:50.470376 (+2273262us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.925578 (+455202us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:50.925969 (+   391us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5225998592}
I20260809 23:19:50.981820 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:50.928541 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:50.920626 30331 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.004s
I20260809 23:19:50.923629 30629 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:50.935400 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.936522 30682 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:50.937889 30531 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:50.902407 30644 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.783s	user 0.014s	sys 0.047s
W20260809 23:19:50.954116 25375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.622s	user 0.016s	sys 0.036s
I20260809 23:19:50.910619 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.999279 30680 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:50.962783 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:50.999634 30531 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:50.962584 24425 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:51.001196 30523 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.001547 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:51.003196 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:50.964751 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.968384 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:51.004156 30712 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:51.004526 30712 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:51.004886 30712 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.007236 24539 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:50.943064 30577 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:50.977631 24544 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:50.984133 24540 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:50.967499 30497 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:50.985018 30652 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:50.985097 30460 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:50.967075 30461 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
W20260809 23:19:50.985929 30739 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:19:50.986325 30757 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260809 23:19:50.987285 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.007234 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2237) took 2353 ms. Trace:
I20260809 23:19:51.046753 24429 rpcz_store.cc:276] 0809 23:19:48.653770 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.653931 (+   161us) service_pool.cc:227] Handling call
0809 23:19:48.701944 (+ 48013us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:48.801551 (+ 99607us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:48.801583 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:48.802034 (+   451us) log.cc:842] Serialized 26 byte log entry
0809 23:19:50.564903 (+1762869us) raft_consensus.cc:1631] Marking committed up to 3
0809 23:19:50.564940 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:50.565015 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:51.006968 (+441953us) raft_consensus.cc:1671] finished
0809 23:19:51.006999 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:51.007165 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47403,"raft.queue_time_us":111576,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":23645,"thread_start_us":1762503,"threads_started":1,"wal-append.queue_time_us":2202782}
I20260809 23:19:50.984454 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:51.049665 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65303.9us
I20260809 23:19:51.009543 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.011219 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.052175 24542 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:50.999693 24545 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:51.012998 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.016047 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.020133 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:51.012914 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44608.3us
I20260809 23:19:51.053967 24539 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:51.054378 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:50.977186 30586 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:51.044265 24546 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:50.974419 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:51.057389 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83072.4us
W20260809 23:19:51.037506 30608 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.800s	user 0.023s	sys 0.047s
I20260809 23:19:50.983090 30593 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:51.027325 30497 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:51.053022 24540 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:51.059916 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:51.062987 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:51.062894 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:50.999336 30644 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.880s	user 0.014s	sys 0.048s
I20260809 23:19:51.087769 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:51.027295 30680 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:51.064092 24544 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:51.059765 24543 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:51.069598 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1102) took 3886 ms (client timeout 2919 ms). Trace:
W20260809 23:19:51.097844 25371 rpcz_store.cc:269] 0809 23:19:47.183285 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:47.184259 (+   974us) service_pool.cc:227] Handling call
0809 23:19:51.069328 (+3885069us) spinlock_profiling.cc:243] Waited 3.53 s on lock 0x7b54002ab7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:51.069509 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8111294848}
I20260809 23:19:51.070782 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:51.080610 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.099511 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.100214 24542 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:51.100509 24540 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:51.082149 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:51.101351 24429 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:51.089370 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.036674 30460 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:51.105479 24545 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:51.081476 30577 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.106153 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:51.108830 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45933.9us
I20260809 23:19:51.059109 30757 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:51.063676 30331 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:51.063833 30645 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:51.022032 30153 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.117064 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:51.117887 24429 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:51.088039 30497 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:51.107877 30745 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260809 23:19:51.108338 30460 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:51.176280 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:51.060032 30523 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
W20260809 23:19:51.067224 30608 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.830s	user 0.023s	sys 0.047s
W20260809 23:19:51.114820 29709 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.007s	sys 0.021s
I20260809 23:19:51.115861 30645 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:51.113386 30680 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:51.113710 30757 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 5 FOLLOWER]: Advancing to term 6
W20260809 23:19:51.207168 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 136358us
W20260809 23:19:51.208209 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148237us
I20260809 23:19:51.083066 30593 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.115123 30753 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:51.207090 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 96721.6us
I20260809 23:19:51.213246 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:51.165625 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.010s	sys 0.016s
W20260809 23:19:51.227993 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.010s	sys 0.017s
W20260809 23:19:51.228410 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 867848us
I20260809 23:19:51.233130 24534 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:51.211158 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:51.257494 25371 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:51.257881 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.258399 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.258788 25369 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:51.259295 24534 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:19:51.257535 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46470.9us
I20260809 23:19:51.262614 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:51.265254 24311 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:51.267198 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.267912 24311 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:51.269778 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.270685 30758 raft_consensus.cc:2741] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260809 23:19:51.270686 24323 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:51.272811 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:51.272709 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.274361 24314 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:51.274477 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.275097 24320 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.275489 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.276201 24311 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:51.279369 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.280287 24317 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:51.281922 24317 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.286577 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:51.291194 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.292670 24317 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:51.294656 25371 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:51.294523 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50128.4us
I20260809 23:19:51.296576 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.297988 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.302067 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:51.304505 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.306569 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.307466 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.307798 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.209123 30645 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:51.308373 24543 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
W20260809 23:19:51.226794 29709 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s	user 0.007s	sys 0.022s
I20260809 23:19:51.309321 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.226053 30449 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:51.322829 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.312045 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.245456 30585 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:51.236148 30753 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:51.308053 25366 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:51.318882 25370 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:51.307273 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:51.313902 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.324707 24543 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:51.235548 30661 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:51.331198 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.331683 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.331851 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.330497 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.302724 30758 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:51.327019 24534 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.327903 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.331907 25368 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:19:51.328751 25371 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:51.334422 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.343864 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:51.345721 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:51.347711 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.547s	user 0.012s	sys 0.057s
I20260809 23:19:51.347563 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.348264 24543 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
W20260809 23:19:51.349722 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.549s	user 0.012s	sys 0.057s
I20260809 23:19:51.349826 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:51.332656 25366 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:51.351424 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.352257 24317 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:51.354915 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.355152 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.355695 24313 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:19:51.353998 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.007s	sys 0.024s
W20260809 23:19:51.357268 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.007s	sys 0.024s
I20260809 23:19:51.311679 30647 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:51.377518 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.368966 25365 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:51.324153 30449 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:51.371773 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.375861 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.384016 24309 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:51.376416 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:51.337957 24378 connection.cc:232] RPC call timeout handler was delayed by -1.4795686721336097s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2919s for a response
I20260809 23:19:51.326799 30585 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:51.325071 30565 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:51.380963 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.328846 30753 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.384613 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:51.385319 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.405814 24309 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:51.331279 30465 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:51.396922 24323 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:51.353487 30758 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:51.383775 25366 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:51.336012 29709 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:51.412657 24269 connection.cc:232] RPC call timeout handler was delayed by -1.5121910250745714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
I20260809 23:19:51.405022 24313 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:51.422397 24309 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.422595 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:51.400723 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.411746 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.310598 30695 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.072s
I20260809 23:19:51.418515 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.423785 24320 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:51.424927 24534 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260809 23:19:51.426564 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:51.427594 25365 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:51.397357 30449 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:19:51.429417 24269 connection.cc:232] RPC call timeout handler was delayed by -1.1399089829064906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260809 23:19:51.464269 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0562089828308672s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23420000000000002s for a response
I20260809 23:19:51.466472 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.467396 24313 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:51.477072 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:51.434720 30763 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.455s	user 0.000s	sys 0.004s
W20260809 23:19:51.435547 30767 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260809 23:19:51.418707 30758 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.439399 24317 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:51.506383 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.506619 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:51.441170 30448 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.782s	user 0.005s	sys 0.006s
I20260809 23:19:51.383288 30680 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:51.444765 30512 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.791s	user 0.024s	sys 0.001s
I20260809 23:19:51.413303 30331 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:51.404547 30585 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.479250 30756 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:51.416374 30565 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.451201 30153 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:51.419450 30589 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.243s	user 0.013s	sys 0.015s
I20260809 23:19:51.458905 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:51.459509 30715 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.479552 25370 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260809 23:19:51.480160 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71542.3us
I20260809 23:19:51.478389 30624 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:19:51.330145 24498 connection.cc:232] RPC call timeout handler was delayed by -2.58045771997422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260809 23:19:51.556849 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:51.554093 30594 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.013s	sys 0.007s
I20260809 23:19:51.401304 30647 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.484151 30522 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:51.540897 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.012s	sys 0.021s
W20260809 23:19:51.513571 30573 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.006s
I20260809 23:19:51.430138 29709 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:51.558094 30684 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.532s	user 0.008s	sys 0.018s
I20260809 23:19:51.557648 24544 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:51.547101 30756 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:51.537407 30331 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:51.559180 30594 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.013s	sys 0.007s
W20260809 23:19:51.558086 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4967577198985964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
I20260809 23:19:51.553874 30494 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:51.509387 30757 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:51.578655 24498 connection.cc:232] RPC call timeout handler was delayed by -2.3311577199492604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
I20260809 23:19:51.516062 30448 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.583750 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:51.585671 24498 connection.cc:232] RPC call timeout handler was delayed by -2.2420577199663967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18930000000000002s for a response
W20260809 23:19:51.590209 24498 connection.cc:232] RPC call timeout handler was delayed by -2.1394577198661864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20070000000000002s for a response
W20260809 23:19:51.590692 24498 connection.cc:232] RPC call timeout handler was delayed by -2.0428153881803155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260809 23:19:51.568085 30373 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:51.522471 30512 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:51.591106 24498 connection.cc:232] RPC call timeout handler was delayed by -1.791715388186276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
W20260809 23:19:51.620342 24498 connection.cc:232] RPC call timeout handler was delayed by -1.5784153880085796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2042s for a response
W20260809 23:19:51.620759 24498 connection.cc:232] RPC call timeout handler was delayed by -1.5103577198460698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2706s for a response
W20260809 23:19:51.621153 24498 connection.cc:232] RPC call timeout handler was delayed by -1.2619577199220657s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2982s for a response
I20260809 23:19:51.628072 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.629079 25365 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 7.
W20260809 23:19:51.570745 30520 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.066s	user 0.038s	sys 0.036s
W20260809 23:19:51.587893 30652 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.557s	user 0.015s	sys 0.003s
I20260809 23:19:51.609377 29709 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.581290 30459 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260809 23:19:51.588917 30494 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.613090 24546 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 5. Current term is 6. Ops: []
I20260809 23:19:51.614265 30757 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.617174 30448 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.566563 30593 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.635663 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:51.562590 30624 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:51.570677 30486 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.613s	user 0.018s	sys 0.060s
I20260809 23:19:51.582589 30331 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.553942 30589 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.664639 30589 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:51.665860 24541 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.661628 30512 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:51.580389 30756 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:51.659129 29709 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:19:51.573114 30573 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.671852 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.619943 30373 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:51.660465 25365 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.660400 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:51.668332 30592 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:19:51.569716 30522 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.618237 30645 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:51.659731 30453 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.014s	sys 0.051s
I20260809 23:19:51.577484 30684 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:51.640107 30520 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.145s	user 0.039s	sys 0.036s
I20260809 23:19:51.647907 30652 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:51.649592 30535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.029s	sys 0.050s
W20260809 23:19:51.673480 30486 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.741s	user 0.018s	sys 0.061s
W20260809 23:19:51.716435 30682 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.721s	user 0.011s	sys 0.002s
W20260809 23:19:51.721293 30712 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.716s	user 0.010s	sys 0.011s
W20260809 23:19:51.720918 30453 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.000s	user 0.014s	sys 0.051s
I20260809 23:19:51.730975 30486 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:51.743527 30567 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.010s	sys 0.005s
W20260809 23:19:51.730780 30535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.029s	sys 0.051s
W20260809 23:19:51.725353 30531 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.722s	user 0.017s	sys 0.005s
I20260809 23:19:51.752713 30712 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.731688 30565 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.765352 30753 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.770650 30758 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.783883 30645 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.785885 30684 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.787663 30624 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.788443 30647 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.797222 30756 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.485868 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.503114 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.431229 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.844568 24316 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260809 23:19:51.846365 24316 pending_rounds.cc:76] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 4
I20260809 23:19:51.846665 24316 pending_rounds.cc:97] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260809 23:19:51.846881 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.507217 24323 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.477941 24311 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:19:51.853582 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.546s (SENT) [suppressed 8 similar messages]
W20260809 23:19:51.510074 24422 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.028s	sys 0.048s
I20260809 23:19:51.844213 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:51.855919 24422 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.416s	user 0.028s	sys 0.049s
I20260809 23:19:51.856352 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
W20260809 23:19:51.563181 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.013s	sys 0.021s
W20260809 23:19:51.857609 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.033s	user 0.019s	sys 0.044s
W20260809 23:19:51.858309 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.033s	user 0.019s	sys 0.045s
I20260809 23:19:51.858451 24430 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260809 23:19:51.858908 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2738) took 1353 ms. Trace:
W20260809 23:19:51.857671 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.280s	user 0.017s	sys 0.037s
I20260809 23:19:51.859443 24423 rpcz_store.cc:276] 0809 23:19:50.505614 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:50.589967 (+ 84353us) service_pool.cc:227] Handling call
0809 23:19:50.616414 (+ 26447us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:50.616693 (+   279us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:50.616728 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:50.617241 (+   513us) log.cc:842] Serialized 26 byte log entry
0809 23:19:51.858054 (+1240813us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:51.858184 (+   130us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:51.858325 (+   141us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:51.858406 (+    81us) raft_consensus.cc:1671] finished
0809 23:19:51.858417 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:51.858617 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1240170,"threads_started":1,"wal-append.queue_time_us":1142662}
W20260809 23:19:51.860720 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.283s	user 0.018s	sys 0.038s
I20260809 23:19:51.862046 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:51.863190 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s	user 0.010s	sys 0.030s
W20260809 23:19:51.863656 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.234s	user 0.010s	sys 0.030s
I20260809 23:19:51.862807 24423 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:51.865614 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2735) took 1402 ms. Trace:
I20260809 23:19:51.863459 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.869679 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:51.879240 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:51.879982 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.884790 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:19:51.656554 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.023s	sys 0.042s
I20260809 23:19:51.847118 24316 raft_consensus.cc:2866] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:51.888376 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.888780 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:51.889849 24535 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:51.889734 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.891029 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:51.891697 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:51.749150 30629 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.003s	sys 0.021s
I20260809 23:19:51.703737 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:51.724021 30377 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.027s	sys 0.045s
I20260809 23:19:51.905597 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:51.893309 24546 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:51.866014 24427 rpcz_store.cc:276] 0809 23:19:50.463163 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:50.587894 (+124731us) service_pool.cc:227] Handling call
0809 23:19:50.588283 (+   389us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:50.588766 (+   483us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:50.588804 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:50.604738 (+ 15934us) log.cc:842] Serialized 52 byte log entry
0809 23:19:51.864956 (+1260218us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:51.865080 (+   124us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:51.865231 (+   151us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:51.865302 (+    71us) raft_consensus.cc:1671] finished
0809 23:19:51.865312 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:51.865561 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1619,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":3110,"thread_start_us":1234765,"threads_started":1,"wal-append.queue_time_us":1154613}
I20260809 23:19:51.949393 24542 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10. Re-sending same reply.
I20260809 23:19:51.950176 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:51.950508 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2235) took 3297 ms. Trace:
I20260809 23:19:51.950786 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:52.008515 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:52.009485 24427 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:52.008694 24545 raft_consensus.cc:2362] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11. Re-sending same reply.
W20260809 23:19:51.952153 30497 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.775s	user 0.011s	sys 0.000s
I20260809 23:19:52.008558 24424 rpcz_store.cc:276] 0809 23:19:48.652975 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.653116 (+   141us) service_pool.cc:227] Handling call
0809 23:19:48.712251 (+ 59135us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.712514 (+   263us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:50.824675 (+2112161us) log.cc:842] Serialized 14 byte log entry
0809 23:19:51.949740 (+1125065us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:51.949907 (+   167us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:51.950032 (+   125us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:51.950169 (+   137us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:51.950396 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87376,"prepare.queue_time_us":2370680,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":93202,"raft.run_cpu_time_us":353,"raft.run_wall_time_us":353,"thread_start_us":3083744,"threads_started":2}
I20260809 23:19:52.010901 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.010738 30497 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.011930 24427 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:52.013252 30497 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.670310 30331 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.015444 24425 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:51.953933 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2232) took 3303 ms. Trace:
I20260809 23:19:52.016489 24435 rpcz_store.cc:276] 0809 23:19:48.651082 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.651241 (+   159us) service_pool.cc:227] Handling call
0809 23:19:48.651581 (+   340us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:48.651807 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:50.577209 (+1925402us) log.cc:842] Serialized 14 byte log entry
0809 23:19:51.953026 (+1375817us) log.cc:842] Serialized 14 byte log entry
0809 23:19:51.953366 (+   340us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:51.953431 (+    65us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:51.953542 (+   111us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:51.953654 (+   112us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:51.953856 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90898,"prepare.queue_time_us":2396474,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":91408,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":248,"thread_start_us":3170013,"threads_started":2}
I20260809 23:19:51.945824 30706 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:51.954183 24544 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:51.957741 24539 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:19:51.958745 24313 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:51.958333 24312 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:51.964000 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:51.965121 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:51.969439 30756 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:51.752038 30567 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.010s	sys 0.005s
I20260809 23:19:51.973526 30150 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:51.973667 30753 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.975389 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:51.976864 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 98134.2us
I20260809 23:19:51.704730 29709 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:51.978163 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.009s	sys 0.030s
I20260809 23:19:51.979461 30647 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:51.979756 30712 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.979593 30661 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:51.981222 30486 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:51.986438 30522 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:51.988260 30758 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:51.988884 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:51.990274 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84485.5us
I20260809 23:19:51.991211 30373 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:51.997279 30453 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:19:52.000447 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.302s	user 0.024s	sys 0.042s
I20260809 23:19:52.041837 25368 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:52.044404 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.009s	sys 0.031s
I20260809 23:19:52.051469 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.052173 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:52.052487 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49961.7us
I20260809 23:19:52.053808 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:52.057523 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.049441 30453 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:52.043435 30647 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:52.065676 24323 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:52.076709 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.081307 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:52.095144 30577 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:52.071079 30647 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:52.087304 24423 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:52.099709 25373 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 6: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:52.101667 25366 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:52.102748 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.103016 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.104061 24309 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:52.104730 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.105131 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:52.107954 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:52.109097 30652 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:52.109524 30652 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:52.109936 30652 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:52.110972 30629 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s	user 0.004s	sys 0.022s
I20260809 23:19:52.111168 30577 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:52.112843 30577 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:52.113745 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2251) took 1348 ms. Trace:
I20260809 23:19:52.114187 24423 rpcz_store.cc:276] 0809 23:19:50.750345 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.994515 (+1244170us) service_pool.cc:227] Handling call
0809 23:19:52.062040 (+ 67525us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:52.098413 (+ 36373us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.098650 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:52.098201 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.4049e+06us
I20260809 23:19:52.115665 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:52.116807 24423 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:52.117915 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.118614 24423 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:19:52.119010 30377 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.148s	user 0.027s	sys 0.048s
I20260809 23:19:52.066188 30573 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:52.115688 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.110572 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:52.100008 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:52.123700 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.127719 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:52.141984 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.148547 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:52.145773 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.065955 30453 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:52.157364 24313 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:52.167444 24317 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:19:52.169552 29709 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:52.153716 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.173481 30764 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:52.176493 24544 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260809 23:19:52.177417 30629 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:19:52.177958 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:52.171504 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.183555 25366 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:19:52.190299 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 336952us
I20260809 23:19:52.178007 30629 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:52.195930 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:52.159219 30373 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:52.188191 30331 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.206419 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:52.206957 30594 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.000s	sys 0.041s
I20260809 23:19:52.207290 24317 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260809 23:19:52.207350 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.013s	sys 0.020s
I20260809 23:19:52.207610 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:52.219398 24539 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
W20260809 23:19:52.208220 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.004s	sys 0.024s
W20260809 23:19:52.208568 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 296338us
W20260809 23:19:52.220417 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.005s	sys 0.021s
W20260809 23:19:52.222399 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.005s	sys 0.021s
I20260809 23:19:52.222849 24544 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:52.223313 30644 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.970s	user 0.011s	sys 0.032s
W20260809 23:19:52.223757 30644 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.011s	sys 0.032s
I20260809 23:19:52.203498 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
W20260809 23:19:52.225478 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.014s	sys 0.020s
I20260809 23:19:52.200380 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:52.224632 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.105s	user 0.014s	sys 0.019s
I20260809 23:19:52.212880 30629 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:19:52.231181 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.001s	sys 0.034s
W20260809 23:19:52.231592 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.004s	sys 0.025s
I20260809 23:19:52.215351 30573 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:52.235607 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.001s	sys 0.034s
I20260809 23:19:52.229579 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1119) took 2992 ms. Trace:
W20260809 23:19:52.240514 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.196s	user 0.014s	sys 0.019s
W20260809 23:19:52.231199 30585 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.612s	user 0.011s	sys 0.007s
I20260809 23:19:52.246631 30585 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.247151 30661 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:52.247802 30661 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:52.248296 30661 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:52.248870 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 797) took 3565 ms (client timeout 3361 ms). Trace:
W20260809 23:19:52.249269 24431 rpcz_store.cc:269] 0809 23:19:48.683092 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.719822 (+ 36730us) service_pool.cc:227] Handling call
0809 23:19:52.248462 (+3528640us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54003eb7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:52.248635 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6101108608}
W20260809 23:19:52.230190 30594 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.000s	sys 0.042s
I20260809 23:19:52.249658 24314 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 10. Ops: []
I20260809 23:19:52.251024 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:52.255810 24313 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [7.7-9.8]
I20260809 23:19:52.256948 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2737) took 1789 ms. Trace:
I20260809 23:19:52.257608 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:52.245419 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88566.5us
I20260809 23:19:52.257365 24430 rpcz_store.cc:276] 0809 23:19:50.467702 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:50.563013 (+ 95311us) service_pool.cc:227] Handling call
0809 23:19:52.256714 (+1693701us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540018d154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:52.256870 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2975099264}
I20260809 23:19:52.214730 30645 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.104s
W20260809 23:19:52.278761 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 127405us
I20260809 23:19:52.279249 30773 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:52.279820 30773 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:52.280191 30773 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:52.280435 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:19:52.282614 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 932795us
I20260809 23:19:52.250746 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.235785 30589 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:52.243799 30453 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:52.237432 25375 rpcz_store.cc:276] 0809 23:19:49.233200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:49.233444 (+   244us) service_pool.cc:227] Handling call
0809 23:19:49.279239 (+ 45795us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:49.279529 (+   290us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:51.014005 (+1734476us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.127761 (+1113756us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:52.139054 (+ 11293us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:52.164074 (+ 25020us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.195598 (+ 31524us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.225037 (+ 29439us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1926008,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":8705,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":278,"thread_start_us":2717437,"threads_started":2}
I20260809 23:19:52.249294 30585 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:52.317328 24534 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.021s	sys 0.011s
I20260809 23:19:52.318532 25370 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260809 23:19:52.331504 24534 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.022s	sys 0.012s
W20260809 23:19:52.333017 24543 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.021s	sys 0.013s
I20260809 23:19:52.292737 30589 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:52.344054 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 177567us
W20260809 23:19:52.346251 24543 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.021s	sys 0.013s
W20260809 23:19:52.334729 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 109455us
I20260809 23:19:52.317425 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.358625 24431 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:52.341140 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:52.360973 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.244315 30573 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.250741 30594 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.095s
I20260809 23:19:52.375062 25373 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:52.385916 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:52.355278 30589 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:52.332469 30712 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:52.403532 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2245) took 1973 ms. Trace:
I20260809 23:19:52.408869 25366 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260809 23:19:52.409278 25366 pending_rounds.cc:76] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 6
W20260809 23:19:52.410308 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107317us
I20260809 23:19:52.417538 30497 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:52.361408 30644 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:52.426674 25372 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:52.432363 25366 pending_rounds.cc:97] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260809 23:19:52.432678 25366 raft_consensus.cc:2866] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:52.407228 24422 rpcz_store.cc:276] 0809 23:19:50.429878 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:50.430499 (+   621us) service_pool.cc:227] Handling call
0809 23:19:50.430868 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:50.431147 (+   279us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:51.856726 (+1425579us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.268760 (+412034us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:52.284319 (+ 15559us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:52.367904 (+ 83585us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.367988 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.403465 (+ 35477us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12593,"prepare.queue_time_us":734693,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":40228,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":27918,"thread_start_us":1775256,"threads_started":2}
I20260809 23:19:52.450883 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2256) took 1651 ms. Trace:
I20260809 23:19:52.451333 24424 rpcz_store.cc:276] 0809 23:19:50.788083 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.011572 (+1223489us) service_pool.cc:227] Handling call
0809 23:19:52.012118 (+   546us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:52.012634 (+   516us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:52.013398 (+   764us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.438558 (+425160us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:52.438625 (+    67us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:52.438683 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.438750 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.438953 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":204,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":17637,"thread_start_us":409443,"threads_started":1}
I20260809 23:19:52.423446 30377 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:52.457998 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.457432 25368 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
W20260809 23:19:52.462960 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77256.8us
I20260809 23:19:52.461251 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.468336 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:52.468466 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:52.459158 30712 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.476158 24431 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:52.475479 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2257) took 1658 ms. Trace:
I20260809 23:19:52.474886 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2740) took 1809 ms. Trace:
W20260809 23:19:52.489620 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 316935us
I20260809 23:19:52.472566 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2264) took 1505 ms. Trace:
I20260809 23:19:52.479326 24427 rpcz_store.cc:276] 0809 23:19:50.817328 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.012837 (+1195509us) service_pool.cc:227] Handling call
0809 23:19:52.013299 (+   462us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:52.014386 (+  1087us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:52.014413 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:52.015118 (+   705us) log.cc:842] Serialized 26 byte log entry
0809 23:19:52.387228 (+372110us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:52.387267 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.387349 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:52.475140 (+ 87791us) raft_consensus.cc:1671] finished
0809 23:19:52.475167 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.475392 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"thread_start_us":371713,"threads_started":1,"wal-append.queue_time_us":456594}
I20260809 23:19:52.471562 24424 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:52.472141 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:52.501358 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:52.509146 30684 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:52.509656 30684 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:52.509509 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:52.509999 30684 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:52.510213 24427 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:52.472368 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:52.496850 24425 rpcz_store.cc:276] 0809 23:19:50.967318 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.002968 (+ 35650us) service_pool.cc:227] Handling call
0809 23:19:52.457405 (+1454437us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003aaf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:52.472478 (+ 15073us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2325781888}
I20260809 23:19:52.523020 24423 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:19:52.524567 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 313728us
I20260809 23:19:52.475152 30712 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:52.543839 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69002.7us
W20260809 23:19:52.546546 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48255.6us
W20260809 23:19:52.568034 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 204876us
W20260809 23:19:52.575718 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 107552us
I20260809 23:19:52.576655 24422 raft_consensus.cc:2354] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:52.479012 24429 rpcz_store.cc:276] 0809 23:19:50.665116 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.204246 (+539130us) service_pool.cc:227] Handling call
0809 23:19:51.248514 (+ 44268us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:51.302073 (+ 53559us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:51.302110 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:51.310281 (+  8171us) log.cc:842] Serialized 26 byte log entry
0809 23:19:52.223011 (+912730us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:52.223089 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.223186 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:52.474579 (+251393us) raft_consensus.cc:1671] finished
0809 23:19:52.474602 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.474803 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9322,"raft.queue_time_us":53458,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":11017,"thread_start_us":853734,"threads_started":1,"wal-append.queue_time_us":1080115}
I20260809 23:19:52.587080 24430 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:52.596449 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:52.602545 24423 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:52.611097 24429 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:52.502414 30644 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.648839 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:52.651719 30565 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:52.652475 30565 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:52.653230 24422 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:52.645413 30784 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260809 23:19:52.655246 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81664.7us
I20260809 23:19:52.655864 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.656150 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:52.656798 30497 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:52.656942 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.657272 30497 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:52.657692 30497 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:52.660964 30644 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.662147 24423 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
W20260809 23:19:52.663898 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49227.1us
I20260809 23:19:52.635485 30520 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
W20260809 23:19:52.674811 30449 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.164s	user 0.018s	sys 0.003s
I20260809 23:19:52.675019 30508 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:52.675395 30449 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.676788 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1020) took 1814 ms. Trace:
I20260809 23:19:52.677187 24422 rpcz_store.cc:276] 0809 23:19:50.862061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.654241 (+1792180us) service_pool.cc:227] Handling call
0809 23:19:52.676703 (+ 22462us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:52.684121 30812 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260809 23:19:52.685712 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:52.685815 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:52.686841 24535 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:52.687079 30331 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:19:52.687749 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:52.688009 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 163221us
I20260809 23:19:52.689334 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:52.690341 24535 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:52.691308 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:52.691529 30813 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260809 23:19:52.694674 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:52.695994 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:52.696825 30531 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:52.697252 30531 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:52.698613 30331 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Stepping down as leader of term 5
I20260809 23:19:52.699080 30331 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:52.700752 30331 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.702198 30331 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 5 FOLLOWER]: Advancing to term 6
I20260809 23:19:52.703231 30805 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:52.705024 30712 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:52.705443 30712 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:52.705570 30805 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:52.705988 30680 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:19:52.707536 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45720.3us
I20260809 23:19:52.711252 24323 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [10.9-12.10]
W20260809 23:19:52.712198 30592 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.949s	user 0.015s	sys 0.001s
I20260809 23:19:52.706761 30712 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:52.669858 30520 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:52.714807 30712 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:52.715487 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80136.6us
I20260809 23:19:52.715260 30520 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.716437 30520 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:52.717093 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:52.685606 30594 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:19:52.649881 30573 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.653837 30565 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:52.755589 24312 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.000s
I20260809 23:19:52.661597 30644 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:52.677575 30449 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.698186 30531 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.706673 30680 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:52.710493 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.690258 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:52.706310 30805 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:52.756031 24312 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:52.754630 30486 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:52.762181 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:19:52.706019 30460 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.514s	user 0.017s	sys 0.000s
I20260809 23:19:52.767225 30109 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:19:52.787997 30820 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:19:52.759732 30644 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:52.776168 30652 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.666s	user 0.016s	sys 0.001s
W20260809 23:19:52.761257 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94702.7us
I20260809 23:19:52.774282 30535 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260809 23:19:52.785728 24430 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260809 23:19:52.795192 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84833.9us
I20260809 23:19:52.795977 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2266) took 1566 ms. Trace:
I20260809 23:19:52.797621 24430 rpcz_store.cc:276] 0809 23:19:51.229673 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.516866 (+1287193us) service_pool.cc:227] Handling call
0809 23:19:52.586889 (+ 70023us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:52.795713 (+208824us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.795920 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:52.789639 30758 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:52.803287 30449 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:19:52.803521 30758 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:52.803953 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:52.804158 30449 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:52.797238 30652 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:52.807425 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 117377us
I20260809 23:19:52.808938 30652 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.809695 30652 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:52.810112 30652 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:52.803987 30758 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:52.813004 30559 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:52.813513 30559 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:52.814525 30559 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.815198 30824 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260809 23:19:52.814538 30798 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.543s	user 0.000s	sys 0.001s
W20260809 23:19:52.808336 30773 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.000s
I20260809 23:19:52.810657 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:52.821071 30773 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.821859 30531 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:52.826048 30494 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.828034 30773 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:52.831032 30460 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.832484 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:52.862919 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59083.3us
I20260809 23:19:52.780529 30680 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:52.835215 24545 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:52.801868 24431 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:52.795111 30644 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:52.876389 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 404325us
I20260809 23:19:52.878146 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 898) took 1364 ms. Trace:
I20260809 23:19:52.822571 30684 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.871027 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 897) took 1389 ms. Trace:
I20260809 23:19:52.878607 24543 rpcz_store.cc:276] 0809 23:19:51.513794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.514023 (+   229us) service_pool.cc:227] Handling call
0809 23:19:51.514434 (+   411us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:51.514692 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:19:52.376159 (+861467us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.877670 (+501511us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:52.877756 (+    86us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:52.877835 (+    79us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.877930 (+    95us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.878082 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":881796,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":121269,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":1240424,"threads_started":3,"wal-append.queue_time_us":358152}
I20260809 23:19:52.880112 24534 rpcz_store.cc:276] 0809 23:19:51.481439 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.483346 (+  1907us) service_pool.cc:227] Handling call
0809 23:19:51.515186 (+ 31840us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:51.515455 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:19:52.387682 (+872227us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.869831 (+482149us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:52.869944 (+   113us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:52.870015 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.870469 (+   454us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:52.870908 (+   439us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31710,"prepare.queue_time_us":952697,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":55,"thread_start_us":1189203,"threads_started":2}
I20260809 23:19:52.881457 30684 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:52.898357 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:19:52.918224 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 459530us
I20260809 23:19:52.924533 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1032) took 1032 ms. Trace:
I20260809 23:19:52.874379 30486 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:52.925704 30486 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:52.926126 30486 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:52.805845 30449 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:52.924850 25365 rpcz_store.cc:276] 0809 23:19:51.892582 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.832092 (+939510us) service_pool.cc:227] Handling call
0809 23:19:52.924473 (+ 92381us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:52.940217 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 202436us
W20260809 23:19:52.902765 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63599.9us
I20260809 23:19:52.944144 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:52.928220 30645 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.021s	sys 0.014s
I20260809 23:19:52.873692 30753 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:52.938421 30453 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.633s	user 0.006s	sys 0.005s
I20260809 23:19:52.899422 30377 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:53.016319 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70082.4us
W20260809 23:19:53.019717 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69450.4us
W20260809 23:19:53.015617 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 359538us
I20260809 23:19:52.936811 30573 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:52.873863 30758 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.025678 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2269) took 1033 ms. Trace:
W20260809 23:19:52.997494 30645 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.021s	sys 0.015s
I20260809 23:19:52.959151 30494 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:53.037187 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93154.6us
I20260809 23:19:53.005999 30508 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:53.040969 24496 leader_election.cc:398] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:53.041296 24496 leader_election.cc:401] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:53.043421 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:53.045202 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 234672us
I20260809 23:19:53.045410 24430 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.025272 30806 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:53.027163 30573 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:53.039885 30577 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.045s	sys 0.049s
I20260809 23:19:53.006515 30373 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260809 23:19:53.062665 30833 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:53.063282 30758 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.064463 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:53.065346 30806 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:53.057595 30577 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.045s	sys 0.049s
I20260809 23:19:53.087603 30806 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:53.067324 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1188) took 1771 ms (client timeout 1859 ms). Trace:
I20260809 23:19:53.049995 30508 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:53.068979 30822 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:19:53.097433 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1044) took 1256 ms. Trace:
I20260809 23:19:53.097668 30508 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.097824 24309 rpcz_store.cc:276] 0809 23:19:51.840964 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.169995 (+329031us) service_pool.cc:227] Handling call
0809 23:19:52.170434 (+   439us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:52.170837 (+   403us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:52.170875 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:52.202519 (+ 31644us) log.cc:842] Serialized 26 byte log entry
0809 23:19:52.539844 (+337325us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:52.539906 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.539993 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:53.097177 (+557184us) raft_consensus.cc:1671] finished
0809 23:19:53.097199 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.097371 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":319279,"threads_started":1,"wal-append.queue_time_us":575316}
W20260809 23:19:53.054721 30629 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.823s	user 0.012s	sys 0.001s
I20260809 23:19:53.100098 30629 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.070106 30845 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260809 23:19:53.075745 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.103994 24430 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.077267 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2262) took 2192 ms. Trace:
I20260809 23:19:52.949829 30652 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.083956 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 902) took 1212 ms. Trace:
I20260809 23:19:53.105235 30629 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.105641 24541 rpcz_store.cc:276] 0809 23:19:51.872065 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.887564 (+ 15499us) service_pool.cc:227] Handling call
0809 23:19:52.435409 (+547845us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:52.455645 (+ 20236us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:52.543483 (+ 87838us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:52.543521 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:52.591129 (+ 47608us) log.cc:842] Serialized 26 byte log entry
0809 23:19:52.916395 (+325266us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:52.916459 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.916561 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:53.083499 (+166938us) raft_consensus.cc:1671] finished
0809 23:19:53.083532 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.083782 (+   250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16038656,"thread_start_us":294677,"threads_started":1,"wal-append.queue_time_us":473467}
W20260809 23:19:53.083148 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62459.7us
I20260809 23:19:53.107613 30805 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.109036 29709 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:53.103397 30377 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.055652 30684 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:53.095109 25369 rpcz_store.cc:269] 0809 23:19:51.296070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.297712 (+  1642us) service_pool.cc:227] Handling call
0809 23:19:53.066961 (+1769249us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:53.067124 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3035791232}
I20260809 23:19:53.059597 30573 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:53.074390 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.104382 30592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:33980 (request call id 5) took 21153 ms. Trace:
I20260809 23:19:53.128626 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:53.105756 30652 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:53.104899 24424 rpcz_store.cc:276] 0809 23:19:50.878053 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.511011 (+1632958us) service_pool.cc:227] Handling call
0809 23:19:52.511493 (+   482us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:52.543176 (+ 31683us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:52.543210 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:52.597586 (+ 54376us) log.cc:842] Serialized 52 byte log entry
0809 23:19:52.927837 (+330251us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:53.006693 (+ 78856us) log.cc:842] Serialized 14 byte log entry
0809 23:19:53.006907 (+   214us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.006997 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:53.070601 (+ 63604us) raft_consensus.cc:1671] finished
0809 23:19:53.070621 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.070757 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4100,"prepare.queue_time_us":79157,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":40962,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":28978,"thread_start_us":278529,"threads_started":1,"wal-append.queue_time_us":453187}
I20260809 23:19:53.131520 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2270) took 1864 ms. Trace:
I20260809 23:19:53.139635 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.136926 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:53.129458 30661 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.881s	user 0.008s	sys 0.007s
I20260809 23:19:53.139544 24316 rpcz_store.cc:276] 0809 23:19:51.263946 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:51.264130 (+   184us) service_pool.cc:227] Handling call
0809 23:19:51.264551 (+   421us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:52.046579 (+782028us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.115066 (+1068487us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54003859d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:53.127971 (+ 12905us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1330273536}
I20260809 23:19:53.132965 30497 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.146699 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.155808 24424 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 14
I20260809 23:19:53.148164 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1046) took 1306 ms. Trace:
I20260809 23:19:53.149322 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:53.150786 30800 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.653s	user 0.001s	sys 0.001s
I20260809 23:19:53.159444 24323 rpcz_store.cc:276] 0809 23:19:51.841765 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.182117 (+340352us) service_pool.cc:227] Handling call
0809 23:19:52.227801 (+ 45684us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:52.711711 (+483910us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.147915 (+436204us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b540019cdd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:53.148107 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1111091456}
I20260809 23:19:53.151381 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:53.209574 24431 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:53.210242 24429 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:53.151736 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:53.135799 30449 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:53.215410 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:53.152046 30645 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:53.238752 24540 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 6: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:53.232669 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:53.236629 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:53.223410 24427 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:53.236426 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 127733us
I20260809 23:19:53.242203 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.143633 30565 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:53.248998 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:53.249660 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:53.200531 30460 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:53.251092 30776 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.040s	user 0.011s	sys 0.007s
I20260809 23:19:53.256009 24543 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260809 23:19:53.268332 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 221624us
I20260809 23:19:53.125873 30833 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
W20260809 23:19:53.241385 30589 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.812s	user 0.007s	sys 0.006s
I20260809 23:19:53.255472 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:53.305244 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.297649 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.217265 30805 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:53.241163 30449 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.308326 25363 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260809 23:19:53.302259 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 243785us
I20260809 23:19:53.159937 30661 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.246343 30645 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:53.320751 24316 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:19:53.303982 25365 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:53.311124 24430 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260809 23:19:53.315858 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45915.3us
I20260809 23:19:53.319095 30839 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:53.327742 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:53.329058 24313 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:53.308285 30520 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.591s	user 0.016s	sys 0.002s
I20260809 23:19:53.338227 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:53.340306 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88757.5us
W20260809 23:19:53.331130 30150 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.277s	user 0.013s	sys 0.001s
I20260809 23:19:53.341223 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:53.349668 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:53.328648 30645 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:53.342976 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:53.358915 25372 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:53.355787 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:53.359922 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:53.360522 24309 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 11
I20260809 23:19:53.327564 30830 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:53.362286 24312 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:53.356447 25369 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:53.364374 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2241) took 4700 ms. Trace:
I20260809 23:19:53.299801 30565 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.388064 24431 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:53.310683 30589 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:53.379287 30647 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.266s	user 0.013s	sys 0.001s
W20260809 23:19:53.377336 30836 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
W20260809 23:19:53.407469 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50603.1us
I20260809 23:19:53.357918 30150 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:19:53.420012 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.413852 30819 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:53.414180 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:53.416627 30865 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260809 23:19:53.319294 30840 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.260s
I20260809 23:19:53.437177 30878 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:19:53.437498 30837 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260809 23:19:53.368244 24428 rpcz_store.cc:276] 0809 23:19:48.656914 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:48.719322 (+ 62408us) service_pool.cc:227] Handling call
0809 23:19:48.816099 (+ 96777us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:49.141821 (+325722us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:49.141861 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:49.142330 (+   469us) log.cc:842] Serialized 26 byte log entry
0809 23:19:51.044355 (+1902025us) spinlock_profiling.cc:243] Waited 7.73 ms on lock 0x7b18003b56c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:52.369009 (+1324654us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:52.583536 (+214527us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.596309 (+ 12773us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:52.602442 (+  6133us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:53.150096 (+547654us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:53.357294 (+207198us) raft_consensus.cc:1671] finished
0809 23:19:53.357313 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.357431 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":461920,"prepare.queue_time_us":22958,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":3021,"raft.queue_time_us":336975,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":37902,"spinlock_wait_cycles":77918720,"thread_start_us":1364142,"threads_started":2,"wal-append.queue_time_us":3212367}
I20260809 23:19:53.442049 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:53.426992 30644 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.550s	user 0.012s	sys 0.000s
I20260809 23:19:53.421752 29709 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:53.421156 30647 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.450776 24427 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:53.425304 30497 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.458678 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.457556 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2260) took 2626 ms. Trace:
I20260809 23:19:53.461203 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.464929 24423 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:53.459880 30833 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:53.466842 30833 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:53.467283 30833 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:53.463169 24435 rpcz_store.cc:276] 0809 23:19:50.831330 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.018635 (+1187305us) service_pool.cc:227] Handling call
0809 23:19:52.053143 (+ 34508us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:52.053968 (+   825us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:19:52.345602 (+291634us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b180045d8a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:19:52.397424 (+ 51822us) log.cc:842] Serialized 14 byte log entry
0809 23:19:52.937910 (+540486us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:53.015730 (+ 77820us) log.cc:842] Serialized 26 byte log entry
0809 23:19:53.015891 (+   161us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:53.015935 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.016022 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:53.457261 (+441239us) raft_consensus.cc:1671] finished
0809 23:19:53.457371 (+   110us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.457497 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125007,"prepare.queue_time_us":343205,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":105274,"raft.run_cpu_time_us":587,"raft.run_wall_time_us":30700,"spinlock_wait_cycles":2770688,"thread_start_us":399989,"threads_started":1,"wal-append.queue_time_us":622674}
I20260809 23:19:53.363994 30839 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.437000 30377 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:19:53.454443 30531 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.002s
I20260809 23:19:53.390491 30830 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.473126 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.475502 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.476413 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.449350 30661 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.496016 24428 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.462268 30712 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:53.505126 24321 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:53.509027 30842 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:53.509783 30870 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260809 23:19:53.511832 24430 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:53.513515 24430 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:19:53.499460 30486 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.000s
I20260809 23:19:53.515318 30486 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.517422 30486 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.519356 24543 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:53.134567 30592 rpcz_store.cc:276] 0809 23:19:31.921970 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:31.922091 (+   121us) service_pool.cc:227] Handling call
0809 23:19:33.064693 (+1142602us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b54006e08a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5dc0 00007fac23866bd2 00007fac23865890 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272
0809 23:19:34.736883 (+1672190us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b58003419b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac2385ba7a 00007fac2385cada 00007fac2385bc8b 00007fac23867084 00007fac23866743 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184
0809 23:19:34.742142 (+  5259us) log.cc:842] Serialized 494 byte log entry
0809 23:19:35.610312 (+868170us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b54006e08a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f1a4f 00007fac237da545 00007fac237df771 00007fac237df6f9 00007fac237df689 00007fac237df651 00007fac237de81d 00007fac29b9c204 00007fac29b9c039 00007fac209386b6
0809 23:19:37.313220 (+1702908us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b54006e08a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f9416 00007fac237f2daa 00007fac237f2014 00007fac237da545 00007fac237df771 00007fac237df6f9 00007fac237df689 00007fac237df651 00007fac237de81d 00007fac29b9c204
0809 23:19:53.057633 (+15744413us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":511002,"raft.queue_time_us":279549,"raft.run_cpu_time_us":64803,"raft.run_wall_time_us":2583867,"spinlock_wait_cycles":3119539456,"thread_start_us":183160,"threads_started":1,"wal-append.queue_time_us":195960,"wal-append.run_cpu_time_us":2965,"wal-append.run_wall_time_us":1009083}
I20260809 23:19:53.520932 24534 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:53.522176 30773 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:53.522581 30773 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:53.522656 30647 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.438041 30845 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260809 23:19:53.511848 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.534993 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.479588 30377 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:53.539243 25371 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:53.541759 24424 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:19:53.501245 30878 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:53.544840 30842 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:53.547770 30866 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260809 23:19:53.548174 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 140287us
I20260809 23:19:53.523495 30773 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.522743 30839 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:53.449980 30644 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.533372 30712 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.561041 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.562384 24435 raft_consensus.cc:2354] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:53.564049 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.535743 30531 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:53.560876 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:53.582119 24534 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:53.568683 24309 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:53.519375 30882 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:53.493863 30819 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:19:53.551868 30867 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260809 23:19:53.501463 30520 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
W20260809 23:19:53.585317 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 243196us
I20260809 23:19:53.586454 24428 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260809 23:19:53.587313 24431 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:19:53.565647 30449 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.592657 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.503486 29709 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.595028 24422 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:53.594213 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.596683 24423 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:53.598568 24424 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
W20260809 23:19:53.588883 30680 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.672s	user 0.007s	sys 0.005s
I20260809 23:19:53.601728 24435 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:53.584751 30882 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:53.590991 30773 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:53.609541 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 195467us
I20260809 23:19:53.579579 30486 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260809 23:19:53.574717 30559 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260809 23:19:53.616848 24431 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260809 23:19:53.615612 24424 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
W20260809 23:19:53.620082 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59792.8us
I20260809 23:19:53.603564 24151 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e4ed6a400f2643da95b8f744ebab6f4c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20260809 23:19:53.600538 30109 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:53.607928 30531 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:53.635195 30874 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
W20260809 23:19:53.635722 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45455.5us
I20260809 23:19:53.640769 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.613354 30589 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:53.646687 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63467.6us
I20260809 23:19:53.644868 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:53.625234 30565 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.652990 24545 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:53.665227 25365 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:53.607551 30680 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.618346 30645 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.629717 30573 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.675052 24422 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:19:53.668823 30585 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:53.676605 30847 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:53.677268 30847 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:53.681726 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.682348 24427 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.644626 30109 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.688941 24422 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 23:19:53.694187 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.689440 30531 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.691480 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.691268 30882 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:53.692808 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67230.6us
I20260809 23:19:53.689684 30585 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:19:53.695639 24422 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.674762 30830 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.671366 30878 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.709000 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:53.709735 24539 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:53.709887 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.712263 24428 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:53.629571 30377 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.699497 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.696578 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.714468 24427 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.714540 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:53.685295 30880 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260809 23:19:53.717061 24427 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:53.713650 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:53.719686 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:53.720556 24543 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:53.725183 24534 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:53.719707 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:53.728971 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:53.732532 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:53.734711 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.735826 24428 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:53.739718 24422 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:19:53.740360 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.730775 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.741503 30331 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:53.741600 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:53.741926 30331 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:53.742287 30331 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:53.714042 30644 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:53.717568 30887 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:19:53.784544 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44300.5us
I20260809 23:19:53.672154 30842 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.743574 24544 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:19:53.796628 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67779.2us
I20260809 23:19:53.725009 29709 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.738528 30805 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260809 23:19:53.800236 30805 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:53.739351 30828 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:19:53.728220 30877 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:53.745810 30860 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:53.785192 30684 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.660s	user 0.012s	sys 0.000s
I20260809 23:19:53.800782 30805 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:53.784000 30377 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:53.784240 30845 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:53.785892 30882 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:53.793530 30585 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:19:53.822609 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:53.819927 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 480556us
I20260809 23:19:53.801167 30497 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:19:53.839376 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129656us
I20260809 23:19:53.839792 24428 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:53.808534 30877 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:53.842559 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:53.836426 30860 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:53.845122 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.840030 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:53.816951 30828 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:53.848073 24422 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:53.826496 30684 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:19:53.704720 30847 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.842212 30497 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:53.853248 30497 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:53.853935 30109 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:53.833612 30845 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:53.866852 24540 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:19:53.865903 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.863598 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:53.814486 30833 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.874616 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:53.878077 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.1641e+06us
I20260809 23:19:53.865451 30645 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:53.867575 30806 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.778s	user 0.008s	sys 0.005s
I20260809 23:19:53.887450 30806 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.889855 30806 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.892669 30833 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:53.893628 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:53.850170 30680 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:53.855810 30585 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:53.868187 30531 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:53.873185 30845 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:53.934123 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:53.915920 30887 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:53.937055 30877 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:53.943941 30909 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260809 23:19:53.945233 30806 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:53.947757 30860 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.916606 30821 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:53.997545 24423 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:19:53.952965 30805 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.954517 30828 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:53.881307 30847 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:53.027765 24425 rpcz_store.cc:276] 0809 23:19:51.991976 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:52.559536 (+567560us) service_pool.cc:227] Handling call
0809 23:19:52.592332 (+ 32796us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:52.635201 (+ 42869us) raft_consensus.cc:1520] Early marking committed up to index 4
0809 23:19:52.651248 (+ 16047us) log.cc:842] Serialized 14 byte log entry
0809 23:19:53.024456 (+373208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:53.025031 (+   575us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:53.025217 (+   186us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.025345 (+   128us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:53.025560 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23110,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":4159,"thread_start_us":367982,"threads_started":1}
I20260809 23:19:53.961485 30585 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:53.977140 30833 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:53.979136 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 128511us
I20260809 23:19:53.987946 24544 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:19:53.988588 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 168351us
W20260809 23:19:53.989193 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 137038us
I20260809 23:19:54.007066 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:53.991602 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 157617us
I20260809 23:19:53.952013 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34970da619114abc942b839d0862a300"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:54.008950 25365 raft_consensus.cc:595] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:54.009027 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:53.999845 24542 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:19:54.010216 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:54.005523 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71540.9us
I20260809 23:19:54.008193 30531 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:54.009328 25365 raft_consensus.cc:603] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:54.010649 30819 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:54.011821 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.012421 30109 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:54.001221 30684 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:53.999639 30821 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.027758 30684 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:54.028187 30764 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:19:54.028592 30764 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.029294 30821 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.029429 30764 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.017186 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.030908 30764 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:54.031031 24535 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:54.033078 30878 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:19:54.028182 30684 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.004833 30833 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:54.003906 30652 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.865s	user 0.020s	sys 0.000s
I20260809 23:19:54.023025 30819 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.035554 30819 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.013137 30877 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:53.999714 30680 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.023583 24321 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:54.040572 30592 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:54.041087 30592 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.024535 30645 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:54.043498 30645 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:54.043887 30645 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.047429 30577 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:54.042098 30592 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.047951 30577 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.048972 30577 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:54.027462 30844 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.392s	user 0.002s	sys 0.002s
I20260809 23:19:54.051616 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:54.053028 30826 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.752s	user 0.001s	sys 0.002s
I20260809 23:19:54.054656 24321 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260809 23:19:54.056216 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.057197 24312 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:54.066663 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 325112us
I20260809 23:19:54.071893 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:54.083582 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.022433 30531 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:54.086972 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76807.1us
I20260809 23:19:54.091799 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.089722 30919 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:54.094411 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.093479 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.098798 24535 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:54.083925 30577 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.089308 30592 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:54.094883 30917 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.098539 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.103508 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.106038 24321 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:54.106519 30497 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.108125 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.111317 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.112908 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.125898 25368 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:54.113680 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.114423 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.114599 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:54.115056 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44897.2us
I20260809 23:19:54.091389 30531 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.115206 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.129772 25363 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:54.116534 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.098361 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.122498 24316 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:54.134230 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.089732 30773 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260809 23:19:54.134825 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.135533 25365 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:54.128016 25372 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:54.135681 25375 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:54.138072 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.134097 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.140022 25370 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.140869 24312 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:54.122864 30922 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.123215 30109 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:54.156080 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.137993 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.159811 25373 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:54.164613 24314 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:54.165612 24323 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:19:54.165731 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76721.9us
W20260809 23:19:54.169219 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58102.5us
I20260809 23:19:54.137727 30878 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:19:54.176348 30845 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.178387 30845 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.179041 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.179728 24320 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:54.139812 30833 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.188076 30821 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:19:54.188414 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94375.8us
I20260809 23:19:54.184737 30373 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260809 23:19:54.189515 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.190865 30589 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:54.193917 30589 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.194367 30589 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.122504 30917 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.146026 30331 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.190761 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.163841 30520 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.187557 25365 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:19:54.204695 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1033) took 2311 ms (client timeout 2614 ms). Trace:
W20260809 23:19:54.205199 25371 rpcz_store.cc:269] 0809 23:19:51.893088 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:53.151350 (+1258262us) service_pool.cc:227] Handling call
0809 23:19:54.204452 (+1053102us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:54.204625 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714696448}
I20260809 23:19:54.208829 30919 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:54.222247 25370 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:19:54.166509 30680 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260809 23:19:54.164474 30109 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.245411 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.209244 30833 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:54.202353 30845 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20260809 23:19:54.220562 30652 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.249470 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.254305 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.254674 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.257248 24544 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.233386 30917 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.165117 30922 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.219455 30824 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.041s
W20260809 23:19:54.255570 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2565648478921503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
I20260809 23:19:54.251888 30520 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:19:54.279731 24378 connection.cc:232] RPC call timeout handler was delayed by -1.1620648480020463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
I20260809 23:19:54.276801 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.250602 30833 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:54.280233 24542 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:54.280570 25321 leader_election.cc:398] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:54.285820 25321 leader_election.cc:401] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260809 23:19:54.287091 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:54.295400 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64019.6us
W20260809 23:19:54.297528 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 127607us
I20260809 23:19:54.277916 30331 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.289371 30833 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.326779 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:54.328289 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 178817us
I20260809 23:19:54.262534 30652 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.295011 30842 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:54.337598 24544 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:19:54.339015 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:54.339980 30927 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:54.340565 30927 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:54.341557 30927 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.343125 30842 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:19:54.343948 30925 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260809 23:19:54.302367 30497 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:54.350917 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49298.8us
I20260809 23:19:54.327885 30520 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:54.303292 30930 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:19:54.363276 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 100874us
I20260809 23:19:54.344143 30842 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.345755 30331 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:19:54.377912 30373 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:54.378350 30373 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.379160 30573 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:54.360394 30819 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.382297 30819 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.387563 30373 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.389566 30373 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:54.365983 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59470.4us
I20260809 23:19:54.346139 30645 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.393087 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.393930 30331 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:54.398244 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.399338 30520 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:54.400830 30520 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:54.400983 30592 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:54.333621 30919 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.401953 30592 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:19:54.369803 30109 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:54.370464 30928 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260809 23:19:54.405246 30109 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:54.405617 30109 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.368853 30922 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.401240 30520 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:54.395213 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.394536 30645 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.415236 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:54.417809 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.418990 24544 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:54.422562 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59107.4us
I20260809 23:19:54.423295 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.423703 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.424875 24542 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:54.424059 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.366843 30930 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.425948 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:54.406589 30109 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.416579 30520 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:54.421784 30922 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:54.433296 30930 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.440426 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.446923 24313 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:54.452037 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.454218 30629 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:54.456132 30712 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.436228 30331 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:54.424335 30932 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.021s
I20260809 23:19:54.443068 30890 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:54.466617 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53381.1us
I20260809 23:19:54.441764 30877 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:54.447999 30922 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:54.470695 30150 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:54.446630 30824 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:19:54.477257 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:54.478410 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 77861.1us
I20260809 23:19:54.465342 30331 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.436789 30589 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.473178 30922 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:19:54.474236 30897 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.776s	user 0.000s	sys 0.014s
I20260809 23:19:54.471361 30109 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:54.484050 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59712.9us
I20260809 23:19:54.493248 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.492105 24534 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:54.484627 30331 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.477680 30652 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.503592 30919 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:54.498667 24542 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:54.505267 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.499487 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.497164 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.479997 30150 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.502556 24546 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:54.506867 24544 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:54.506809 24435 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:54.479832 30877 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:54.466323 30684 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.509215 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.517100 24544 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.505404 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.522442 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.522809 24544 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:54.513891 24534 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:54.523643 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.523964 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.524320 24544 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:54.524662 24534 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:54.515760 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.525377 24543 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:54.516500 24539 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11.
I20260809 23:19:54.480176 30824 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.471750 30531 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.532507 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.534808 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.539839 24535 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:54.533999 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.514201 30847 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:54.542575 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.540660 24545 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:19:54.515405 30842 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:54.547753 30842 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:54.550551 30842 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:54.551106 30589 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:54.556037 24539 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260809 23:19:54.556726 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:19:54.557305 24316 raft_consensus.cc:595] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.545990 30847 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.558857 30847 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:19:54.559451 30939 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
W20260809 23:19:54.561388 30888 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.041s	user 0.004s	sys 0.000s
I20260809 23:19:54.562700 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2292) took 1423 ms. Trace:
I20260809 23:19:54.562551 30877 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.563144 24311 rpcz_store.cc:276] 0809 23:19:53.138738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:53.141705 (+  2967us) service_pool.cc:227] Handling call
0809 23:19:53.223878 (+ 82173us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:53.304595 (+ 80717us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:53.304613 (+    18us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:53.343758 (+ 39145us) log.cc:842] Serialized 26 byte log entry
0809 23:19:53.499753 (+155995us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:19:53.499794 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:53.499861 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:54.560805 (+1060944us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:54.562287 (+  1482us) raft_consensus.cc:1671] finished
0809 23:19:54.562312 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:54.562532 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30870912,"thread_start_us":80374,"threads_started":1,"wal-append.queue_time_us":136781}
I20260809 23:19:54.565137 30842 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.566690 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.569480 30842 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:54.572822 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79633.8us
I20260809 23:19:54.576870 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.577059 30680 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:54.578049 30577 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:54.578414 30577 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.579134 30629 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:54.581306 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:54.504449 30919 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.547837 30712 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:19:54.604415 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 122961us
I20260809 23:19:54.590667 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.583690 24430 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:54.597189 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.598551 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:54.594811 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62360.9us
I20260809 23:19:54.611207 24424 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.617463 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.603790 30589 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260809 23:19:54.619266 30944 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.592953 30842 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:19:54.622092 30531 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.575316 30652 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20260809 23:19:54.597576 30520 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.624387 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.624893 30520 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.625244 24430 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260809 23:19:54.627151 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.597021 30824 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.627713 24424 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:19:54.542462 30833 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.633291 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.633689 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.635388 24430 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.635737 25370 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.638653 24542 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:54.642083 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.643404 24424 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.589671 30932 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:54.646499 25375 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.646281 25370 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:54.657387 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.541s (SENT)
I20260809 23:19:54.658020 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.602288 30919 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.660171 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.645838 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.663080 24424 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.612078 30878 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:54.661212 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.613137 30644 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:54.670624 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.671465 24542 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:54.671640 24546 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:54.672554 24431 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [10.9-12.10]
I20260809 23:19:54.673324 24428 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:54.674309 24427 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:54.699473 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.674026 30331 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.671475 24422 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:54.582373 30922 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.675763 30860 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:54.676698 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.677641 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.678705 30684 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.679157 30833 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.679628 24317 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:54.719538 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.720491 25371 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11.
I20260809 23:19:54.680543 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.681840 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.721647 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 923) took 1214 ms. Trace:
I20260809 23:19:54.682709 30486 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.683084 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:54.723248 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.680162 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.724251 24309 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:54.631434 30373 raft_consensus.cc:2741] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:19:54.688333 30573 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:19:54.689240 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.591223 30680 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260809 23:19:54.690106 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.691879 25363 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:54.692654 25369 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260809 23:19:54.695803 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.696134 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.579165 30577 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.734715 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.709952 30150 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.684155 24321 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 23:19:54.700121 25370 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:54.702397 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111827us
I20260809 23:19:54.704874 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.583616 30449 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.081s
I20260809 23:19:54.664531 30824 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:54.708988 30860 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.717510 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.718047 30497 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.762089 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.762185 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.683255 30520 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.763861 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.727524 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:54.765192 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83410.2us
I20260809 23:19:54.730475 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.722025 24540 rpcz_store.cc:276] 0809 23:19:53.507618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:53.507791 (+   173us) service_pool.cc:227] Handling call
0809 23:19:53.522153 (+ 14362us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:53.867289 (+345136us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:54.721384 (+854095us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:54.721586 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":787599232}
I20260809 23:19:54.732872 24313 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:54.668607 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:54.707043 30878 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:54.741050 24270 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11.
I20260809 23:19:54.709589 30806 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.738844 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.739113 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.768276 24323 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.768399 24311 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:54.743009 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47304.9us
I20260809 23:19:54.709154 30932 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.744997 24545 raft_consensus.cc:2354] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:54.769587 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.716702 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.770113 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.767931 24270 leader_election.cc:401] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:19:54.758762 25375 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:54.680137 24312 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 23:19:54.735291 30833 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:54.774245 24431 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.783300 25368 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:54.715188 30712 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260809 23:19:54.744515 30890 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.825104 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.784709 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.785162 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.739303 30920 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:54.822558 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.828699 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.824105 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.738991 30934 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:54.830670 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.831521 24317 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:54.832105 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:54.836066 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73956.2us
W20260809 23:19:54.836915 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49785.1us
I20260809 23:19:54.723820 30644 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.759423 30652 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.864305 24422 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:54.739404 30805 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:54.859323 30922 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.897006 30922 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.763064 30917 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.897748 24311 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:54.898020 30917 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.840607 25375 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:54.898746 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.735046 30531 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260809 23:19:54.860318 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.899958 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.732148 30486 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.904907 30917 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.905773 24430 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:54.906080 25372 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:19:54.820791 30819 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.784252 30589 raft_consensus.cc:2741] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260809 23:19:54.739007 30684 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260809 23:19:54.912659 30684 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:19:54.913021 30684 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:54.913798 30684 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.863700 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.864841 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260809 23:19:54.915247 30773 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:19:54.865487 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.863292 30953 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260809 23:19:54.917325 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.865962 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:54.917539 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.917944 25372 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:54.785271 30806 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.744807 30845 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.096s
I20260809 23:19:54.919234 30684 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:54.920197 30806 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.842293 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:54.770893 30497 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.921188 24424 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:54.843446 30930 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.922276 30497 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.759827 30150 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.923183 30930 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.872490 30950 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260809 23:19:54.764909 30645 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
I20260809 23:19:54.926041 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.925935 30940 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:19:54.927584 30948 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:19:54.929666 30948 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:54.931494 30773 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:54.932214 30773 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:54.934075 30854 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:54.934533 30854 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.935606 30854 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.936939 30854 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.875236 30860 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.878379 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.879410 24309 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:19:54.879141 30932 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.880501 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.882177 30644 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.882319 30847 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.882828 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.883345 30652 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:54.884294 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 143440us
I20260809 23:19:54.823514 30833 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:54.952834 30833 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:54.945315 30821 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260809 23:19:54.954403 30821 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:54.838166 30920 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:54.885689 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53662.6us
I20260809 23:19:54.886407 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.956538 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.887552 24431 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260809 23:19:54.831817 30887 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:54.932593 30773 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.892421 24422 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:19:54.900923 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.861620 30331 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.903266 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:54.910107 30819 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:54.910730 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.968044 24323 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:19:54.955211 30920 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:54.907826 25370 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:19:54.917428 25371 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:19:54.873126 30841 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260809 23:19:54.923352 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:54.973693 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.924614 30150 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:54.907964 24311 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:54.941208 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 275634us
I20260809 23:19:54.944552 24321 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:54.991127 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.985736 24317 raft_consensus.cc:2362] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10. Re-sending same reply.
I20260809 23:19:54.992645 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:54.947147 30680 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11.
I20260809 23:19:54.950668 30577 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:54.998303 24312 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260809 23:19:54.998775 24312 pending_rounds.cc:76] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 8
I20260809 23:19:54.951757 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:54.999022 24312 pending_rounds.cc:97] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 10.9
W20260809 23:19:54.952026 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71606.5us
I20260809 23:19:54.999720 24542 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:19:54.952189 30652 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:54.954741 30821 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:54.889065 25325 leader_election.cc:398] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:55.023077 25325 leader_election.cc:401] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260809 23:19:54.910943 30878 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:54.970384 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.974097 25369 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260809 23:19:54.975459 24320 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260809 23:19:55.024159 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53864us
I20260809 23:19:55.011595 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2285) took 1010 ms. Trace:
I20260809 23:19:54.980302 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.027680 24543 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 23:19:54.975544 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:54.979200 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:54.946122 30944 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:19:55.030297 30944 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:55.030526 24546 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:55.031517 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:55.031248 30944 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.032351 24546 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:55.032825 24534 pending_rounds.cc:76] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 6
I20260809 23:19:55.032990 30944 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:55.033296 24534 pending_rounds.cc:97] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260809 23:19:54.989529 24540 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:55.033686 24534 raft_consensus.cc:2866] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:55.033828 24546 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:55.034788 24540 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:19:54.989638 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:54.959506 30887 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:54.962363 30845 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:19:55.043228 25371 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
W20260809 23:19:55.021003 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 248329us
I20260809 23:19:54.973526 30573 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:54.999317 24312 raft_consensus.cc:2866] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:55.036748 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:55.040005 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 144840us
I20260809 23:19:55.041703 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:54.991290 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:55.046310 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 155576us
W20260809 23:19:55.047202 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71743.7us
I20260809 23:19:54.993978 30377 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:54.965216 30953 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260809 23:19:55.049667 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.072113 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:54.974891 30449 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
I20260809 23:19:55.072518 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:55.079111 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:55.026216 24425 rpcz_store.cc:276] 0809 23:19:54.001377 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.567187 (+565810us) service_pool.cc:227] Handling call
0809 23:19:54.567608 (+   421us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:54.567876 (+   268us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:54.639590 (+ 71714us) log.cc:842] Serialized 14 byte log entry
0809 23:19:54.890383 (+250793us) spinlock_profiling.cc:243] Waited 45.3 ms on lock 0x7b1800370c60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:19:54.995251 (+104868us) log.cc:842] Serialized 14 byte log entry
0809 23:19:55.011179 (+ 15928us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:55.011248 (+    69us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:55.011309 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.011397 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:55.011530 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7619,"prepare.queue_time_us":170199,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":14780,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":209460224,"thread_start_us":311754,"threads_started":3,"wal-append.queue_time_us":253842}
I20260809 23:19:55.048488 30845 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:54.995630 30331 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260809 23:19:55.069226 30927 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:55.089951 24535 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:55.044057 30841 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.043s
I20260809 23:19:55.052459 30486 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.076079 30890 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:55.050449 30531 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:55.081159 30377 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:55.081733 30845 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:55.084340 30805 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:55.162564 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:55.151062 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66538.4us
I20260809 23:19:55.155249 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.157296 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.097352 30684 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:55.172179 24427 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:55.160024 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.161811 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:19:55.164952 24321 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:55.050742 30652 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.095s
I20260809 23:19:55.167301 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:55.168213 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.169430 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:55.069530 30573 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.173192 24312 raft_consensus.cc:595] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:55.174829 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285884us
I20260809 23:19:55.166273 24535 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260809 23:19:55.082965 30629 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260809 23:19:55.097326 30764 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260809 23:19:55.178238 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:55.179822 24424 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:55.180083 24312 raft_consensus.cc:603] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:55.185184 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:55.185274 24317 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:55.184388 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 184124us
I20260809 23:19:55.179596 24313 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:55.072803 30773 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.198904 24316 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:55.199223 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:55.198549 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:55.162416 30805 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:55.107930 30806 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260809 23:19:55.197067 24424 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260809 23:19:55.203938 24541 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:55.206436 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:55.204409 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 134020us
I20260809 23:19:55.210057 24269 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11.
I20260809 23:19:55.210188 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:55.210790 24427 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:55.210970 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:55.212215 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:55.212805 25370 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 6.
I20260809 23:19:55.213709 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.215047 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:55.214720 25366 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260809 23:19:55.215921 25371 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260809 23:19:55.216043 25366 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 9
I20260809 23:19:55.216425 25366 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260809 23:19:55.216712 25366 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:55.207909 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:55.205680 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45767.3us
I20260809 23:19:55.172145 30684 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:55.241204 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.241513 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:55.242214 24424 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:55.249830 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.249519 30712 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:55.255764 30712 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.259874 30712 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.260949 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.203351 30644 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:55.267323 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 157939us
I20260809 23:19:55.165552 30890 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260809 23:19:55.198523 30900 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:55.127990 30821 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.207909 30805 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:55.302677 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:55.284080 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.184149 30377 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260809 23:19:55.188731 30150 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260809 23:19:55.283181 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:55.253549 30684 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:55.271952 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.321872 30684 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.254101 30773 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:55.177961 30920 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:55.310865 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94668.8us
I20260809 23:19:55.332947 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.286196 25375 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:55.218266 30331 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.086s
I20260809 23:19:55.239094 30449 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260809 23:19:55.335304 25375 raft_consensus.cc:595] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:55.303535 30764 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
W20260809 23:19:55.340830 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68990.8us
I20260809 23:19:55.340934 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:55.344866 25375 raft_consensus.cc:603] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:55.156208 30887 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.352214 24309 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:55.347330 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.357340 24543 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:55.357438 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:55.361332 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.362057 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:55.360716 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.364266 24535 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:55.346542 30821 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:55.338135 30819 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.365713 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:55.340734 30974 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:55.328017 30652 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260809 23:19:55.346771 30764 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:55.354110 30917 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:55.445760 30984 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:55.446668 30847 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:55.635710 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 386882us
I20260809 23:19:55.353778 30486 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.107s
I20260809 23:19:55.491235 30712 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:55.609887 30878 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.364266 30922 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:55.446574 30644 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260809 23:19:55.623986 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 334124us
W20260809 23:19:55.619161 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 253506us
I20260809 23:19:55.663307 24316 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:55.671146 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:55.619659 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.003s	sys 0.007s
W20260809 23:19:55.637400 25323 leader_election.cc:334] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.651s (SENT)
I20260809 23:19:55.446707 30824 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.170s
W20260809 23:19:55.649238 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 292166us
I20260809 23:19:55.672139 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:55.640666 30520 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.003s
I20260809 23:19:55.364238 30972 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.468s
W20260809 23:19:55.671983 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.003s	sys 0.008s
W20260809 23:19:55.654136 25373 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.005s	sys 0.014s
I20260809 23:19:55.623243 30974 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:55.636063 30449 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.623602 30980 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.606678 30997 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.224s
I20260809 23:19:55.771206 24311 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260809 23:19:55.786064 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2303) took 1145 ms. Trace:
I20260809 23:19:55.718317 24542 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 7. Current term is 8. Ops: []
I20260809 23:19:55.682811 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:55.679535 24309 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:19:55.837487 25373 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.011s	sys 0.015s
W20260809 23:19:55.839793 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 157077us
W20260809 23:19:55.838194 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 845546us
I20260809 23:19:55.842382 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:19:55.845768 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.007s	sys 0.024s
W20260809 23:19:55.850358 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.007s	sys 0.024s
W20260809 23:19:55.850728 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 517689us
I20260809 23:19:55.765727 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2294) took 1090 ms. Trace:
I20260809 23:19:55.851544 24428 rpcz_store.cc:276] 0809 23:19:54.675046 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.706702 (+ 31656us) service_pool.cc:227] Handling call
0809 23:19:54.727427 (+ 20725us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:54.891236 (+163809us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:54.891278 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:54.891329 (+    51us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:54.891377 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.765480 (+874103us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:55.765532 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:55.765662 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376175360}
I20260809 23:19:55.854166 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.855665 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:55.855814 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.644330 30573 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:55.858464 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:55.748951 30978 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
W20260809 23:19:55.777411 31010 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
W20260809 23:19:55.781442 30833 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.828s	user 0.013s	sys 0.006s
W20260809 23:19:55.728183 30993 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.165s	user 0.000s	sys 0.009s
W20260809 23:19:55.773269 30845 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.001s
W20260809 23:19:55.773847 30854 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.836s	user 0.024s	sys 0.031s
W20260809 23:19:55.748472 30971 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.352s	user 0.000s	sys 0.001s
W20260809 23:19:55.798475 30960 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.923s	user 0.002s	sys 0.000s
W20260809 23:19:55.728624 30964 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.835s	user 0.010s	sys 0.002s
W20260809 23:19:55.762030 30970 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.867s	user 0.002s	sys 0.000s
W20260809 23:19:55.772353 30577 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.735s	user 0.014s	sys 0.003s
W20260809 23:19:55.781785 31004 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.165s	user 0.000s	sys 0.002s
I20260809 23:19:55.648442 30712 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:55.876681 30845 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.889385 30712 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:55.891413 30845 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.896754 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1057) took 1202 ms. Trace:
I20260809 23:19:55.838970 24320 rpcz_store.cc:276] 0809 23:19:54.640808 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.641258 (+   450us) service_pool.cc:227] Handling call
0809 23:19:54.677114 (+ 35856us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:55.043490 (+366376us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.785769 (+742279us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:55.785987 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685200128}
I20260809 23:19:55.671911 30629 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:55.865180 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.899315 30680 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.762620 30950 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:55.783732 30373 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:55.900313 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.900323 30373 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:55.901814 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.676105 30847 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.902601 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2288) took 1334 ms. Trace:
I20260809 23:19:55.902654 30847 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.748143 30991 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.395s
I20260809 23:19:55.891453 24542 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:55.796504 30949 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:55.787328 30922 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.904758 30949 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:55.681807 30520 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.905262 30847 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.690996 30917 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.724746 30920 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.770586 31020 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.694553 30684 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:55.905689 30949 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.906220 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:55.690765 30585 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:19:55.853431 30930 raft_consensus.cc:2741] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260809 23:19:55.748930 30988 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:19:55.865535 30944 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.832s	user 0.011s	sys 0.013s
I20260809 23:19:55.871999 30833 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.881712 30942 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:55.673107 30764 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.879236 30645 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:55.769131 30805 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.861810 30150 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:55.885568 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44570.2us
I20260809 23:19:55.868289 30821 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.673625 30878 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.887809 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:55.878348 30573 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.929145 30878 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:55.930366 25365 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:55.897158 25366 rpcz_store.cc:276] 0809 23:19:54.694081 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.915955 (+221874us) service_pool.cc:227] Handling call
0809 23:19:54.916263 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:55.279417 (+363154us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.896512 (+617095us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:55.896695 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":685512576}
I20260809 23:19:55.901546 30373 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.931442 24435 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260809 23:19:55.931831 30373 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:55.933671 30150 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:55.934217 30150 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.903788 30934 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:55.787213 30994 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:55.935743 30994 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:55.936602 30994 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.760417 31018 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:55.937830 30150 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:55.938055 31018 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.789958 30984 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12.
I20260809 23:19:55.908890 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.953912 24543 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:19:55.770706 30592 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:55.955193 30712 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.955571 30592 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.957839 30712 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:55.958195 30592 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:55.890636 30860 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:55.858536 30819 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260809 23:19:55.876327 30109 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.913328 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1049) took 1223 ms. Trace:
I20260809 23:19:55.856839 30449 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:55.861594 30806 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260809 23:19:55.860653 30980 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:55.906826 24534 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:55.938678 30150 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.925734 30645 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:55.950157 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.949442 30497 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:55.926046 30974 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:55.952858 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51160.9us
I20260809 23:19:55.988297 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:55.931857 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:55.911396 30585 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:55.968595 24539 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:55.960678 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:19:55.993785 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 356865us
I20260809 23:19:55.903143 24429 rpcz_store.cc:276] 0809 23:19:54.568144 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.569942 (+  1798us) service_pool.cc:227] Handling call
0809 23:19:54.570302 (+   360us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:54.570663 (+   361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:54.570702 (+    39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:54.571322 (+   620us) log.cc:842] Serialized 52 byte log entry
0809 23:19:54.706669 (+135347us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:54.972052 (+265383us) log.cc:842] Serialized 14 byte log entry
0809 23:19:54.972837 (+   785us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:54.972933 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:55.797745 (+824812us) spinlock_profiling.cc:243] Waited 54.7 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:55.902292 (+104547us) raft_consensus.cc:1671] finished
0809 23:19:55.902400 (+   108us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:55.902540 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":173580,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":4090,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":8569,"spinlock_wait_cycles":125739904,"thread_start_us":373332,"threads_started":2,"wal-append.queue_time_us":187697}
W20260809 23:19:55.995296 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.784s (SENT)
I20260809 23:19:55.995745 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:55.979321 30806 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:55.920549 30845 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:19:55.949730 30773 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.003s
I20260809 23:19:55.914131 30988 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.014233 30988 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:55.972739 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:55.906461 31020 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.017882 31020 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:19:55.982009 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.019714 30645 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.972345 25373 rpcz_store.cc:276] 0809 23:19:54.687251 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.690794 (+  3543us) service_pool.cc:227] Handling call
0809 23:19:54.695683 (+  4889us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:54.696225 (+   542us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:54.696258 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:54.705315 (+  9057us) log.cc:842] Serialized 52 byte log entry
0809 23:19:54.967196 (+261881us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:55.840384 (+873188us) log.cc:842] Serialized 14 byte log entry
0809 23:19:55.845165 (+  4781us) log.cc:842] Serialized 14 byte log entry
0809 23:19:55.845361 (+   196us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.845456 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:55.910550 (+ 65094us) raft_consensus.cc:1671] finished
0809 23:19:55.910577 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:55.910758 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19833,"prepare.queue_time_us":805735,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":24306,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":14599,"spinlock_wait_cycles":16811648,"thread_start_us":1055837,"threads_started":2,"wal-append.queue_time_us":322259}
I20260809 23:19:55.980609 30764 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:55.961450 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.022406 30585 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:55.979688 30573 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.024123 30585 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:19:56.024049 30573 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.027650 30573 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.028959 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.030460 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:55.990818 30497 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.033478 30497 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.034312 30806 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:56.038573 31020 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:55.974498 25369 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:19:56.039172 30934 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:19:56.039713 30934 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.040108 30934 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:55.980273 30109 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.047159 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.049218 30109 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.001223 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:56.052022 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50900.6us
I20260809 23:19:56.052902 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.008919 24320 raft_consensus.cc:2354] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:56.011217 30592 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.989851 30917 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.012328 30819 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:55.985123 30150 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.021430 30645 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:56.054623 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58832.9us
W20260809 23:19:56.055171 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52428.5us
I20260809 23:19:56.011482 30712 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:56.056542 30993 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260809 23:19:56.057921 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.075172 31015 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:55.949571 30805 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.088248 31015 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.089390 31015 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.090385 30150 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.075995 30331 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:56.085619 30847 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:56.091315 30331 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.091554 30847 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:56.092303 31002 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.476s	user 0.002s	sys 0.000s
I20260809 23:19:56.086027 30764 raft_consensus.cc:979] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower ab5aefaeb3d14775b30a74a2a6d3357f to start an election
I20260809 23:19:56.093657 30331 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.076193 30712 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.095074 30917 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:56.096030 30712 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.085687 30592 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.096881 30592 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:56.058182 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85543.3us
I20260809 23:19:56.099648 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.057647 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:56.060837 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72652.1us
I20260809 23:19:56.010504 24534 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:56.063553 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:56.102896 24423 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:56.057615 30920 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.083561 24431 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:19:56.106137 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:56.083737 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.059443 30922 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:56.086787 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:55.987371 30980 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:56.102276 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.100448 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2301) took 1026 ms. Trace:
I20260809 23:19:56.106858 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.050448 30930 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260809 23:19:56.112344 24430 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:56.113950 24422 rpcz_store.cc:276] 0809 23:19:55.074494 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.074839 (+   345us) service_pool.cc:227] Handling call
0809 23:19:55.075220 (+   381us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:55.084769 (+  9549us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:55.084814 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:55.137698 (+ 52884us) log.cc:842] Serialized 26 byte log entry
0809 23:19:55.302888 (+165190us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:55.303483 (+   595us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:55.303567 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:55.795164 (+491597us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b48001701d8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:19:56.091718 (+296554us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:56.100229 (+  8511us) raft_consensus.cc:1671] finished
0809 23:19:56.100251 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:56.100393 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9079,"raft.queue_time_us":13082,"raft.run_cpu_time_us":3192,"raft.run_wall_time_us":696997,"spinlock_wait_cycles":126846976,"thread_start_us":124367,"threads_started":1,"wal-append.queue_time_us":218639}
I20260809 23:19:56.118688 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.118805 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:56.120325 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.127269 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:56.108400 30882 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.232s	user 0.017s	sys 0.012s
I20260809 23:19:56.139458 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.104702 30592 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.136382 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.124670 30842 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:19:56.143438 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56772.4us
I20260809 23:19:56.116550 31026 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:56.151362 25373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:19:56.117080 30764 raft_consensus.cc:1067] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Signalling peer ab5aefaeb3d14775b30a74a2a6d3357f to start an election
W20260809 23:19:56.153100 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50664.6us
I20260809 23:19:56.132594 31022 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:56.131374 30992 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:56.157541 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.160418 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:56.145015 30953 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.057s	user 0.004s	sys 0.012s
I20260809 23:19:56.165153 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.130961 30629 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:56.218233 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:56.166440 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:56.219650 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53295.1us
W20260809 23:19:56.151407 30882 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.270s	user 0.017s	sys 0.012s
W20260809 23:19:56.221568 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.929s (SENT)
W20260809 23:19:56.224035 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66613.8us
I20260809 23:19:56.224426 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:56.224740 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.092s (SENT)
I20260809 23:19:56.166790 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:56.142508 30773 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.142328 31020 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.173120 25366 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:19:56.175197 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1068) took 1433 ms. Trace:
I20260809 23:19:56.175782 30983 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:56.145686 31032 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.157092 30712 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:19:55.950134 30577 consensus_queue.cc:1159] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 9 status { last_received { term: 8 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260809 23:19:56.216877 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51693us
I20260809 23:19:56.217725 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:56.157318 30833 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.225492 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.224713 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.227438 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.242879 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.217502 30680 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:56.243229 24431 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:19:56.244668 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:56.214890 30953 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.004s	sys 0.013s
I20260809 23:19:56.232161 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:56.168516 31022 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.234241 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.245970 25368 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 6.
I20260809 23:19:56.246001 24423 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:56.246897 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.247726 25368 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:56.170289 31026 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.248739 30821 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:19:56.249351 31026 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.236652 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.242733 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.251721 25371 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260809 23:19:56.252059 25371 pending_rounds.cc:76] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 8
I20260809 23:19:56.252276 25371 pending_rounds.cc:97] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:19:56.252763 25371 raft_consensus.cc:2866] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:56.226316 30449 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.176795 31034 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
I20260809 23:19:56.227918 25372 rpcz_store.cc:276] 0809 23:19:54.741779 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.767876 (+1026097us) service_pool.cc:227] Handling call
0809 23:19:55.768249 (+   373us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:55.768644 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:19:55.768967 (+   323us) log.cc:842] Serialized 14 byte log entry
0809 23:19:56.128029 (+359062us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:56.129871 (+  1842us) log.cc:842] Serialized 26 byte log entry
0809 23:19:56.130351 (+   480us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:56.130772 (+   421us) log.cc:842] Serialized 14 byte log entry
0809 23:19:56.140229 (+  9457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:56.140303 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:56.174996 (+ 34693us) raft_consensus.cc:1671] finished
0809 23:19:56.175016 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:56.175141 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35254,"prepare.queue_time_us":378,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":6463,"raft.queue_time_us":37095,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":33152,"spinlock_wait_cycles":27238784,"thread_start_us":342013,"threads_started":1,"wal-append.queue_time_us":402696}
I20260809 23:19:56.252075 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.231885 30930 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:56.275250 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.264432 24543 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 23:19:56.230558 30847 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.252255 30497 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:19:56.267750 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:56.226958 30109 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:56.284207 24423 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:56.270680 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.270776 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.272363 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.231019 30629 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:56.233531 30773 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:56.288246 31037 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260809 23:19:56.288900 30629 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.289901 30773 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.224794 30652 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260809 23:19:56.231709 30983 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:56.233513 30920 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:56.231164 30934 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.300735 25373 raft_consensus.cc:595] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.301234 25373 raft_consensus.cc:603] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:19:56.267521 30940 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.293582 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.302284 24543 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:56.294697 24378 leader_election.cc:398] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:56.297537 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 952) took 1096 ms. Trace:
I20260809 23:19:56.302776 24378 leader_election.cc:401] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260809 23:19:56.268354 31022 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:56.304658 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 144330us
I20260809 23:19:56.300017 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.279874 30953 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.307335 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:56.239681 30997 consensus_queue.cc:1159] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 9 status { last_received { term: 8 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:56.278795 30922 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:56.309016 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.310305 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.319787 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.320803 24309 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:56.312398 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.313817 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:56.325946 24320 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:56.315248 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.236253 31020 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.333120 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.298056 30917 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:56.296298 31034 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.298175 30497 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:56.302913 24545 rpcz_store.cc:276] 0809 23:19:55.201125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.202139 (+  1014us) service_pool.cc:227] Handling call
0809 23:19:55.215379 (+ 13240us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:55.324077 (+108698us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:55.324114 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:55.788390 (+464276us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:19:55.788900 (+   510us) log.cc:842] Serialized 26 byte log entry
0809 23:19:56.064656 (+275756us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:56.064706 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:56.064820 (+   114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:56.290551 (+225731us) raft_consensus.cc:1671] finished
0809 23:19:56.290577 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:56.297467 (+  6890us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":147476,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":80992,"spinlock_wait_cycles":9471872,"thread_start_us":383564,"threads_started":2,"wal-append.queue_time_us":357626}
I20260809 23:19:56.304517 30824 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.304797 30878 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:56.332100 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:56.345237 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.309110 30109 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:56.347304 24311 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:56.319288 30150 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.310029 30331 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:56.296936 31018 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.351507 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.352414 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.356287 25371 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:56.298028 30847 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260809 23:19:56.363476 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57168.1us
I20260809 23:19:56.336128 30991 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.239130 31032 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.357618 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.342160 30773 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:56.230770 30980 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.342572 30950 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:56.317716 30940 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.308316 30920 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.322608 30953 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.386664 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.387948 30819 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.388609 31033 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:56.389648 31009 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.390084 30373 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.390184 31038 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.353753 30878 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.323253 30927 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260809 23:19:56.402395 25371 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:56.408802 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.379088 30331 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:56.379316 30652 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:56.411420 31033 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:56.459208 25370 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.379753 30589 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.460768 24309 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:56.510022 31054 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:56.510588 31054 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.511693 31061 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 23:19:56.512789 31054 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.513381 31043 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.138s
I20260809 23:19:56.514207 30773 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:56.465929 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.486377 30373 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.515460 31049 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.516059 30920 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.515990 31049 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.516855 30373 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.466848 31020 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.463557 30833 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260809 23:19:56.516865 30652 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.462188 30842 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.490610 30878 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:56.518680 30652 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:56.519627 31046 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260809 23:19:56.521315 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.521932 24425 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:56.503190 30331 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.378304 30991 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:56.471797 31048 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260809 23:19:56.501896 30680 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.539439 30680 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.542026 30680 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.476821 30940 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.476567 30988 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.551959 25366 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:56.477510 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.553093 24321 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:56.553357 24320 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:56.408901 30819 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.480345 31032 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.409739 30806 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.483490 24312 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:56.484014 24311 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:19:56.486004 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2773) took 1319 ms. Trace:
I20260809 23:19:56.465946 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:56.490615 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 117111us
I20260809 23:19:56.559433 24435 rpcz_store.cc:276] 0809 23:19:55.166580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.166780 (+   200us) service_pool.cc:227] Handling call
0809 23:19:55.187824 (+ 21044us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:55.931963 (+744139us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:56.485649 (+553686us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:56.485924 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1709475584}
I20260809 23:19:56.559994 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.459122 24316 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260809 23:19:56.460031 30890 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.510957 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.562969 24309 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:19:56.563354 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.513674 30934 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.360177 30150 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.518368 31049 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.467459 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.565742 24317 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:56.566249 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.567277 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.567828 24424 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:56.568622 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:56.569057 31011 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.937s	user 0.003s	sys 0.000s
I20260809 23:19:56.569661 24311 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 8: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:56.570114 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 237096us
I20260809 23:19:56.570537 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.467607 31034 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.525656 25370 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260809 23:19:56.526432 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.522763 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.533293 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.573426 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.518421 31033 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:56.534662 31059 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260809 23:19:56.534847 31031 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:19:56.535383 30953 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:56.535908 31045 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
I20260809 23:19:56.485785 31018 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:56.587308 31018 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.588841 24428 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 7: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:56.475023 30497 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.544718 24314 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:56.545648 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.589908 24422 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:56.548331 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:56.462373 30930 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:56.378611 30983 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.554342 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.554598 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:56.555486 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.562002 24313 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:56.353130 30824 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.564021 24316 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:56.522760 30331 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:56.573901 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47627.6us
I20260809 23:19:56.660884 24320 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:19:56.577606 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44555.3us
I20260809 23:19:56.663228 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.578190 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.578864 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.578955 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.577104 30927 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:56.583583 31049 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.588662 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.543452 30991 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.591882 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.603590 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.566852 30150 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.576107 30890 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.589150 30486 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:56.667539 30930 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.614863 30842 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.661494 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:56.690018 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.691265 30917 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:56.691792 30917 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:56.692093 30712 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:56.566871 24427 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:56.693009 24431 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:19:56.692811 30917 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.693428 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.659260 30988 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.694125 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:56.658483 30974 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.694949 30974 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.486366 31009 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.705173 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.668592 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.668785 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.669770 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:56.706081 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95138.4us
W20260809 23:19:56.671008 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63112.4us
W20260809 23:19:56.671366 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 117124us
I20260809 23:19:56.675478 30991 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:19:56.676108 30965 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.747s	user 0.003s	sys 0.000s
I20260809 23:19:56.675946 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:56.721629 24544 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:56.678097 31018 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:56.678346 30887 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:56.666631 30927 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.678994 30531 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:19:56.679188 31067 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:56.681559 31022 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.743980 31022 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.667196 30983 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:56.737280 30531 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:56.747715 30531 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.748189 30890 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.748742 30531 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:56.748806 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1048) took 2062 ms. Trace:
I20260809 23:19:56.749282 25365 rpcz_store.cc:276] 0809 23:19:54.686547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.689851 (+  3304us) service_pool.cc:227] Handling call
0809 23:19:56.748607 (+2058756us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54006004d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:56.748750 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2699077888}
I20260809 23:19:56.685882 30819 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.688577 30486 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.697038 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:56.758868 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61965.5us
I20260809 23:19:56.759462 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:56.760193 25365 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:56.664610 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.696978 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:56.707324 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.703965 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.761022 24427 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:56.712821 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.714265 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.724301 24309 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:56.762876 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:56.732317 31032 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260809 23:19:56.765419 30920 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:56.736159 30860 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:56.766402 24544 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:19:56.769460 31072 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.770056 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2553) took 1670 ms. Trace:
I20260809 23:19:56.770059 31072 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.770594 30585 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:56.771099 31021 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.736647 30877 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:56.772495 30877 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.772019 31021 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.772938 31072 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.773048 24544 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260809 23:19:56.773496 30877 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:56.775864 31068 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260809 23:19:56.775606 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.737959 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:56.778666 24425 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:56.704824 30150 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:56.766243 31067 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:56.779141 30150 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.770081 31030 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.766764 30486 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:56.780191 31030 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.780578 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:56.781440 30150 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.666576 30644 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.781775 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.782332 31030 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.782096 30644 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.766218 30920 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260809 23:19:56.690917 30930 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.750586 25363 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:19:56.687583 30806 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:56.782905 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.705906 30992 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:56.752130 24540 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:56.777549 30988 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:56.651291 31054 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260809 23:19:56.721050 30991 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:56.767791 24545 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:56.709599 31049 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260809 23:19:56.770615 24539 rpcz_store.cc:276] 0809 23:19:55.099190 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.103089 (+  3899us) service_pool.cc:227] Handling call
0809 23:19:55.155240 (+ 52151us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:56.000320 (+845080us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:56.769817 (+769497us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54001d6854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:56.769990 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1849197824}
I20260809 23:19:56.778469 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:56.770237 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 105756us
I20260809 23:19:56.741360 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:56.692859 30712 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:56.789683 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:56.748756 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.249s	user 0.022s	sys 0.038s
W20260809 23:19:56.757813 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51078.2us
W20260809 23:19:56.822992 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.374s	user 0.023s	sys 0.038s
I20260809 23:19:56.823257 25365 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:56.822367 24320 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:56.825346 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.739041 30927 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:56.790436 24546 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:19:56.811934 25372 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:56.830773 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.831105 31073 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.831727 30377 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:56.832167 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:56.832401 30377 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.832945 24546 raft_consensus.cc:2354] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
I20260809 23:19:56.791294 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:56.814949 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:56.768169 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.784278 30644 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.827431 24320 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:56.827325 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:56.827661 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 86403.3us
I20260809 23:19:56.823616 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:56.852306 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.753495 31018 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260809 23:19:56.791957 31021 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:56.843464 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:56.855597 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87539.5us
I20260809 23:19:56.818194 30991 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:56.826745 30331 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.831785 31073 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.847008 30805 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Leader election won for term 7
I20260809 23:19:56.869033 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:56.840027 30988 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:56.843464 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.815593 30974 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:56.838301 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47116.6us
I20260809 23:19:56.857792 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:56.957175 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 99512.4us
I20260809 23:19:56.851603 25365 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:56.860338 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:56.860594 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:56.780196 31067 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:56.828663 30992 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
W20260809 23:19:56.963840 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 126484us
I20260809 23:19:56.913038 31027 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:56.917325 24323 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:56.920948 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.926227 30919 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:19:56.936323 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:19:56.945252 30841 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:56.948583 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:56.961483 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:56.911700 24429 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
W20260809 23:19:56.969321 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108809us
I20260809 23:19:56.838657 30900 raft_consensus.cc:2741] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260809 23:19:56.971011 24309 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:19:56.971453 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64398.6us
I20260809 23:19:56.856185 31030 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:19:57.069357 24430 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 23:19:56.989028 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.868335 31043 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:56.971766 30854 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:56.966598 31033 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:56.996995 30821 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260809 23:19:56.992187 30917 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260809 23:19:56.952617 30988 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:56.996385 30919 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:57.016268 30805 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:57.031519 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:19:57.094966 30878 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.576s	user 0.015s	sys 0.000s
I20260809 23:19:56.900002 30842 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:19:57.033241 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71889.7us
I20260809 23:19:57.114581 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:57.115422 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:19:57.116278 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:57.039219 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:57.117769 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 78691.1us
I20260809 23:19:57.034749 30764 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:57.056684 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.129631 24539 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:56.964861 31032 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:19:56.964442 30980 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:56.964916 30890 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.074537 24546 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:19:57.023033 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:56.972090 30992 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:57.017164 31018 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:57.147764 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:56.974475 31027 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:57.032068 30841 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:57.102990 31056 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
W20260809 23:19:57.103938 31085 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260809 23:19:57.107474 30953 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:57.111828 31073 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.087806 31043 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.122990 25370 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:57.131278 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:57.173499 24320 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:57.174685 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:57.175441 24320 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:57.133402 24422 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:57.093360 31030 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:57.137521 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130465us
I20260809 23:19:57.097759 30377 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:57.181583 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:19:57.194533 25325 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 2.011s (SENT) [suppressed 32 similar messages]
I20260809 23:19:57.201721 25370 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260809 23:19:57.191035 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:57.150149 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 546591us
I20260809 23:19:57.153158 24546 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:57.190382 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.098s	sys 0.132s
W20260809 23:19:57.207353 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.098s	sys 0.140s
I20260809 23:19:57.155381 24311 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:57.158836 24321 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260809 23:19:57.160465 25368 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260809 23:19:57.207975 25368 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 8
I20260809 23:19:57.208225 25368 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:19:57.208477 25368 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:57.208833 25368 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260809 23:19:57.209177 25368 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:57.136945 30983 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:57.127807 30854 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:57.128944 30764 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.126377 30842 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.165436 31033 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.141506 30934 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:57.141645 30819 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:57.135020 30980 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.136096 30845 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.166742 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.292755 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:57.179471 24380 leader_election.cc:398] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 8 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9.
W20260809 23:19:57.295135 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 115855us
I20260809 23:19:57.145711 30585 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:19:57.206704 24540 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260809 23:19:57.162484 30953 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:57.135473 30878 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.166658 31009 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:57.182256 31043 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.215088 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:57.149257 30890 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.188969 30331 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:57.253068 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45695.7us
I20260809 23:19:57.355657 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:57.253911 30934 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.208844 31018 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:57.212373 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:57.290364 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:57.423924 24546 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:57.263335 31083 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:19:57.211982 30841 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.292234 24320 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:19:57.427541 24543 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:57.380380 24311 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:19:57.251724 30764 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.386464 30819 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.386049 25370 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:57.391238 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.249011 30842 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.343437 30980 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:19:57.396250 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 953) took 2190 ms (client timeout 2011 ms). Trace:
I20260809 23:19:57.214583 30992 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:57.447168 24535 rpcz_store.cc:269] 0809 23:19:55.206053 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.206197 (+   144us) service_pool.cc:227] Handling call
0809 23:19:57.396038 (+2189841us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.396182 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":355419008}
W20260809 23:19:57.447647 31094 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:19:57.448546 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.429207 31038 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.449695 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:57.451208 31087 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.159s	user 0.003s	sys 0.000s
I20260809 23:19:57.451624 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.423318 31018 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.452409 30920 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:57.399145 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:57.452893 31018 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:57.453006 30920 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:57.453729 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54711.4us
I20260809 23:19:57.218839 30773 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.170s
I20260809 23:19:57.213788 30644 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.137s
I20260809 23:19:57.455122 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:57.456148 30644 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:19:57.459990 30842 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.460216 30953 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.461472 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1073) took 1144 ms. Trace:
I20260809 23:19:57.461860 25365 rpcz_store.cc:276] 0809 23:19:56.317251 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.973317 (+656066us) service_pool.cc:227] Handling call
0809 23:19:57.055057 (+ 81740us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:57.055408 (+   351us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:57.055448 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:57.075713 (+ 20265us) log.cc:842] Serialized 26 byte log entry
0809 23:19:57.241286 (+165573us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:57.241337 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:57.241440 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:57.461214 (+219774us) raft_consensus.cc:1671] finished
0809 23:19:57.461247 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:57.461409 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":126537,"threads_started":1,"wal-append.queue_time_us":343492}
I20260809 23:19:57.463702 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:57.267899 30497 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.117s
I20260809 23:19:57.343796 30585 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.424014 24542 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 6 FOLLOWER]: Advancing to term 7
I20260809 23:19:57.465596 30585 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:19:57.447537 30992 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:57.425024 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:57.215608 30983 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:19:57.391904 30877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.040s	sys 0.062s
W20260809 23:19:57.386013 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2688) took 2372 ms (client timeout 1871 ms). Trace:
W20260809 23:19:57.484396 24541 rpcz_store.cc:269] 0809 23:19:54.980280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.990369 (+ 10089us) service_pool.cc:227] Handling call
0809 23:19:57.352065 (+2361696us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540029cf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.352206 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356753280}
I20260809 23:19:57.432140 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:57.433848 24321 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:19:57.393002 30930 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.207537 30805 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:57.470045 30927 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.598s	user 0.006s	sys 0.005s
I20260809 23:19:57.351282 30845 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:57.438454 31100 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260809 23:19:57.439935 31099 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260809 23:19:57.518254 24535 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:57.452736 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.522305 24544 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260809 23:19:57.522684 24544 pending_rounds.cc:76] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:19:57.524454 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.526113 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.455186 31033 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.533721 24546 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:57.424701 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:57.543498 24540 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260809 23:19:57.479622 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285114us
I20260809 23:19:57.545379 24542 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7.
I20260809 23:19:57.401261 30331 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:19:57.483022 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58093.9us
I20260809 23:19:57.547555 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.549365 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:57.550505 30377 raft_consensus.cc:2741] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260809 23:19:57.490255 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58236.9us
I20260809 23:19:57.432578 31032 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260809 23:19:57.401463 31073 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.461964 30953 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:57.431298 30934 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.489780 31053 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:19:57.497447 30919 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.478628 30819 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.500442 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1221) took 1272 ms. Trace:
I20260809 23:19:57.590461 25366 rpcz_store.cc:276] 0809 23:19:56.228205 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.236738 (+  8533us) service_pool.cc:227] Handling call
0809 23:19:57.500229 (+1263491us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.500379 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":622572288}
I20260809 23:19:57.450297 31038 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.506608 25370 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:57.591747 24534 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:19:57.592284 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:57.488507 30877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.041s	sys 0.063s
I20260809 23:19:57.494769 30920 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:57.510275 30949 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:19:57.512524 30531 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.763s	user 0.011s	sys 0.001s
I20260809 23:19:57.558743 24422 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:34784
I20260809 23:19:57.483968 30983 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:57.250248 31027 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.565668 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.567557 24544 pending_rounds.cc:97] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260809 23:19:57.570528 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:57.571767 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.621618 24544 raft_consensus.cc:2866] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:57.494971 30150 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:57.571393 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1078) took 1039 ms. Trace:
W20260809 23:19:57.625507 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55177.8us
I20260809 23:19:57.387374 31030 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.506444 30824 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.595074 24316 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260809 23:19:57.593168 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74454.6us
I20260809 23:19:57.598004 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:57.595822 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.600872 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:57.635576 25370 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:57.601647 24423 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:19:57.605814 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:57.557751 30878 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:57.560354 30377 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:57.608685 31018 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.559381 31083 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260809 23:19:57.611603 30819 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:57.654866 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.653638 25370 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:57.596334 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:57.564339 30974 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.623314 24422 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:19:57.659726 24422 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:57.494270 30930 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.621652 24542 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
W20260809 23:19:57.639366 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1141) took 2450 ms (client timeout 1649 ms). Trace:
I20260809 23:19:57.640923 25371 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260809 23:19:57.641306 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45608.8us
I20260809 23:19:57.641300 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.641822 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.518965 30927 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.560304 31073 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.614154 30890 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.696372 24435 raft_consensus.cc:2396] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260809 23:19:57.587423 30497 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20260809 23:19:57.625769 24317 rpcz_store.cc:276] 0809 23:19:56.531835 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.566286 (+ 34451us) service_pool.cc:227] Handling call
0809 23:19:56.566746 (+   460us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:56.577492 (+ 10746us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:56.577528 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:56.580497 (+  2969us) log.cc:842] Serialized 26 byte log entry
0809 23:19:56.672579 (+ 92082us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:56.713501 (+ 40922us) log.cc:842] Serialized 14 byte log entry
0809 23:19:56.722147 (+  8646us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:56.915737 (+193590us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:56.915805 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:57.567607 (+651802us) spinlock_profiling.cc:243] Waited 4.21 ms on lock 0x7b18003a1800. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:57.571079 (+  3472us) raft_consensus.cc:1671] finished
0809 23:19:57.571112 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:57.571309 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8618,"prepare.queue_time_us":49990,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":12906,"raft.queue_time_us":14395,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":21630,"spinlock_wait_cycles":39777280,"thread_start_us":10943,"threads_started":1,"wal-append.queue_time_us":176095}
I20260809 23:19:57.656996 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:57.640197 31092 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:57.662436 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.632254 30860 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.589797 31111 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20260809 23:19:57.692919 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:57.680027 25369 rpcz_store.cc:269] 0809 23:19:55.180141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:55.187285 (+  7144us) service_pool.cc:227] Handling call
0809 23:19:57.630393 (+2443108us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.630511 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1695394688}
I20260809 23:19:57.693564 24540 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:57.680496 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.731091 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:19:57.731318 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50903.9us
I20260809 23:19:57.656761 25366 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:19:57.635089 31102 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.705299 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:57.653028 30805 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:19:57.640416 30150 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:19:57.731976 25369 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
W20260809 23:19:57.727464 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65128.6us
I20260809 23:19:57.726475 24422 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.741961 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:57.674780 30842 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:57.742453 30887 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.996s	user 0.010s	sys 0.005s
I20260809 23:19:57.763077 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.737478 31009 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:19:57.739136 31061 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.202s	user 0.020s	sys 0.032s
I20260809 23:19:57.744865 30953 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:19:57.767925 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114761us
I20260809 23:19:57.693691 30974 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:57.768940 30486 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.989s	user 0.010s	sys 0.003s
I20260809 23:19:57.789678 30531 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:57.801154 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2779) took 1757 ms (client timeout 1812 ms). Trace:
W20260809 23:19:57.818921 24428 rpcz_store.cc:269] 0809 23:19:56.044269 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.046768 (+  2499us) service_pool.cc:227] Handling call
0809 23:19:57.800895 (+1754127us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b540018c254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.801081 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243276160}
I20260809 23:19:57.787776 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 967) took 1209 ms. Trace:
I20260809 23:19:57.820362 24545 rpcz_store.cc:276] 0809 23:19:56.577999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.578727 (+   728us) service_pool.cc:227] Handling call
0809 23:19:57.787585 (+1208858us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.787701 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235614464}
I20260809 23:19:57.800491 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:57.822144 24317 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:19:57.796370 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1152) took 1485 ms. Trace:
I20260809 23:19:57.822962 25371 rpcz_store.cc:276] 0809 23:19:56.311243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.595333 (+284090us) service_pool.cc:227] Handling call
0809 23:19:57.631202 (+1035869us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:57.705904 (+ 74702us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:57.796170 (+ 90266us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b540065c550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:57.796298 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2237814528}
W20260809 23:19:57.783084 31061 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.246s	user 0.020s	sys 0.033s
I20260809 23:19:57.768785 30940 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.755646 31092 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:57.816651 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:57.825752 24316 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:57.817318 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:57.826448 24320 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:57.827298 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:57.827970 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:57.812656 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.828846 24435 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:57.828899 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:57.830188 24545 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:57.740214 30860 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.846887 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:57.846873 24316 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:57.852771 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:57.853685 24321 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:57.806162 30953 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:57.852880 24320 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:57.857498 25371 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:19:57.858110 24435 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260809 23:19:57.827030 30486 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.861960 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:57.862553 31117 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260809 23:19:57.863615 30486 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:19:57.863849 31082 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.769s	user 0.003s	sys 0.000s
I20260809 23:19:57.813688 30806 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:19:57.861744 31121 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260809 23:19:57.759549 30998 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.086s
I20260809 23:19:57.857123 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:57.850474 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:57.875739 25370 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:57.785928 30764 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.873718 30486 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:57.811731 31009 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:57.827382 30845 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.848485 30497 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.872937 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:57.759948 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:19:57.870714 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 165360us
W20260809 23:19:57.936095 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 181422us
I20260809 23:19:57.797315 30980 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:57.876857 24428 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:57.753363 31102 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.832190 31092 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:57.888810 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98267.7us
I20260809 23:19:57.866025 30953 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:57.813326 30920 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
W20260809 23:19:57.858865 30712 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.034s	user 0.012s	sys 0.000s
I20260809 23:19:57.935850 24320 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:57.878865 30922 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.529s	user 0.012s	sys 0.009s
I20260809 23:19:57.870379 31038 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:57.830880 30531 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.897794 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:57.934620 30991 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.970s	user 0.004s	sys 0.008s
I20260809 23:19:57.881883 31132 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:57.941180 30629 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:57.930362 31026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.413s	user 0.013s	sys 0.026s
I20260809 23:19:57.989559 24320 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:57.929509 30824 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:57.995692 24423 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:57.999609 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:57.981083 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.848295 30940 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.855019 30150 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:57.982860 30922 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.639s	user 0.013s	sys 0.009s
I20260809 23:19:58.034375 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:57.991678 30712 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:57.967096 30845 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:57.973933 31092 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:57.961530 30806 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:58.077571 30806 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.034545 30991 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.081452 30991 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:19:58.094048 30988 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.967s	user 0.011s	sys 0.001s
I20260809 23:19:58.096716 30764 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.096948 30983 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.098328 24425 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:58.098080 31009 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:57.973811 30805 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:58.036262 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55306us
I20260809 23:19:57.981616 31018 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.990644 31111 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:57.878290 31136 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.038632 31132 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.038538 31073 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.048213 30845 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:58.037125 31026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.520s	user 0.013s	sys 0.026s
I20260809 23:19:58.032874 30927 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:57.957500 30486 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.039897 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.068383 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:58.111001 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 264205us
I20260809 23:19:58.136988 24320 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:57.960928 31125 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260809 23:19:58.039203 30997 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.035566 30824 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:58.036414 30930 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:57.981184 30497 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.147368 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.097705 30764 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.120198 31066 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:19:58.046357 30922 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:19:58.151166 31033 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.621s	user 0.011s	sys 0.000s
I20260809 23:19:58.172926 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.173677 24311 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:58.177309 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.094648 30988 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.140108 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.141495 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:58.182801 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.184135 24546 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:58.187708 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:58.142853 24540 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:58.148190 24320 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.103752 31143 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:58.081044 30806 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.152992 30531 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:58.155323 31124 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260809 23:19:58.155481 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:58.156618 30980 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.039471 30629 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.034909 31145 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.126s
I20260809 23:19:58.163249 31175 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.163591 31132 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.164294 30920 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.225526 30983 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.165647 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.114637 30712 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260809 23:19:58.134096 30845 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:19:58.170445 30949 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.000s
I20260809 23:19:58.147697 30824 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:19:58.170990 30331 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.624s	user 0.011s	sys 0.000s
I20260809 23:19:58.148061 30997 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.171537 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.275707 24311 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:58.280052 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.280638 24323 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:58.172472 31033 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:58.211093 30922 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:58.179673 31043 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.000s
W20260809 23:19:58.217257 25323 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.849s (SENT)
I20260809 23:19:58.170325 30764 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.176422 31141 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
I20260809 23:19:58.137526 31111 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.195327 30860 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:58.179977 30927 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:58.313511 30927 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.315475 30927 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.212545 30974 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.224972 31143 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:58.316376 30764 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:58.317209 30992 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.827s	user 0.007s	sys 0.003s
I20260809 23:19:58.316780 31143 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.263242 31147 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.274531 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:58.131233 31136 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.280544 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.178849 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.281750 30331 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:58.283298 30647 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:58.284107 30629 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.285482 30997 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.286026 30486 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:58.287233 31112 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.377s	user 0.001s	sys 0.001s
I20260809 23:19:58.330159 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1074) took 1990 ms. Trace:
W20260809 23:19:58.289604 30585 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.823s	user 0.004s	sys 0.007s
I20260809 23:19:58.269604 31092 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.214052 30805 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.276638 30920 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.271271 31175 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.290822 30497 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.312598 31111 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.318214 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:58.314695 25375 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:19:58.268849 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 185986us
I20260809 23:19:58.320449 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.293800 24323 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:58.323606 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.321868 30647 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:58.297600 30934 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
W20260809 23:19:58.333086 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65084.8us
I20260809 23:19:58.330593 25373 rpcz_store.cc:276] 0809 23:19:56.340052 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.979849 (+639797us) service_pool.cc:227] Handling call
0809 23:19:57.118191 (+138342us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:57.348050 (+229859us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:57.348097 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:57.348692 (+   595us) log.cc:842] Serialized 26 byte log entry
0809 23:19:57.615954 (+267262us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:19:57.745612 (+129658us) log.cc:842] Serialized 14 byte log entry
0809 23:19:57.827794 (+ 82182us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.297540 (+469746us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:58.297598 (+    58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:58.329828 (+ 32230us) raft_consensus.cc:1671] finished
0809 23:19:58.329944 (+   116us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:58.330092 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":203383,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":5109,"raft.queue_time_us":101666,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":9681,"spinlock_wait_cycles":392074496,"thread_start_us":364156,"threads_started":3,"wal-append.queue_time_us":343401}
I20260809 23:19:58.279379 31026 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260809 23:19:58.332934 30585 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.346400 31175 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:58.348934 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 131727us
I20260809 23:19:58.350781 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1064) took 3615 ms. Trace:
I20260809 23:19:58.355443 25363 rpcz_store.cc:276] 0809 23:19:54.735211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.938615 (+203404us) service_pool.cc:227] Handling call
0809 23:19:56.750261 (+1811646us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:56.786281 (+ 36020us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.350395 (+1564114us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54006004d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.350700 (+   305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4104405376}
I20260809 23:19:58.356653 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:58.358560 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:58.350096 24316 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:58.350871 24323 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260809 23:19:58.352474 30486 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:58.343158 24427 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:19:58.331238 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.349172 30592 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.360733 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.364532 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:58.365710 25373 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:58.366282 24316 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260809 23:19:58.361410 25363 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:58.320735 31043 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.352799 30949 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.369740 24321 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:58.370769 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:58.319326 31147 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.375030 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.376502 24316 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:58.366266 30992 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:58.377463 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.377585 30992 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.378090 25363 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:58.377086 30949 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.365182 30860 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:58.381302 25365 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:58.380600 24316 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260809 23:19:58.369303 31081 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:58.364563 31175 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:58.376562 30486 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:58.367235 31150 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:58.377956 30647 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.364300 30592 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.410231 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:58.416015 24428 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:19:58.410450 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:58.411849 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.411885 24316 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:19:58.380962 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:58.376029 31111 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:58.354820 30934 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.415719 31175 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.433166 31175 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.434120 30992 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:58.434450 30585 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.424571 30647 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:58.417846 25363 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:58.429716 24422 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.420059 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.424124 31147 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.448932 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:58.449707 24428 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:19:58.451078 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71366.9us
I20260809 23:19:58.418005 31150 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.411267 31081 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:58.423019 31043 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.457823 31147 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:58.460188 31147 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:58.460580 31147 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.459899 31081 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.461403 30934 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.461586 31081 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:58.421527 30949 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.463842 31030 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.464409 31030 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.451232 31136 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:58.468750 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 281139us
W20260809 23:19:58.469415 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2793) took 1940 ms (client timeout 1960 ms). Trace:
I20260809 23:19:58.467724 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:19:58.469817 24431 rpcz_store.cc:269] 0809 23:19:56.528750 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.572385 (+ 43635us) service_pool.cc:227] Handling call
0809 23:19:58.469250 (+1896865us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.469359 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51270400}
W20260809 23:19:58.453209 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 968) took 1874 ms (client timeout 1835 ms). Trace:
W20260809 23:19:58.475317 24543 rpcz_store.cc:269] 0809 23:19:56.578512 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.578626 (+   114us) service_pool.cc:227] Handling call
0809 23:19:58.452954 (+1874328us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.453135 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1753193728}
I20260809 23:19:58.477141 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:58.443871 30992 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.379339 30824 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.497604 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:58.481913 24428 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:58.501492 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:58.505548 24428 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:58.509320 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:58.510634 24428 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:58.491104 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.521575 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:58.520057 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.522557 24545 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:19:58.519670 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.478763 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.523991 24543 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
W20260809 23:19:58.524225 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47204.6us
I20260809 23:19:58.527489 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:58.528065 24431 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.520358 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:58.528246 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:58.529074 24534 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:58.529479 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:58.530025 25373 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:58.523721 24541 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:58.524519 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:58.528852 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.511554 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:58.531872 24542 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:58.454243 30377 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.544415 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:58.548065 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 198439us
W20260809 23:19:58.549258 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 194610us
I20260809 23:19:58.550187 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:58.552536 25372 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260809 23:19:58.553212 24316 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:58.555087 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:58.469653 30531 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:58.479029 31175 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:58.560407 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:58.490304 30842 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:58.574584 24428 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:19:58.577713 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 65005.8us
I20260809 23:19:58.580308 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.522783 30486 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:19:58.557768 31070 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.000s	sys 0.032s
I20260809 23:19:58.479444 31111 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:58.589531 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.593379 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:58.544903 30497 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:58.458647 30953 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.561200 30531 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:58.590860 31034 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.481s	user 0.010s	sys 0.034s
I20260809 23:19:58.551234 31092 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.596139 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:58.620208 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1075) took 2154 ms. Trace:
I20260809 23:19:58.599197 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.607406 24424 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:58.595599 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:58.571357 31015 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:58.572233 30920 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.571476 31026 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:58.519765 30992 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.612227 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1071) took 2137 ms. Trace:
W20260809 23:19:58.592803 31070 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.000s	sys 0.033s
I20260809 23:19:58.536253 31185 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:19:58.558332 30377 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.614321 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:58.580574 31054 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:19:58.573704 31187 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:58.586071 30842 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.621028 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:58.598711 24428 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:19:58.584350 30331 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:58.598628 30860 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:58.650830 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 848) took 2142 ms. Trace:
I20260809 23:19:58.598044 31125 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:58.682015 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2753) took 1893 ms (client timeout 1988 ms). Trace:
W20260809 23:19:58.664999 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68893.4us
I20260809 23:19:58.722383 24545 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:19:58.621572 25368 rpcz_store.cc:276] 0809 23:19:56.503740 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.028535 (+524795us) service_pool.cc:227] Handling call
0809 23:19:57.029015 (+   480us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:57.218410 (+189395us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.619948 (+1401538us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:58.620147 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":301124352}
W20260809 23:19:58.788909 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 174689us
I20260809 23:19:58.680112 24422 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20260809 23:19:58.721063 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 125544us
I20260809 23:19:58.678781 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1081) took 1060 ms. Trace:
I20260809 23:19:58.793591 25363 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.679446 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2320) took 1069 ms. Trace:
I20260809 23:19:58.641865 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.726063 24431 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:19:58.789440 24429 rpcz_store.cc:276] 0809 23:19:56.508070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.588777 (+ 80707us) service_pool.cc:227] Handling call
0809 23:19:56.589045 (+   268us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:57.027139 (+438094us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.650477 (+1623338us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:58.650743 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":741356288}
I20260809 23:19:58.717509 24314 rpcz_store.cc:276] 0809 23:19:56.460773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.460887 (+   114us) service_pool.cc:227] Handling call
0809 23:19:58.598281 (+2137394us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.598403 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67364608}
W20260809 23:19:58.808269 24309 rpcz_store.cc:269] 0809 23:19:56.788690 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.788830 (+   140us) service_pool.cc:227] Handling call
0809 23:19:58.681778 (+1892948us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540019d7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.681915 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306325760}
I20260809 23:19:58.810142 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:58.658223 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2307) took 2544 ms. Trace:
I20260809 23:19:58.638662 24316 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:58.869127 24430 rpcz_store.cc:276] 0809 23:19:56.113733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.281147 (+167414us) service_pool.cc:227] Handling call
0809 23:19:56.281552 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:57.069918 (+788366us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.657991 (+1588073us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:58.658162 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1810634112}
I20260809 23:19:58.870195 24431 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260809 23:19:58.617537 31034 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.518s	user 0.011s	sys 0.034s
I20260809 23:19:58.811707 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.617020 31009 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:58.883597 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75429.4us
W20260809 23:19:58.644289 31186 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:19:58.894054 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:58.875617 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:19:58.885569 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:58.619419 30531 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.780277 30942 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:58.892187 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 82125.6us
I20260809 23:19:58.881531 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.905597 24309 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:58.645612 31102 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.717584 31187 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:58.666262 30647 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:58.910677 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2696) took 2084 ms. Trace:
I20260809 23:19:58.785928 31021 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.897255 24423 rpcz_store.cc:276] 0809 23:19:57.609676 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.637005 (+ 27329us) service_pool.cc:227] Handling call
0809 23:19:58.679195 (+1042190us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.679351 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781434240}
I20260809 23:19:58.640323 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:19:58.898995 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2695) took 2072 ms. Trace:
I20260809 23:19:58.916504 24535 rpcz_store.cc:276] 0809 23:19:56.826469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.452697 (+626228us) service_pool.cc:227] Handling call
0809 23:19:57.518079 (+ 65382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:57.556736 (+ 38657us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.898781 (+1342045us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b54001d5e50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.898915 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":149651200}
I20260809 23:19:58.913532 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:58.898298 31125 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:58.899571 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.657377 30592 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:19:58.902424 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1216) took 2684 ms (client timeout 2121 ms). Trace:
W20260809 23:19:58.922254 25375 rpcz_store.cc:269] 0809 23:19:56.217849 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.217989 (+   140us) service_pool.cc:227] Handling call
0809 23:19:58.902244 (+2684255us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.902355 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4740599168}
I20260809 23:19:58.612255 30497 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:58.675550 31090 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.535s	user 0.002s	sys 0.001s
I20260809 23:19:58.657377 30953 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.907372 24430 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:58.649227 24164 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:19:58.936337 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.669914 30988 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:58.867611 31026 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:58.811697 31184 raft_consensus.cc:2741] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:19:58.950271 30842 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.953572 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.954572 24316 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:19:58.956089 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.957173 30940 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.960132 31125 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:58.960618 31125 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.961899 25373 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:19:58.921066 31021 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.964633 31021 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.965503 30919 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:58.966390 30940 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:58.967116 30940 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:58.967801 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54363.4us
I20260809 23:19:58.968642 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:58.969306 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.969486 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:58.969959 30940 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:58.966400 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2791) took 2506 ms. Trace:
I20260809 23:19:58.970883 31132 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:19:58.971088 24424 rpcz_store.cc:276] 0809 23:19:56.460162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.568985 (+108823us) service_pool.cc:227] Handling call
0809 23:19:56.569396 (+   411us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:58.784657 (+2215261us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.966153 (+181496us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:58.966328 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4686500608}
I20260809 23:19:58.790014 30932 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:58.658676 30824 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.972560 31102 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:58.973081 31102 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:58.973474 31102 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:58.972565 30932 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:58.975278 31150 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.975719 30932 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:58.975867 31150 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:58.976403 31150 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.647464 30994 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:58.982805 31066 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:58.612705 31111 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.983317 31066 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:58.912226 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.892555 25370 rpcz_store.cc:276] 0809 23:19:57.618626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.656682 (+ 38056us) service_pool.cc:227] Handling call
0809 23:19:58.678598 (+1021916us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.678715 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409233152}
I20260809 23:19:58.915205 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:58.986500 24321 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:19:58.986429 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:58.790280 31147 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.679535 30845 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.003875 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:58.617769 30998 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.294s
I20260809 23:19:58.594547 31175 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.910380 30942 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.008817 31175 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.009917 30942 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:58.927424 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.010608 25375 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:59.011502 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.012378 30845 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.012090 30932 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.939683 30377 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.012647 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.013366 31111 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:58.917047 30847 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:19:59.014767 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1159) took 2353 ms. Trace:
I20260809 23:19:59.015139 24311 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:59.017155 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2765) took 1026 ms. Trace:
I20260809 23:19:58.674400 30486 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.911718 31187 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:58.928583 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1225) took 1382 ms. Trace:
I20260809 23:19:58.900513 31009 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:58.785032 31015 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.797474 30331 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:58.933722 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:58.916975 30764 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:19:59.112005 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1230) took 1478 ms. Trace:
I20260809 23:19:58.972011 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:58.912142 24539 rpcz_store.cc:276] 0809 23:19:56.826639 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.471332 (+644693us) service_pool.cc:227] Handling call
0809 23:19:57.802688 (+331356us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:57.831592 (+ 28904us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:57.831639 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:57.856059 (+ 24420us) log.cc:842] Serialized 26 byte log entry
0809 23:19:58.110080 (+254021us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:58.110119 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.110190 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:58.261296 (+151106us) raft_consensus.cc:1671] finished
0809 23:19:58.261326 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:58.902369 (+641043us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b5400115150. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:58.910606 (+  8237us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18600,"raft.queue_time_us":16050,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":21805,"spinlock_wait_cycles":713187712,"thread_start_us":220071,"threads_started":1,"wal-append.queue_time_us":372966}
I20260809 23:19:59.113001 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2315) took 1593 ms. Trace:
I20260809 23:19:59.106122 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1022) took 1065 ms. Trace:
I20260809 23:19:59.113451 24425 rpcz_store.cc:276] 0809 23:19:57.519433 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.528629 (+  9196us) service_pool.cc:227] Handling call
0809 23:19:57.528998 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:58.098725 (+569727us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.112677 (+1013952us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b540018cc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.112937 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1308904192}
I20260809 23:19:59.113719 25365 rpcz_store.cc:276] 0809 23:19:57.633706 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.692589 (+ 58883us) service_pool.cc:227] Handling call
0809 23:19:59.111711 (+1419122us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.111924 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1495440256}
I20260809 23:19:59.022629 25366 rpcz_store.cc:276] 0809 23:19:57.545763 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.732596 (+186833us) service_pool.cc:227] Handling call
0809 23:19:57.733017 (+   421us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.240323 (+507306us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b180030f2e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac23858f6b 00007fac2386378b
0809 23:19:58.262056 (+ 21733us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:58.262095 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:58.262678 (+   583us) log.cc:842] Serialized 26 byte log entry
0809 23:19:58.366698 (+104020us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:19:58.366729 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:58.366796 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:58.928289 (+561493us) raft_consensus.cc:1671] finished
0809 23:19:58.928312 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:58.928527 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30936,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":15509,"spinlock_wait_cycles":4065152,"thread_start_us":151066,"threads_started":2,"wal-append.queue_time_us":181807}
I20260809 23:19:58.988780 25363 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:58.992609 25370 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.113268 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.117499 24423 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:58.993023 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:59.118607 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 125673us
I20260809 23:19:58.617517 31030 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.991525 24321 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:19:58.997620 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75978.4us
I20260809 23:19:59.114302 24435 rpcz_store.cc:276] 0809 23:19:57.969456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.969780 (+   324us) service_pool.cc:227] Handling call
0809 23:19:59.034301 (+1064521us) spinlock_profiling.cc:243] Waited 76.5 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.034843 (+   542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175936384}
I20260809 23:19:59.123172 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:58.962638 31125 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.125393 24546 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:19:58.998697 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:58.973655 30449 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260809 23:19:59.004920 25373 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:59.006917 24314 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:19:59.007079 31147 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:58.914477 30647 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:59.017590 24311 rpcz_store.cc:276] 0809 23:19:57.991043 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.280309 (+289266us) service_pool.cc:227] Handling call
0809 23:19:59.016932 (+736623us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.017088 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1596779392}
I20260809 23:19:59.013746 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.015192 25372 rpcz_store.cc:276] 0809 23:19:56.661802 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.108586 (+446784us) service_pool.cc:227] Handling call
0809 23:19:58.425330 (+1316744us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:19:58.425522 (+   192us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.553028 (+127506us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.014512 (+461484us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.014707 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312432768}
I20260809 23:19:59.021644 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.150198 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:59.047134 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:19:59.103344 25325 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.008s	sys 0.017s
I20260809 23:19:59.105629 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:19:59.152012 25325 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.009s	sys 0.018s
W20260809 23:19:59.093650 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.013s	sys 0.025s
I20260809 23:19:58.901881 31043 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:59.113184 24535 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:19:59.128014 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.65298e+06us
I20260809 23:19:59.155524 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:59.131801 25370 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:19:59.133152 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2319) took 1536 ms (client timeout 1657 ms). Trace:
I20260809 23:19:58.937760 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:19:59.150400 24541 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.151536 25365 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:19:58.970120 24544 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260809 23:19:59.152555 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.013s	sys 0.025s
I20260809 23:19:59.017882 31150 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.122856 24321 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.172681 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:19:59.152356 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 679058us
I20260809 23:19:59.042727 30942 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:59.030587 30877 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:59.156888 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:58.924907 30860 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:59.160530 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.011s	sys 0.011s
I20260809 23:19:59.161670 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:59.163183 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:59.163601 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 164903us
I20260809 23:19:59.167693 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1084) took 1185 ms. Trace:
W20260809 23:19:59.164842 24427 rpcz_store.cc:269] 0809 23:19:57.596862 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.597085 (+   223us) service_pool.cc:227] Handling call
0809 23:19:59.132941 (+1535856us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54003b9f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.133078 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1581520640}
I20260809 23:19:59.170187 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1170) took 1142 ms. Trace:
I20260809 23:19:59.175453 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:19:59.125209 31032 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.181591 25365 rpcz_store.cc:276] 0809 23:19:57.982346 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.148999 (+1166653us) service_pool.cc:227] Handling call
0809 23:19:59.151376 (+  2377us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:59.167518 (+ 16142us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.167651 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:19:59.182296 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.011s	sys 0.011s
I20260809 23:19:59.177977 24309 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:59.184767 25370 rpcz_store.cc:276] 0809 23:19:58.027490 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.986114 (+958624us) service_pool.cc:227] Handling call
0809 23:19:59.170114 (+184000us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:19:59.186214 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.154290 30647 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.187024 24309 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 7.
I20260809 23:19:59.017336 31102 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.190912 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.190670 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:59.199898 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.201000 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:59.163383 31053 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:59.196661 31136 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:59.196521 31012 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.179569 31049 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260809 23:19:59.184571 30932 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.179736 30764 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:59.172133 31164 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:59.180421 31132 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:19:59.214327 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.185652 30974 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:19:59.227147 24321 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:19:59.157800 30377 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260809 23:19:59.179564 31111 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:59.179507 31125 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:59.264781 31125 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:59.265259 31125 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.266223 31015 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.229892 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.267266 24544 raft_consensus.cc:2362] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 7. Re-sending same reply.
I20260809 23:19:59.268494 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.269208 24544 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:19:59.269655 31102 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.270035 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.231653 24543 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260809 23:19:59.271091 24543 pending_rounds.cc:76] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 5
I20260809 23:19:59.271363 24543 pending_rounds.cc:97] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260809 23:19:59.271682 24543 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:19:59.272888 31009 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:59.176461 30842 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260809 23:19:59.273240 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.274001 30842 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:59.274250 24543 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:59.274497 30842 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:59.275632 30842 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:59.212672 31195 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
W20260809 23:19:59.232372 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70778.2us
I20260809 23:19:59.278515 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:59.280843 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:59.187619 30629 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:19:59.244678 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43846.2us
I20260809 23:19:59.250013 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.250943 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.251543 31030 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:59.251746 31049 consensus_queue.cc:1225] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 23:19:59.252915 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.167762 31198 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.117s
I20260809 23:19:59.294502 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:59.195892 30992 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:59.175752 30878 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260809 23:19:59.165634 30847 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260809 23:19:59.288079 24535 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.247774 30934 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:19:59.297958 30842 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.331311 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.332516 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.333720 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.186151 30877 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.198348 30948 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.088s
I20260809 23:19:59.163596 31043 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.312935 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.299762 30449 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.287758 30932 raft_consensus.cc:2741] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260809 23:19:59.247125 30845 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:59.242548 31053 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:59.319855 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:59.185825 30942 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.321118 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.321409 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.321738 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.322347 30988 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.317966 30878 consensus_queue.cc:1225] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260809 23:19:59.291901 31164 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260809 23:19:59.247249 31012 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.318316 30940 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:19:59.325095 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.317798 31033 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:59.304481 31102 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260809 23:19:59.325974 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2579) took 1510 ms. Trace:
I20260809 23:19:59.166631 31026 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.230440 31136 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:59.335717 24534 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:19:59.308977 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.343205 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.345564 24430 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:19:59.286772 31068 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260809 23:19:59.346203 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.000s	sys 0.042s
I20260809 23:19:59.292187 31030 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:59.350582 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.371981 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.377076 24543 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.376704 24423 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:59.313496 31070 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.118s
W20260809 23:19:59.397579 30684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.274s	user 0.027s	sys 0.031s
I20260809 23:19:59.418148 31132 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:19:59.450260 30684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.345s	user 0.028s	sys 0.043s
I20260809 23:19:59.450387 31132 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.437695 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2326) took 1584 ms. Trace:
I20260809 23:19:59.422902 31141 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260809 23:19:59.451452 24317 rpcz_store.cc:276] 0809 23:19:57.860598 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.434176 (+573578us) service_pool.cc:227] Handling call
0809 23:19:58.434799 (+   623us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.558400 (+123601us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:58.558431 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:58.571615 (+ 13184us) log.cc:842] Serialized 26 byte log entry
0809 23:19:59.182770 (+611155us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:59.182811 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.183009 (+   198us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:59.437503 (+254494us) raft_consensus.cc:1671] finished
0809 23:19:59.437523 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:59.437643 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120082,"raft.queue_time_us":133494,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":9278,"thread_start_us":570615,"threads_started":1,"wal-append.queue_time_us":855784}
I20260809 23:19:59.379189 31038 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:19:59.453935 31038 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.456338 31038 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.457144 30684 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:59.403585 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.457551 30684 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:59.458366 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.458415 30684 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.459451 31070 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:19:59.459843 31070 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:59.459887 30684 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.460493 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:59.460731 31070 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.461350 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.462000 31070 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.520217 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:19:59.464781 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:59.465505 31021 consensus_queue.cc:1159] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 10 status { last_received { term: 9 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:19:59.346099 30934 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:59.415851 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2314) took 1617 ms. Trace:
I20260809 23:19:59.467680 31012 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.468315 25370 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:59.459041 24428 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.403951 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.535363 24313 rpcz_store.cc:276] 0809 23:19:57.798008 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.811350 (+ 13342us) service_pool.cc:227] Handling call
0809 23:19:58.683201 (+871851us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.683596 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:58.683624 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:58.684069 (+   445us) log.cc:842] Serialized 26 byte log entry
0809 23:19:59.144049 (+459980us) raft_consensus.cc:1631] Marking committed up to 4
0809 23:19:59.144104 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.144190 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:59.414553 (+270363us) raft_consensus.cc:1671] finished
0809 23:19:59.414577 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:59.415578 (+  1001us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540019d7d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.415789 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7438,"raft.queue_time_us":392,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":1910640384,"thread_start_us":459551,"threads_started":1,"wal-append.queue_time_us":728605}
I20260809 23:19:59.537305 24431 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.522141 24321 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:19:59.523623 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2799) took 1160 ms. Trace:
I20260809 23:19:59.529002 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:59.523699 24535 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:59.529891 24541 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:59.534279 31027 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:19:59.530506 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:59.583016 24422 rpcz_store.cc:276] 0809 23:19:58.363188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.535976 (+172788us) service_pool.cc:227] Handling call
0809 23:19:58.679871 (+143895us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.906548 (+226677us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.523366 (+616818us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400287dd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:19:59.523550 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":330186496}
I20260809 23:19:59.530705 24540 rpcz_store.cc:276] 0809 23:19:57.815816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.815976 (+   160us) service_pool.cc:227] Handling call
0809 23:19:57.816257 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:58.202242 (+385985us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.325732 (+1123490us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400148554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.325914 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":750075264}
I20260809 23:19:59.420509 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1236) took 1274 ms. Trace:
I20260809 23:19:59.462090 24429 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:19:59.532182 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.001s	sys 0.047s
I20260809 23:19:59.612469 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:59.612895 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 826149us
W20260809 23:19:59.376274 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56526.6us
I20260809 23:19:59.460089 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.618225 24423 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:59.339596 30805 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.533890 31049 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:19:59.484910 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.376698 30644 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:19:59.415791 31081 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.953s	user 0.008s	sys 0.002s
I20260809 23:19:59.578264 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.645351 24535 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
I20260809 23:19:59.448940 30942 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:59.585266 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.534675 30991 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:59.586091 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:19:59.449596 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.601230 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2775) took 1017 ms. Trace:
W20260809 23:19:59.523007 30994 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.544s	user 0.002s	sys 0.008s
I20260809 23:19:59.620940 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:59.611617 25365 rpcz_store.cc:276] 0809 23:19:58.145699 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.183083 (+1037384us) service_pool.cc:227] Handling call
0809 23:19:59.183446 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:59.419663 (+236217us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:59.419696 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:19:59.419740 (+    44us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:19:59.419782 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.420279 (+   497us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.420329 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:59.420455 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":542718848}
I20260809 23:19:59.676457 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:59.677102 25365 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.622648 24534 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:19:59.679587 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.431501 30497 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.612077 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.633038 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.695922 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.696668 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.695971 24422 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:19:59.638098 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.420217 30877 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:19:59.640468 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.037s	sys 0.066s
W20260809 23:19:59.723573 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.038s	sys 0.066s
I20260809 23:19:59.412633 30984 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:19:59.644381 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.431268 30331 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:19:59.548879 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:19:59.554971 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 104124us
I20260809 23:19:59.643110 31111 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.375953 30845 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:59.371249 30652 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.849s	user 0.024s	sys 0.037s
I20260809 23:19:59.634583 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.744822 24540 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:19:59.679016 30841 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.657017 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.752170 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:19:59.537246 31136 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.537762 31030 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.722349 30819 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.433696 31067 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.417639 31032 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:59.427395 31053 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.550382 24164 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:19:59.535234 30948 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:59.536266 30940 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:59.667184 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.770473 24539 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
W20260809 23:19:59.668177 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 82261.7us
I20260809 23:19:59.647663 30805 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:59.755100 30531 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.843s	user 0.009s	sys 0.002s
W20260809 23:19:59.681756 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60954.5us
I20260809 23:19:59.691597 24428 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:19:59.636209 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:59.750288 30845 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.721668 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.701855 25365 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:19:59.723239 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.634158 31049 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:19:59.723937 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 688479us
W20260809 23:19:59.770162 31197 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.435s	user 0.003s	sys 0.001s
W20260809 23:19:59.641920 24545 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.012s	sys 0.098s
W20260809 23:19:59.789851 24545 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.012s	sys 0.099s
I20260809 23:19:59.686337 24431 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:19:59.794517 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.733528 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:19:59.795459 24431 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260809 23:19:59.796226 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.037s	sys 0.086s
I20260809 23:19:59.738216 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:19:59.797449 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.037s	sys 0.087s
I20260809 23:19:59.646649 31061 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:59.405218 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.745165 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:19:59.804487 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59410us
I20260809 23:19:59.732664 24435 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:19:59.747977 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.808112 24425 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.748462 31111 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.750517 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:19:59.815932 24534 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.625945 30974 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260809 23:19:59.539548 31033 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.540549 30647 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:19:59.750065 30652 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.228s	user 0.024s	sys 0.037s
I20260809 23:19:59.679792 31102 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:19:59.644322 30988 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.756300 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:19:59.756349 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:59.832995 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76739.2us
I20260809 23:19:59.399652 31043 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.723879 30942 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:19:59.403144 30764 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.765048 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:59.732370 30984 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:59.668633 24321 rpcz_store.cc:276] 0809 23:19:58.583928 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.895771 (+311843us) service_pool.cc:227] Handling call
0809 23:19:59.226807 (+331036us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b580034eeb0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23864228 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c
0809 23:19:59.595780 (+368973us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":121669504}
I20260809 23:19:59.675578 31009 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260809 23:19:59.776299 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.756795 30934 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.741508 30497 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:19:59.699250 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.779031 25373 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:19:59.780504 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:19:59.783967 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:59.856267 25365 raft_consensus.cc:2354] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:19:59.786387 30531 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.786226 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:19:59.679922 30991 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.676036 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.867197 24423 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260809 23:19:59.775434 30940 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
W20260809 23:19:59.816680 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83224.4us
I20260809 23:19:59.877660 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:19:59.820510 25372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.016s	sys 0.029s
W20260809 23:19:59.879251 25372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.016s	sys 0.029s
I20260809 23:19:59.822731 24546 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:19:59.880295 30988 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.788244 31049 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:19:59.881592 31209 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260809 23:19:59.882826 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2767) took 1741 ms. Trace:
I20260809 23:19:59.883235 24316 rpcz_store.cc:276] 0809 23:19:58.141027 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.957559 (+816532us) service_pool.cc:227] Handling call
0809 23:19:58.958185 (+   626us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.958725 (+   540us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:58.958767 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:58.959325 (+   558us) log.cc:842] Serialized 26 byte log entry
0809 23:19:59.376796 (+417471us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:59.376838 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.376913 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:59.882627 (+505714us) raft_consensus.cc:1671] finished
0809 23:19:59.882646 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:59.882769 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4431,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":417058,"threads_started":1,"wal-append.queue_time_us":815613}
I20260809 23:19:59.845422 30942 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:19:59.884917 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.885114 30942 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:19:59.886029 30991 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:19:59.888736 30847 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:19:59.847949 30984 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:19:59.868016 30497 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.892663 30497 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.876929 31032 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:19:59.894614 31032 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:19:59.895823 30805 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.895509 31032 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.829859 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.898164 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2764) took 1936 ms. Trace:
I20260809 23:19:59.769361 30819 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.899443 24312 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:59.898561 24323 rpcz_store.cc:276] 0809 23:19:57.961725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.360645 (+398920us) service_pool.cc:227] Handling call
0809 23:19:58.360955 (+   310us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.553496 (+192541us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:58.553541 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:58.574441 (+ 20900us) log.cc:842] Serialized 26 byte log entry
0809 23:19:59.167247 (+592806us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:19:59.167283 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.167345 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:19:59.749781 (+582436us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:59.897876 (+148095us) raft_consensus.cc:1671] finished
0809 23:19:59.897898 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:19:59.898108 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":196578,"raft.queue_time_us":210425,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":8993,"spinlock_wait_cycles":2798336,"thread_start_us":592468,"threads_started":1,"wal-append.queue_time_us":759911}
I20260809 23:19:59.899991 30647 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.900437 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.901456 24323 raft_consensus.cc:2362] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11. Re-sending same reply.
I20260809 23:19:59.829348 24435 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:19:59.904268 31111 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:19:59.837801 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:59.906912 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69183.6us
I20260809 23:19:59.907680 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:19:59.826768 30860 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.000s	sys 0.014s
I20260809 23:19:59.802809 31083 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:19:59.844646 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.012s	sys 0.039s
I20260809 23:19:59.802803 31038 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:19:59.803081 30878 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:19:59.846599 24313 raft_consensus.cc:2354] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
W20260809 23:19:59.916891 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.012s	sys 0.039s
I20260809 23:19:59.803316 31184 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:19:59.815786 31176 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:19:59.774399 30948 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.667481 30644 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.921422 24314 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:19:59.851611 24534 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:19:59.852675 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.961036 24321 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:19:59.838306 31066 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.849s	user 0.010s	sys 0.001s
I20260809 23:19:59.776064 31067 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.824602 31012 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:19:59.857195 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.977166 25365 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:19:59.913293 31081 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:19:59.857743 24430 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:19:59.858613 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 78171.2us
I20260809 23:19:59.837358 31147 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:19:59.868395 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1174) took 1594 ms. Trace:
I20260809 23:19:59.984073 25370 rpcz_store.cc:276] 0809 23:19:58.273653 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.198498 (+924845us) service_pool.cc:227] Handling call
0809 23:19:59.198861 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:59.536164 (+337303us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.868141 (+331977us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.868329 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":618915200}
I20260809 23:19:59.818822 31198 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260809 23:19:59.849725 31102 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:19:59.908624 24435 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:19:59.909485 24422 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:19:59.818564 31061 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:59.855711 30585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.008s	sys 0.008s
I20260809 23:20:00.003641 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:00.004779 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:59.865334 30486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.013s	sys 0.007s
I20260809 23:19:59.912075 24431 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:19:59.922761 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:59.961447 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.006168 24534 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260809 23:20:00.006229 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83595.6us
I20260809 23:19:59.961861 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:00.007117 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45362.5us
I20260809 23:20:00.008703 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:19:59.927349 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:19:59.903657 31187 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.795s	user 0.010s	sys 0.000s
I20260809 23:19:59.968987 31184 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:19:59.969945 31083 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.974422 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.958528 30647 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260809 23:19:59.784283 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:19:59.921509 31127 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.006s	sys 0.015s
I20260809 23:19:59.998539 31059 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:19:59.916471 30860 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.000s	sys 0.015s
I20260809 23:19:59.982062 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:19:59.975155 31125 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.709s	user 0.011s	sys 0.000s
I20260809 23:19:59.985569 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:19:59.983816 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:19:59.992103 24424 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:19:59.993086 24543 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:00.003352 31147 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.011922 31012 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:19:59.994849 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1234) took 1854 ms. Trace:
I20260809 23:20:00.052368 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:00.052649 25373 rpcz_store.cc:276] 0809 23:19:58.140764 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.143609 (+1002845us) service_pool.cc:227] Handling call
0809 23:19:59.143996 (+   387us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:59.858132 (+714136us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.994572 (+136440us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:19:59.994784 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1452741376}
I20260809 23:19:59.995806 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:19:59.995949 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:19:59.999128 24539 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260809 23:19:59.947984 30878 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:19:59.996956 30917 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.009s	sys 0.018s
I20260809 23:19:59.841181 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:00.057950 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:19:59.950886 30497 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260809 23:20:00.013751 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:00.061767 24534 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.976999 31066 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:19:59.981088 31081 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:19:59.975770 25375 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260809 23:20:00.024307 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 240129us
I20260809 23:20:00.029922 31185 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:00.025755 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.062472 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:00.005328 30585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.009s	sys 0.009s
I20260809 23:20:00.070896 31150 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.071619 30819 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.071425 31150 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.072325 30567 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:00.072978 30567 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:00.073396 30567 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:00.074036 31150 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.028057 30890 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:00.074730 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:00.035636 30841 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:00.028916 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:00.030428 24422 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:00.011404 30486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.013s	sys 0.007s
I20260809 23:20:00.081079 24431 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:19:59.950822 31038 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.084982 25366 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260809 23:20:00.059577 30917 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.009s	sys 0.018s
I20260809 23:20:00.057265 31147 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.041344 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.101279 24544 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:19:59.919781 30974 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:19:59.993191 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.019s	sys 0.026s
I20260809 23:19:59.947758 30805 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260809 23:20:00.027297 30842 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.000s
I20260809 23:20:00.013599 31111 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:00.107076 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.020s	sys 0.026s
I20260809 23:20:00.048449 25363 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:00.025254 31127 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.009s	sys 0.017s
W20260809 23:20:00.041376 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59407.4us
I20260809 23:20:00.108597 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:00.051311 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1075) took 3588 ms (client timeout 1849 ms). Trace:
I20260809 23:20:00.051995 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:00.051365 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.060389 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:00.014660 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:00.112682 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52404us
I20260809 23:20:00.113385 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:00.068188 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:00.069087 30644 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.069736 30497 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:00.075348 24427 raft_consensus.cc:2362] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11. Re-sending same reply.
I20260809 23:20:00.125202 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.028277 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:00.079313 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:00.031136 31059 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:00.013211 31187 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.092715 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.040453 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:00.028283 30920 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260809 23:20:00.100330 24534 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:00.078102 31185 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:00.048645 31125 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.119848 24535 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:00.112457 24323 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:00.204339 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.204674 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1167) took 2649 ms. Trace:
I20260809 23:20:00.076004 31012 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.048424 30331 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.050s
I20260809 23:20:00.124038 30841 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:00.111135 24312 rpcz_store.cc:269] 0809 23:19:56.462455 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:56.567025 (+104570us) service_pool.cc:227] Handling call
0809 23:20:00.051136 (+3484111us) spinlock_profiling.cc:243] Waited 3.33 s on lock 0x7b540019dcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.051238 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7661397376}
I20260809 23:20:00.082624 30998 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.245386 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1232) took 2246 ms. Trace:
I20260809 23:20:00.086512 31033 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.157661 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:00.128820 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.126721 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.101758 30819 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.145460 24431 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:00.194401 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:00.261446 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67163.8us
I20260809 23:20:00.261907 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.262841 24534 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:00.206153 31185 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:00.110176 31026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.003s	sys 0.022s
I20260809 23:20:00.089838 31102 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:00.198029 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.015s	sys 0.020s
W20260809 23:20:00.190115 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 154942us
W20260809 23:20:00.252285 31211 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.503s	user 0.001s	sys 0.002s
I20260809 23:20:00.105000 30890 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:00.226087 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.109637 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2596) took 1172 ms. Trace:
I20260809 23:20:00.120038 30629 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.133888 30644 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:00.108433 30805 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:00.234148 25368 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:00.205013 25371 rpcz_store.cc:276] 0809 23:19:57.555162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.868747 (+313585us) service_pool.cc:227] Handling call
0809 23:19:57.896860 (+ 28113us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:19:57.944245 (+ 47385us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:57.960243 (+ 15998us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:19:57.973412 (+ 13169us) log.cc:842] Serialized 52 byte log entry
0809 23:19:59.155956 (+1182544us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b18002e2040. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:19:59.797910 (+641954us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:19:59.797966 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.798066 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:00.188307 (+390241us) raft_consensus.cc:1671] finished
0809 23:20:00.188327 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:00.204607 (+ 16280us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40070,"raft.queue_time_us":65368,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":47304,"spinlock_wait_cycles":4059008,"thread_start_us":641727,"threads_started":1,"wal-append.queue_time_us":2049380}
I20260809 23:20:00.234181 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1092) took 1243 ms. Trace:
I20260809 23:20:00.294085 24542 rpcz_store.cc:276] 0809 23:19:58.937066 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.938332 (+  1266us) service_pool.cc:227] Handling call
0809 23:19:58.951882 (+ 13550us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:58.952143 (+   261us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:59.339436 (+387293us) log.cc:842] Serialized 14 byte log entry
0809 23:20:00.107753 (+768317us) log.cc:842] Serialized 14 byte log entry
0809 23:20:00.109342 (+  1589us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:00.109384 (+    42us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:00.109436 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.109495 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:00.109588 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":692443,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":85,"raft.queue_time_us":13562,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":13173,"spinlock_wait_cycles":100613760,"thread_start_us":1154639,"threads_started":2}
I20260809 23:20:00.319326 24314 rpcz_store.cc:276] 0809 23:19:58.990874 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.143883 (+153009us) service_pool.cc:227] Handling call
0809 23:20:00.233980 (+1090097us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.234124 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1703211776}
W20260809 23:20:00.198837 31132 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.678s	user 0.010s	sys 0.003s
I20260809 23:20:00.185566 31175 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:00.241715 24425 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:00.242298 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1233) took 2101 ms. Trace:
I20260809 23:20:00.243836 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1096) took 1252 ms. Trace:
I20260809 23:20:00.245931 25366 rpcz_store.cc:276] 0809 23:19:57.998720 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.145446 (+1146726us) service_pool.cc:227] Handling call
0809 23:19:59.155752 (+ 10306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.091474 (+935722us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.245086 (+153612us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:00.245310 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2131912960}
I20260809 23:20:00.252278 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2717) took 1016 ms. Trace:
I20260809 23:20:00.124230 30497 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:00.195786 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:00.273239 30994 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:00.267814 24430 pending_rounds.cc:76] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 5
I20260809 23:20:00.322968 30805 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:20:00.268846 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 456368us
I20260809 23:20:00.203572 30991 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:00.339108 24430 pending_rounds.cc:97] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260809 23:20:00.340675 24430 raft_consensus.cc:2866] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:00.197391 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.205021 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:00.277298 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.015s	sys 0.020s
I20260809 23:20:00.155584 30531 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.338905 30994 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:00.342259 24323 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:00.346681 30994 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.348637 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:00.351742 31229 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.251s	user 0.000s	sys 0.000s
I20260809 23:20:00.322113 31125 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.289316 30984 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.256939 30942 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.240229 30841 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:00.267874 31227 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260809 23:20:00.238965 31067 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.128940 31049 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.239710 31012 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.247949 24317 raft_consensus.cc:2932] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:00.302448 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.376256 25370 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:00.277516 30449 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.899s	user 0.001s	sys 0.010s
I20260809 23:20:00.326632 25363 rpcz_store.cc:276] 0809 23:19:58.140474 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.126301 (+985827us) service_pool.cc:227] Handling call
0809 23:19:59.126670 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.119293 (+992623us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.242013 (+122720us) spinlock_profiling.cc:243] Waited 916 ms on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:00.242227 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2107841280}
I20260809 23:20:00.330520 24539 rpcz_store.cc:276] 0809 23:19:59.236141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.771336 (+535195us) service_pool.cc:227] Handling call
0809 23:19:59.790463 (+ 19127us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.054760 (+264297us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.251979 (+197219us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540029cf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:00.252203 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":478860544}
I20260809 23:20:00.331409 25375 rpcz_store.cc:276] 0809 23:19:58.991126 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.013537 (+ 22411us) service_pool.cc:227] Handling call
0809 23:20:00.243613 (+1230076us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.243768 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1898776320}
W20260809 23:20:00.382166 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.021s	sys 0.027s
I20260809 23:20:00.327770 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:00.382503 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.021s	sys 0.027s
W20260809 23:20:00.382916 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55263.3us
I20260809 23:20:00.383395 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2597) took 1444 ms. Trace:
W20260809 23:20:00.276341 31026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.004s	sys 0.022s
I20260809 23:20:00.276655 30819 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.265096 24546 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:00.440762 24543 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:00.441146 24543 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:00.292794 31059 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:00.317687 31225 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260809 23:20:00.337831 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:00.338214 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:00.227636 30919 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:00.339036 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:00.278228 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:00.365814 24321 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 10. Ops: []
I20260809 23:20:00.337785 30497 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:00.326690 31083 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.260655 30998 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.437966 30449 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:00.337353 31184 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:00.323148 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:00.527253 24534 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:00.379437 31012 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.380298 24544 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:00.332635 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:00.383742 24545 rpcz_store.cc:276] 0809 23:19:58.938695 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.938884 (+   189us) service_pool.cc:227] Handling call
0809 23:19:58.939187 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:58.939380 (+   193us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:59.790320 (+850940us) log.cc:842] Serialized 14 byte log entry
0809 23:20:00.382902 (+592582us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:00.382973 (+    71us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:00.383026 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.383219 (+   193us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:00.383333 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":702762,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"thread_start_us":1431030,"threads_started":2}
I20260809 23:20:00.366175 30860 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:00.444059 24541 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:20:00.155251 31081 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.446274 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:00.334998 31132 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.449683 24542 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260809 23:20:00.334092 30629 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.374043 30841 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:00.442082 24543 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.442821 31067 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260809 23:20:00.445314 31175 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.085s
I20260809 23:20:00.376677 31066 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.437755 30942 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.446763 30647 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.499598 24425 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:00.509264 30567 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.343781 30890 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.383136 31054 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:00.501210 31136 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.726s	user 0.008s	sys 0.002s
W20260809 23:20:00.501106 30845 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.702s	user 0.010s	sys 0.000s
W20260809 23:20:00.500942 30940 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.000s
I20260809 23:20:00.531818 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:00.563934 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:00.822728 24314 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:00.571620 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:00.613474 31081 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:00.617662 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:00.831987 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 214429us
I20260809 23:20:00.584012 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1094) took 1593 ms. Trace:
I20260809 23:20:00.833448 25371 rpcz_store.cc:276] 0809 23:19:58.990619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.513982 (+1523363us) service_pool.cc:227] Handling call
0809 23:20:00.583354 (+ 69372us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.583613 (+   259us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.583649 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:00.583705 (+    56us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:00.583759 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.583825 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:00.583943 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:00.674780 24429 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:00.573985 24539 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
W20260809 23:20:00.768414 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1172) took 2559 ms (client timeout 3225 ms). Trace:
W20260809 23:20:00.836520 25368 rpcz_store.cc:269] 0809 23:19:58.208609 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.810739 (+602130us) service_pool.cc:227] Handling call
0809 23:20:00.768001 (+1957262us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.768299 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3112032128}
I20260809 23:20:00.511507 30974 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:00.597966 31027 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.965s	user 0.009s	sys 0.000s
I20260809 23:20:00.786952 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1100) took 1490 ms. Trace:
W20260809 23:20:00.597783 31226 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.474s	user 0.000s	sys 0.001s
I20260809 23:20:00.511708 30919 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:00.754961 30934 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.018s	sys 0.040s
I20260809 23:20:00.594007 30819 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.572225 30900 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:00.785756 31083 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.789920 31054 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:00.817413 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.008s	sys 0.020s
W20260809 23:20:00.939941 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.008s	sys 0.020s
I20260809 23:20:00.529307 31147 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.817006 30845 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.567564 30860 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:00.817216 31111 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.010s	sys 0.029s
I20260809 23:20:00.791283 30764 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:00.781040 30647 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:00.582477 30948 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:00.607590 30991 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:00.762398 31032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.021s	sys 0.034s
I20260809 23:20:00.824949 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:00.501693 30585 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
W20260809 23:20:00.608907 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47435.5us
I20260809 23:20:00.601168 30878 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.829789 25366 raft_consensus.cc:2354] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:00.598959 30984 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:00.567997 31030 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.794s	user 0.000s	sys 0.011s
I20260809 23:20:00.838573 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:00.588414 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2727) took 1118 ms. Trace:
I20260809 23:20:00.835879 24534 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:00.844740 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:00.828173 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.012s	sys 0.016s
W20260809 23:20:01.017544 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.013s	sys 0.016s
W20260809 23:20:01.017585 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 179104us
W20260809 23:20:01.018453 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 905063us
I20260809 23:20:00.584535 31061 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:00.960462 31152 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.756060 31198 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260809 23:20:00.931241 24539 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:00.937074 25371 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:00.888103 31053 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.012s	sys 0.022s
I20260809 23:20:00.938506 31083 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.780193 31176 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:00.930091 30934 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.019s	sys 0.041s
I20260809 23:20:00.890573 25368 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:00.833534 24425 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:20:00.882211 24540 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.013s	sys 0.027s
I20260809 23:20:00.578718 31187 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:00.948562 30567 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.955523 24316 raft_consensus.cc:2354] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:20:00.956809 30845 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:00.592499 31038 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:00.861884 30942 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:00.842670 31237 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:00.960654 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.012s	sys 0.044s
W20260809 23:20:01.067675 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.014s	sys 0.046s
I20260809 23:20:01.003157 30991 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.946081 25370 rpcz_store.cc:276] 0809 23:19:59.296998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.995719 (+698721us) service_pool.cc:227] Handling call
0809 23:20:00.786668 (+790949us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.786860 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":739282688}
W20260809 23:20:00.946779 31111 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.010s	sys 0.030s
I20260809 23:20:01.002499 24545 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:00.813606 31136 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.612799 31049 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:00.958081 30860 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:01.019963 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:00.592808 30998 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.022460 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:00.850517 24321 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:01.020982 30900 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:01.016639 24546 rpcz_store.cc:276] 0809 23:19:59.469806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.880481 (+410675us) service_pool.cc:227] Handling call
0809 23:20:00.257758 (+377277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.440720 (+182962us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:00.588169 (+147449us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400115150. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:00.588343 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":866980736}
I20260809 23:20:01.014348 30585 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:00.843883 30940 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.026566 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.035780 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1239) took 1842 ms. Trace:
I20260809 23:20:01.130231 25371 rpcz_store.cc:276] 0809 23:19:59.193244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.854860 (+1661616us) service_pool.cc:227] Handling call
0809 23:20:00.936821 (+ 81961us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.035465 (+ 98644us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.035713 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:01.132097 24534 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:20:01.036211 24539 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:00.621555 31185 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:00.578898 30920 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.435s
W20260809 23:20:00.995865 31127 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.009s	sys 0.017s
W20260809 23:20:01.004388 30497 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.000s
W20260809 23:20:01.050686 24540 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.015s	sys 0.028s
I20260809 23:20:01.049372 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1241) took 1841 ms. Trace:
I20260809 23:20:01.055685 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:01.040992 31053 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.013s	sys 0.024s
I20260809 23:20:01.143831 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.169227 24546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 11
I20260809 23:20:01.139515 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.059751 24309 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:01.025630 30984 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.889767 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:01.057615 31253 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260809 23:20:01.042290 31270 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:00.995283 30764 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:01.078469 24317 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.702s	user 0.011s	sys 0.000s
W20260809 23:20:01.066637 31245 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260809 23:20:01.085899 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1235) took 2944 ms. Trace:
I20260809 23:20:01.202133 24424 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:01.024690 31030 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.066212 31125 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.083777 31111 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:01.116226 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.012s	sys 0.021s
W20260809 23:20:01.219429 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.013s	sys 0.022s
I20260809 23:20:01.116653 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.220285 24545 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 10
W20260809 23:20:01.199007 31070 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.662s	user 0.010s	sys 0.004s
W20260809 23:20:01.125073 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.005s	sys 0.029s
W20260809 23:20:01.222064 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.005s	sys 0.029s
W20260809 23:20:01.222352 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 904546us
I20260809 23:20:01.125681 24429 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260809 23:20:01.130739 24423 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.012s	sys 0.023s
W20260809 23:20:01.223203 24423 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.013s	sys 0.023s
I20260809 23:20:01.223912 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2336) took 1804 ms. Trace:
W20260809 23:20:01.003584 31032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.990s	user 0.022s	sys 0.035s
I20260809 23:20:01.136852 24313 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:01.069423 30845 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:01.147310 25368 rpcz_store.cc:276] 0809 23:19:59.208118 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.852881 (+1644763us) service_pool.cc:227] Handling call
0809 23:20:00.853472 (+   591us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.049048 (+195576us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.049303 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:01.148854 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 128970us
I20260809 23:20:01.233326 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:01.148849 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.023s	sys 0.014s
I20260809 23:20:01.055673 31176 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.168000 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112506us
W20260809 23:20:01.172569 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.026s	sys 0.029s
W20260809 23:20:01.237444 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.891s	user 0.026s	sys 0.030s
W20260809 23:20:01.237754 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 891596us
W20260809 23:20:01.141562 30994 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.011s	sys 0.026s
W20260809 23:20:01.175092 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.832s	user 0.001s	sys 0.040s
W20260809 23:20:01.238695 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.001s	sys 0.040s
I20260809 23:20:01.238605 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:01.133196 24539 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.241765 24539 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:01.039090 31152 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.191845 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.009s	sys 0.036s
W20260809 23:20:01.243129 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.010s	sys 0.036s
I20260809 23:20:01.190752 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1245) took 1898 ms. Trace:
I20260809 23:20:01.243968 25371 rpcz_store.cc:276] 0809 23:19:59.292599 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.131438 (+1838839us) service_pool.cc:227] Handling call
0809 23:20:01.189298 (+ 57860us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.190086 (+   788us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.190142 (+    56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.190214 (+    72us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:01.190279 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.190394 (+   115us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.190650 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1483,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":9379}
I20260809 23:20:01.211705 31132 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.174165 31175 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.054s
W20260809 23:20:01.201673 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.013s	user 0.021s	sys 0.045s
I20260809 23:20:01.171195 30497 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.202081 25372 rpcz_store.cc:276] 0809 23:19:58.141041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.150145 (+1009104us) service_pool.cc:227] Handling call
0809 23:19:59.173577 (+ 23432us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:59.173802 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:59.173828 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:59.194600 (+ 20772us) log.cc:842] Serialized 26 byte log entry
0809 23:19:59.879665 (+685065us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:19:59.879706 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:19:59.879789 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:00.596549 (+716760us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:01.076268 (+479719us) raft_consensus.cc:1671] finished
0809 23:20:01.076298 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.085810 (+  9512us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116838272,"thread_start_us":630020,"threads_started":1,"wal-append.queue_time_us":854607}
I20260809 23:20:01.123416 30900 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:01.064749 30942 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:01.075037 31237 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.149461 31027 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.213701 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:01.149747 31127 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.009s	sys 0.017s
I20260809 23:20:01.198444 30998 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:01.305220 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.007s	sys 0.040s
W20260809 23:20:01.228088 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.029s	sys 0.058s
W20260809 23:20:01.306687 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.029s	sys 0.059s
W20260809 23:20:01.231118 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.020s	sys 0.060s
I20260809 23:20:01.058720 30919 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.231519 24311 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:01.234093 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.023s	sys 0.015s
I20260809 23:20:01.192127 31260 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:01.234949 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1104) took 1155 ms. Trace:
I20260809 23:20:01.224300 24423 rpcz_store.cc:276] 0809 23:19:59.419223 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.876035 (+456812us) service_pool.cc:227] Handling call
0809 23:19:59.879532 (+  3497us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:19:59.879736 (+   204us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:19:59.894189 (+ 14453us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.223532 (+1329343us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.223595 (+    63us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:01.223653 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.223717 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.223856 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":563,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":70234,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":79,"thread_start_us":1318801,"threads_started":2,"wal-append.queue_time_us":919165}
I20260809 23:20:01.086465 30629 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.251056 24545 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:01.040395 30974 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.274185 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.085s	user 0.021s	sys 0.046s
I20260809 23:20:01.275221 25372 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:20:01.320021 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1182) took 1704 ms. Trace:
I20260809 23:20:01.320338 25372 rpcz_store.cc:276] 0809 23:19:59.615502 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.274603 (+1659101us) service_pool.cc:227] Handling call
0809 23:20:01.274985 (+   382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.319768 (+ 44783us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.319965 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:01.280813 24498 leader_election.cc:398] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:20:01.208093 31270 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.193580 30984 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.276626 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.017s	sys 0.041s
I20260809 23:20:01.272769 31067 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260809 23:20:01.289175 25369 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260809 23:20:01.255151 31102 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.971s	user 0.015s	sys 0.042s
W20260809 23:20:01.289930 31256 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260809 23:20:01.245587 30847 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:01.227317 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:01.242033 30994 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.889s	user 0.012s	sys 0.026s
W20260809 23:20:01.307415 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.020s	sys 0.060s
I20260809 23:20:01.285218 30900 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:01.220090 31125 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.307907 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.007s	sys 0.040s
I20260809 23:20:01.287917 30942 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:01.312879 24321 rpcz_store.cc:276] 0809 23:20:00.079853 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.233754 (+1153901us) service_pool.cc:227] Handling call
0809 23:20:01.234118 (+   364us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.234556 (+   438us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.234635 (+    79us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.234676 (+    41us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.234715 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.234771 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.234879 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":13321}
I20260809 23:20:01.241734 31147 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:01.307389 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.009s	sys 0.025s
W20260809 23:20:01.381528 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.009s	sys 0.025s
W20260809 23:20:01.382283 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.10403e+06us
I20260809 23:20:01.383237 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.384745 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.325745 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.385852 24545 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:01.308400 31251 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260809 23:20:01.330825 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:01.334419 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.017s	sys 0.041s
I20260809 23:20:01.335232 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.327421 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:01.338073 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:01.390012 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62763.9us
I20260809 23:20:01.337517 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:01.315701 30805 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.947s	user 0.008s	sys 0.003s
I20260809 23:20:01.185945 24546 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:01.400031 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.339143 30920 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:01.282790 30845 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.271884 31267 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.094s
I20260809 23:20:01.307958 30647 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.362519 25370 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:01.315680 30998 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:01.348704 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.021s	sys 0.037s
W20260809 23:20:01.412900 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.022s	sys 0.037s
W20260809 23:20:01.337352 31102 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.053s	user 0.017s	sys 0.044s
W20260809 23:20:01.390826 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52850.7us
I20260809 23:20:01.413980 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:01.315279 31252 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260809 23:20:01.392095 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:01.398319 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:01.377736 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.013s	sys 0.041s
I20260809 23:20:01.305871 31027 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.404417 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:01.406553 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:01.407130 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76407.4us
I20260809 23:20:01.421514 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.422295 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.423045 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2763) took 3477 ms. Trace:
I20260809 23:20:01.423098 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.424729 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.425848 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:01.409590 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.426900 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.427410 24546 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:01.354393 30449 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.357937 31136 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.360651 31175 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:01.418135 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.014s	sys 0.041s
I20260809 23:20:01.404774 30934 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:01.378736 30991 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.410147 30920 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:01.354883 30847 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.423540 24320 rpcz_store.cc:276] 0809 23:19:57.940826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:58.363271 (+422445us) service_pool.cc:227] Handling call
0809 23:19:58.363680 (+   409us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:58.580565 (+216885us) raft_consensus.cc:1520] Early marking committed up to index 5
0809 23:19:58.612363 (+ 31798us) log.cc:842] Serialized 14 byte log entry
0809 23:19:58.684355 (+ 71992us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b18003a1c80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:19:59.409263 (+724908us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b18003a1c80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:01.239306 (+1830043us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:01.241942 (+  2636us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.242111 (+   169us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:01.242147 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.242214 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.418033 (+175819us) raft_consensus.cc:1671] finished
0809 23:20:01.418053 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.418188 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":216430,"prepare.queue_time_us":19698,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":220717,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":11388,"spinlock_wait_cycles":100789888,"thread_start_us":1829663,"threads_started":2,"wal-append.queue_time_us":1960333}
I20260809 23:20:01.447132 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1109) took 1060 ms. Trace:
I20260809 23:20:01.431942 24321 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:20:01.431665 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:01.448788 24423 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:01.448299 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1106) took 1356 ms. Trace:
I20260809 23:20:01.347081 31070 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260809 23:20:01.405001 30994 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.408648 30940 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:01.421818 30877 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:01.306209 30585 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.411427 30567 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260809 23:20:01.440951 25368 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:01.432365 24546 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:01.410631 31147 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:01.442767 24545 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:01.407801 31032 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.449443 24321 rpcz_store.cc:276] 0809 23:20:00.091895 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.430068 (+1338173us) service_pool.cc:227] Handling call
0809 23:20:01.430339 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.447971 (+ 17632us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.448231 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:01.430085 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.337086 31111 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.450317 24320 rpcz_store.cc:276] 0809 23:20:00.386376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.446159 (+1059783us) service_pool.cc:227] Handling call
0809 23:20:01.446519 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.446729 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.446753 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.446797 (+    44us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:01.446852 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.446912 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.447064 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:01.431576 30948 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:01.423677 31185 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.429834 31187 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:20:01.420428 25375 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.007s	sys 0.019s
W20260809 23:20:01.519375 25375 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.077s	user 0.007s	sys 0.019s
I20260809 23:20:01.436893 30449 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.431474 31049 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:20:01.454602 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 238768us
I20260809 23:20:01.412154 30805 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.464772 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 223037us
I20260809 23:20:01.466061 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.442812 31030 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.443804 31136 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:01.472781 24545 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:01.445418 30764 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.474193 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2729) took 2003 ms. Trace:
I20260809 23:20:01.445214 31083 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.457450 30991 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:01.453565 31127 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:01.482345 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55514.8us
I20260809 23:20:01.521628 30948 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.492111 24423 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:01.456166 30934 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:01.503777 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73795.8us
I20260809 23:20:01.439349 30647 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:01.524142 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.472043 30940 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:01.465149 31132 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:01.527511 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:01.528350 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:01.480034 30877 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:01.480626 30497 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.472280 31176 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.540211 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.572526 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.448275 31280 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
W20260809 23:20:01.541301 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 101774us
I20260809 23:20:01.537786 24540 rpcz_store.cc:276] 0809 23:19:59.470439 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.277667 (+807228us) service_pool.cc:227] Handling call
0809 23:20:00.277929 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.278113 (+   184us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:01.146415 (+868302us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.473818 (+327403us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.473873 (+    55us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:01.473925 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.473996 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.474133 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":470,"prepare.queue_time_us":953211,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":14244,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":35473,"thread_start_us":1128119,"threads_started":2}
I20260809 23:20:01.545485 24425 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:01.521731 30994 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.534720 24545 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.545667 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106714us
I20260809 23:20:01.545347 31136 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:01.580071 24545 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:01.550148 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.453866 31290 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.523689 31237 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.521539 31185 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:01.527486 31102 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.553027 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2782) took 1935 ms. Trace:
W20260809 23:20:01.602620 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52558us
I20260809 23:20:01.602752 24323 rpcz_store.cc:276] 0809 23:19:59.617394 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.346853 (+729459us) service_pool.cc:227] Handling call
0809 23:20:00.347182 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:00.347378 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:01.308815 (+961437us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.550997 (+242182us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.551372 (+   375us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.552733 (+  1361us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.552833 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.552969 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9612,"prepare.queue_time_us":949957,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":5010,"raft.queue_time_us":4515,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":9726,"thread_start_us":1197870,"threads_started":2}
I20260809 23:20:01.557693 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.481012 31147 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.533756 31150 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.562817 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:01.566360 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:01.615653 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49421us
I20260809 23:20:01.569813 24539 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260809 23:20:01.574678 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:01.576264 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.576682 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:01.576212 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.617403 24539 pending_rounds.cc:76] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 10
I20260809 23:20:01.496488 30998 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.618080 24539 pending_rounds.cc:97] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260809 23:20:01.618630 24539 raft_consensus.cc:2866] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:01.571715 30991 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:01.597802 24545 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260809 23:20:01.628945 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66207.6us
I20260809 23:20:01.629268 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.518795 30920 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.630455 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53830.5us
I20260809 23:20:01.620208 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.623029 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1001) took 1320 ms. Trace:
I20260809 23:20:01.587257 30940 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:01.627470 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62337.5us
I20260809 23:20:01.578866 31125 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.576725 30647 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:01.638768 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:01.636175 24539 rpcz_store.cc:276] 0809 23:20:00.303066 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.839262 (+536196us) service_pool.cc:227] Handling call
0809 23:20:00.931095 (+ 91833us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.622406 (+691311us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.622980 (+   574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:01.645530 24540 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:20:01.553771 31127 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.599764 30984 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.653080 24498 leader_election.cc:398] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:01.677904 24498 leader_election.cc:401] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260809 23:20:01.653663 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.580235 31270 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.608834 31267 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:01.682000 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.689826 30449 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.614168 31027 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:01.690862 31282 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.684804 31176 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260809 23:20:01.692117 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2738) took 1617 ms. Trace:
I20260809 23:20:01.692571 24535 rpcz_store.cc:276] 0809 23:20:00.075121 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.589596 (+514475us) service_pool.cc:227] Handling call
0809 23:20:00.590056 (+   460us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.639182 (+ 49126us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.639230 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.639817 (+   587us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.352141 (+712324us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:01.352196 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.352287 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.691899 (+339612us) raft_consensus.cc:1671] finished
0809 23:20:01.691922 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.692057 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48556,"raft.queue_time_us":49944,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":34712,"thread_start_us":711972,"threads_started":1,"wal-append.queue_time_us":933711}
I20260809 23:20:01.693595 30805 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.601505 30567 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260809 23:20:01.694258 30805 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:01.694660 30805 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:01.694918 30567 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:01.695051 30805 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:01.695415 30567 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:01.692232 30841 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.915s	user 0.006s	sys 0.006s
I20260809 23:20:01.699268 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1242) took 2467 ms. Trace:
I20260809 23:20:01.575083 31132 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.703904 25370 rpcz_store.cc:276] 0809 23:19:59.231616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.070786 (+1839170us) service_pool.cc:227] Handling call
0809 23:20:01.071170 (+   384us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.405651 (+334481us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.699021 (+293370us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540064c8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:01.699211 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":668951424}
I20260809 23:20:01.561415 30948 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.619585 31290 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:20:01.620298 31237 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.638998 30497 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:01.639508 30764 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.696295 30567 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.620137 30845 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:01.715924 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 81413.5us
I20260809 23:20:01.702333 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:01.723037 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:01.665560 30878 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:01.743608 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:01.744910 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77573.1us
I20260809 23:20:01.746181 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:01.733292 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51419.5us
W20260809 23:20:01.697424 31054 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.742s	user 0.013s	sys 0.000s
I20260809 23:20:01.699349 31026 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:01.679566 30940 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:01.749284 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:01.758404 25370 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:01.627727 31150 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.750458 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80381.4us
I20260809 23:20:01.614755 31102 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.782169 24323 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:20:01.615072 31185 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.783384 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.784668 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.703635 30920 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:01.738873 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:01.759840 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:01.726462 30497 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:01.786590 25370 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 8: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:01.787547 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:01.767805 31291 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
I20260809 23:20:01.751206 31290 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:01.770747 31289 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.238s	user 0.001s	sys 0.001s
I20260809 23:20:01.722484 30841 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.771236 30647 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.772637 31032 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:01.689862 30984 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.067s
I20260809 23:20:01.785904 30567 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:01.744159 30764 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.776958 31081 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.905s	user 0.011s	sys 0.000s
I20260809 23:20:01.678700 30934 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.777187 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:01.777658 31264 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260809 23:20:01.779247 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:01.780135 31127 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.745088 31083 raft_consensus.cc:2741] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260809 23:20:01.756256 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:01.856292 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77127.8us
W20260809 23:20:01.788730 31275 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260809 23:20:01.703769 31296 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.086s
I20260809 23:20:01.791146 24423 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:01.768596 31302 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
W20260809 23:20:01.793448 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 153432us
I20260809 23:20:01.796698 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:01.834388 24534 pending_rounds.cc:76] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:01.869725 24534 pending_rounds.cc:97] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:20:01.870107 24534 raft_consensus.cc:2866] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:01.714223 31267 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.791453 30497 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:01.796399 30940 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:01.854732 30860 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.685s	user 0.012s	sys 0.000s
I20260809 23:20:01.875706 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:01.745038 31237 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.797363 31295 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
I20260809 23:20:01.703706 30877 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:01.837750 31310 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260809 23:20:01.846903 25373 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:01.847409 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:01.889761 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.752784 31054 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:01.780295 30991 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.856758 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 100597us
I20260809 23:20:01.858494 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:01.737062 31136 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.865687 24323 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:01.847406 31038 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:01.933408 31147 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:01.933972 30841 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:01.934242 31059 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:01.886588 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:01.935997 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:01.891571 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2345) took 2286 ms (client timeout 2230 ms). Trace:
I20260809 23:20:01.891857 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:01.855046 30819 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:01.864641 31081 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:01.864377 31248 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:01.911556 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.865239 30845 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:01.891301 31111 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:01.873668 31305 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:01.941222 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2822) took 2177 ms. Trace:
I20260809 23:20:01.945461 24430 rpcz_store.cc:276] 0809 23:19:59.763412 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.267296 (+503884us) service_pool.cc:227] Handling call
0809 23:20:00.267613 (+   317us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:00.361769 (+ 94156us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.361813 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:00.362809 (+   996us) log.cc:842] Serialized 52 byte log entry
0809 23:20:01.351226 (+988417us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:01.351280 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.351379 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.941029 (+589650us) raft_consensus.cc:1671] finished
0809 23:20:01.941049 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.941169 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":4706,"thread_start_us":903730,"threads_started":1,"wal-append.queue_time_us":1088478}
I20260809 23:20:01.836601 31290 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:01.937187 24313 rpcz_store.cc:269] 0809 23:19:59.605105 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.022205 (+417100us) service_pool.cc:227] Handling call
0809 23:20:01.891345 (+1869140us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:01.891501 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2165061888}
I20260809 23:20:01.937702 25370 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:01.951215 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:01.951411 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93338.1us
I20260809 23:20:01.952399 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:01.938102 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1175) took 3646 ms. Trace:
I20260809 23:20:01.939718 24545 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:01.936695 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:01.953329 25363 rpcz_store.cc:276] 0809 23:19:58.291497 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.440489 (+2148992us) service_pool.cc:227] Handling call
0809 23:20:00.440944 (+   455us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.441375 (+   431us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.441415 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.506765 (+ 65350us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.388420 (+881655us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:01.388475 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.388565 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.937882 (+549317us) raft_consensus.cc:1671] finished
0809 23:20:01.937906 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.938038 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5621,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":41775,"thread_start_us":881145,"threads_started":1,"wal-append.queue_time_us":969877}
I20260809 23:20:01.949056 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2342) took 2305 ms. Trace:
I20260809 23:20:01.956746 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.957952 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.958505 24320 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:01.960253 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:01.962090 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2735) took 2013 ms. Trace:
I20260809 23:20:01.955441 24422 rpcz_store.cc:276] 0809 23:19:59.643797 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.129248 (+485451us) service_pool.cc:227] Handling call
0809 23:20:00.129678 (+   430us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.130151 (+   473us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.130191 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.130710 (+   519us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.319782 (+1189072us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:01.563506 (+243724us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.868500 (+304994us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.868720 (+   220us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.948729 (+ 80009us) raft_consensus.cc:1671] finished
0809 23:20:01.948750 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.948988 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30236,"prepare.queue_time_us":295832,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":260136,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":26059,"thread_start_us":1644582,"threads_started":3,"wal-append.queue_time_us":1402751}
I20260809 23:20:01.962762 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.963526 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.962472 24541 rpcz_store.cc:276] 0809 23:19:59.949180 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.568822 (+619642us) service_pool.cc:227] Handling call
0809 23:20:00.772121 (+203299us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.774355 (+  2234us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.774400 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.775019 (+   619us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.413304 (+638285us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:01.413361 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.413439 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:01.961886 (+548447us) raft_consensus.cc:1671] finished
0809 23:20:01.961910 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:01.962032 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1732,"raft.queue_time_us":16006,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":30553,"thread_start_us":637937,"threads_started":1,"wal-append.queue_time_us":777198}
I20260809 23:20:01.963685 24430 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:01.959054 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:01.968205 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:01.969651 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:01.969841 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:01.888195 31032 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260809 23:20:01.974606 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:01.976188 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:01.977445 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:01.977617 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.888037 30860 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.980583 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:02.003149 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:01.974606 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:01.886519 30940 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.006916 24313 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:02.007448 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:02.007651 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.980386 25370 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:02.004009 24541 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:02.005393 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.009857 24430 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:02.010952 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.981405 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.933449 31278 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:02.011739 24430 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:02.014505 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.014537 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.023916 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:02.026429 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.026562 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.028194 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.030913 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:02.062485 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.063215 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:02.037539 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.162016 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:01.959564 31111 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.064036 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.143080 31102 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.133075 31318 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:02.144060 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:01.953359 30845 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.197826 24422 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:02.029742 24430 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:20:02.037676 31323 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:20:01.949286 31248 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.144315 30991 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:01.943929 30819 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:01.948647 30652 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260809 23:20:02.075450 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.127663 24164 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 8 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:01.979250 30984 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:02.040625 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.245905 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:02.031504 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2783) took 2407 ms. Trace:
I20260809 23:20:02.246842 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.063041 31125 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:02.182330 25363 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:20:02.013959 31339 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260809 23:20:02.132004 31264 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260809 23:20:02.037475 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.078475 31185 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.057240 31322 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.199s
I20260809 23:20:01.955719 31305 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.144768 30940 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.035081 30994 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:02.271215 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:02.274195 25363 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:02.246912 24314 rpcz_store.cc:276] 0809 23:19:59.623827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.207679 (+1583852us) service_pool.cc:227] Handling call
0809 23:20:01.224395 (+ 16716us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.284140 (+ 59745us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.284186 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:01.284239 (+    53us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.284290 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.031243 (+746953us) spinlock_profiling.cc:243] Waited 59.5 ms on lock 0x7b5400253ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.031302 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.031443 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136853248}
I20260809 23:20:02.253744 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.191181 31150 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:02.199596 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 162148us
I20260809 23:20:02.190614 31296 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:02.245453 30984 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:02.218524 30991 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:02.264803 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.197686 31334 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260809 23:20:02.208179 31032 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:02.252324 31282 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:02.302649 31296 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.277200 24422 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:02.221297 30948 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.259191 30764 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.303304 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:02.271582 30994 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:02.304494 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.307009 24314 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:02.279721 31185 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:02.313937 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 129428us
I20260809 23:20:02.222416 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.294883 30991 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:02.324020 30845 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.324663 30845 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:02.329144 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:02.332401 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.333278 24423 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:02.289718 30940 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.321213 24430 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:02.295289 30984 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:02.311154 31147 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
W20260809 23:20:02.352849 31320 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.376s	user 0.001s	sys 0.000s
I20260809 23:20:02.345844 31290 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.288517 31237 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.280562 30449 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.316696 30994 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:02.348407 30805 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.006s
I20260809 23:20:02.346838 30948 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:02.368913 31330 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260809 23:20:02.371937 31313 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260809 23:20:02.360284 31147 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:02.362792 30860 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.373713 31346 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:20:02.334862 31185 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:02.387424 30900 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.036s	user 0.013s	sys 0.000s
W20260809 23:20:02.376765 31270 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.020s	sys 0.032s
I20260809 23:20:02.359264 30940 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.486438 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:02.397797 31272 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260809 23:20:02.534577 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2358) took 1138 ms. Trace:
I20260809 23:20:02.535269 24422 rpcz_store.cc:276] 0809 23:20:01.396464 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.312235 (+915771us) service_pool.cc:227] Handling call
0809 23:20:02.534448 (+222213us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:02.396018 31270 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.020s	sys 0.033s
W20260809 23:20:02.508421 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.004s	sys 0.006s
I20260809 23:20:02.511149 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:02.515291 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1099) took 3218 ms. Trace:
I20260809 23:20:02.525358 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1177) took 4213 ms. Trace:
I20260809 23:20:02.374028 30805 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.530052 30860 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.399735 31111 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.372988 30449 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:02.563022 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52471.7us
I20260809 23:20:02.564801 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:02.566524 30449 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:02.389611 30920 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.001s
I20260809 23:20:02.421227 31147 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.573446 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1272) took 2047 ms. Trace:
W20260809 23:20:02.507401 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.001s	sys 0.009s
W20260809 23:20:02.402590 31287 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.959s	user 0.002s	sys 0.000s
I20260809 23:20:02.592392 31292 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:02.592945 31292 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:02.593364 31292 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:02.396804 31338 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260809 23:20:02.593477 31290 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:02.401113 31347 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260809 23:20:02.599036 30819 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.537655 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:02.553700 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.005s	sys 0.007s
I20260809 23:20:02.558745 24423 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:02.560354 30805 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.561034 30942 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.147s	user 0.012s	sys 0.001s
I20260809 23:20:02.531618 30900 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.567186 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2344) took 2962 ms. Trace:
W20260809 23:20:02.568715 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2342) took 2964 ms (client timeout 1783 ms). Trace:
I20260809 23:20:02.574860 31334 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.575977 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:02.486491 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.000s	sys 0.010s
I20260809 23:20:02.545513 25371 rpcz_store.cc:276] 0809 23:19:59.296683 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.245178 (+1948495us) service_pool.cc:227] Handling call
0809 23:20:01.290944 (+ 45766us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:01.295700 (+  4756us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.295728 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:01.307062 (+ 11334us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.554192 (+247130us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:01.845459 (+291267us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.845662 (+   203us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.845748 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.514562 (+668814us) raft_consensus.cc:1671] finished
0809 23:20:02.514624 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.515020 (+   396us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":843260,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":19158,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":11493,"thread_start_us":447736,"threads_started":2,"wal-append.queue_time_us":937851}
I20260809 23:20:02.580209 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2344) took 2857 ms. Trace:
I20260809 23:20:02.581688 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2828) took 1355 ms. Trace:
I20260809 23:20:02.545794 25375 rpcz_store.cc:276] 0809 23:19:58.311805 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.441808 (+2130003us) service_pool.cc:227] Handling call
0809 23:20:00.442206 (+   398us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:01.519769 (+1077563us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.519799 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:01.520372 (+   573us) log.cc:842] Serialized 52 byte log entry
0809 23:20:01.839182 (+318810us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:01.839234 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.839318 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.524759 (+685441us) raft_consensus.cc:1671] finished
0809 23:20:02.524800 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.525161 (+   361us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61058,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":17505,"spinlock_wait_cycles":1856896,"thread_start_us":1366397,"threads_started":2,"wal-append.queue_time_us":840561}
I20260809 23:20:02.549022 31363 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.610767 24425 rpcz_store.cc:276] 0809 23:20:01.226395 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.227007 (+   612us) service_pool.cc:227] Handling call
0809 23:20:02.581222 (+1354215us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.581507 (+   285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467563776}
I20260809 23:20:02.611498 30942 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.611802 31363 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.612473 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.612789 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.613550 24425 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:02.614323 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.614550 31363 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.583138 30940 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.615056 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.000s	sys 0.011s
I20260809 23:20:02.586691 31185 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.615238 31334 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:02.591554 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.001s	sys 0.010s
W20260809 23:20:02.616730 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 657711us
I20260809 23:20:02.617015 24425 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260809 23:20:02.617951 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2834) took 1333 ms. Trace:
I20260809 23:20:02.598578 24312 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:02.618583 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.585734 30860 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.608799 24309 rpcz_store.cc:276] 0809 23:19:59.604840 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.328314 (+723474us) service_pool.cc:227] Handling call
0809 23:20:00.351208 (+ 22894us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.171562 (+820354us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.566575 (+1395013us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.566971 (+   396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1628969344}
I20260809 23:20:02.628791 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.629098 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.629845 30860 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.630100 24309 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:20:02.631568 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2740) took 1307 ms. Trace:
I20260809 23:20:02.632017 24540 rpcz_store.cc:276] 0809 23:20:01.323995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.652059 (+328064us) service_pool.cc:227] Handling call
0809 23:20:01.652371 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:01.653378 (+  1007us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.653416 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:01.653843 (+   427us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.766796 (+112953us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:01.766850 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.766982 (+   132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.631000 (+864018us) raft_consensus.cc:1671] finished
0809 23:20:02.631045 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.631343 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31318,"raft.queue_time_us":15443,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":17203,"thread_start_us":53593,"threads_started":1,"wal-append.queue_time_us":303292}
I20260809 23:20:02.635452 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.636567 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:02.620296 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.621497 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2617) took 1382 ms. Trace:
W20260809 23:20:02.639938 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75558.6us
I20260809 23:20:02.640264 24545 rpcz_store.cc:276] 0809 23:20:01.239220 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.632654 (+393434us) service_pool.cc:227] Handling call
0809 23:20:01.632916 (+   262us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.961969 (+329053us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.621013 (+659044us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.621328 (+   315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":704961792}
I20260809 23:20:02.644058 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.645000 24545 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:02.646484 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:02.610697 24424 rpcz_store.cc:276] 0809 23:19:59.732108 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.255899 (+523791us) service_pool.cc:227] Handling call
0809 23:20:00.256178 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.202834 (+946656us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.579586 (+1376752us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54003ebcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.579972 (+   386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2169764736}
I20260809 23:20:02.622545 30900 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.624102 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1247) took 2500 ms (client timeout 2188 ms). Trace:
I20260809 23:20:02.624894 24425 rpcz_store.cc:276] 0809 23:20:01.284880 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.615763 (+1330883us) service_pool.cc:227] Handling call
0809 23:20:02.616453 (+   690us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:02.617638 (+  1185us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.617881 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:02.609198 24316 rpcz_store.cc:269] 0809 23:19:59.604265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.884625 (+280360us) service_pool.cc:227] Handling call
0809 23:20:02.568111 (+2683486us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.568447 (+   336us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2460381056}
I20260809 23:20:02.630635 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.619475 24422 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:02.590710 24321 rpcz_store.cc:276] 0809 23:20:00.525773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.483476 (+957703us) service_pool.cc:227] Handling call
0809 23:20:02.572037 (+1088561us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.573189 (+  1152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896615296}
I20260809 23:20:02.641464 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.634611 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:02.618517 31334 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.657192 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:02.658419 31334 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:20:02.659129 31334 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.660137 31334 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:02.661787 31132 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.619004 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.655072 31362 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:02.655843 24323 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:02.652679 25366 rpcz_store.cc:269] 0809 23:20:00.123234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.332318 (+209084us) service_pool.cc:227] Handling call
0809 23:20:02.623629 (+2291311us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.623866 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":671955712}
I20260809 23:20:02.668402 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.669736 31152 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:02.674973 30847 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.676028 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57277.9us
W20260809 23:20:02.677205 31255 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.301s	user 0.000s	sys 0.002s
I20260809 23:20:02.678050 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.679221 31363 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260809 23:20:02.680524 31258 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.304s	user 0.000s	sys 0.003s
W20260809 23:20:02.684224 31259 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.309s	user 0.005s	sys 0.000s
I20260809 23:20:02.686148 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2340) took 3080 ms. Trace:
I20260809 23:20:02.687433 31363 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:02.688489 31175 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.689116 31175 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:02.689630 31054 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:02.686583 24428 rpcz_store.cc:276] 0809 23:19:59.605878 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.907066 (+301188us) service_pool.cc:227] Handling call
0809 23:19:59.907478 (+   412us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:19:59.908300 (+   822us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:59.908333 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:19:59.920367 (+ 12034us) log.cc:842] Serialized 26 byte log entry
0809 23:20:00.108380 (+188013us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b18003666a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:20:00.940332 (+831952us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:01.429001 (+488669us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.604239 (+175238us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.604749 (+   510us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.683176 (+1078427us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:02.685733 (+  2557us) raft_consensus.cc:1671] finished
0809 23:20:02.685752 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.685942 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13591,"prepare.queue_time_us":580974,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":40,"raft.queue_time_us":168059,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":41507,"spinlock_wait_cycles":253037568,"thread_start_us":1462809,"threads_started":3,"wal-append.queue_time_us":1191781}
I20260809 23:20:02.690902 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.693236 24428 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:02.694813 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.626098 30942 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.699046 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:02.665887 31362 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:02.705946 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
W20260809 23:20:02.708002 25372 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.000s	sys 0.013s
I20260809 23:20:02.703750 31362 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:02.711349 25372 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.000s	sys 0.013s
I20260809 23:20:02.714414 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2836) took 1425 ms. Trace:
W20260809 23:20:02.714819 31247 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.772s	user 0.003s	sys 0.001s
I20260809 23:20:02.715198 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2345) took 2968 ms. Trace:
I20260809 23:20:02.714983 24422 rpcz_store.cc:276] 0809 23:20:01.289236 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.664198 (+1374962us) service_pool.cc:227] Handling call
0809 23:20:02.708491 (+ 44293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.713871 (+  5380us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.713920 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.714061 (+   141us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:02.714106 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.714177 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.714318 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:02.710129 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:02.713083 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:02.720160 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2837) took 1403 ms. Trace:
I20260809 23:20:02.720496 24425 rpcz_store.cc:276] 0809 23:20:01.317209 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.699164 (+1381955us) service_pool.cc:227] Handling call
0809 23:20:02.712152 (+ 12988us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.719863 (+  7711us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.719889 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.719946 (+    57us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:02.719990 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.720043 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.720120 (+    77us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:02.715641 24431 rpcz_store.cc:276] 0809 23:19:59.747069 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.263579 (+516510us) service_pool.cc:227] Handling call
0809 23:20:00.263959 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.264329 (+   370us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.264461 (+   132us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.274914 (+ 10453us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.243495 (+968581us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:01.484730 (+241235us) log.cc:842] Serialized 14 byte log entry
0809 23:20:02.714313 (+1229583us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b18001899a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7
0809 23:20:02.714578 (+   265us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.714687 (+   109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.714761 (+    74us) raft_consensus.cc:1671] finished
0809 23:20:02.714866 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.715037 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":298628,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":4723,"raft.queue_time_us":154121,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":48893,"spinlock_wait_cycles":8263552,"thread_start_us":1267024,"threads_started":3,"wal-append.queue_time_us":1915376}
I20260809 23:20:02.729439 31292 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.734603 25371 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:02.735898 24546 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:02.743711 24546 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:02.747452 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:02.755610 31292 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.776381 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1108) took 1112 ms. Trace:
I20260809 23:20:02.777483 25375 rpcz_store.cc:276] 0809 23:20:01.667233 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.641348 (+974115us) service_pool.cc:227] Handling call
0809 23:20:02.647473 (+  6125us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.724772 (+ 77299us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.724804 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.739512 (+ 14708us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:02.773201 (+ 33689us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.773320 (+   119us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.773508 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21703,"raft.queue_time_us":16064,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":1913}
I20260809 23:20:02.774994 24323 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:02.781852 31336 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:02.782407 31336 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:02.782887 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 82238.8us
I20260809 23:20:02.783108 30805 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:02.786175 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:02.786629 30497 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.896s	user 0.021s	sys 0.001s
W20260809 23:20:02.787696 31378 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260809 23:20:02.787317 30497 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.790351 24431 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:02.792608 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 879) took 1186 ms. Trace:
I20260809 23:20:02.792851 24431 rpcz_store.cc:276] 0809 23:20:01.606486 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.732169 (+1125683us) service_pool.cc:227] Handling call
0809 23:20:02.774895 (+ 42726us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.792407 (+ 17512us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.792565 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:02.794757 24540 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260809 23:20:02.782855 31336 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:02.800521 31111 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.801311 31111 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.801913 31267 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.808252 31127 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.809154 31127 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.812472 24430 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:02.809078 30567 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.964s	user 0.011s	sys 0.013s
I20260809 23:20:02.814549 31127 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:02.772048 30984 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.819084 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2354) took 1488 ms. Trace:
I20260809 23:20:02.820938 24428 rpcz_store.cc:276] 0809 23:20:01.319231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.709725 (+1390494us) service_pool.cc:227] Handling call
0809 23:20:02.773504 (+ 63779us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.804642 (+ 31138us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.804672 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.807236 (+  2564us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:02.807290 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.807362 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.807442 (+    80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:02.821056 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.822968 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.810505 30942 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:02.830256 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:02.832650 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 884955us
I20260809 23:20:02.838258 24540 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260809 23:20:02.809247 30994 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:02.839358 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 893127us
I20260809 23:20:02.850318 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.854401 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.859315 31316 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:02.862344 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.863083 31049 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:02.863791 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1107) took 1227 ms. Trace:
I20260809 23:20:02.864231 25371 rpcz_store.cc:276] 0809 23:20:01.636808 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.619874 (+983066us) service_pool.cc:227] Handling call
0809 23:20:02.863554 (+243680us) spinlock_profiling.cc:243] Waited 94 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.863704 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216173952}
I20260809 23:20:02.865428 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.867089 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 8.
I20260809 23:20:02.867909 31296 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.868183 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:02.869176 25375 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:02.870098 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.871646 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:02.873857 31027 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:02.874549 31027 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:02.879941 30878 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:02.880334 30994 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:02.858698 24164 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 7, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 7 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:02.866767 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:02.897635 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49177.8us
W20260809 23:20:02.903854 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 237987us
I20260809 23:20:02.911469 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.912268 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 7.
I20260809 23:20:02.918633 24535 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:02.922646 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.921370 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.925431 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.930253 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:02.934132 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:02.897243 30805 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:02.939802 24314 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:02.951742 24428 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:02.955215 31125 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.959793 31125 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.965503 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2604) took 3425 ms. Trace:
I20260809 23:20:02.965898 24542 rpcz_store.cc:276] 0809 23:19:59.539765 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.329035 (+789270us) service_pool.cc:227] Handling call
0809 23:20:00.360407 (+ 31372us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.840026 (+479619us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.964975 (+2124949us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.965305 (+   330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199106944}
I20260809 23:20:02.967785 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1169) took 5028 ms. Trace:
I20260809 23:20:02.933861 31296 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:02.968171 25369 rpcz_store.cc:276] 0809 23:19:57.939213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:57.998056 (+ 58843us) service_pool.cc:227] Handling call
0809 23:19:58.039532 (+ 41476us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.334062 (+3294530us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.967245 (+1633183us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.967577 (+   332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7472657664}
I20260809 23:20:02.968802 31296 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.971655 31175 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.974005 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:02.975756 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1181) took 3360 ms (client timeout 2581 ms). Trace:
W20260809 23:20:02.976156 25373 rpcz_store.cc:269] 0809 23:19:59.614878 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.067880 (+453002us) service_pool.cc:227] Handling call
0809 23:20:02.975390 (+2907510us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:02.975564 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3984605952}
I20260809 23:20:02.976913 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2829) took 1750 ms. Trace:
I20260809 23:20:02.977267 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.951397 30994 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260809 23:20:02.955729 30878 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:02.954067 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:02.979506 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:02.955997 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:02.952749 31260 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.380s	user 0.007s	sys 0.005s
I20260809 23:20:02.914305 31362 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:02.974593 31175 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:02.977187 31059 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.042s	user 0.011s	sys 0.012s
I20260809 23:20:02.979115 24543 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.011675 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.996358 25363 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:02.977654 31278 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:03.017012 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1099) took 3546 ms. Trace:
W20260809 23:20:03.017266 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96426.2us
I20260809 23:20:03.017515 24317 rpcz_store.cc:276] 0809 23:19:59.464287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.528283 (+ 63996us) service_pool.cc:227] Handling call
0809 23:19:59.608514 (+ 80231us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:19:59.637447 (+ 28933us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:19:59.637510 (+    63us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:19:59.757349 (+119839us) log.cc:842] Serialized 546 byte log entry
0809 23:20:00.247797 (+490448us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.197718 (+949921us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.464357 (+266639us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.464552 (+   195us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.464626 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.710995 (+1246369us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:02.973394 (+262399us) spinlock_profiling.cc:243] Waited 8.94 ms on lock 0x7b240019bae8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:20:02.978887 (+  5493us) raft_consensus.cc:1671] finished
0809 23:20:02.978909 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.011068 (+ 32159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":505,"prepare.queue_time_us":134015,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":51599,"raft.run_cpu_time_us":535,"raft.run_wall_time_us":67587,"spinlock_wait_cycles":121992960,"thread_start_us":687925,"threads_started":2,"wal-append.queue_time_us":1011262}
I20260809 23:20:03.021400 31339 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:03.024149 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:03.025524 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.026854 24539 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:03.027647 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2831) took 1784 ms. Trace:
I20260809 23:20:03.028738 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:02.983482 31111 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.028056 24430 rpcz_store.cc:276] 0809 23:20:01.243217 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.486767 (+1243550us) service_pool.cc:227] Handling call
0809 23:20:02.487655 (+   888us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.812953 (+325298us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.027355 (+214402us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:03.027544 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731233536}
W20260809 23:20:02.998628 31260 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.562s	user 0.007s	sys 0.006s
W20260809 23:20:03.025537 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45309.6us
I20260809 23:20:03.028136 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:03.000308 30497 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.011181 30900 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:03.053213 24424 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11.
I20260809 23:20:03.048538 25371 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:03.055356 25373 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260809 23:20:03.056239 24316 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:03.056988 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:03.041956 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:03.057657 25375 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:03.060738 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1185) took 1280 ms. Trace:
I20260809 23:20:03.061250 25371 rpcz_store.cc:276] 0809 23:20:01.774194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.979090 (+1204896us) service_pool.cc:227] Handling call
0809 23:20:03.044691 (+ 65601us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.054212 (+  9521us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.054555 (+   343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:03.062920 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1190) took 1287 ms. Trace:
I20260809 23:20:03.063529 25373 rpcz_store.cc:276] 0809 23:20:01.775619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.978309 (+1202690us) service_pool.cc:227] Handling call
0809 23:20:03.037518 (+ 59209us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:03.062653 (+ 25135us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.062869 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:03.052942 31263 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.517s	user 0.005s	sys 0.000s
I20260809 23:20:03.067380 24544 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260809 23:20:03.078783 24544 pending_rounds.cc:76] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 7
I20260809 23:20:03.067793 30567 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:03.068143 31384 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.252s	user 0.000s	sys 0.002s
I20260809 23:20:03.068406 30948 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:03.069278 31027 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.070494 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:03.071113 31288 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.600s	user 0.007s	sys 0.002s
I20260809 23:20:03.075049 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1098) took 3718 ms. Trace:
I20260809 23:20:03.076088 30878 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.065465 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:03.101253 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8234758088365197s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260809 23:20:03.102159 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9340656348504126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
W20260809 23:20:03.102526 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1057486690115184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21880000000000002s for a response
I20260809 23:20:03.052839 30900 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:03.103214 30900 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.103457 31059 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:03.104053 31059 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.065948 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:02.978549 24429 rpcz_store.cc:276] 0809 23:20:01.226814 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.229824 (+  3010us) service_pool.cc:227] Handling call
0809 23:20:01.230149 (+   325us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.314875 (+ 84726us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:01.477548 (+162673us) log.cc:842] Serialized 14 byte log entry
0809 23:20:02.877636 (+1400088us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b1800412ae0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:20:02.951617 (+ 73981us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.975041 (+ 23424us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:02.975232 (+   191us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.976509 (+  1277us) spinlock_profiling.cc:243] Waited 84.4 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.976554 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:02.976721 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68130,"prepare.queue_time_us":264156,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":56022,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":29082,"spinlock_wait_cycles":210623616,"thread_start_us":276957,"threads_started":2,"wal-append.queue_time_us":412710}
I20260809 23:20:03.083245 31187 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
W20260809 23:20:03.098284 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 78768.5us
I20260809 23:20:03.094159 25368 rpcz_store.cc:276] 0809 23:19:59.296326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.232164 (+1935838us) service_pool.cc:227] Handling call
0809 23:20:01.232465 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:01.472050 (+239585us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.974803 (+1502753us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:03.014660 (+ 39857us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":478993152}
I20260809 23:20:03.055806 31026 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:03.083568 30567 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:03.069229 31083 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260809 23:20:03.083854 30948 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:03.056030 31296 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.093048 24544 pending_rounds.cc:97] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260809 23:20:03.126274 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:03.128400 25371 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:03.093822 30497 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:03.130235 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.077966 30984 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.134150 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.124890 24544 raft_consensus.cc:2866] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:03.142225 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:03.083514 31125 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.143293 24430 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:03.140542 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:03.125025 30948 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.121642 30567 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.151738 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:03.152957 24314 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:03.160728 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:03.165216 25375 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:03.135771 30497 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:03.180569 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53790.8us
I20260809 23:20:03.179674 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:03.179646 24429 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:03.185232 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:03.176290 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 883) took 1554 ms. Trace:
I20260809 23:20:03.181660 25368 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:03.196331 25373 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:03.199434 24321 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260809 23:20:03.199939 30991 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.762s	user 0.018s	sys 0.005s
I20260809 23:20:03.200433 30991 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.201624 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1237) took 4926 ms. Trace:
I20260809 23:20:03.172063 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.159292 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:03.195997 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:03.212494 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 371786us
I20260809 23:20:03.218114 24541 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:03.201992 25365 rpcz_store.cc:276] 0809 23:19:58.274711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.124232 (+1849521us) service_pool.cc:227] Handling call
0809 23:20:00.124590 (+   358us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.124817 (+   227us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.124852 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.125317 (+   465us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.068158 (+942841us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:01.068214 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.068314 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:02.874143 (+1805829us) spinlock_profiling.cc:243] Waited 97.7 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:03.201111 (+326968us) raft_consensus.cc:1671] finished
0809 23:20:03.201140 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.201438 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":224656256,"thread_start_us":942440,"threads_started":1,"wal-append.queue_time_us":1190735}
W20260809 23:20:03.227905 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 120187us
I20260809 23:20:03.221009 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1116) took 1420 ms. Trace:
I20260809 23:20:03.220333 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.223593 24424 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:03.203774 25370 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:03.247579 25373 rpcz_store.cc:276] 0809 23:20:01.800529 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.064985 (+1264456us) service_pool.cc:227] Handling call
0809 23:20:03.143463 (+ 78478us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.196764 (+ 53301us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.220931 (+ 24167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:03.144917 31059 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.250345 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:03.179451 31390 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.264s	user 0.002s	sys 0.000s
I20260809 23:20:03.214255 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1121) took 1218 ms. Trace:
I20260809 23:20:03.217722 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 998) took 3439 ms. Trace:
I20260809 23:20:03.218883 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2343) took 3562 ms. Trace:
I20260809 23:20:03.220352 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.244863 24541 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:03.251058 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:03.205021 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1097) took 4023 ms (client timeout 2224 ms). Trace:
I20260809 23:20:03.187201 30497 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:03.207065 31147 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:20:03.229506 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2732) took 3281 ms. Trace:
I20260809 23:20:03.303321 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:20:03.279102 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90208.8us
I20260809 23:20:03.164258 30819 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:03.252342 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56770.4us
I20260809 23:20:03.265985 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2840) took 1635 ms. Trace:
I20260809 23:20:03.300197 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:03.297581 24311 rpcz_store.cc:269] 0809 23:19:59.181340 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:59.185913 (+  4573us) service_pool.cc:227] Handling call
0809 23:20:03.143008 (+3957095us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:03.204512 (+ 61504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4535390080}
I20260809 23:20:03.216163 24164 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:03.317528 24424 rpcz_store.cc:276] 0809 23:20:01.612961 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.222789 (+1609828us) service_pool.cc:227] Handling call
0809 23:20:03.223224 (+   435us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.247911 (+ 24687us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.248120 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:03.297269 24543 rpcz_store.cc:276] 0809 23:19:59.778535 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.370964 (+592429us) service_pool.cc:227] Handling call
0809 23:20:00.440653 (+ 69689us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.969904 (+2529251us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b58008b07b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:20:03.217297 (+247393us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:03.217327 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:03.217390 (+    63us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:03.217442 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.217530 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.217659 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21305,"spinlock_wait_cycles":1389700736}
I20260809 23:20:03.128070 31026 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:03.181418 31334 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.002s
I20260809 23:20:03.276404 25371 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:03.249977 31247 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260809 23:20:03.307757 24544 rpcz_store.cc:276] 0809 23:19:59.948328 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.561155 (+612827us) service_pool.cc:227] Handling call
0809 23:20:00.599699 (+ 38544us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.199618 (+2599919us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.228683 (+ 29065us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b54001fc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:03.229424 (+   741us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5615197440}
I20260809 23:20:03.311978 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:03.203949 31283 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:03.321818 24316 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:03.220464 31248 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.270718 30942 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:03.202852 30991 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.230247 30984 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:03.335641 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:03.344555 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.345144 24543 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:03.345378 24544 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:03.320029 31053 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:03.342154 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.338985 25321 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11.
I20260809 23:20:03.353386 25321 leader_election.cc:401] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260809 23:20:03.218525 31184 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:03.300768 31316 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:03.308038 31362 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:03.348403 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85906.3us
I20260809 23:20:03.351336 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.361999 25373 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:03.357751 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.215360 31336 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:03.367914 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51081.2us
I20260809 23:20:03.344566 31187 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:03.347746 31083 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:03.352608 31053 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:03.329532 31334 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.356959 30819 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:20:03.381574 30984 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:03.393368 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 267389us
I20260809 23:20:03.398149 24425 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:03.403872 31397 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260809 23:20:03.404095 30841 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:03.363692 31184 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.393589 31053 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:03.407653 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50042.2us
I20260809 23:20:03.408643 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.390262 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:03.406199 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 102506us
I20260809 23:20:03.397886 31083 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:03.364765 31059 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:03.416301 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2838) took 2075 ms. Trace:
I20260809 23:20:03.388540 31339 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:03.304443 24313 rpcz_store.cc:276] 0809 23:20:01.995692 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.479368 (+483676us) service_pool.cc:227] Handling call
0809 23:20:02.480160 (+   792us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:02.566574 (+ 86414us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.566618 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:02.567407 (+   789us) log.cc:842] Serialized 52 byte log entry
0809 23:20:02.625633 (+ 58226us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:02.632783 (+  7150us) log.cc:842] Serialized 14 byte log entry
0809 23:20:02.633337 (+   554us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.634495 (+  1158us) spinlock_profiling.cc:243] Waited 85.7 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.634557 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.213808 (+579251us) raft_consensus.cc:1671] finished
0809 23:20:03.213927 (+   119us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.214068 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6518,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":3748,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":320,"spinlock_wait_cycles":197077248,"thread_start_us":57746,"threads_started":1,"wal-append.queue_time_us":144753}
I20260809 23:20:03.422165 25366 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:03.398371 30991 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:03.417338 24422 rpcz_store.cc:276] 0809 23:20:01.340680 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.723843 (+1383163us) service_pool.cc:227] Handling call
0809 23:20:02.751421 (+ 27578us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:02.751822 (+   401us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.751851 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:02.752458 (+   607us) log.cc:842] Serialized 26 byte log entry
0809 23:20:02.780493 (+ 28035us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:02.780544 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.780644 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.416083 (+635439us) raft_consensus.cc:1671] finished
0809 23:20:03.416104 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.416238 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":27492,"threads_started":1,"wal-append.queue_time_us":83778}
I20260809 23:20:03.425793 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2841) took 1721 ms. Trace:
I20260809 23:20:03.428794 24429 rpcz_store.cc:276] 0809 23:20:01.704666 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.316386 (+1611720us) service_pool.cc:227] Handling call
0809 23:20:03.393743 (+ 77357us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.425425 (+ 31682us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.425452 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:03.425494 (+    42us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:03.425535 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.425602 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.425719 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:03.402216 30819 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:03.433270 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:03.395450 31248 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260809 23:20:03.441605 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.443215 31416 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:03.443481 31270 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.068s
I20260809 23:20:03.444175 30647 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:20:03.444404 24540 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:03.444599 30890 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260809 23:20:03.405838 31187 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.443238 31198 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:03.428759 31362 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:03.440109 31386 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.371s	user 0.000s	sys 0.002s
I20260809 23:20:03.297077 24435 rpcz_store.cc:276] 0809 23:19:59.656090 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.197103 (+541013us) service_pool.cc:227] Handling call
0809 23:20:00.237297 (+ 40194us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.322026 (+ 84729us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.322056 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.336379 (+ 14323us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.307088 (+970709us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:01.604003 (+296915us) log.cc:842] Serialized 14 byte log entry
0809 23:20:01.616567 (+ 12564us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:02.814983 (+1198416us) spinlock_profiling.cc:243] Waited 84.3 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:02.815064 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.070180 (+255116us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:03.194357 (+124177us) raft_consensus.cc:1671] finished
0809 23:20:03.194383 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.218717 (+ 24334us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11679,"prepare.queue_time_us":247491,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":4590,"raft.queue_time_us":78949,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":22248,"spinlock_wait_cycles":346138240,"thread_start_us":1174743,"threads_started":2,"wal-append.queue_time_us":1066309}
I20260809 23:20:03.456694 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.440510 24541 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.437752 30819 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:03.465476 31021 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:03.415344 31316 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:03.468976 24535 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:03.380240 31283 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:03.476511 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53398.4us
I20260809 23:20:03.473397 24498 leader_election.cc:398] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:20:03.477759 24498 leader_election.cc:401] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:03.479195 31337 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260809 23:20:03.482358 31175 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:03.483458 30629 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:03.484256 31026 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.484793 30629 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:03.485203 30629 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:03.485947 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2365) took 1178 ms. Trace:
I20260809 23:20:03.487149 24422 rpcz_store.cc:276] 0809 23:20:02.307486 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.435655 (+1128169us) service_pool.cc:227] Handling call
0809 23:20:03.473189 (+ 37534us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.485554 (+ 12365us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.485584 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:03.485648 (+    64us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:03.485698 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.485767 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.485880 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:03.490031 24435 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260809 23:20:03.490171 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 81647.5us
I20260809 23:20:03.472749 31021 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:03.501531 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:03.495325 24424 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:03.515194 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.506121 31026 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10.
I20260809 23:20:03.518085 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:03.524546 24429 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:03.468699 31198 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:03.506984 31021 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:03.509538 30984 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.462455 31416 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.541705 31316 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:03.542407 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 86516.8us
I20260809 23:20:03.543216 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1111) took 1835 ms. Trace:
I20260809 23:20:03.547297 24435 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260809 23:20:03.493389 30890 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260809 23:20:03.552098 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:03.539004 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.551540 24541 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:03.553932 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:03.554658 24316 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.194967 24431 rpcz_store.cc:276] 0809 23:20:01.607966 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.003747 (+1395781us) service_pool.cc:227] Handling call
0809 23:20:03.055681 (+ 51934us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.077088 (+ 21407us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:03.077382 (+   294us) log.cc:842] Serialized 14 byte log entry
0809 23:20:03.156325 (+ 78943us) log.cc:842] Serialized 14 byte log entry
0809 23:20:03.162182 (+  5857us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:03.162397 (+   215us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:03.162480 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.162559 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.162674 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31759,"prepare.queue_time_us":614,"raft.queue_time_us":39018,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":35570,"spinlock_wait_cycles":53846912,"thread_start_us":70700,"threads_started":1}
I20260809 23:20:03.555320 31416 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.559216 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.559427 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1125) took 1028 ms. Trace:
I20260809 23:20:03.559610 31187 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.560134 24546 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260809 23:20:03.551980 25363 rpcz_store.cc:276] 0809 23:20:01.707670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.274860 (+567190us) service_pool.cc:227] Handling call
0809 23:20:03.537490 (+1262630us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b5400631354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:03.543132 (+  5642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1151929216}
I20260809 23:20:03.559856 24321 rpcz_store.cc:276] 0809 23:20:02.531539 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.660692 (+129153us) service_pool.cc:227] Handling call
0809 23:20:03.163804 (+503112us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.199832 (+ 36028us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.559230 (+359398us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400243bd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:03.559366 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1130226432}
I20260809 23:20:03.513427 30900 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.526808 31334 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.573276 24430 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:03.486270 30629 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.526387 31027 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:03.533113 31339 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:03.569548 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.535249 31336 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.529995 31083 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.581106 25363 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:03.581759 24498 leader_election.cc:398] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:03.593039 24498 leader_election.cc:401] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:03.593182 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:03.553969 31277 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.114s	user 0.002s	sys 0.008s
I20260809 23:20:03.587492 24430 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:03.578513 31337 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:03.604570 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:03.581486 24316 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:03.609766 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 104918us
I20260809 23:20:03.614317 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:03.606885 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 157569us
I20260809 23:20:03.616324 25371 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:03.620565 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:03.608904 31081 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.026s	sys 0.011s
W20260809 23:20:03.624534 31081 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.026s	sys 0.012s
I20260809 23:20:03.628109 25371 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:03.633201 31322 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:03.633934 25371 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:03.634754 31362 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:03.635285 25371 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:03.636569 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.576256 30948 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.635267 31362 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:03.607208 31337 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:03.642001 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 129603us
I20260809 23:20:03.643311 31083 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:03.649484 31300 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:03.649968 31300 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:03.651595 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1188) took 1866 ms. Trace:
I20260809 23:20:03.651326 30890 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:03.613322 31027 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.659087 30890 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:03.659644 30890 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:03.661263 31027 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.651521 31152 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:20:03.663669 31152 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:03.664806 31152 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.668545 30948 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.653435 25369 rpcz_store.cc:276] 0809 23:20:01.774997 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.969307 (+1194310us) service_pool.cc:227] Handling call
0809 23:20:02.969710 (+   403us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:02.970225 (+   515us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.970255 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:03.006547 (+ 36292us) log.cc:842] Serialized 52 byte log entry
0809 23:20:03.193767 (+187220us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:03.193821 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.193916 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.641051 (+447135us) raft_consensus.cc:1671] finished
0809 23:20:03.641078 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.641237 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":906,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"thread_start_us":163773,"threads_started":1,"wal-append.queue_time_us":451689}
I20260809 23:20:03.682497 25369 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260809 23:20:03.683422 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58456.8us
I20260809 23:20:03.684489 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.686110 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.688704 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2348) took 3460 ms. Trace:
I20260809 23:20:03.692656 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.694028 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.689188 24427 rpcz_store.cc:276] 0809 23:20:00.228184 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.298009 (+ 69825us) service_pool.cc:227] Handling call
0809 23:20:00.298556 (+   547us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:00.300081 (+  1525us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:00.300125 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:00.300628 (+   503us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.219838 (+919210us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.219880 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.219943 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.506199 (+2286256us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:03.688167 (+181968us) raft_consensus.cc:1671] finished
0809 23:20:03.688194 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.688517 (+   323us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4175,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7022,"spinlock_wait_cycles":182268672,"thread_start_us":918597,"threads_started":1,"wal-append.queue_time_us":1139100}
I20260809 23:20:03.699705 25363 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:03.703404 31415 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260809 23:20:03.703826 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.705222 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.707350 31361 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:03.707902 31361 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:03.708256 31361 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:03.708567 31288 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.490s	user 0.004s	sys 0.000s
I20260809 23:20:03.709021 31413 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:03.709721 31413 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:03.652505 31337 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:03.726573 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 152892us
I20260809 23:20:03.732650 31278 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:03.734186 31278 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:20:03.735095 31278 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:03.735427 31278 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:03.736310 31278 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.738054 31278 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.738569 30497 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.650898 31300 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:03.748616 31300 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:03.711093 31413 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:03.752688 30567 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.001s
I20260809 23:20:03.753343 30567 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.762517 30819 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.764101 25365 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:03.759277 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73266.1us
I20260809 23:20:03.765938 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:03.697943 30629 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:03.725993 31125 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:03.767498 31125 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:03.767941 31125 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.699025 31267 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:20:03.759670 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 123902us
W20260809 23:20:03.762971 24314 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.000s
I20260809 23:20:03.774456 24314 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:03.734688 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.716143 31152 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:03.809952 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.812608 24541 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260809 23:20:03.765996 30819 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.813086 24541 pending_rounds.cc:76] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 7
I20260809 23:20:03.826960 24541 pending_rounds.cc:97] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260809 23:20:03.827307 24541 raft_consensus.cc:2866] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:03.848834 24430 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:03.852273 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57518.7us
I20260809 23:20:03.801393 31083 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.043s
I20260809 23:20:03.853070 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:03.870467 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2629) took 1921 ms. Trace:
I20260809 23:20:03.850066 30900 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.850802 30948 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:03.816797 31053 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:03.884155 30567 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.887781 30919 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:03.871186 24541 rpcz_store.cc:276] 0809 23:20:01.942621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.062966 (+120345us) service_pool.cc:227] Handling call
0809 23:20:02.162503 (+ 99537us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.217859 (+1055356us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54002e1e20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:03.854821 (+636962us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.863609 (+  8788us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1840053504}
I20260809 23:20:03.848186 30819 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:03.863023 31339 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:03.904352 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51375.3us
I20260809 23:20:03.906190 30567 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:03.898538 31021 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.914114 24540 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:03.894601 31248 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:20:03.910532 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 175985us
I20260809 23:20:03.897377 30647 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:20:03.912210 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:03.913120 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:03.865468 31059 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260809 23:20:03.916121 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:03.906392 31339 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.919544 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:03.906975 30567 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.936360 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.937342 30920 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:03.939441 30920 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:03.947719 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:03.951226 24422 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:03.948274 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1102) took 4483 ms. Trace:
I20260809 23:20:03.949766 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:03.899922 30948 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:03.952811 24312 rpcz_store.cc:276] 0809 23:19:59.465126 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:00.251396 (+786270us) service_pool.cc:227] Handling call
0809 23:20:01.892001 (+1640605us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.626783 (+734782us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.194500 (+567717us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:03.948049 (+753549us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b540043b7d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:03.948203 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5253151360}
I20260809 23:20:03.924782 30497 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.967921 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:03.968688 24314 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:03.970259 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:03.942844 30652 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:03.974637 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:03.957332 31059 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260809 23:20:03.979511 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:03.965746 31316 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:03.986042 24316 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:03.980960 24543 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:04.004220 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:04.005069 24164 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 12, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 12 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:20:03.991981 30991 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.007s
I20260809 23:20:04.007385 30991 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11.
I20260809 23:20:03.990509 31021 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:03.988433 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.009531 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 887) took 2395 ms. Trace:
I20260809 23:20:04.009896 24428 rpcz_store.cc:276] 0809 23:20:01.614535 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.316130 (+1701595us) service_pool.cc:227] Handling call
0809 23:20:03.356566 (+ 40436us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:03.401287 (+ 44721us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.401361 (+    74us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:03.420970 (+ 19609us) log.cc:842] Serialized 26 byte log entry
0809 23:20:03.773381 (+352411us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:03.773423 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.773489 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.009298 (+235809us) raft_consensus.cc:1671] finished
0809 23:20:04.009323 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.009469 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1129,"raft.queue_time_us":16544,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":12054,"thread_start_us":306623,"threads_started":1,"wal-append.queue_time_us":536232}
I20260809 23:20:04.011436 24317 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260809 23:20:04.012436 30948 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.013003 30948 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.013439 30948 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.014225 31361 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.015431 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2632) took 1978 ms. Trace:
I20260809 23:20:04.015733 31305 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:03.995776 31362 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:04.017591 31349 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.750s	user 0.003s	sys 0.008s
I20260809 23:20:04.015918 24545 rpcz_store.cc:276] 0809 23:20:02.036629 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.701214 (+664585us) service_pool.cc:227] Handling call
0809 23:20:02.747462 (+ 46248us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:02.876150 (+128688us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:02.876180 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:02.876834 (+   654us) log.cc:842] Serialized 52 byte log entry
0809 23:20:02.901692 (+ 24858us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:03.043816 (+142124us) log.cc:842] Serialized 14 byte log entry
0809 23:20:03.060425 (+ 16609us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.060501 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:03.405836 (+345335us) raft_consensus.cc:1671] finished
0809 23:20:03.405868 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.015207 (+609339us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b540029ca50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:04.015347 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143066,"prepare.queue_time_us":167161,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":145422,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7151,"spinlock_wait_cycles":67261824,"thread_start_us":128991,"threads_started":2,"wal-append.queue_time_us":433520}
I20260809 23:20:04.019071 31362 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.019408 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2356) took 2675 ms. Trace:
I20260809 23:20:04.019855 24425 rpcz_store.cc:276] 0809 23:20:01.344342 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.721801 (+1377459us) service_pool.cc:227] Handling call
0809 23:20:02.777864 (+ 56063us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.403446 (+625582us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.019092 (+615646us) spinlock_profiling.cc:243] Waited 620 ms on lock 0x7b540023da54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.019330 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1426093312}
W20260809 23:20:03.972836 31147 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.000s
I20260809 23:20:03.996755 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:04.022962 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 117485us
I20260809 23:20:04.023316 24539 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11.
I20260809 23:20:03.994910 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:03.996235 24546 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260809 23:20:04.006810 24164 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 7, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:04.020387 30567 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:03.997570 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1126) took 1024 ms. Trace:
I20260809 23:20:04.030503 25368 rpcz_store.cc:276] 0809 23:20:02.972906 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.344541 (+371635us) service_pool.cc:227] Handling call
0809 23:20:03.344924 (+   383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.479325 (+134401us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.479364 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:03.479434 (+    70us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:03.479506 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:03.997287 (+517781us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54002ab7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:03.997347 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:03.997505 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":299573120}
I20260809 23:20:04.033123 24543 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:03.976176 31292 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.103s	user 0.003s	sys 0.019s
W20260809 23:20:03.990998 24378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.581s (SENT) [suppressed 15 similar messages]
I20260809 23:20:03.979732 31059 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:04.038718 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.061969 24314 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.021149 31361 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.061496 24541 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:04.099015 24164 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 8 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:20:04.103299 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 337569us
I20260809 23:20:04.103549 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:04.104426 25368 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:04.058815 31292 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.185s	user 0.005s	sys 0.019s
I20260809 23:20:04.123623 24164 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.128547 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.129365 24540 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.126497 31316 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:04.130235 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1103) took 2520 ms. Trace:
I20260809 23:20:04.130787 31316 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:04.132674 31049 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260809 23:20:04.133646 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.134462 31361 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:04.136175 31361 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:04.136528 31361 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:04.137281 31428 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260809 23:20:04.133833 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.098481 31059 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:20:04.170106 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:04.171149 25368 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:04.190907 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2833) took 2913 ms. Trace:
I20260809 23:20:04.189425 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.188784 31049 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20260809 23:20:04.203001 31049 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:04.203472 31049 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:04.204658 31049 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.206632 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2794) took 2329 ms. Trace:
I20260809 23:20:04.207132 24320 rpcz_store.cc:276] 0809 23:20:01.777774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.252600 (+474826us) service_pool.cc:227] Handling call
0809 23:20:02.252886 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.310017 (+ 57131us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:02.310374 (+   357us) log.cc:842] Serialized 14 byte log entry
0809 23:20:02.511381 (+201007us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:02.511518 (+   137us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:02.511656 (+   138us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.036484 (+1524828us) spinlock_profiling.cc:243] Waited 56.9 ms on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.036542 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.102772 (+ 66230us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":5183,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":18844,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":331,"spinlock_wait_cycles":130884992,"thread_start_us":196965,"threads_started":1,"wal-append.queue_time_us":290627,"wal-append.run_cpu_time_us":2307,"wal-append.run_wall_time_us":1186583}
I20260809 23:20:04.210398 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.211428 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.212334 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.243173 24539 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.217773 31300 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:04.057982 31125 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.220100 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:04.232370 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1114) took 2489 ms (client timeout 2787 ms). Trace:
I20260809 23:20:04.237069 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2638) took 1092 ms. Trace:
I20260809 23:20:04.238363 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.192468 24423 rpcz_store.cc:276] 0809 23:20:01.277179 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.489157 (+1211978us) service_pool.cc:227] Handling call
0809 23:20:02.489651 (+   494us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:02.609200 (+119549us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.153462 (+1544262us) spinlock_profiling.cc:243] Waited 68.4 ms on lock 0x7b5400260cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.190812 (+ 37350us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":157384576}
I20260809 23:20:04.118597 30567 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:04.257925 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.252202 24535 rpcz_store.cc:276] 0809 23:20:03.144717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.344205 (+199488us) service_pool.cc:227] Handling call
0809 23:20:03.403029 (+ 58824us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.469557 (+ 66528us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.236778 (+767221us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.236995 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38295808}
I20260809 23:20:04.270179 24422 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:04.062892 30890 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.130625 25372 rpcz_store.cc:276] 0809 23:20:01.609720 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.610900 (+  1180us) service_pool.cc:227] Handling call
0809 23:20:01.611236 (+   336us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:01.835554 (+224318us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.835597 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:01.842207 (+  6610us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.897184 (+ 54977us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:02.724303 (+827119us) log.cc:842] Serialized 14 byte log entry
0809 23:20:02.794252 (+ 69949us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.119184 (+1324932us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54006004d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.119232 (+    48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.119273 (+    41us) raft_consensus.cc:1671] finished
0809 23:20:04.129985 (+ 10712us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.130167 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":504300,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":7828,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":7282,"spinlock_wait_cycles":514928384,"thread_start_us":805353,"threads_started":2,"wal-append.queue_time_us":522698,"wal-append.run_cpu_time_us":4214,"wal-append.run_wall_time_us":1429872}
W20260809 23:20:04.254980 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 377619us
W20260809 23:20:04.263622 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129871us
I20260809 23:20:04.271771 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.268716 24424 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:04.277683 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83922.8us
I20260809 23:20:04.276865 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.137526 31361 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:04.251745 25370 rpcz_store.cc:269] 0809 23:20:01.742560 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.037202 (+294642us) service_pool.cc:227] Handling call
0809 23:20:04.232053 (+2194851us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b540064c8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:04.232297 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2184038656}
I20260809 23:20:04.197923 31053 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.294054 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:04.295421 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.291450 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.299667 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1197) took 1385 ms. Trace:
I20260809 23:20:04.300007 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:04.300047 25375 rpcz_store.cc:276] 0809 23:20:02.914373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.274541 (+360168us) service_pool.cc:227] Handling call
0809 23:20:03.274919 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.352748 (+ 77829us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:03.394019 (+ 41271us) log.cc:842] Serialized 14 byte log entry
0809 23:20:03.928255 (+534236us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b18003d9600. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:20:04.256100 (+327845us) log.cc:842] Serialized 14 byte log entry
0809 23:20:04.299325 (+ 43225us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:04.299378 (+    53us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:04.299431 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.299491 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.299614 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66497,"prepare.queue_time_us":332637,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":5955,"raft.queue_time_us":27930,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":14864768,"thread_start_us":488167,"threads_started":2,"wal-append.queue_time_us":406461}
W20260809 23:20:04.305902 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1124) took 1660 ms (client timeout 1779 ms). Trace:
W20260809 23:20:04.306326 25366 rpcz_store.cc:269] 0809 23:20:02.645199 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:02.667921 (+ 22722us) service_pool.cc:227] Handling call
0809 23:20:04.305694 (+1637773us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:04.305843 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1721914112}
I20260809 23:20:04.270881 24314 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:04.290154 24429 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:20:04.316637 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.272429 31336 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:04.280347 31049 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:04.282126 30942 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.328842 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.333205 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:04.329152 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:04.332372 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.348085 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.350618 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1199) took 1306 ms. Trace:
I20260809 23:20:04.351104 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.347214 30917 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.355628 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.351148 25363 rpcz_store.cc:276] 0809 23:20:03.044073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.607559 (+563486us) service_pool.cc:227] Handling call
0809 23:20:03.626337 (+ 18778us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.713626 (+ 87289us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.350330 (+636704us) spinlock_profiling.cc:243] Waited 69 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.350548 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158654720}
I20260809 23:20:04.328666 30948 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.359395 31392 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
W20260809 23:20:04.368448 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48536.7us
I20260809 23:20:04.365481 30585 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:04.374481 30585 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:04.374966 30585 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:04.374328 24164 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.376287 31437 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.376215 24164 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.376960 31283 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.376837 31437 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.377842 24164 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:20:04.379658 31442 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260809 23:20:04.380290 24165 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 340) took 43090 ms (client timeout 15000 ms). Trace:
W20260809 23:20:04.380703 24165 rpcz_store.cc:269] 0809 23:19:21.296911 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:21.316537 (+ 19626us) service_pool.cc:227] Handling call
0809 23:20:04.369566 (+43053029us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:19:44.005413 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:19:44.177948 (+172535us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:19:44.178013 (+    65us) write_op.cc:435] Acquired schema lock
0809 23:19:44.178027 (+    14us) tablet.cc:660] Decoding operations
0809 23:19:44.193343 (+ 15316us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:19:44.193465 (+   122us) write_op.cc:641] Partition lock acquired for write op
0809 23:19:44.193504 (+    39us) tablet.cc:683] Acquiring locks for 24 operations
0809 23:19:44.698353 (+504849us) tablet.cc:699] Row locks acquired
0809 23:19:44.709910 (+ 11557us) write_op.cc:260] PREPARE: finished
0809 23:19:44.783088 (+ 73178us) write_op.cc:270] Start()
0809 23:19:44.785837 (+  2749us) write_op.cc:276] Timestamp: P: 1786317584724057 usec, L: 0
0809 23:19:44.817321 (+ 31484us) op_driver.cc:331] REPLICATION: starting
0809 23:19:44.918202 (+100881us) log.cc:842] Serialized 9339 byte log entry
0809 23:19:46.914452 (+1996250us) spinlock_profiling.cc:243] Waited 7.38 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:19:47.821914 (+907462us) op_driver.cc:444] REPLICATION: finished
0809 23:19:50.606888 (+2784974us) write_op.cc:301] APPLY: starting
0809 23:19:50.623742 (+ 16854us) tablet.cc:1365] starting BulkCheckPresence
0809 23:19:51.782652 (+1158910us) tablet.cc:1368] finished BulkCheckPresence
0809 23:19:51.782730 (+    78us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:19:53.054453 (+1271723us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:19:53.087525 (+ 33072us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=0,mrs_lookups=0
0809 23:19:53.087572 (+    47us) write_op.cc:312] APPLY: finished
0809 23:19:53.088476 (+   904us) log.cc:842] Serialized 212 byte log entry
0809 23:19:53.145286 (+ 56810us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:53.353402 (+208116us) write_op.cc:454] Released schema lock
0809 23:19:53.386312 (+ 32910us) write_op.cc:341] FINISH: Updating metrics
0809 23:19:53.430232 (+ 43920us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:19:53.430271 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":892247,"reported_tablets":46,"child_traces":[["op",{"apply.queue_time_us":2684455,"apply.run_cpu_time_us":63772,"apply.run_wall_time_us":3057629,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104351,"num_ops":24,"prepare.queue_time_us":537920,"prepare.run_cpu_time_us":18270,"prepare.run_wall_time_us":959929,"raft.queue_time_us":730326,"raft.run_cpu_time_us":70949,"raft.run_wall_time_us":2436586,"replication_time_us":2977931,"spinlock_wait_cycles":16969984,"thread_start_us":2896555,"threads_started":5,"wal-append.queue_time_us":287259,"wal-append.run_cpu_time_us":52611,"wal-append.run_wall_time_us":4824764}]]}
I20260809 23:20:04.389415 31026 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:04.336788 31053 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:04.392787 24164 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:20:04.394037 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61774us
I20260809 23:20:04.399418 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.400403 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.359314 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:04.324033 30567 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.406013 25370 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260809 23:20:04.410295 25363 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260809 23:20:04.412720 25325 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 10. Current term is 11.
I20260809 23:20:04.412295 24164 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.365630 31361 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:04.420301 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68292.5us
I20260809 23:20:04.366262 30991 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.417932 24316 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:04.372942 31392 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:04.336313 30942 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.389881 31026 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:04.422606 31390 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260809 23:20:04.446005 25370 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:04.444602 31300 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:04.420017 30585 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.409458 31437 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.457016 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.429747 31283 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:04.441457 30860 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:04.406649 31021 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260809 23:20:04.463100 25375 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:04.432519 31292 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260809 23:20:04.430421 31053 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.411219 30819 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260809 23:20:04.479278 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.370834 30917 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.481799 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1253) took 1560 ms. Trace:
I20260809 23:20:04.491571 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:04.491086 25365 rpcz_store.cc:276] 0809 23:20:02.921625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.278796 (+357171us) service_pool.cc:227] Handling call
0809 23:20:03.315508 (+ 36712us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.764655 (+449147us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.481576 (+716921us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.481743 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1030864128}
W20260809 23:20:04.492018 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53983.6us
I20260809 23:20:04.462564 31283 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:04.483631 31198 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.495321 31444 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:04.496317 31444 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:04.496670 31444 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:04.497494 31444 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.494524 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.503161 24540 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:04.504465 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2628) took 2720 ms. Trace:
I20260809 23:20:04.504626 31413 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:04.505131 31413 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:04.505591 31413 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:04.504839 24534 rpcz_store.cc:276] 0809 23:20:01.784348 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:01.784472 (+   124us) service_pool.cc:227] Handling call
0809 23:20:01.834173 (+ 49701us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:01.870734 (+ 36561us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:01.870768 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:01.874664 (+  3896us) log.cc:842] Serialized 26 byte log entry
0809 23:20:01.972568 (+ 97904us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:01.972615 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:01.972680 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.016801 (+2044121us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:04.500397 (+483596us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b24003c01e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:20:04.501031 (+   634us) raft_consensus.cc:1671] finished
0809 23:20:04.501064 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.504283 (+  3219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63598720,"thread_start_us":96213,"threads_started":1,"wal-append.queue_time_us":327584}
I20260809 23:20:04.506166 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:04.507567 31443 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260809 23:20:04.508024 31444 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:04.485461 31059 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.513564 31053 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.515126 24534 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 7. Current term is 8. Ops: []
I20260809 23:20:04.516656 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2375) took 1726 ms. Trace:
I20260809 23:20:04.558341 24435 rpcz_store.cc:276] 0809 23:20:02.790407 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.552478 (+762071us) service_pool.cc:227] Handling call
0809 23:20:03.583709 (+ 31231us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:03.901324 (+317615us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.901355 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:03.956764 (+ 55409us) log.cc:842] Serialized 26 byte log entry
0809 23:20:04.271917 (+315153us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:04.271951 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.272019 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.516451 (+244432us) raft_consensus.cc:1671] finished
0809 23:20:04.516472 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.516600 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16404,"raft.queue_time_us":36446,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":42903,"thread_start_us":562708,"threads_started":2,"wal-append.queue_time_us":500811}
I20260809 23:20:04.559397 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.520598 31448 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:04.560469 31448 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:04.559360 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64908.6us
I20260809 23:20:04.561055 31448 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:04.562443 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.522315 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.523519 24424 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:04.523744 31059 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.525050 24535 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:04.488474 31184 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.584157 31334 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:04.584378 31026 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.483012 31422 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:04.587527 31422 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.589844 31422 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.591303 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.592320 24424 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:04.592161 31021 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:04.593581 31316 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.594025 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.594252 31316 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.596619 30917 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.596715 31021 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:04.487316 31441 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:04.495425 31283 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.597548 31316 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.597664 31021 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:04.479653 31437 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:04.551925 30647 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:04.598920 31437 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.526430 30585 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.527599 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1133) took 1004 ms. Trace:
I20260809 23:20:04.600407 25369 rpcz_store.cc:276] 0809 23:20:03.522754 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.714555 (+191801us) service_pool.cc:227] Handling call
0809 23:20:03.775921 (+ 61366us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:03.776331 (+   410us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.776366 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:03.805896 (+ 29530us) log.cc:842] Serialized 26 byte log entry
0809 23:20:04.190178 (+384282us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:04.190216 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.190289 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.510777 (+320488us) raft_consensus.cc:1671] finished
0809 23:20:04.510799 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.510954 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":341674,"threads_started":1,"wal-append.queue_time_us":541835}
I20260809 23:20:04.602730 31437 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:04.487026 30629 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.720s	user 0.015s	sys 0.000s
I20260809 23:20:04.603550 30567 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.603423 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:04.604866 31337 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.771s	user 0.011s	sys 0.000s
I20260809 23:20:04.605289 31337 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.607066 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:04.607785 25369 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:04.456658 31009 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:04.440234 30991 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.480185 31392 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.611481 31392 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:04.557312 30942 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.612396 30991 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.538338 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:04.556690 24431 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:20:04.525166 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.562753 31152 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:04.522049 24423 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:04.562093 31448 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.563086 30860 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:04.572737 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1133) took 1003 ms. Trace:
I20260809 23:20:04.540808 30890 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.575631 24422 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:04.575733 24309 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:04.576062 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.576885 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.577193 31315 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.577858 24323 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:04.501936 31125 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:04.536399 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 545422us
I20260809 23:20:04.650413 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.615331 24313 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:04.614259 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.597316 31441 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:04.640597 24321 rpcz_store.cc:276] 0809 23:20:03.569747 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.571568 (+1001821us) service_pool.cc:227] Handling call
0809 23:20:04.572026 (+   458us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:04.572296 (+   270us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.572340 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:04.572393 (+    53us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:04.572445 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.572513 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.572633 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:04.575357 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:04.555572 24164 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 3 to 10, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.644992 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.646845 24544 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:04.625633 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:04.647069 25368 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:04.646189 30585 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20260809 23:20:04.652550 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.623769 25372 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:04.653126 30991 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:04.654698 24314 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:04.655040 24424 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:04.659813 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.660791 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.660816 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:04.663265 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.618525 31026 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.662554 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:04.627867 31059 consensus_queue.cc:1159] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 10 status { last_received { term: 9 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:20:04.665729 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.671241 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.680047 24535 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:04.674188 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.701778 24314 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:20:04.666103 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:04.652685 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 388825us
I20260809 23:20:04.663964 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1202) took 1511 ms. Trace:
I20260809 23:20:04.683547 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:04.661761 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:04.666129 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1131) took 1115 ms. Trace:
I20260809 23:20:04.704520 31260 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:04.673352 30860 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.707406 25371 rpcz_store.cc:276] 0809 23:20:03.152274 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.648407 (+496133us) service_pool.cc:227] Handling call
0809 23:20:03.648738 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:03.693265 (+ 44527us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:04.267762 (+574497us) log.cc:842] Serialized 14 byte log entry
0809 23:20:04.629566 (+361804us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:04.629637 (+    71us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:04.629706 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.663561 (+ 33855us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:04.663635 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.663838 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":235148,"prepare.queue_time_us":237833,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":11076,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":18588,"spinlock_wait_cycles":101678336,"thread_start_us":663701,"threads_started":2}
I20260809 23:20:04.714316 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.715689 31448 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:04.716988 31260 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:04.709774 24313 rpcz_store.cc:276] 0809 23:20:03.550782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.550919 (+   137us) service_pool.cc:227] Handling call
0809 23:20:04.665960 (+1115041us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:04.666053 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2321779456}
I20260809 23:20:04.718043 31260 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Stepping down as leader of term 7
I20260809 23:20:04.718429 31260 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:04.719250 31270 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:04.719937 31270 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:04.720083 30991 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:20:04.720546 30991 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:04.720878 30991 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:04.720925 24323 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:04.710485 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.723104 24321 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 8.
I20260809 23:20:04.721637 31038 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:04.738582 31422 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:04.739194 31422 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.741597 31422 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:04.742805 24544 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260809 23:20:04.743932 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.746167 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.748337 30567 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:04.748726 31449 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260809 23:20:04.748513 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:04.749205 30567 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.750454 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1134) took 1214 ms. Trace:
I20260809 23:20:04.730532 31175 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260809 23:20:04.750878 25373 rpcz_store.cc:276] 0809 23:20:03.535654 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:03.944861 (+409207us) service_pool.cc:227] Handling call
0809 23:20:03.945232 (+   371us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:03.945709 (+   477us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:03.945746 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:03.946290 (+   544us) log.cc:842] Serialized 26 byte log entry
0809 23:20:04.329709 (+383419us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:04.329757 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.329831 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:04.750181 (+420350us) raft_consensus.cc:1671] finished
0809 23:20:04.750206 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:04.750388 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3159,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6390,"thread_start_us":361149,"threads_started":1,"wal-append.queue_time_us":714079}
I20260809 23:20:04.665742 31444 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260809 23:20:04.666308 31184 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260809 23:20:04.754827 31184 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:04.659479 31315 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.755314 31260 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.756966 31260 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:20:04.757741 31315 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:04.762220 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.763115 31038 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:04.764828 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.766781 24541 pending_rounds.cc:76] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:04.767177 24541 pending_rounds.cc:97] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260809 23:20:04.767513 24541 raft_consensus.cc:2866] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:04.755316 31184 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:04.771039 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 120591us
I20260809 23:20:04.772181 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.772390 31184 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.774262 31184 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:04.770500 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108812us
I20260809 23:20:04.775189 31316 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:04.776007 31316 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:04.779098 24539 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:04.778919 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.663300 30942 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:04.789172 24309 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:04.677917 31248 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.749s	user 0.008s	sys 0.002s
I20260809 23:20:04.783486 25369 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:20:04.794245 24317 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.010s	sys 0.006s
W20260809 23:20:04.795244 24317 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.011s	sys 0.007s
I20260809 23:20:04.675951 31337 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.646695 30629 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
I20260809 23:20:04.657569 31152 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.123s
I20260809 23:20:04.799355 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.806663 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.799193 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.789892 24164 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.821440 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:04.822490 25369 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 7 FOLLOWER]: Advancing to term 8
I20260809 23:20:04.820043 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:04.823318 30942 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:04.823660 30567 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:04.825505 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:04.774845 31038 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:04.656446 31009 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:04.828215 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 154283us
I20260809 23:20:04.789608 30647 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:04.830565 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:04.830463 24164 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:04.672107 31441 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.836694 30819 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:04.838958 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:04.854419 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.843385 31337 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:04.843518 31283 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:04.798822 31248 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:04.799391 31175 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260809 23:20:04.721050 31270 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.876284 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.816810 30917 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260809 23:20:04.672644 31021 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.864029 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:04.863314 24164 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:04.838438 30991 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.874993 24540 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:04.872749 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.861291 30819 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:04.898454 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:04.898782 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:04.850996 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:04.885329 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:04.872270 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 478811us
W20260809 23:20:04.901319 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50469.3us
I20260809 23:20:04.903837 31413 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:04.905820 30890 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:04.869601 31125 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.026s
I20260809 23:20:04.906436 30629 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260809 23:20:04.877812 31441 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:04.910306 31339 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:04.913640 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.915319 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:04.916388 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:04.877936 31038 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:04.921900 24164 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 10, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:04.925675 24540 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:04.920742 25369 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 8.
I20260809 23:20:04.948961 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:04.953702 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68480us
I20260809 23:20:04.911791 30819 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:04.963181 24309 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:04.956434 31125 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:04.964885 31448 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:04.965621 30629 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:04.966323 31175 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:04.911922 30948 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:04.943908 30919 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:04.923950 31413 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:04.996793 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80476.1us
I20260809 23:20:04.942591 30942 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260809 23:20:05.036484 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:05.040985 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72864.5us
I20260809 23:20:05.010637 30919 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:05.092425 24164 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:05.099210 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:05.099992 24545 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:04.969506 31337 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
I20260809 23:20:05.097510 30919 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:05.041764 31448 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.098971 31038 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.155385 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.023s	sys 0.072s
W20260809 23:20:05.159744 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.024s	sys 0.073s
I20260809 23:20:05.155927 24164 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:05.079795 31413 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260809 23:20:05.004228 31125 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.101390 30917 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260809 23:20:05.179121 31466 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.303s	user 0.001s	sys 0.000s
W20260809 23:20:05.181123 31459 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260809 23:20:05.183328 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2811) took 1120 ms. Trace:
I20260809 23:20:05.189953 24164 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 9 to 12, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 12 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:05.186473 24545 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:05.205979 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:05.162385 31283 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.162138 31336 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.183727 24311 rpcz_store.cc:276] 0809 23:20:04.063404 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.063576 (+   172us) service_pool.cc:227] Handling call
0809 23:20:04.119464 (+ 55888us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.194299 (+ 74835us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.194346 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.194895 (+   549us) log.cc:842] Serialized 26 byte log entry
0809 23:20:04.544328 (+349433us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:04.544384 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:04.544477 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.183118 (+638641us) raft_consensus.cc:1671] finished
0809 23:20:05.183138 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.183270 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28925,"raft.queue_time_us":43828,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":35760,"thread_start_us":261446,"threads_started":1,"wal-append.queue_time_us":573009}
W20260809 23:20:05.212757 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.022s	sys 0.040s
W20260809 23:20:05.214905 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.022s	sys 0.040s
I20260809 23:20:05.167444 31009 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.217305 24164 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:05.252190 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:05.259441 24423 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:05.217234 30991 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.267795 31471 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:05.268554 31471 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:05.269593 31471 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:05.271108 31413 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:05.279524 30819 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:05.220707 31336 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:05.287518 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.017s	sys 0.021s
W20260809 23:20:05.288884 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.017s	sys 0.021s
W20260809 23:20:05.303006 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 388154us
I20260809 23:20:05.299222 24164 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 9, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 9 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:20:05.292862 31026 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.587s	user 0.004s	sys 0.008s
I20260809 23:20:05.281862 31413 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:05.335371 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.008s	sys 0.013s
W20260809 23:20:05.335791 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.008s	sys 0.013s
W20260809 23:20:05.341182 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.011s	sys 0.010s
W20260809 23:20:05.341677 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.011s	sys 0.010s
W20260809 23:20:05.332692 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.022s	sys 0.021s
I20260809 23:20:05.292567 31009 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:05.348026 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.022s	sys 0.021s
W20260809 23:20:05.352088 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.013s	sys 0.010s
W20260809 23:20:05.352545 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.013s	sys 0.010s
W20260809 23:20:05.352903 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 576448us
W20260809 23:20:05.351533 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 724350us
W20260809 23:20:05.355736 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.012s	sys 0.029s
W20260809 23:20:05.358765 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.012s	sys 0.029s
I20260809 23:20:05.324258 31260 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.395792 24164 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:20:05.367218 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.000s	sys 0.020s
I20260809 23:20:05.349874 31316 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:05.414031 24164 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 8, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:20:05.432950 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.007s	sys 0.020s
W20260809 23:20:05.433405 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.008s	sys 0.020s
I20260809 23:20:05.403376 31448 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:05.432062 24164 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 2 to 10. New cstate: current_term: 10 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:05.365314 31009 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:05.412743 30991 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:05.419063 31316 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:05.412011 31283 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:05.429906 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.016s	sys 0.015s
W20260809 23:20:05.483824 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.001s	sys 0.023s
W20260809 23:20:05.455735 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.002s	sys 0.020s
I20260809 23:20:05.404940 31260 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.490531 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.017s	sys 0.015s
W20260809 23:20:05.490840 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.002s	sys 0.020s
I20260809 23:20:05.493130 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:05.490957 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.001s	sys 0.024s
W20260809 23:20:05.501112 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.008s	sys 0.041s
W20260809 23:20:05.501477 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.008s	sys 0.041s
W20260809 23:20:05.501778 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 600999us
W20260809 23:20:05.489168 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.011s	sys 0.022s
W20260809 23:20:05.503823 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.012s	sys 0.022s
I20260809 23:20:05.504467 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:05.512617 31336 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:05.514302 31336 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.516130 31009 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.480708 31049 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260809 23:20:05.516389 31437 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:05.479142 31413 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.518079 31260 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:05.519196 31184 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.730s	user 0.006s	sys 0.006s
I20260809 23:20:05.519731 30819 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:05.520174 31413 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:05.520630 31413 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.490473 31125 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:05.529376 24423 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:20:05.534464 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:05.535516 24539 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:05.503849 31270 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.014s	sys 0.036s
I20260809 23:20:05.522544 31413 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:05.576807 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72423.1us
W20260809 23:20:05.527100 30860 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.014s	sys 0.023s
W20260809 23:20:05.590967 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.003s	sys 0.045s
W20260809 23:20:05.593089 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.003s	sys 0.045s
I20260809 23:20:05.527348 31026 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:05.601964 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.012s	sys 0.023s
W20260809 23:20:05.602610 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.012s	sys 0.024s
W20260809 23:20:05.571039 31270 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.014s	sys 0.036s
I20260809 23:20:05.484988 31448 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.085s
I20260809 23:20:05.609929 31471 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:05.613116 31339 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:05.615231 31496 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:05.617375 30948 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:05.653491 30948 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:05.657256 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2784) took 1040 ms. Trace:
I20260809 23:20:05.623638 31049 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:05.532197 31248 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.514752 31336 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:05.625458 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1135) took 1038 ms. Trace:
I20260809 23:20:05.661441 25375 rpcz_store.cc:276] 0809 23:20:04.586781 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.588634 (+  1853us) service_pool.cc:227] Handling call
0809 23:20:04.589033 (+   399us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.599533 (+ 10500us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.599579 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.600087 (+   508us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.216800 (+616713us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:05.216839 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.217110 (+   271us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.625175 (+408065us) raft_consensus.cc:1671] finished
0809 23:20:05.625204 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.625383 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10000,"raft.queue_time_us":12266,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":616302,"threads_started":1,"wal-append.queue_time_us":975697}
I20260809 23:20:05.537565 31283 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.627027 31009 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Leader election won for term 8
W20260809 23:20:05.663934 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.012s	sys 0.039s
I20260809 23:20:05.489912 30991 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.632463 31316 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.592348 30860 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.014s	sys 0.023s
I20260809 23:20:05.604485 31026 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.639560 31339 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:05.603504 30890 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.005s	sys 0.029s
I20260809 23:20:05.657867 24543 rpcz_store.cc:276] 0809 23:20:04.616881 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.617531 (+   650us) service_pool.cc:227] Handling call
0809 23:20:04.617911 (+   380us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.641586 (+ 23675us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:04.673624 (+ 32038us) log.cc:842] Serialized 14 byte log entry
0809 23:20:04.716136 (+ 42512us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b18003b5b40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:04.719988 (+  3852us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b18003b5b40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:05.326425 (+606437us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:05.331652 (+  5227us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.331793 (+   141us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:05.331822 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.331886 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.656907 (+325021us) raft_consensus.cc:1671] finished
0809 23:20:05.656941 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.657169 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13047,"prepare.queue_time_us":31850,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":58406,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":45065,"spinlock_wait_cycles":112756736,"thread_start_us":569827,"threads_started":1,"wal-append.queue_time_us":981004}
I20260809 23:20:05.631824 31184 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:05.667769 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.012s	sys 0.039s
W20260809 23:20:05.701911 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 651341us
W20260809 23:20:05.669518 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.009s	sys 0.028s
I20260809 23:20:05.660171 31448 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:20:05.669138 30919 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.704365 31448 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:05.705303 31448 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.705775 30991 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.706538 31448 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:05.684018 31026 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:05.669772 31009 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:05.661280 31049 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:05.718353 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.007s	sys 0.015s
W20260809 23:20:05.722791 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.007s	sys 0.016s
W20260809 23:20:05.702342 30890 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.006s	sys 0.029s
W20260809 23:20:05.722399 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.026s	sys 0.046s
I20260809 23:20:05.706691 30919 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:05.724225 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.010s	sys 0.029s
I20260809 23:20:05.724768 24311 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:05.725858 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.013s	sys 0.011s
I20260809 23:20:05.729590 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:05.661013 31248 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.730821 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.026s	sys 0.046s
I20260809 23:20:05.749140 24309 raft_consensus.cc:2354] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:05.701561 31184 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:05.724602 31021 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.022s	sys 0.028s
W20260809 23:20:05.749670 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.013s	sys 0.012s
I20260809 23:20:05.758607 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:05.763932 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:05.754472 31038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.020s	sys 0.031s
I20260809 23:20:05.730603 30890 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:05.757462 31152 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:05.762049 31021 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.023s	sys 0.028s
I20260809 23:20:05.781006 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:05.723371 31270 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260809 23:20:05.762190 31316 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:05.787284 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:05.789810 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:05.790586 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:05.808632 24311 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:05.775667 31038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.020s	sys 0.031s
I20260809 23:20:05.807974 24309 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
W20260809 23:20:05.781867 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.003s	sys 0.028s
W20260809 23:20:05.812508 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.004s	sys 0.028s
I20260809 23:20:05.810050 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:05.813192 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 607298us
I20260809 23:20:05.816203 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:05.836061 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.028s	sys 0.046s
W20260809 23:20:05.838559 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.028s	sys 0.047s
I20260809 23:20:05.813589 24321 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:20:05.840685 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 926389us
I20260809 23:20:05.841653 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:05.843312 24539 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:20:05.843597 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:05.844103 24311 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260809 23:20:05.844482 24309 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260809 23:20:05.845266 24542 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260809 23:20:05.845781 24542 pending_rounds.cc:76] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:05.846024 24542 pending_rounds.cc:97] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260809 23:20:05.846257 24542 raft_consensus.cc:2866] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:20:05.848315 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.017s	sys 0.034s
W20260809 23:20:05.848630 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.017s	sys 0.034s
W20260809 23:20:05.852936 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.030s	sys 0.017s
W20260809 23:20:05.853355 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.030s	sys 0.018s
I20260809 23:20:05.854076 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:05.810990 30917 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:05.861632 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:05.863420 24321 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:05.875142 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:05.877669 24323 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:05.879436 25369 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:05.879699 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:05.883507 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:05.885109 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:05.765527 31339 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.901810 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:05.903728 25370 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:05.904137 24312 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:05.853545 31152 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.856199 31175 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:05.909366 25369 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:20:05.885854 25372 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:05.908156 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:05.904676 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:05.920965 25365 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 9: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:05.919423 31441 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.033s	sys 0.015s
I20260809 23:20:05.919489 31152 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:05.919616 31175 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.878865 30942 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:05.873317 30991 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:05.891184 31496 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:05.930378 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:05.915920 31260 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:05.980666 31260 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:05.923861 31441 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.033s	sys 0.015s
I20260809 23:20:05.933885 31283 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:05.923946 31270 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:05.936645 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
W20260809 23:20:05.937569 31486 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.164s	user 0.000s	sys 0.004s
I20260809 23:20:05.937949 30991 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.939100 31479 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
W20260809 23:20:05.939751 31480 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260809 23:20:05.939895 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:05.940435 31490 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
W20260809 23:20:05.941247 31494 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260809 23:20:05.905596 30919 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:05.919775 30890 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.817090 31184 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.942391 24323 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 9.
I20260809 23:20:05.920861 31336 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:05.943252 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2791) took 1016 ms. Trace:
I20260809 23:20:05.948886 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 915) took 1170 ms. Trace:
I20260809 23:20:05.990221 31339 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:05.854501 31009 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.960551 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1141) took 1067 ms. Trace:
I20260809 23:20:05.963452 31315 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260809 23:20:06.018637 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.016s	sys 0.010s
I20260809 23:20:05.967403 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.029968 24314 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:05.967973 31049 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:05.913625 30917 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:05.968379 30942 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.969079 31495 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260809 23:20:05.969781 24312 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:05.975812 25372 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:05.979904 24321 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: [9.6-12.7]
I20260809 23:20:05.936597 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1040) took 1154 ms. Trace:
I20260809 23:20:05.986793 25369 raft_consensus.cc:595] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:06.043061 25369 raft_consensus.cc:603] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:06.020277 24546 rpcz_store.cc:276] 0809 23:20:04.927335 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.927529 (+   194us) service_pool.cc:227] Handling call
0809 23:20:04.939503 (+ 11974us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.955278 (+ 15775us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.955324 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.995910 (+ 40586us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.603144 (+607234us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:05.603198 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.603286 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.943037 (+339751us) raft_consensus.cc:1671] finished
0809 23:20:05.943059 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.943193 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":141,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":6264,"thread_start_us":558539,"threads_started":1,"wal-append.queue_time_us":902518}
I20260809 23:20:05.971145 30860 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:05.989522 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:05.952613 30948 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:05.992933 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.000s	sys 0.024s
I20260809 23:20:05.984938 31260 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:05.990873 31248 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.033717 31021 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:06.027185 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.020692 24422 rpcz_store.cc:276] 0809 23:20:04.778240 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.778399 (+   159us) service_pool.cc:227] Handling call
0809 23:20:04.778726 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.779109 (+   383us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.779142 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.779572 (+   430us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.495099 (+715527us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:05.495140 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.495228 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.948626 (+453398us) raft_consensus.cc:1671] finished
0809 23:20:05.948652 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.948802 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":959,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":8621,"thread_start_us":715127,"threads_started":1,"wal-append.queue_time_us":1033968}
I20260809 23:20:05.929519 31175 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:06.031080 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.017s	sys 0.010s
I20260809 23:20:06.025954 25368 rpcz_store.cc:276] 0809 23:20:04.892757 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.895999 (+  3242us) service_pool.cc:227] Handling call
0809 23:20:04.896336 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.897463 (+  1127us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:04.897494 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.915599 (+ 18105us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.595656 (+680057us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:05.595714 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.595809 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.960309 (+364500us) raft_consensus.cc:1671] finished
0809 23:20:05.960335 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.960488 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9317,"raft.queue_time_us":1246,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":595542,"threads_started":1,"wal-append.queue_time_us":990821}
I20260809 23:20:06.035879 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2865) took 1271 ms. Trace:
I20260809 23:20:06.007128 31270 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:06.006543 31283 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:06.046741 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.000s	sys 0.024s
I20260809 23:20:06.046715 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.026567 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.055814 24312 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:20:05.998514 31496 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:06.056915 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 594338us
I20260809 23:20:06.041745 24544 rpcz_store.cc:276] 0809 23:20:04.782631 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.782754 (+   123us) service_pool.cc:227] Handling call
0809 23:20:04.799498 (+ 16744us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.799923 (+   425us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.799960 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.850263 (+ 50303us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.504294 (+654031us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:05.504334 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.504410 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.936383 (+431973us) raft_consensus.cc:1671] finished
0809 23:20:05.936404 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:05.936539 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":17007,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":7033,"thread_start_us":620568,"threads_started":1,"wal-append.queue_time_us":937623}
I20260809 23:20:06.056298 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:06.057938 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.053431 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.046432 31441 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.052219 24428 rpcz_store.cc:276] 0809 23:20:04.764593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.780682 (+ 16089us) service_pool.cc:227] Handling call
0809 23:20:04.815205 (+ 34523us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.815674 (+   469us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.815713 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.826641 (+ 10928us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.495539 (+668898us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:05.495574 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.496293 (+   719us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:05.936918 (+440625us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:06.035497 (+ 98579us) raft_consensus.cc:1671] finished
0809 23:20:06.035533 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.035801 (+   268us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":3763,"spinlock_wait_cycles":5898624,"thread_start_us":660741,"threads_started":1,"wal-append.queue_time_us":894787}
I20260809 23:20:06.059589 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:06.052747 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 652347us
I20260809 23:20:06.046432 30860 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:06.066116 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1257) took 1874 ms. Trace:
I20260809 23:20:06.070484 24312 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:06.068231 25363 rpcz_store.cc:276] 0809 23:20:04.191507 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.439304 (+247797us) service_pool.cc:227] Handling call
0809 23:20:04.461600 (+ 22296us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.503996 (+ 42396us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.504069 (+    73us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.642694 (+138625us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.359190 (+716496us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:05.359229 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.359303 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.065664 (+706361us) raft_consensus.cc:1671] finished
0809 23:20:06.065703 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.065948 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":622323,"threads_started":1,"wal-append.queue_time_us":1014792}
I20260809 23:20:06.051057 31184 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:06.050849 31049 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.077708 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.053196 31021 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:06.080715 24321 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260809 23:20:06.059326 31470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.029s	sys 0.062s
W20260809 23:20:06.064016 30585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.008s	sys 0.015s
W20260809 23:20:06.097513 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.065s	user 0.017s	sys 0.052s
W20260809 23:20:06.097824 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.093s	user 0.017s	sys 0.052s
W20260809 23:20:06.083009 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.003s	sys 0.018s
W20260809 23:20:06.098632 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.003s	sys 0.019s
I20260809 23:20:06.084353 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.088572 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1037) took 1324 ms. Trace:
I20260809 23:20:06.089268 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2789) took 1311 ms. Trace:
I20260809 23:20:06.090433 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:06.081588 25373 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.005s	sys 0.020s
I20260809 23:20:06.068173 31038 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:06.068361 30860 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:06.123569 24312 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:06.124598 25373 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.005s	sys 0.020s
I20260809 23:20:06.122977 24541 rpcz_store.cc:276] 0809 23:20:04.764085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.766324 (+  2239us) service_pool.cc:227] Handling call
0809 23:20:04.766619 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.768449 (+  1830us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.768476 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.768846 (+   370us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.336255 (+567409us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:05.380384 (+ 44129us) log.cc:842] Serialized 14 byte log entry
0809 23:20:05.412330 (+ 31946us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.412428 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.088378 (+675950us) raft_consensus.cc:1671] finished
0809 23:20:06.088398 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.088511 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31471,"prepare.queue_time_us":43690,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":31603,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":2762,"thread_start_us":567018,"threads_started":1,"wal-append.queue_time_us":917538}
I20260809 23:20:06.138072 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1261) took 1351 ms. Trace:
I20260809 23:20:06.142385 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1140) took 1568 ms. Trace:
I20260809 23:20:06.142791 24320 rpcz_store.cc:276] 0809 23:20:04.573717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.665030 (+ 91313us) service_pool.cc:227] Handling call
0809 23:20:04.665354 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:04.665550 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:05.004798 (+339248us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.142071 (+1137273us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:06.142123 (+    52us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:06.142173 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.142239 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.142332 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43834,"prepare.queue_time_us":503881,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"thread_start_us":1432012,"threads_started":2}
I20260809 23:20:06.138427 25373 rpcz_store.cc:276] 0809 23:20:04.787461 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.787609 (+   148us) service_pool.cc:227] Handling call
0809 23:20:04.800458 (+ 12849us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:04.821568 (+ 21110us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:04.833421 (+ 11853us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.137632 (+1304211us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:06.137705 (+    73us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:06.137786 (+    81us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.137892 (+   106us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.138020 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12215,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":20680,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":1303774,"threads_started":2,"wal-append.queue_time_us":981946}
W20260809 23:20:06.135601 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.021s	sys 0.036s
W20260809 23:20:06.146994 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.022s	sys 0.036s
I20260809 23:20:06.150544 24321 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:06.091359 31470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.030s	sys 0.062s
W20260809 23:20:06.151664 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67398.6us
I20260809 23:20:06.152642 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:06.130798 30585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.008s	sys 0.015s
I20260809 23:20:06.058915 31270 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.088430 31413 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:06.062182 31283 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.189323 31283 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.190820 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.193245 30917 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.194746 31496 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.196424 25368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:20:06.197059 25368 raft_consensus.cc:595] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:06.197427 25368 raft_consensus.cc:603] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:06.197947 30991 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:06.198181 30942 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:06.199357 31505 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.184s	user 0.002s	sys 0.000s
I20260809 23:20:06.200783 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1140) took 1310 ms. Trace:
I20260809 23:20:06.201798 31026 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.123041 24535 rpcz_store.cc:276] 0809 23:20:04.778290 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.778416 (+   126us) service_pool.cc:227] Handling call
0809 23:20:04.778802 (+   386us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.779197 (+   395us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.779223 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.779681 (+   458us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.434076 (+654395us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:05.434152 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.434252 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.089037 (+654785us) raft_consensus.cc:1671] finished
0809 23:20:06.089057 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.089212 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17709,"raft.queue_time_us":988,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8551,"spinlock_wait_cycles":26880,"thread_start_us":653826,"threads_started":1,"wal-append.queue_time_us":998422}
I20260809 23:20:06.081991 31315 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
I20260809 23:20:06.077558 31441 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.166900 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:06.171429 31521 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260809 23:20:06.171639 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.171902 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.172331 31270 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.173527 31524 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:06.173671 25363 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:06.175273 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:06.164373 31448 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:06.158326 31038 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:06.185413 31021 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.203531 30948 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:06.085367 31049 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.237617 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:06.233588 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58396.2us
I20260809 23:20:06.238559 24546 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20260809 23:20:06.220113 31518 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260809 23:20:06.201184 25371 rpcz_store.cc:276] 0809 23:20:04.889951 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.895926 (+  5975us) service_pool.cc:227] Handling call
0809 23:20:04.941737 (+ 45811us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.958211 (+ 16474us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.958256 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:04.958737 (+   481us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.744284 (+785547us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:05.744340 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.746118 (+  1778us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.164987 (+418869us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:06.200482 (+ 35495us) raft_consensus.cc:1671] finished
0809 23:20:06.200505 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.200720 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1239,"raft.queue_time_us":140,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":17763,"spinlock_wait_cycles":49275648,"thread_start_us":674532,"threads_started":1,"wal-append.queue_time_us":982368}
I20260809 23:20:06.146719 30567 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:06.146606 31361 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:20:06.211601 24317 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.021s	sys 0.036s
I20260809 23:20:06.155658 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:06.251696 24317 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.021s	sys 0.036s
W20260809 23:20:06.215478 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 425758us
I20260809 23:20:06.216992 24311 raft_consensus.cc:1203] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Deduplicated request from leader. Original: 8.7->[9.8-9.8]   Dedup: 9.8->[]
I20260809 23:20:06.220784 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.221221 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:06.253006 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.253472 24539 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:06.223048 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.232183 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:06.205224 31514 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260809 23:20:06.252374 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97015us
I20260809 23:20:06.262755 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.251421 24540 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260809 23:20:06.257899 24311 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260809 23:20:06.265723 24323 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:06.223484 31448 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:06.270598 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.272883 24320 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:06.242102 31524 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:20:06.303568 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.305126 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.306783 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.307530 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.247526 31021 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:06.314116 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.307822 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.249667 31038 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:06.317008 24422 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:06.318063 24312 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.257656 31270 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.345574 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.351784 24312 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:06.346154 30917 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:06.352098 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.279114 30942 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election won for term 9
W20260809 23:20:06.354436 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49413.8us
I20260809 23:20:06.279717 31448 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:06.356551 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:06.372704 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.376274 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.310089 30948 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:06.388057 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.398208 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.399494 24323 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.397337 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.396687 24425 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:06.408741 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.313957 31530 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:06.413793 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.411046 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.402835 24311 raft_consensus.cc:2362] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15. Re-sending same reply.
I20260809 23:20:06.414698 24542 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260809 23:20:06.415084 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:20:06.415678 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.419087 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1044) took 1613 ms. Trace:
I20260809 23:20:06.423974 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.424355 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.425158 24431 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:06.428985 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.419565 24534 rpcz_store.cc:276] 0809 23:20:04.805351 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.888912 (+ 83561us) service_pool.cc:227] Handling call
0809 23:20:04.921647 (+ 32735us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:04.944357 (+ 22710us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:04.944406 (+    49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:05.003089 (+ 58683us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.723219 (+720130us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:05.723257 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.723328 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.418762 (+695434us) raft_consensus.cc:1671] finished
0809 23:20:06.418792 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.419012 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22356,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7748,"thread_start_us":598644,"threads_started":1,"wal-append.queue_time_us":935493}
I20260809 23:20:06.417662 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.434832 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.435205 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.440965 31503 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:06.441483 31510 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:06.441342 31537 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:06.443287 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.446178 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.444116 24423 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:06.450421 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:06.448770 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.454342 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.402640 31260 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.442174 31026 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.451285 24320 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:06.461400 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.409058 30919 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:06.467459 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.476859 24311 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
W20260809 23:20:06.447232 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.010s	sys 0.035s
W20260809 23:20:06.477389 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.010s	sys 0.035s
I20260809 23:20:06.479421 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.481907 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.485399 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1142) took 1569 ms. Trace:
W20260809 23:20:06.489476 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.035s	sys 0.022s
W20260809 23:20:06.491154 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.036s	sys 0.022s
I20260809 23:20:06.378839 31270 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.365332 30917 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.493098 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2810) took 2464 ms. Trace:
I20260809 23:20:06.508049 24323 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260809 23:20:06.403556 31125 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.075s
I20260809 23:20:06.360414 31038 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.518853 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.520143 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.511288 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.511915 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.488162 25366 rpcz_store.cc:276] 0809 23:20:04.916321 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.949033 (+ 32712us) service_pool.cc:227] Handling call
0809 23:20:04.999884 (+ 50851us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:05.007120 (+  7236us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:05.007151 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:05.035751 (+ 28600us) log.cc:842] Serialized 26 byte log entry
0809 23:20:05.754226 (+718475us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:05.754282 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:05.754366 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:06.204515 (+450149us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:06.467150 (+262635us) raft_consensus.cc:1671] finished
0809 23:20:06.467177 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.485303 (+ 18126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6834,"raft.queue_time_us":7427,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":32795,"spinlock_wait_cycles":8240000,"thread_start_us":672221,"threads_started":1,"wal-append.queue_time_us":918737}
I20260809 23:20:06.513275 24317 rpcz_store.cc:276] 0809 23:20:04.029183 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:04.029425 (+   242us) service_pool.cc:227] Handling call
0809 23:20:04.097723 (+ 68298us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:04.118818 (+ 21095us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:04.796658 (+677840us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.489975 (+1693317us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.491638 (+  1663us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:06.491706 (+    68us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:06.491779 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.491860 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.493032 (+  1172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9120,"prepare.queue_time_us":970583,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":132309,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":6847,"thread_start_us":2169264,"threads_started":4,"wal-append.queue_time_us":934587}
I20260809 23:20:06.522099 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.522845 24317 raft_consensus.cc:2354] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:06.524428 24428 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:20:06.524794 31270 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.525810 31270 raft_consensus.cc:2741] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260809 23:20:06.526067 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.527165 24317 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.528760 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.531286 24311 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:06.533155 24317 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12.
I20260809 23:20:06.606734 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.462699 31503 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:06.609800 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 340045us
I20260809 23:20:06.404327 31536 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.126s
I20260809 23:20:06.437649 31175 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.437523 31152 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.623445 24323 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:06.506418 31315 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:06.482694 31542 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:06.631106 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.624083 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.442199 31510 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.528203 31497 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:06.651767 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 348274us
I20260809 23:20:06.527100 31270 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.079s
I20260809 23:20:06.529484 30919 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:06.646076 24425 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:06.646132 24429 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:06.652719 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:06.641058 31490 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260809 23:20:06.482981 31260 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.612844 31441 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:06.648103 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:06.629369 31125 consensus_queue.cc:1225] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260809 23:20:06.474637 31530 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.169s
I20260809 23:20:06.640714 31540 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.640413 31026 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:06.673878 24542 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:06.636837 31549 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:06.636677 31545 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:06.644165 31437 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:06.681521 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.679347 31049 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.688438 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.657475 31526 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:06.701586 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.681136 30585 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:06.696133 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.691183 24323 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.692633 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.693140 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.661942 30942 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:06.694099 24312 raft_consensus.cc:2362] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11. Re-sending same reply.
W20260809 23:20:06.710466 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 244962us
I20260809 23:20:06.704375 24542 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:06.664561 30948 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:06.703437 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 222164us
I20260809 23:20:06.703619 24311 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.656747 31533 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.702400 24320 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:06.632447 31548 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.167s
W20260809 23:20:06.699848 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51856.8us
I20260809 23:20:06.711341 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.681270 31026 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:06.685395 31009 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:06.687204 31579 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:06.721520 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.705433 30585 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:06.710806 31531 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.342s	user 0.000s	sys 0.001s
I20260809 23:20:06.689164 31537 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:06.719106 24312 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:06.750306 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.751799 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.726740 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:06.731523 24422 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:06.754876 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.716502 30948 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:06.710038 31339 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.679s	user 0.007s	sys 0.006s
I20260809 23:20:06.733448 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2795) took 1200 ms. Trace:
I20260809 23:20:06.730911 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:06.722119 31502 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.392s	user 0.001s	sys 0.001s
I20260809 23:20:06.664494 31497 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.711908 31125 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:20:06.724370 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46571.7us
I20260809 23:20:06.792596 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:06.792922 24422 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:06.793556 31587 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:06.794171 31587 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:06.794466 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.795182 24422 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:06.795136 31587 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.795593 30942 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.796269 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.796845 24422 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:20:06.797642 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.798170 24422 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:06.748605 31579 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.799621 31038 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:06.761072 25368 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:06.731032 31300 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.801009 31579 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.801429 31510 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:06.801219 31300 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:06.757925 31534 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.341s	user 0.000s	sys 0.001s
I20260809 23:20:06.709683 31526 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.803303 31300 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.804042 31526 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:06.795360 31152 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:06.784020 31522 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.805733 31152 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:06.805814 30860 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.000s
I20260809 23:20:06.806318 24422 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:06.771328 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:06.807196 31184 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.725s	user 0.009s	sys 0.002s
I20260809 23:20:06.741452 31503 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.807897 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.809515 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.810242 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.730065 30919 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.773727 30890 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:06.779179 25369 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:06.778293 24545 rpcz_store.cc:276] 0809 23:20:05.532922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:05.533078 (+   156us) service_pool.cc:227] Handling call
0809 23:20:05.576301 (+ 43223us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:06.096717 (+520416us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:06.367065 (+270348us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.723117 (+356052us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:06.723212 (+    95us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:06.733158 (+  9946us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.733271 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:06.733392 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":417,"prepare.queue_time_us":551215,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":2031,"raft.queue_time_us":47265,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":11272,"thread_start_us":1008316,"threads_started":3}
I20260809 23:20:06.812074 31584 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.812834 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:06.813169 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:06.813477 24545 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:06.701020 31437 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.814303 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.725992 31441 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.815172 31548 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:20:06.777797 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46987.4us
I20260809 23:20:06.732663 31542 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:20:06.686278 31540 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.800218 31038 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:06.736147 31248 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:06.805974 31522 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:06.795334 31471 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.154s	user 0.015s	sys 0.000s
I20260809 23:20:06.784328 30948 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:06.810658 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.811358 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.812620 31584 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.813731 24534 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260809 23:20:06.732177 31545 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:06.773239 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:06.788956 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:06.853430 24313 raft_consensus.cc:2396] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260809 23:20:06.772939 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:06.818704 24431 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:06.821067 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:06.826200 31558 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260809 23:20:06.825925 31152 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:06.839107 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.847335 24534 pending_rounds.cc:76] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 9
I20260809 23:20:06.857681 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:06.889680 24534 pending_rounds.cc:97] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260809 23:20:06.882329 24427 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:06.883081 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 125184us
I20260809 23:20:06.883937 24311 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.890071 24534 raft_consensus.cc:2866] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:06.890609 24422 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:06.884537 25370 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:06.886293 24545 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:06.893077 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:06.722146 31533 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.819849 31542 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:06.921491 31542 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.815629 31548 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:06.923013 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.734575 31009 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.952530 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.953600 24545 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:06.953840 24316 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:06.951964 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:06.926101 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 105109us
I20260809 23:20:06.938552 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:06.939998 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.003s	sys 0.019s
W20260809 23:20:06.958753 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.004s	sys 0.019s
W20260809 23:20:06.959584 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 603064us
I20260809 23:20:06.956792 24535 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:06.942126 31573 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:06.929786 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:06.967420 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:06.947602 31548 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:06.928598 24431 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:06.823433 31038 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:06.962778 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:06.963057 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:06.968534 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.964763 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:06.962946 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.972229 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:06.977726 24422 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:06.852901 31545 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:06.979650 31545 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.980990 31545 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:06.928825 31175 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:06.947652 31470 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.005s	sys 0.001s
I20260809 23:20:06.815184 31441 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:06.984707 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.984753 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:06.776410 31537 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.806527 30860 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:06.962982 30917 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:06.933189 31524 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.001s	sys 0.004s
I20260809 23:20:06.927414 31533 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:06.999827 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:07.005143 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:07.007551 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44870.4us
W20260809 23:20:07.011060 24424 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.002s
I20260809 23:20:07.011545 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:07.026108 24428 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260809 23:20:07.026188 24431 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:07.027524 24424 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:06.997589 31470 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.006s	sys 0.002s
W20260809 23:20:06.944573 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.006s	sys 0.012s
I20260809 23:20:07.013379 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:06.968123 31542 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:07.031177 24544 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:07.030797 24431 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12.
W20260809 23:20:06.983915 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.005s	sys 0.005s
W20260809 23:20:07.017220 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87548.5us
I20260809 23:20:06.844148 31248 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:07.029063 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.007s	sys 0.012s
I20260809 23:20:06.947185 31592 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:07.029924 24535 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:06.995175 31038 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:07.079450 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 117656us
I20260809 23:20:07.070488 24424 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260809 23:20:07.072852 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.006s	sys 0.005s
I20260809 23:20:07.068782 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:07.073361 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73596.5us
I20260809 23:20:07.083680 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:07.076494 25365 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:07.074153 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48688.5us
I20260809 23:20:07.085335 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:06.999086 31527 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:07.019961 31300 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:07.030572 31501 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260809 23:20:07.098670 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:07.106034 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:07.091558 24425 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:20:07.100741 31021 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.781s	user 0.005s	sys 0.006s
I20260809 23:20:07.022334 31579 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:07.107329 24431 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:07.032814 31500 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:07.076432 31125 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:07.029417 31524 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:07.033021 31547 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:07.023746 31260 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:07.117172 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:07.085700 31437 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:07.133002 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:07.137984 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:07.093107 31549 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:07.156386 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52605.7us
I20260809 23:20:07.103430 31605 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260809 23:20:07.130784 31520 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:07.205050 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2818) took 1389 ms (client timeout 1775 ms). Trace:
I20260809 23:20:06.988255 31573 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:07.093101 31615 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:07.139065 31579 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:07.124238 31175 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:07.109992 30917 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:07.221909 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2377) took 1049 ms. Trace:
I20260809 23:20:07.331419 24321 rpcz_store.cc:276] 0809 23:20:06.172699 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.461143 (+288444us) service_pool.cc:227] Handling call
0809 23:20:07.221732 (+760589us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:07.221852 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1474205952}
W20260809 23:20:07.236732 24313 rpcz_store.cc:269] 0809 23:20:05.815213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:05.815987 (+   774us) service_pool.cc:227] Handling call
0809 23:20:07.204807 (+1388820us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:07.204974 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2383127808}
I20260809 23:20:07.216018 31184 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:07.104817 31300 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:07.339977 24429 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.674s	user 0.011s	sys 0.000s
W20260809 23:20:07.367782 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 152847us
I20260809 23:20:07.159822 31604 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:07.211830 31413 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:07.380244 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 274276us
I20260809 23:20:07.380537 24429 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260809 23:20:07.370785 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 287191us
I20260809 23:20:07.442943 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:07.103520 31527 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:07.445639 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:07.442009 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:07.221863 31498 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:07.427840 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:07.490165 24424 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:07.491743 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63971.4us
W20260809 23:20:07.492978 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51083.7us
I20260809 23:20:07.493970 24496 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12.
I20260809 23:20:07.494323 24496 leader_election.cc:401] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:07.331017 31175 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:07.529407 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:07.368000 31049 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:07.372092 31283 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:07.099802 31592 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:07.529462 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:07.541641 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:07.375385 31184 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:07.442818 31260 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:07.375051 31615 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:07.447448 31604 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:07.451875 31579 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:07.441417 31413 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:07.156630 31547 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:07.627539 31623 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
W20260809 23:20:07.541649 31518 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260809 23:20:07.565376 31184 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:07.628366 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2655) took 1410 ms. Trace:
I20260809 23:20:07.701752 24541 rpcz_store.cc:276] 0809 23:20:06.217927 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.218046 (+   119us) service_pool.cc:227] Handling call
0809 23:20:06.218407 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.218786 (+   379us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:20:06.218811 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.228104 (+  9293us) log.cc:842] Serialized 26 byte log entry
0809 23:20:07.081777 (+853673us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:07.081833 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.081921 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:07.628122 (+546201us) raft_consensus.cc:1671] finished
0809 23:20:07.628144 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:07.628288 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80223,"raft.queue_time_us":244065,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":39711,"thread_start_us":779423,"threads_started":1,"wal-append.queue_time_us":1211774}
I20260809 23:20:07.613670 31599 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:07.629031 31617 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.298s	user 0.000s	sys 0.000s
I20260809 23:20:07.629320 30890 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:07.630093 31500 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:07.630483 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1214) took 1696 ms. Trace:
I20260809 23:20:07.642076 31497 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:07.545921 31049 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:07.375290 30917 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:07.544874 31175 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:07.544031 31522 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:07.659947 31604 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:07.547775 31283 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:07.712903 25365 rpcz_store.cc:276] 0809 23:20:05.934478 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:05.975098 (+ 40620us) service_pool.cc:227] Handling call
0809 23:20:07.630204 (+1655106us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540055f854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:07.630406 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2373067520}
I20260809 23:20:07.644096 31628 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.014s
I20260809 23:20:07.244184 31549 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:07.426321 31336 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.191s
W20260809 23:20:07.847990 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.058s	sys 0.158s
W20260809 23:20:07.896682 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.059s	sys 0.158s
W20260809 23:20:07.897034 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 801378us
W20260809 23:20:07.838128 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.008s	sys 0.041s
W20260809 23:20:07.898162 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.008s	sys 0.042s
I20260809 23:20:07.702527 31441 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:07.902143 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2398) took 1130 ms. Trace:
I20260809 23:20:07.724665 30991 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:20:07.838032 31594 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260809 23:20:07.731395 30860 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:07.828351 31049 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:07.721812 31009 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:07.828707 31152 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:07.896036 31501 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
I20260809 23:20:07.924844 25365 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260809 23:20:07.908775 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.013s	sys 0.031s
W20260809 23:20:07.962862 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.014s	sys 0.031s
W20260809 23:20:07.963528 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 825566us
I20260809 23:20:07.964816 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:07.676813 31260 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:07.915903 24427 rpcz_store.cc:276] 0809 23:20:06.771774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.982297 (+210523us) service_pool.cc:227] Handling call
0809 23:20:06.982539 (+   242us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:06.982724 (+   185us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:07.210255 (+227531us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.901663 (+691408us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:07.901736 (+    73us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:07.901819 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.901906 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:07.902072 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2952,"prepare.queue_time_us":506879,"prepare.run_cpu_time_us":4124,"prepare.run_wall_time_us":91029,"raft.queue_time_us":11686,"thread_start_us":907005,"threads_started":2}
I20260809 23:20:07.834218 31540 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:07.558637 31524 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:07.855363 31283 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:07.974171 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:07.976470 25372 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:08.023157 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2806) took 1081 ms. Trace:
W20260809 23:20:07.934366 31316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.091s	user 0.004s	sys 0.007s
W20260809 23:20:08.026176 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61417.4us
I20260809 23:20:08.042394 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:08.042671 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 94885us
I20260809 23:20:08.045159 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2808) took 1103 ms. Trace:
I20260809 23:20:08.045535 24541 rpcz_store.cc:276] 0809 23:20:06.941287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.720764 (+779477us) service_pool.cc:227] Handling call
0809 23:20:07.721251 (+   487us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.721518 (+   267us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:07.721841 (+   323us) log.cc:842] Serialized 14 byte log entry
0809 23:20:08.042185 (+320344us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:08.043349 (+  1164us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:08.044698 (+  1349us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.044886 (+   188us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.045087 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5876,"prepare.queue_time_us":717,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"thread_start_us":311965,"threads_started":1}
I20260809 23:20:08.046861 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:07.984575 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:08.049316 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:08.040664 24535 rpcz_store.cc:276] 0809 23:20:06.940665 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.504884 (+564219us) service_pool.cc:227] Handling call
0809 23:20:07.564565 (+ 59681us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.615272 (+ 50707us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:07.631112 (+ 15840us) log.cc:842] Serialized 14 byte log entry
0809 23:20:08.021533 (+390421us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:08.021599 (+    66us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:08.021652 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.021732 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.021893 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45630,"prepare.queue_time_us":30922,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":344342,"threads_started":2}
I20260809 23:20:07.929435 31537 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:07.959431 31536 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.002s	sys 0.005s
I20260809 23:20:07.923489 31565 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:07.905267 31576 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:07.934911 31315 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:08.104004 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2868) took 1330 ms. Trace:
I20260809 23:20:08.104538 24431 rpcz_store.cc:276] 0809 23:20:06.773304 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.078083 (+304779us) service_pool.cc:227] Handling call
0809 23:20:07.107084 (+ 29001us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.121811 (+ 14727us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.103746 (+981935us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b540017f7d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:08.103929 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4745088}
W20260809 23:20:08.107491 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.024s	sys 0.028s
W20260809 23:20:08.107900 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.025s	sys 0.028s
W20260809 23:20:08.108237 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 571069us
I20260809 23:20:08.116744 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:07.984678 31009 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:08.050879 31316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.207s	user 0.005s	sys 0.007s
I20260809 23:20:08.122982 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:08.132206 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:08.052932 31623 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:20:08.034418 30890 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:08.171748 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:08.225809 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54145.1us
I20260809 23:20:08.172757 24535 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:08.175268 31530 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:20:08.175792 31593 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.852s	user 0.001s	sys 0.001s
I20260809 23:20:08.178201 31569 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:08.131664 31565 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.117132 31315 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:08.116333 31576 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.181326 31591 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
W20260809 23:20:08.102705 31536 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.003s	sys 0.005s
I20260809 23:20:08.423630 31591 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:08.191907 31655 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260809 23:20:08.196049 30917 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:08.196149 31592 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:08.197688 31654 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:20:08.199529 31260 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:08.228986 31573 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:08.203097 31549 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:08.205338 31660 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260809 23:20:08.205691 31026 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:08.207381 31607 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.641s	user 0.002s	sys 0.000s
I20260809 23:20:07.980749 31575 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.290s
W20260809 23:20:08.543082 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 500729us
I20260809 23:20:08.211416 24313 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:07.974890 31496 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.146s
I20260809 23:20:08.361003 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:08.134272 31594 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260809 23:20:08.376082 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:08.551517 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190670us
W20260809 23:20:08.377635 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.048s	sys 0.097s
I20260809 23:20:07.981894 31152 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.208734 31546 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:08.362047 31510 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:08.476624 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:08.578471 24311 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:08.478192 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1161) took 1769 ms. Trace:
I20260809 23:20:08.200515 31527 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:08.399544 31530 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:08.397770 31315 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:08.490396 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.718s	user 0.022s	sys 0.073s
W20260809 23:20:08.508962 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.002s	sys 0.001s
W20260809 23:20:08.674515 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.002s	sys 0.002s
W20260809 23:20:08.517889 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.000s	sys 0.003s
I20260809 23:20:08.479568 31596 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 12
I20260809 23:20:08.208688 31441 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:08.093339 31579 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.235836 31009 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.448501 31300 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:08.544651 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:08.535164 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:07.981716 31615 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.550379 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:08.552093 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.048s	sys 0.097s
I20260809 23:20:08.557332 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2824) took 1029 ms. Trace:
I20260809 23:20:08.557914 24535 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
W20260809 23:20:08.615643 31548 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.606s	user 0.002s	sys 0.011s
I20260809 23:20:08.503938 31592 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:08.535953 31448 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 2.124s	user 0.023s	sys 0.004s
W20260809 23:20:08.567070 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.002s	sys 0.002s
W20260809 23:20:08.749817 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.078s	user 0.002s	sys 0.002s
W20260809 23:20:08.565991 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.002s	sys 0.002s
W20260809 23:20:08.752882 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.048s	user 0.002s	sys 0.002s
W20260809 23:20:08.753280 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.79018e+06us
I20260809 23:20:08.523564 30942 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:08.564220 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.029s	user 0.003s	sys 0.002s
W20260809 23:20:08.763312 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.228s	user 0.003s	sys 0.003s
W20260809 23:20:08.566507 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.450s	user 0.000s	sys 0.005s
I20260809 23:20:08.579530 24320 rpcz_store.cc:276] 0809 23:20:06.708641 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.717792 (+  9151us) service_pool.cc:227] Handling call
0809 23:20:06.735889 (+ 18097us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:06.756553 (+ 20664us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.756587 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:06.757133 (+   546us) log.cc:842] Serialized 52 byte log entry
0809 23:20:06.939903 (+182770us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:06.940243 (+   340us) log.cc:842] Serialized 14 byte log entry
0809 23:20:06.967289 (+ 27046us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.967372 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:08.477936 (+1510564us) raft_consensus.cc:1671] finished
0809 23:20:08.477965 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.478123 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42528,"prepare.queue_time_us":350,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":70501,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":14883,"thread_start_us":165411,"threads_started":1,"wal-append.queue_time_us":374967}
W20260809 23:20:08.614192 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 377715us
W20260809 23:20:08.675864 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.000s	sys 0.004s
W20260809 23:20:08.764258 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.648s	user 0.000s	sys 0.005s
I20260809 23:20:08.764824 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:08.765713 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.75226e+06us
I20260809 23:20:08.675525 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2805) took 1735 ms. Trace:
I20260809 23:20:08.510591 30917 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.089s
W20260809 23:20:08.647445 31339 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.684s	user 0.004s	sys 0.009s
I20260809 23:20:08.564397 31549 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
W20260809 23:20:08.714903 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 170308us
W20260809 23:20:08.716658 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.27105e+06us
W20260809 23:20:08.525763 31470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.377s	user 0.000s	sys 0.003s
I20260809 23:20:08.716537 24313 rpcz_store.cc:276] 0809 23:20:07.528181 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.529134 (+   953us) service_pool.cc:227] Handling call
0809 23:20:08.210890 (+681756us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:08.211191 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:08.557043 (+345852us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.557262 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44923264}
W20260809 23:20:08.677030 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.905s	user 0.033s	sys 0.084s
I20260809 23:20:08.694842 25365 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:08.619814 31496 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:08.688829 31649 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
W20260809 23:20:08.764690 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.80523e+06us
I20260809 23:20:08.795313 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:08.812621 24535 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:08.813691 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:08.377856 31587 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.579s	user 0.022s	sys 0.052s
W20260809 23:20:08.771322 24545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.003s	sys 0.031s
W20260809 23:20:08.815129 24545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.102s	user 0.003s	sys 0.032s
I20260809 23:20:08.676658 31527 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:08.695271 31605 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:08.767402 24542 rpcz_store.cc:276] 0809 23:20:06.940320 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.206516 (+266196us) service_pool.cc:227] Handling call
0809 23:20:07.206879 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.207130 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:07.428057 (+220927us) log.cc:842] Serialized 14 byte log entry
0809 23:20:08.674997 (+1246940us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:08.675081 (+    84us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:08.675145 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.675268 (+   123us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.675456 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49681,"prepare.queue_time_us":233679,"prepare.run_cpu_time_us":3664,"prepare.run_wall_time_us":19633,"raft.queue_time_us":730832,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":27574,"thread_start_us":1233240,"threads_started":2}
I20260809 23:20:08.780938 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:08.781257 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:08.847805 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80906.6us
I20260809 23:20:08.848616 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:08.692307 31009 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:08.794484 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:08.816794 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2801) took 2031 ms. Trace:
I20260809 23:20:08.855795 24544 rpcz_store.cc:276] 0809 23:20:06.785155 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.069697 (+284542us) service_pool.cc:227] Handling call
0809 23:20:07.104132 (+ 34435us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.947371 (+843239us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:07.947417 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:07.947469 (+    52us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:07.947514 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.816523 (+869009us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:08.816582 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.816734 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1938603776}
I20260809 23:20:08.715737 31573 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:08.735843 31569 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:08.815893 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1062) took 1890 ms. Trace:
I20260809 23:20:08.865453 24545 rpcz_store.cc:276] 0809 23:20:06.925545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.128270 (+202725us) service_pool.cc:227] Handling call
0809 23:20:07.183539 (+ 55269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.212683 (+ 29144us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:07.523987 (+311304us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.711957 (+187970us) log.cc:842] Serialized 14 byte log entry
0809 23:20:08.815507 (+1103550us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:08.815567 (+    60us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:08.815633 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.815710 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.815836 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250594,"prepare.queue_time_us":329979,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":85,"raft.queue_time_us":1494329,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":155022,"thread_start_us":1289537,"threads_started":2}
W20260809 23:20:08.763690 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.2698e+06us
I20260809 23:20:08.715135 31336 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:08.647637 31547 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:08.827414 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:08.850430 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69589.6us
I20260809 23:20:08.869321 24535 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:08.761468 31300 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:08.794133 31339 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.839s	user 0.005s	sys 0.009s
W20260809 23:20:08.794916 31470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.655s	user 0.000s	sys 0.004s
I20260809 23:20:08.557554 30890 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:08.822606 31587 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.024s	user 0.023s	sys 0.052s
I20260809 23:20:08.690352 31441 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.881059 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1150) took 3007 ms. Trace:
I20260809 23:20:08.891649 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:08.892879 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:08.822983 31527 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:08.823057 31316 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:08.894783 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2663) took 2648 ms. Trace:
W20260809 23:20:08.864552 31638 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.466s	user 0.001s	sys 0.000s
W20260809 23:20:08.864259 31636 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.349s	user 0.001s	sys 0.000s
I20260809 23:20:08.885028 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:08.892983 31651 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:08.906749 24427 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:08.959786 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:08.910686 24431 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260809 23:20:08.920298 24539 rpcz_store.cc:276] 0809 23:20:06.245966 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.268975 (+ 23009us) service_pool.cc:227] Handling call
0809 23:20:06.269333 (+   358us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.682782 (+413449us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.682809 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.683252 (+   443us) log.cc:842] Serialized 26 byte log entry
0809 23:20:06.937773 (+254521us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:06.937827 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:06.937923 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:08.894553 (+1956630us) raft_consensus.cc:1671] finished
0809 23:20:08.894580 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.894723 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10321,"raft.queue_time_us":240726,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":88029,"thread_start_us":631338,"threads_started":2,"wal-append.queue_time_us":638478}
I20260809 23:20:08.904492 24309 rpcz_store.cc:276] 0809 23:20:05.873978 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:05.888851 (+ 14873us) service_pool.cc:227] Handling call
0809 23:20:05.914249 (+ 25398us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.081819 (+167570us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:06.444962 (+363143us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.012437 (+567475us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:07.034903 (+ 22466us) log.cc:842] Serialized 26 byte log entry
0809 23:20:07.035118 (+   215us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:07.035163 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.203614 (+168451us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:07.203665 (+    51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:08.864467 (+1660802us) raft_consensus.cc:1671] finished
0809 23:20:08.880774 (+ 16307us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:08.880977 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89898,"prepare.queue_time_us":633807,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":5694,"raft.queue_time_us":24830,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":21728,"spinlock_wait_cycles":374006528,"thread_start_us":786611,"threads_started":2,"wal-append.queue_time_us":2230276}
I20260809 23:20:08.966774 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:08.879802 31624 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.447s	user 0.000s	sys 0.001s
I20260809 23:20:08.762063 31592 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.738160 31548 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.864938 31549 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:08.565164 31591 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.780500 31448 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:08.969002 31175 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.070s	user 0.013s	sys 0.000s
I20260809 23:20:08.951833 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:08.952416 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:08.874938 31009 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:08.688676 30585 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.874271 31569 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:08.886988 31547 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:08.736132 31530 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.863932 31605 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:08.922204 31649 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
W20260809 23:20:09.061661 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 267746us
I20260809 23:20:08.896406 31270 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:08.845263 31496 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:08.794463 30942 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.994031 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:08.846238 31546 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20260809 23:20:08.919386 31527 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:08.955884 30948 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 2.027s	user 0.009s	sys 0.006s
I20260809 23:20:08.928359 31587 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:08.879390 31573 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:08.886605 31300 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.039249 24541 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:08.889457 31336 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.009696 31651 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260809 23:20:09.052381 24546 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:09.055873 25365 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:09.077409 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:09.201915 31648 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:09.073199 30917 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:09.167258 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.042431 24429 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:20:09.211185 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 259423us
I20260809 23:20:09.261050 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:09.100934 31152 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.032611 31175 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.201553 31609 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.689s	user 0.009s	sys 0.012s
I20260809 23:20:09.073963 30585 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:09.043818 31569 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:20:09.289443 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 278418us
I20260809 23:20:09.045611 31009 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.371335 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:09.211311 31529 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.211185 25375 raft_consensus.cc:1203] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Deduplicated request from leader. Original: 8.7->[10.8-10.8]   Dedup: 10.8->[]
W20260809 23:20:09.211534 31645 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.652s	user 0.001s	sys 0.001s
I20260809 23:20:09.052289 31547 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.317067 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.017s (SENT) [suppressed 13 similar messages]
I20260809 23:20:09.212456 31615 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:09.387055 31615 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:09.387353 31441 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.387531 31615 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:09.387841 31300 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.387956 31584 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.388507 31300 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:09.388890 31300 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:09.389211 31300 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:09.389603 31049 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.403s	user 0.004s	sys 0.007s
I20260809 23:20:09.379992 25375 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260809 23:20:09.390020 31049 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.376268 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1054) took 2941 ms. Trace:
I20260809 23:20:09.391254 24534 rpcz_store.cc:276] 0809 23:20:06.434891 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.435152 (+   261us) service_pool.cc:227] Handling call
0809 23:20:06.475566 (+ 40414us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:06.930979 (+455413us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.370811 (+2439832us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:09.376199 (+  5388us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":776800128}
I20260809 23:20:09.393234 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2396) took 2622 ms. Trace:
I20260809 23:20:09.393635 24422 rpcz_store.cc:276] 0809 23:20:06.771194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.978320 (+207126us) service_pool.cc:227] Handling call
0809 23:20:06.978638 (+   318us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:07.018433 (+ 39795us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:07.018478 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:07.019186 (+   708us) log.cc:842] Serialized 52 byte log entry
0809 23:20:07.501081 (+481895us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:07.501132 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.501211 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.393007 (+1891796us) raft_consensus.cc:1671] finished
0809 23:20:09.393031 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.393162 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45035,"raft.queue_time_us":89834,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":20370,"thread_start_us":407527,"threads_started":1,"wal-append.queue_time_us":1193738}
W20260809 23:20:09.380105 31038 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 2.272s	user 0.014s	sys 0.002s
I20260809 23:20:09.395344 31652 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.212482 31668 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:09.398764 31620 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.638s	user 0.001s	sys 0.001s
I20260809 23:20:09.224913 31565 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.380883 31536 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.384262 31175 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.272014 31576 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.384758 31592 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.400790 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1148) took 2676 ms. Trace:
I20260809 23:20:09.381145 30860 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.401394 30860 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.401183 25373 rpcz_store.cc:276] 0809 23:20:06.724630 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.724762 (+   132us) service_pool.cc:227] Handling call
0809 23:20:06.745404 (+ 20642us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:06.816377 (+ 70973us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:06.885338 (+ 68961us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.633877 (+748539us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b180024f640. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:07.652446 (+ 18569us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.919654 (+267208us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:07.919712 (+    58us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:07.919760 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.400578 (+1480818us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:09.400623 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.400735 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":523101,"prepare.queue_time_us":118190,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":1144812,"raft.run_cpu_time_us":483,"raft.run_wall_time_us":180200,"spinlock_wait_cycles":191353472}
I20260809 23:20:09.357385 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1027) took 2512 ms. Trace:
I20260809 23:20:09.388527 31579 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:09.407178 24428 rpcz_store.cc:276] 0809 23:20:06.844667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.367806 (+523139us) service_pool.cc:227] Handling call
0809 23:20:09.356696 (+1988890us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:09.357293 (+   597us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":474185088}
I20260809 23:20:09.407766 31579 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:09.408300 31579 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:09.395347 31038 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.383400 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.363153 31628 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.411474 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2872) took 2637 ms. Trace:
I20260809 23:20:09.354734 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1149) took 2628 ms. Trace:
I20260809 23:20:09.348012 30942 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.413328 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2670) took 1369 ms. Trace:
I20260809 23:20:09.412101 25371 rpcz_store.cc:276] 0809 23:20:06.725904 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.735138 (+  9234us) service_pool.cc:227] Handling call
0809 23:20:06.761237 (+ 26099us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:06.761445 (+   208us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:06.761777 (+   332us) log.cc:842] Serialized 14 byte log entry
0809 23:20:08.747733 (+1985956us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:09.354052 (+606319us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b18002a7e40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7
0809 23:20:09.354286 (+   234us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:09.354353 (+    67us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:09.354428 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.354519 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.354666 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":766034,"prepare.queue_time_us":1443,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":899011,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":9549,"spinlock_wait_cycles":16195840,"thread_start_us":390045,"threads_started":2,"wal-append.queue_time_us":330781}
I20260809 23:20:09.413750 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2391) took 2678 ms. Trace:
I20260809 23:20:09.413663 24541 rpcz_store.cc:276] 0809 23:20:08.043903 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.083008 (+ 39105us) service_pool.cc:227] Handling call
0809 23:20:08.087452 (+  4444us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:09.165078 (+1077626us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.413105 (+248027us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400147b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:09.413273 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2188371712}
I20260809 23:20:09.414322 24435 rpcz_store.cc:276] 0809 23:20:06.735699 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.746804 (+ 11105us) service_pool.cc:227] Handling call
0809 23:20:06.771471 (+ 24667us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.771806 (+   335us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.771831 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.772241 (+   410us) log.cc:842] Serialized 26 byte log entry
0809 23:20:08.796407 (+2024166us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:08.796444 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:08.796528 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.413552 (+617024us) raft_consensus.cc:1671] finished
0809 23:20:09.413571 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.413699 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":48592,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":22390,"thread_start_us":2023664,"threads_started":1,"wal-append.queue_time_us":2639657}
I20260809 23:20:09.416320 31688 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:20:09.418008 31677 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.607s	user 0.000s	sys 0.001s
I20260809 23:20:09.386435 31691 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.420812 31413 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.423117 31630 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.732s	user 0.015s	sys 0.009s
I20260809 23:20:09.411863 24424 rpcz_store.cc:276] 0809 23:20:06.774440 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.099169 (+324729us) service_pool.cc:227] Handling call
0809 23:20:07.457969 (+358800us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:07.510242 (+ 52273us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.411281 (+1901039us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400061650. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:09.411418 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":776780544}
I20260809 23:20:09.424024 31537 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:20:09.386044 24430 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2382) took 2971 ms (client timeout 2307 ms). Trace:
W20260809 23:20:09.251626 31638 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.184s	user 0.006s	sys 0.000s
I20260809 23:20:09.425727 31530 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.427611 31248 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.427915 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1158) took 2744 ms. Trace:
I20260809 23:20:09.428283 31248 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:09.428501 24323 rpcz_store.cc:276] 0809 23:20:06.683112 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.713655 (+ 30543us) service_pool.cc:227] Handling call
0809 23:20:06.714113 (+   458us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:06.714543 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.714581 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:06.720111 (+  5530us) log.cc:842] Serialized 52 byte log entry
0809 23:20:07.132073 (+411962us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:07.215617 (+ 83544us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.215803 (+   186us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.215884 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.427472 (+2211588us) raft_consensus.cc:1671] finished
0809 23:20:09.427509 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.427819 (+   310us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":89328,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":111787,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":51380,"thread_start_us":357964,"threads_started":1,"wal-append.queue_time_us":913105}
I20260809 23:20:09.166432 30948 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.431337 31573 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.399950 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1048) took 3585 ms. Trace:
I20260809 23:20:09.433005 24543 rpcz_store.cc:276] 0809 23:20:05.814437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:05.814636 (+   199us) service_pool.cc:227] Handling call
0809 23:20:05.844585 (+ 29949us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.242154 (+397569us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.242186 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.276766 (+ 34580us) log.cc:842] Serialized 26 byte log entry
0809 23:20:07.071713 (+794947us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:07.071791 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.071917 (+   126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.399710 (+2327793us) raft_consensus.cc:1671] finished
0809 23:20:09.399734 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.399892 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":194049,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":2445,"thread_start_us":1068391,"threads_started":2,"wal-append.queue_time_us":2213440}
I20260809 23:20:09.386125 31547 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:09.425628 24430 rpcz_store.cc:269] 0809 23:20:06.414363 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.414509 (+   146us) service_pool.cc:227] Handling call
0809 23:20:09.385816 (+2971307us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:09.385952 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1534830464}
I20260809 23:20:09.386528 31712 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.090056 31270 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.209751 31587 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.331548 31545 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 2.350s	user 0.022s	sys 0.000s
I20260809 23:20:09.422011 31413 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:09.471837 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.895s	user 0.013s	sys 0.019s
W20260809 23:20:09.411707 31637 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
W20260809 23:20:09.486261 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.005s	sys 0.000s
I20260809 23:20:09.449855 30860 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.391746 31605 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.419867 31691 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.473050 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.013s	sys 0.019s
I20260809 23:20:09.395906 31652 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.375883 31678 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:09.397426 31668 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.400585 31536 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.411489 31628 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.346331 31681 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:09.192502 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.472463 31696 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:09.481915 31283 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.442s	user 0.008s	sys 0.005s
W20260809 23:20:09.406494 31510 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.791s	user 0.005s	sys 0.001s
I20260809 23:20:09.487175 31049 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.508127 31662 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:09.602025 31610 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 23:20:06.449062 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45888 (local address 127.21.255.2:42099)
0809 23:20:08.726977 (+2277915us) server_negotiation.cc:206] Beginning negotiation
0809 23:20:08.737340 (+ 10363us) server_negotiation.cc:399] Waiting for connection header
0809 23:20:08.778484 (+ 41144us) server_negotiation.cc:407] Connection header received
0809 23:20:08.863319 (+ 84835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 23:20:08.875482 (+ 12163us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 23:20:08.977753 (+102271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 23:20:09.031088 (+ 53335us) server_negotiation.cc:226] Negotiated authn=SASL
0809 23:20:09.385241 (+354153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45888: BlockingRecv error: recv got EOF from 127.0.0.1:45888 (error 108)
Metrics: {"server-negotiator.queue_time_us":1678571,"thread_start_us":372599,"threads_started":1}
W20260809 23:20:09.521181 31339 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.017s	sys 0.048s
I20260809 23:20:09.506206 30948 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:09.348573 31635 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.611s	user 0.000s	sys 0.006s
I20260809 23:20:09.325667 31549 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.609905 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:09.610751 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.023s	sys 0.058s
W20260809 23:20:09.522313 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.006s	sys 0.000s
I20260809 23:20:09.432619 31529 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.620868 25369 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:09.679428 31536 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.683632 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2388) took 3039 ms. Trace:
W20260809 23:20:09.670691 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.023s	sys 0.058s
I20260809 23:20:09.684165 24423 rpcz_store.cc:276] 0809 23:20:06.644721 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.644882 (+   161us) service_pool.cc:227] Handling call
0809 23:20:06.672339 (+ 27457us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.682734 (+ 10395us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.682765 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.699912 (+ 17147us) log.cc:842] Serialized 26 byte log entry
0809 23:20:07.139955 (+440043us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:07.139997 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.140065 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.683323 (+2543258us) raft_consensus.cc:1671] finished
0809 23:20:09.683351 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.683556 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7810,"raft.queue_time_us":22948,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":6311,"spinlock_wait_cycles":51584,"thread_start_us":400461,"threads_started":1,"wal-append.queue_time_us":2911282}
W20260809 23:20:09.670894 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 704183us
W20260809 23:20:09.671219 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61480.7us
I20260809 23:20:09.344127 31548 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:09.626776 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.011s	sys 0.032s
I20260809 23:20:09.578270 31500 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.604523 31448 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.485690 31712 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.577126 31009 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.610580 31499 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.615952 31678 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.669445 31724 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
W20260809 23:20:09.703390 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.011s	sys 0.032s
I20260809 23:20:09.748626 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:09.626989 31651 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260809 23:20:09.627540 31336 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.709578 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:09.711859 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:09.719890 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.503s	user 0.008s	sys 0.016s
W20260809 23:20:09.769625 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.553s	user 0.008s	sys 0.016s
W20260809 23:20:09.669781 31339 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.017s	sys 0.048s
I20260809 23:20:09.660715 31510 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:09.735025 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:09.652351 31681 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:09.704882 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.683796 31496 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:09.762202 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.002s	sys 0.023s
I20260809 23:20:09.762305 24313 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:09.766785 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 395518us
I20260809 23:20:09.651366 31283 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:09.794154 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.018s	sys 0.044s
I20260809 23:20:09.797113 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2828) took 2150 ms. Trace:
W20260809 23:20:09.801517 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.002s	sys 0.023s
W20260809 23:20:09.809513 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.01502e+06us
I20260809 23:20:09.810832 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:09.802726 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.018s	sys 0.044s
W20260809 23:20:09.815011 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 930011us
W20260809 23:20:09.753225 31647 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.014s	sys 0.008s
W20260809 23:20:09.804921 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 487851us
I20260809 23:20:09.802587 24316 rpcz_store.cc:276] 0809 23:20:07.647111 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.647340 (+   229us) service_pool.cc:227] Handling call
0809 23:20:08.215863 (+568523us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:08.216105 (+   242us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:08.216374 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:08.216729 (+   355us) log.cc:842] Serialized 14 byte log entry
0809 23:20:09.796702 (+1579973us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:09.796761 (+    59us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:09.796821 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.796908 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.797042 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26627,"prepare.queue_time_us":670,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":10959,"raft.queue_time_us":27577,"spinlock_wait_cycles":24290560,"thread_start_us":1552940,"threads_started":1}
I20260809 23:20:09.824077 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:09.786036 31021 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:09.831321 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.832474 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:09.807684 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.011s	sys 0.026s
I20260809 23:20:09.838951 25365 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:20:09.839731 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.012s	sys 0.026s
W20260809 23:20:09.841737 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 764474us
I20260809 23:20:09.836356 31529 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.839040 24316 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:09.848322 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:09.848641 31704 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260809 23:20:09.849129 31604 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.936s	user 0.014s	sys 0.000s
I20260809 23:20:09.850028 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:09.850732 31184 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 2.117s	user 0.011s	sys 0.004s
I20260809 23:20:09.850819 24323 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:09.851257 31184 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.851603 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:09.849390 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:09.852608 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:09.853252 31184 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.854552 31339 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.855007 31283 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.855120 31339 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.856032 31691 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.681560 31662 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.857161 31339 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.858183 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.859092 31546 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.859788 25369 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:20:09.859584 31546 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:09.861204 30991 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.938s	user 0.011s	sys 0.003s
I20260809 23:20:09.861657 30991 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:09.861747 31546 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.862236 31316 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:09.862689 31592 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:09.864709 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:09.936946 31592 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.937455 31592 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:09.937515 31125 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260809 23:20:09.865381 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63414.7us
I20260809 23:20:09.866076 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:09.867628 31725 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260809 23:20:09.868850 31681 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:09.944948 30585 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.568s	user 0.007s	sys 0.003s
I20260809 23:20:09.945761 24317 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:09.946327 31471 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.849515 31604 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.946841 31471 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.948626 31604 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.949198 31604 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:09.949913 31471 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.949712 31604 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.899151 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.891573 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:09.951053 25373 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 8.
W20260809 23:20:09.951325 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79682.7us
I20260809 23:20:09.954242 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:09.954851 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:09.955152 24320 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:09.955703 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:09.844225 31587 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.858778 31688 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.957105 31688 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:09.808736 31668 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.958992 31038 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:09.959285 31688 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.959882 31688 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:20:09.960177 31688 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:09.960465 31688 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:09.962287 31648 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:09.962838 31648 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:09.964841 31569 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.543s	user 0.004s	sys 0.004s
I20260809 23:20:09.965811 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:09.867794 31270 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.838562 31021 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:09.838713 31510 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:09.849593 31568 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 23:20:06.313915 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.255.2:42099 (local address 127.0.0.1:45888)
0809 23:20:06.768160 (+454245us) negotiation.cc:107] Waiting for socket to connect
0809 23:20:06.768248 (+    88us) client_negotiation.cc:174] Beginning negotiation
0809 23:20:06.768723 (+   475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 23:20:09.031538 (+2262815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 23:20:09.037946 (+  6408us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 23:20:09.119897 (+ 81951us) client_negotiation.cc:189] Negotiated authn=SASL
0809 23:20:09.383486 (+263589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 23:20:09.383634 (+   148us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 23:20:09.384225 (+   591us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.21.255.2:42099: sent 0 of 16 requested bytes
Metrics: {"client-negotiator.queue_time_us":440397,"thread_start_us":344291,"threads_started":1}
I20260809 23:20:09.894088 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:09.895046 31657 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.834s	user 0.000s	sys 0.002s
I20260809 23:20:09.896654 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2661) took 3651 ms. Trace:
I20260809 23:20:10.007648 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:10.008226 24546 rpcz_store.cc:276] 0809 23:20:06.245339 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.263139 (+ 17800us) service_pool.cc:227] Handling call
0809 23:20:06.302887 (+ 39748us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:09.209258 (+2906371us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.896401 (+687143us) spinlock_profiling.cc:243] Waited 2.75 s on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:09.896593 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6322685824}
I20260809 23:20:09.887944 25375 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:20:09.901038 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:10.011922 25368 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:20:09.905406 31596 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.216s	user 0.012s	sys 0.002s
W20260809 23:20:09.824947 31542 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 2.751s	user 0.018s	sys 0.001s
I20260809 23:20:09.799914 31727 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:09.926736 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:09.928881 31546 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:09.844609 31605 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:09.825645 31647 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.015s	sys 0.009s
I20260809 23:20:10.026350 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.027107 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:09.929700 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:09.859639 31652 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.934285 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:09.890343 31712 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:09.993746 31510 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:09.937659 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73042.7us
W20260809 23:20:09.944239 24356 heartbeater.cc:644] Failed to heartbeat to 127.21.255.62:37921 (1 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.255.62:37921 timed out after 15.000s (SENT)
I20260809 23:20:09.955817 31260 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:09.956354 24323 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:20:09.875576 31723 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260809 23:20:09.748817 31499 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:09.726696 31500 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.045322 31510 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:10.060968 31500 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.066190 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:09.954885 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2833) took 1490 ms. Trace:
I20260809 23:20:10.068594 31499 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.068761 24312 rpcz_store.cc:276] 0809 23:20:08.464985 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.622570 (+157585us) service_pool.cc:227] Handling call
0809 23:20:08.725707 (+103137us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:08.992151 (+266444us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:09.080044 (+ 87893us) log.cc:842] Serialized 14 byte log entry
0809 23:20:09.461346 (+381302us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:09.461418 (+    72us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:09.461474 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.954217 (+492743us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b18004724e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:20:09.954670 (+   453us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:09.954711 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:09.954818 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22865,"prepare.queue_time_us":404530,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":50,"raft.queue_time_us":22984,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":16020,"spinlock_wait_cycles":636901376,"thread_start_us":428005,"threads_started":2,"wal-append.queue_time_us":646870}
W20260809 23:20:10.070904 31703 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.184s	user 0.001s	sys 0.002s
I20260809 23:20:09.988276 31604 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:09.989298 24321 raft_consensus.cc:2354] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:09.989467 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:10.075996 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1063) took 1321 ms. Trace:
I20260809 23:20:10.076413 24544 rpcz_store.cc:276] 0809 23:20:08.754546 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.857044 (+102498us) service_pool.cc:227] Handling call
0809 23:20:08.887483 (+ 30439us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:08.901622 (+ 14139us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:08.901661 (+    39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:08.902263 (+   602us) log.cc:842] Serialized 52 byte log entry
0809 23:20:09.260027 (+357764us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:09.260081 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.260168 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:10.075762 (+815594us) raft_consensus.cc:1671] finished
0809 23:20:10.075781 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:10.075941 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":365899,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":72085,"thread_start_us":246716,"threads_started":1,"wal-append.queue_time_us":626166}
I20260809 23:20:10.044436 31260 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:10.078017 31260 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:10.028266 31546 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.048153 31021 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.085172 31021 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:10.085314 31546 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.039566 31668 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.087147 24430 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:10.087510 31668 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.089617 31668 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:10.047644 31545 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260809 23:20:10.021972 31596 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12.
I20260809 23:20:10.019495 31587 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.095405 31587 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.097879 31587 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.071022 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:09.954288 31615 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.566s	user 0.015s	sys 0.004s
I20260809 23:20:09.994966 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:09.995936 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.063s	sys 0.082s
W20260809 23:20:09.998299 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.020s	sys 0.025s
W20260809 23:20:10.110535 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.063s	sys 0.082s
I20260809 23:20:09.999802 25369 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:10.000133 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:10.111913 25373 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:10.017863 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.018987 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:09.936928 31628 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.029668 24428 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:09.993412 31270 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:09.888172 31678 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:10.058409 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.20851e+06us
I20260809 23:20:09.965271 31569 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:10.055038 31738 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260809 23:20:10.031584 31413 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.001s
I20260809 23:20:10.017503 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260809 23:20:09.897709 31599 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 2.175s	user 0.012s	sys 0.002s
I20260809 23:20:10.607514 31599 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:10.033319 24309 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:10.655763 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.656486 24544 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:10.054808 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1159) took 3367 ms. Trace:
I20260809 23:20:10.660132 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.031931 31339 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.659906 24317 rpcz_store.cc:276] 0809 23:20:06.687326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.715650 (+ 28324us) service_pool.cc:227] Handling call
0809 23:20:06.731650 (+ 16000us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:09.946108 (+3214458us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:10.054559 (+108451us) spinlock_profiling.cc:243] Waited 3.21 s on lock 0x7b54003868d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:10.054738 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7390935936}
I20260809 23:20:10.661134 31339 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.662946 24534 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:10.664331 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:10.665172 31339 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.668711 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.669329 24535 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:10.109488 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.672555 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.672924 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:10.673585 24424 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:10.674062 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.674739 24535 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:10.675869 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.103587 24425 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:10.677525 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.678956 24534 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:10.106215 24430 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:10.681806 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.115679 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:10.744321 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 628742us
I20260809 23:20:10.074831 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2884) took 1182 ms. Trace:
I20260809 23:20:10.767864 24429 rpcz_store.cc:276] 0809 23:20:08.891848 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.892616 (+   768us) service_pool.cc:227] Handling call
0809 23:20:10.074508 (+1181892us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400062554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:10.074634 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311701760}
W20260809 23:20:10.032341 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61354.4us
W20260809 23:20:10.110694 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.020s	sys 0.025s
I20260809 23:20:10.113967 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:20:10.301868 31755 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
W20260809 23:20:10.303921 24380 connection.cc:232] RPC call timeout handler was delayed by -2.3942435749340802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
W20260809 23:20:10.366341 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2825) took 2837 ms (client timeout 1566 ms). Trace:
W20260809 23:20:10.401830 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.1408e+06us
I20260809 23:20:10.495440 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:10.501129 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.194s	user 0.031s	sys 0.150s
I20260809 23:20:10.508543 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:10.531762 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.015s	sys 0.019s
W20260809 23:20:10.800316 24321 rpcz_store.cc:269] 0809 23:20:07.528632 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:07.529204 (+   572us) service_pool.cc:227] Handling call
0809 23:20:10.366124 (+2836920us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:10.366270 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5561491328}
I20260809 23:20:10.801687 24428 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:10.803216 31470 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:10.595079 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:10.613097 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.002s	sys 0.044s
I20260809 23:20:10.803839 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.801878 24498 leader_election.cc:398] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:10.804443 24498 leader_election.cc:401] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
W20260809 23:20:10.804589 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.003s	sys 0.045s
W20260809 23:20:10.804979 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 813822us
I20260809 23:20:10.805043 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.805416 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:10.806252 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.574379 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.616686 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:10.807175 24423 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:10.808130 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:10.809212 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.809728 31413 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:10.802073 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.495s	user 0.032s	sys 0.151s
I20260809 23:20:10.638602 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.609099 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:20:10.802608 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 294135us
I20260809 23:20:10.814314 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:10.654414 24543 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:10.657898 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.659330 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.676517 24535 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:10.682685 24430 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:10.824100 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.824872 24430 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:10.819473 31727 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260809 23:20:10.825834 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.826329 31727 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:10.698730 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.021s	sys 0.046s
I20260809 23:20:10.701417 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.822707 31546 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.498345 30629 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.494894 31021 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:10.832084 31021 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.834040 31021 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.281548 31038 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.835362 31038 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.304303 31175 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.285202 31260 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.830212 30629 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.842101 31260 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.303287 30919 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.126096 31270 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.844558 30919 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.846016 24431 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:10.176955 31300 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.787s	user 0.007s	sys 0.006s
I20260809 23:20:10.300690 31569 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.540159 31591 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:10.207406 31592 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.293883 31361 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:10.901224 31361 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:10.214097 31510 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.936571 31510 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.495829 31520 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:10.078290 31315 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.360s	user 0.013s	sys 0.000s
W20260809 23:20:10.216584 31527 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.008s	user 0.013s	sys 0.000s
I20260809 23:20:10.938635 31315 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.939442 31527 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.941121 31315 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.302307 31542 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.948959 31542 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:10.051486 31579 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.643s	user 0.014s	sys 0.004s
I20260809 23:20:10.951800 31579 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.954020 31579 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:10.301216 31633 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.408s	user 0.003s	sys 0.000s
I20260809 23:20:10.301899 31661 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.957612 31661 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.959741 31661 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.499001 31681 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:10.961020 31681 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:10.650154 31720 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.763s	user 0.004s	sys 0.000s
W20260809 23:20:10.427896 31663 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.539s	user 0.003s	sys 0.003s
I20260809 23:20:10.964604 24544 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:10.212070 31701 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260809 23:20:10.966593 31701 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:10.967064 31701 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:10.967201 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:10.537925 31662 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:10.176736 31688 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.971449 31688 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.219156 31648 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.969185 31542 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.975163 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.700273 24424 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:10.717561 31026 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:10.977126 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.978526 24430 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:20:10.722910 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.747438 31684 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:10.979240 24539 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:10.979281 31684 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:10.979871 31684 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:10.980165 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.981268 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:10.981763 31648 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.979895 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.984239 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:10.984855 24430 raft_consensus.cc:2362] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12. Re-sending same reply.
I20260809 23:20:10.985878 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.986478 24430 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
W20260809 23:20:10.760102 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.020s	sys 0.010s
W20260809 23:20:10.992442 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.021s	sys 0.011s
I20260809 23:20:10.994009 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1230) took 1150 ms. Trace:
I20260809 23:20:10.775710 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.995383 24429 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:10.996419 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.997341 24429 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260809 23:20:10.998060 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:10.802770 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.015s	sys 0.019s
I20260809 23:20:11.002250 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2677) took 2444 ms. Trace:
I20260809 23:20:11.002624 24542 rpcz_store.cc:276] 0809 23:20:08.557456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.845472 (+288016us) service_pool.cc:227] Handling call
0809 23:20:08.845800 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:08.846025 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:09.748818 (+902793us) log.cc:842] Serialized 14 byte log entry
0809 23:20:11.001872 (+1253054us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.001933 (+    61us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:11.001992 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.002072 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.002194 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1084644,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":97,"raft.queue_time_us":122,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":77,"thread_start_us":2123464,"threads_started":2,"wal-append.queue_time_us":494959}
I20260809 23:20:10.806560 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:11.005568 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 199059us
I20260809 23:20:10.816828 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.006466 24428 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:11.006500 24498 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12.
I20260809 23:20:11.007316 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.818482 31604 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260809 23:20:11.007923 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:10.813365 31549 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:11.009231 31549 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:10.817262 31470 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:10.810348 31413 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.816792 24541 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:11.009965 31470 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.010426 31470 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.011075 31690 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:10.678701 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.011580 31413 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:11.012180 24425 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260809 23:20:11.013082 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.013659 24425 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.007925 24428 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:10.821525 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.014963 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.015450 31591 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:11.015340 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 193913us
I20260809 23:20:11.016031 31591 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.016557 31591 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.016831 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:10.688923 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.023177 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:10.827764 31546 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.023962 31591 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.025417 31546 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.830581 31537 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:10.839783 31175 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.027525 31175 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.842862 30629 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.827416 24545 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:11.028772 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.029379 24545 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:11.030205 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.030381 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:11.030978 24545 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260809 23:20:10.847204 24423 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:10.848104 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:10.837016 31764 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:11.032488 24431 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:10.853832 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:11.034072 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 180364us
I20260809 23:20:11.034837 30629 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:11.035255 30629 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.035610 24317 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:20:11.035953 24427 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:11.036250 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.031757 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.037177 24545 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:10.889220 31763 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.037784 31763 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.038530 31413 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:11.039224 31413 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:11.039556 31413 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.039894 31413 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.040161 31763 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.846910 30919 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.893635 24535 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:11.041118 24321 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:11.041637 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.042482 24535 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260809 23:20:11.010728 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.042989 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:11.043264 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.044965 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.045231 24535 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260809 23:20:10.838871 31038 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:10.899282 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.046044 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.046465 24534 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:11.047312 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.899379 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.050020 24543 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:11.052105 24543 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:11.052264 31591 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:10.899436 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46356us
I20260809 23:20:11.053566 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.054280 24312 raft_consensus.cc:2354] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:10.899380 31339 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:11.055297 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.055732 31413 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.056643 24541 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:11.057117 31764 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.057644 31764 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:10.902210 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 129061us
I20260809 23:20:10.850631 31300 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.904410 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.060000 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 156970us
I20260809 23:20:11.055950 25371 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:11.060731 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:10.907678 24422 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:11.061976 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:10.924600 24170 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:20:10.929859 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 341) took 25262 ms (client timeout 15000 ms). Trace:
I20260809 23:20:10.933933 31592 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.905819 31361 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.937994 31520 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.064605 30919 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:10.827481 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.022s	sys 0.046s
W20260809 23:20:11.073503 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.18212e+06us
I20260809 23:20:10.942528 25372 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:11.075448 24423 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260809 23:20:11.076032 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2415) took 1002 ms. Trace:
I20260809 23:20:11.076331 24423 rpcz_store.cc:276] 0809 23:20:10.073689 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.074726 (+1001037us) service_pool.cc:227] Handling call
0809 23:20:11.075099 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.075801 (+   702us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.075982 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:11.077111 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1225) took 2713 ms. Trace:
I20260809 23:20:11.077447 25372 rpcz_store.cc:276] 0809 23:20:08.363652 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.366852 (+  3200us) service_pool.cc:227] Handling call
0809 23:20:08.423852 (+ 57000us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.075172 (+2651320us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.076906 (+  1734us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:11.077056 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5791318912}
I20260809 23:20:10.942857 31527 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.956967 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1216) took 4228 ms. Trace:
W20260809 23:20:10.517956 31682 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.631s	user 0.000s	sys 0.003s
I20260809 23:20:10.965669 31681 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.124220 31615 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.169183 31615 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:10.968420 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:10.758291 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:10.990000 24540 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:20:11.170140 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 411904us
I20260809 23:20:11.170790 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.171092 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2800) took 4390 ms (client timeout 1843 ms). Trace:
I20260809 23:20:11.171591 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.171525 24540 rpcz_store.cc:269] 0809 23:20:06.780547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.811132 (+ 30585us) service_pool.cc:227] Handling call
0809 23:20:11.170871 (+4359739us) spinlock_profiling.cc:243] Waited 4.14 s on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.170997 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9528097664}
I20260809 23:20:11.172668 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.173255 31648 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:11.173791 31648 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:11.174029 31763 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:11.174584 31763 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:11.175387 24543 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:20:11.175475 31763 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:10.991269 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.178810 24546 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:11.179147 24435 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:11.179544 31579 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:11.179702 31549 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.180003 31579 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.180301 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2889) took 1248 ms. Trace:
I20260809 23:20:11.180449 31579 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.180763 24435 rpcz_store.cc:276] 0809 23:20:09.931708 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:09.932383 (+   675us) service_pool.cc:227] Handling call
0809 23:20:09.932793 (+   410us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.179562 (+1246769us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.180075 (+   513us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:11.180248 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2865853184}
I20260809 23:20:11.174739 31648 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.182019 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.182616 24435 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.180214 31549 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.172935 24544 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:20:11.185153 31549 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:10.990144 31684 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.579351 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2830) took 2344 ms. Trace:
I20260809 23:20:11.005627 31026 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:11.187515 24545 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:11.008500 31604 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:11.188180 31604 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:11.188884 31684 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.189179 31604 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.190450 31604 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:10.826815 31727 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:11.190389 31684 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:11.019752 31542 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.191581 31684 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:11.191859 31542 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.191979 31684 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:11.192320 31542 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:10.827972 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.193252 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 365341us
I20260809 23:20:11.020385 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.021548 31470 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:10.846691 31270 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.037014 31765 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:11.191532 31727 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.205051 31727 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.040344 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.060264 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.207291 24313 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:10.909147 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1157) took 1792 ms. Trace:
I20260809 23:20:11.208025 25366 rpcz_store.cc:276] 0809 23:20:09.116605 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:09.202248 (+ 85643us) service_pool.cc:227] Handling call
0809 23:20:09.344610 (+142362us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:09.344885 (+   275us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:09.354814 (+  9929us) log.cc:842] Serialized 14 byte log entry
0809 23:20:10.908657 (+1553843us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:10.908711 (+    54us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:10.908867 (+   156us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:10.908962 (+    95us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:10.909089 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":586970,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":105253,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":1553395,"threads_started":2,"wal-append.queue_time_us":821410}
I20260809 23:20:11.210397 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.063894 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:11.211339 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 147521us
I20260809 23:20:11.064314 24429 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:11.214375 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.215112 24429 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:11.065930 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:11.216516 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 150670us
I20260809 23:20:11.217377 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:11.066797 30919 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.067919 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.068126 31520 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.068549 31300 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:11.068789 31021 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:11.069726 31592 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:11.062911 24164 rpcz_store.cc:269] 0809 23:19:45.667891 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:45.668171 (+   280us) service_pool.cc:227] Handling call
0809 23:20:10.929750 (+25261579us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:20:06.371755 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:20:06.695211 (+323456us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:20:06.707397 (+ 12186us) write_op.cc:435] Acquired schema lock
0809 23:20:06.707415 (+    18us) tablet.cc:660] Decoding operations
0809 23:20:06.792804 (+ 85389us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:20:06.792925 (+   121us) write_op.cc:641] Partition lock acquired for write op
0809 23:20:06.792954 (+    29us) tablet.cc:683] Acquiring locks for 27 operations
0809 23:20:07.157632 (+364678us) tablet.cc:699] Row locks acquired
0809 23:20:07.217274 (+ 59642us) write_op.cc:260] PREPARE: finished
0809 23:20:07.428927 (+211653us) write_op.cc:270] Start()
0809 23:20:07.554266 (+125339us) write_op.cc:276] Timestamp: P: 1786317607371515 usec, L: 0
0809 23:20:07.643143 (+ 88877us) op_driver.cc:331] REPLICATION: starting
0809 23:20:07.644366 (+  1223us) log.cc:842] Serialized 10537 byte log entry
0809 23:20:09.930705 (+2286339us) op_driver.cc:444] REPLICATION: finished
0809 23:20:10.287082 (+356377us) write_op.cc:301] APPLY: starting
0809 23:20:10.287234 (+   152us) tablet.cc:1365] starting BulkCheckPresence
0809 23:20:10.318118 (+ 30884us) tablet.cc:1368] finished BulkCheckPresence
0809 23:20:10.318212 (+    94us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:20:10.342715 (+ 24503us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:20:10.343269 (+   554us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=54,key_file_lookups=54,delta_file_lookups=0,mrs_lookups=0
0809 23:20:10.343344 (+    75us) write_op.cc:312] APPLY: finished
0809 23:20:10.344171 (+   827us) log.cc:842] Serialized 236 byte log entry
0809 23:20:10.344419 (+   248us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:10.344656 (+   237us) write_op.cc:454] Released schema lock
0809 23:20:10.344768 (+   112us) write_op.cc:341] FINISH: Updating metrics
0809 23:20:10.345813 (+  1045us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:10.345838 (+    25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7101950,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":346021,"apply.run_cpu_time_us":33090,"apply.run_wall_time_us":61062,"cfile_cache_hit":110,"cfile_cache_hit_bytes":116879,"num_ops":27,"prepare.queue_time_us":625904,"prepare.run_cpu_time_us":40511,"prepare.run_wall_time_us":1870033,"raft.queue_time_us":526838,"raft.run_cpu_time_us":22556,"raft.run_wall_time_us":311566,"replication_time_us":2287420,"thread_start_us":1517824,"threads_started":4,"wal-append.queue_time_us":1747748}]]}
I20260809 23:20:11.066480 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.242568 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 176142us
I20260809 23:20:11.071478 31690 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260809 23:20:11.250013 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 123534us
I20260809 23:20:11.072425 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2406) took 1009 ms. Trace:
I20260809 23:20:11.079300 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.080147 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.069727 24170 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 6 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.251176 24422 rpcz_store.cc:276] 0809 23:20:10.062670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:10.065675 (+  3005us) service_pool.cc:227] Handling call
0809 23:20:11.072235 (+1006560us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54003b9f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.072360 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1932732672}
I20260809 23:20:11.251914 24311 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:11.081292 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.055658 31339 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:11.035584 30629 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:11.255653 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:11.085884 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:10.994407 25365 rpcz_store.cc:276] 0809 23:20:09.844025 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:09.845042 (+  1017us) service_pool.cc:227] Handling call
0809 23:20:09.845451 (+   409us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:09.966250 (+120799us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:09.966281 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:09.966920 (+   639us) log.cc:842] Serialized 52 byte log entry
0809 23:20:10.992923 (+1026003us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:10.993366 (+   443us) log.cc:842] Serialized 14 byte log entry
0809 23:20:10.993608 (+   242us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:10.993714 (+   106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:10.993766 (+    52us) raft_consensus.cc:1671] finished
0809 23:20:10.993781 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:10.993939 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":128145,"prepare.queue_time_us":5926,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":127879,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":14985,"thread_start_us":970063,"threads_started":1,"wal-append.queue_time_us":251342}
I20260809 23:20:11.093762 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:11.099954 24425 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:11.258944 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.259514 24425 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:11.068668 31361 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:11.104609 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.264577 25372 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:11.264832 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 172580us
I20260809 23:20:11.105139 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.107035 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2896) took 1016 ms. Trace:
I20260809 23:20:11.265750 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:11.265631 24427 rpcz_store.cc:276] 0809 23:20:10.090339 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:10.638293 (+547954us) service_pool.cc:227] Handling call
0809 23:20:11.106818 (+468525us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.106957 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":516704640}
I20260809 23:20:11.266824 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.234709 31765 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:11.111498 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2412) took 1038 ms. Trace:
I20260809 23:20:11.239444 31662 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:11.267656 24431 rpcz_store.cc:276] 0809 23:20:10.072829 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.034309 (+961480us) service_pool.cc:227] Handling call
0809 23:20:11.084219 (+ 49910us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.084354 (+   135us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:11.084378 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.084405 (+    27us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:11.084431 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.084463 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.111271 (+ 26808us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b540017f2d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.111426 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114152704}
I20260809 23:20:11.268932 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.120042 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.120848 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.121533 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.123260 25369 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260809 23:20:11.082237 31688 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.243315 31778 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:11.124389 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.059355 31413 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:11.125203 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55167.9us
I20260809 23:20:11.126281 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.138166 31701 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.139176 24534 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:11.126626 31503 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.105232 31527 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.029s
I20260809 23:20:11.152601 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2416) took 1078 ms. Trace:
I20260809 23:20:11.153111 31764 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:10.982266 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.187198 24314 rpcz_store.cc:276] 0809 23:20:08.234522 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:08.234805 (+   283us) service_pool.cc:227] Handling call
0809 23:20:08.395962 (+161157us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:08.465475 (+ 69513us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:08.511327 (+ 45852us) log.cc:842] Serialized 14 byte log entry
0809 23:20:09.396070 (+884743us) spinlock_profiling.cc:243] Waited 62.1 ms on lock 0x7b1800428140. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39
0809 23:20:09.397387 (+  1317us) spinlock_profiling.cc:243] Waited 874 us on lock 0x7b1800428140. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:09.759936 (+362549us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:09.792322 (+ 32386us) log.cc:842] Serialized 26 byte log entry
0809 23:20:09.792535 (+   213us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:09.792861 (+   326us) log.cc:842] Serialized 14 byte log entry
0809 23:20:09.816631 (+ 23770us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:09.816716 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.940922 (+124206us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b1800428140. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:10.579054 (+638132us) raft_consensus.cc:1671] finished
0809 23:20:10.579080 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:10.579258 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":168479,"prepare.queue_time_us":194190,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":935,"raft.queue_time_us":190901,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":50480,"spinlock_wait_cycles":203710336,"thread_start_us":345819,"threads_started":1,"wal-append.queue_time_us":1558298}
I20260809 23:20:11.203711 31470 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.219904 25366 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:11.228193 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:11.228228 24430 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:20:11.229228 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.081027 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.234086 30919 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.234148 31026 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.253620 24321 raft_consensus.cc:2354] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:11.241307 24429 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:11.255160 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.318778 24424 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:11.319052 24422 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:11.078658 25363 rpcz_store.cc:276] 0809 23:20:06.728223 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:06.729019 (+   796us) service_pool.cc:227] Handling call
0809 23:20:06.756054 (+ 27035us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:06.778072 (+ 22018us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:06.778102 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:06.778509 (+   407us) log.cc:842] Serialized 26 byte log entry
0809 23:20:07.094113 (+315604us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:07.116816 (+ 22703us) log.cc:842] Serialized 14 byte log entry
0809 23:20:07.836989 (+720173us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:07.837138 (+   149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:09.892589 (+2055451us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b1800333400. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:10.273549 (+380960us) spinlock_profiling.cc:243] Waited 50 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:10.956734 (+683185us) raft_consensus.cc:1671] finished
0809 23:20:10.956758 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:10.956907 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":688422,"prepare.queue_time_us":62134,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":40,"raft.queue_time_us":867818,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":75177,"spinlock_wait_cycles":186540800,"thread_start_us":292256,"threads_started":1,"wal-append.queue_time_us":573209}
I20260809 23:20:11.262387 24170 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:11.263134 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.110306 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2410) took 1038 ms. Trace:
I20260809 23:20:11.233609 31779 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:11.124238 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.271965 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.320552 25365 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:11.320683 25368 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
W20260809 23:20:11.320866 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57936.9us
W20260809 23:20:11.320984 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49144.8us
I20260809 23:20:11.269949 25375 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:11.321406 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.273026 24427 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:11.241811 31777 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:11.268254 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.253741 31270 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:11.244005 31500 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:11.247143 31592 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.256637 31300 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:11.284142 25371 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:11.254241 31339 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:11.328333 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.287554 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.330695 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.331115 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.288877 24425 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:11.269567 31175 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:11.303462 24423 rpcz_store.cc:276] 0809 23:20:10.074053 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.077124 (+1003071us) service_pool.cc:227] Handling call
0809 23:20:11.077454 (+   330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:11.077793 (+   339us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:11.077819 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:11.078278 (+   459us) log.cc:842] Serialized 26 byte log entry
0809 23:20:11.121529 (+ 43251us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:11.121578 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.121644 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:11.152387 (+ 30743us) raft_consensus.cc:1671] finished
0809 23:20:11.152410 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.152542 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4930,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":16032,"thread_start_us":40805,"threads_started":1,"wal-append.queue_time_us":54565}
I20260809 23:20:11.306746 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:11.192875 31684 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.312240 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.312317 24320 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:11.312448 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84349.3us
I20260809 23:20:11.311748 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.256524 31615 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
W20260809 23:20:11.315892 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86848.6us
I20260809 23:20:11.272655 31765 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.299830 31785 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.272787 24431 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:11.324198 31779 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.320343 24428 rpcz_store.cc:276] 0809 23:20:10.072234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.015810 (+943576us) service_pool.cc:227] Handling call
0809 23:20:11.108558 (+ 92748us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.108784 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:11.108820 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.108873 (+    53us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:11.108919 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.108990 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.110109 (+  1119us) spinlock_profiling.cc:243] Waited 92.4 ms on lock 0x7b54003322d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.110235 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212559232}
W20260809 23:20:11.324090 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55958.8us
I20260809 23:20:11.379914 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.336282 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.333134 31270 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:11.338693 31500 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:11.381045 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44821.9us
I20260809 23:20:11.378245 31779 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:11.381821 24496 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:11.381497 31270 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.382552 31260 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:11.382664 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.383051 31260 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.338369 24170 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 8, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 8 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.323494 31764 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.383637 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:11.383987 31764 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.384074 31546 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.384490 31764 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.383576 31260 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:11.384656 24170 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.346778 25369 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:11.325448 31662 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:11.322029 25365 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:11.387259 31662 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.278666 31591 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260809 23:20:11.357918 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:11.387728 31662 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.383385 31690 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:11.388545 31690 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:11.287734 31778 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.389628 31690 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.391304 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.384538 31546 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.392391 25370 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:11.318230 31026 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.323369 31763 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:11.393154 31026 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.393445 31763 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:11.394150 31763 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.382975 31500 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.360006 24535 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:20:11.354687 31765 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.310446 31413 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260809 23:20:11.396751 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.358847 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.397493 24535 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:11.327708 31592 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:11.397724 31413 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:11.366772 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.398921 31413 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:11.399667 24542 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:11.372321 25368 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:11.310549 31503 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.367406 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.373898 25375 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:11.407775 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.408555 25375 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260809 23:20:11.410957 25375 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:11.411535 31503 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:11.412177 31503 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:11.412709 31503 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:11.413648 31503 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.415977 31778 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:11.421725 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.422575 25375 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
W20260809 23:20:11.374596 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 99263us
I20260809 23:20:11.360266 24422 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:11.354578 31785 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:11.342072 31625 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:11.328246 31777 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.382154 24496 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260809 23:20:11.392728 31546 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.386289 24170 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 9 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:20:11.371959 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8618866552133113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
I20260809 23:20:11.325403 31681 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:11.399716 31413 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:11.342285 31175 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:11.464655 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2897) took 1372 ms. Trace:
I20260809 23:20:11.427290 31774 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260809 23:20:11.427484 31785 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.497095 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.466238 31520 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:11.427767 31625 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.428521 31701 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.468578 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.430115 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:11.527554 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59074.5us
I20260809 23:20:11.434890 24535 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260809 23:20:11.528985 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:11.430794 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49105.8us
I20260809 23:20:11.471921 24424 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:11.444414 31681 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.473592 24311 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260809 23:20:11.473716 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.445348 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.475332 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.448793 31648 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.427009 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.447113 24423 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:11.447468 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.444486 31413 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:11.543049 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.543344 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2425) took 1041 ms. Trace:
I20260809 23:20:11.444432 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:11.471598 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 113620us
I20260809 23:20:11.545120 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:11.374174 31579 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:11.432149 24498 connection.cc:232] RPC call timeout handler was delayed by -2.6359866550192237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1945s for a response
W20260809 23:20:11.554381 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4451866550371051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
I20260809 23:20:11.493490 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.498780 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.555138 25370 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.555413 24314 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.499212 24170 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 7, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 7 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:11.557770 24170 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.559839 24170 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 9 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.561321 24170 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 4 to 8, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:11.505780 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.562805 25370 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:11.507047 24313 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:20:11.349298 31021 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.112s
I20260809 23:20:11.333448 31300 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.532114 24424 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:11.533394 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.574332 24309 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.533921 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.445148 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.539367 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.427721 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.575137 24541 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:11.492849 31175 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:11.577455 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.578219 24311 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:11.579107 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.492281 24428 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
W20260809 23:20:11.544512 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 244620us
I20260809 23:20:11.544569 24323 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:11.543740 24423 rpcz_store.cc:276] 0809 23:20:10.502035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.345694 (+843659us) service_pool.cc:227] Handling call
0809 23:20:11.446868 (+101174us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.543100 (+ 96232us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.543289 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:11.547497 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.494024 24430 rpcz_store.cc:276] 0809 23:20:10.092020 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.083154 (+991134us) service_pool.cc:227] Handling call
0809 23:20:11.218188 (+135034us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:11.315489 (+ 97301us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.464393 (+148904us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540018c250. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.464582 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257565056}
I20260809 23:20:11.565084 24314 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:11.526526 31625 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:11.540560 31681 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.599720 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.584488 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:11.605711 25375 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:11.472798 31762 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.103s
I20260809 23:20:11.604131 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.608174 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.539378 31500 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:11.503600 31520 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:11.588259 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.423877 31026 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.589684 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:11.626353 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.596145 31549 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.544046 31569 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:11.422904 31763 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.602768 24309 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:11.602597 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.609424 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.611361 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.607403 24539 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:11.608820 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:11.577498 31527 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:11.595208 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.661497 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.661700 25369 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:11.635522 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.493978 31339 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.646569 24544 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:11.646605 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.627142 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.655323 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:11.616062 31591 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:11.675992 24534 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260809 23:20:11.656800 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:11.674899 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 128423us
I20260809 23:20:11.677888 25375 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:11.656937 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.546504 31260 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.682140 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.602372 31783 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:11.703089 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:11.696812 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:11.708341 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:11.650279 31569 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:11.711738 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:11.711874 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 136502us
I20260809 23:20:11.712970 24422 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:11.715399 24435 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:20:11.695330 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:11.647655 31413 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.722819 31765 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.723743 31569 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.724879 31765 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:11.725028 31569 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:11.725363 31026 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:11.714280 25405 debug-util.cc:398] Leaking SignalData structure 0x7b080028cea0 after lost signal to thread 22536
W20260809 23:20:11.727263 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800336b40 after lost signal to thread 24220
I20260809 23:20:11.727957 31774 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:11.728364 31774 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:11.729039 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08000f5100 after lost signal to thread 24355
I20260809 23:20:11.729511 31661 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:11.729944 31661 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:11.730608 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08002a8d60 after lost signal to thread 24469
I20260809 23:20:11.731940 31549 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:11.732091 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2427) took 1228 ms. Trace:
I20260809 23:20:11.687606 31789 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20260809 23:20:11.732429 24428 rpcz_store.cc:276] 0809 23:20:10.503791 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.378507 (+874716us) service_pool.cc:227] Handling call
0809 23:20:11.475157 (+ 96650us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.585634 (+110477us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.731899 (+146265us) spinlock_profiling.cc:243] Waited 96.2 ms on lock 0x7b540023f350. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:11.732031 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":221250432}
I20260809 23:20:11.734025 24428 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260809 23:20:11.736392 31527 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:11.737013 31527 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:11.739248 25372 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:11.741601 24435 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:11.610098 31662 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:11.744783 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08000495a0 after lost signal to thread 24579
W20260809 23:20:11.785188 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08003f2580 after lost signal to thread 25410
W20260809 23:20:11.786561 25405 debug-util.cc:398] Leaking SignalData structure 0x7b080006dee0 after lost signal to thread 31623
I20260809 23:20:11.710738 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:11.717563 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:11.703923 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:11.807430 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90046us
W20260809 23:20:11.808005 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 104222us
W20260809 23:20:11.804263 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46977.9us
I20260809 23:20:11.747277 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:11.808725 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:11.809221 24539 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:11.768208 24428 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [9.9-12.10]
W20260809 23:20:11.807463 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 214474us
I20260809 23:20:11.812299 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:11.605671 31787 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.141s
I20260809 23:20:11.819322 24544 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:11.818378 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:11.806393 24430 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:11.848517 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 940) took 1018 ms. Trace:
I20260809 23:20:11.849094 24430 rpcz_store.cc:276] 0809 23:20:10.830262 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.710463 (+880201us) service_pool.cc:227] Handling call
0809 23:20:11.805939 (+ 95476us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.847855 (+ 41916us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.848435 (+   580us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:11.714146 31762 raft_consensus.cc:2741] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
I20260809 23:20:11.670693 31701 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.126s
W20260809 23:20:11.858598 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 255110us
I20260809 23:20:11.747238 31783 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:11.853052 31780 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.411s	user 0.000s	sys 0.001s
I20260809 23:20:11.686748 31591 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.820252 31765 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:11.881240 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:11.895920 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:11.880965 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 326906us
W20260809 23:20:11.860284 31782 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.446s	user 0.001s	sys 0.000s
I20260809 23:20:11.888295 24544 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:11.884166 31761 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260809 23:20:11.705868 31786 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.211s
I20260809 23:20:11.717396 31520 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.924396 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2428) took 1420 ms. Trace:
I20260809 23:20:11.926473 24422 rpcz_store.cc:276] 0809 23:20:10.504099 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.437748 (+933649us) service_pool.cc:227] Handling call
0809 23:20:11.438726 (+   978us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.743173 (+304447us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.924137 (+180964us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:11.924324 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":630087424}
I20260809 23:20:11.953547 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2418) took 1466 ms. Trace:
I20260809 23:20:11.955051 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2424) took 1453 ms. Trace:
I20260809 23:20:11.966710 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2419) took 1479 ms. Trace:
I20260809 23:20:11.956326 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:11.898945 31793 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:11.953776 31549 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:11.961982 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:11.967594 24427 rpcz_store.cc:276] 0809 23:20:10.487632 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.325806 (+838174us) service_pool.cc:227] Handling call
0809 23:20:11.326096 (+   290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.326301 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:11.547291 (+220990us) log.cc:842] Serialized 14 byte log entry
0809 23:20:11.966326 (+419035us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.966384 (+    58us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:11.966438 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.966508 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.966652 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66826,"prepare.queue_time_us":250680,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"thread_start_us":559052,"threads_started":2}
I20260809 23:20:11.954072 24429 rpcz_store.cc:276] 0809 23:20:10.487391 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.325686 (+838295us) service_pool.cc:227] Handling call
0809 23:20:11.325974 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.401140 (+ 75166us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:11.613292 (+212152us) log.cc:842] Serialized 14 byte log entry
0809 23:20:11.858591 (+245299us) log.cc:842] Serialized 14 byte log entry
0809 23:20:11.922412 (+ 63821us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b1800312760. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:11.922582 (+   170us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.922636 (+    54us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:11.922695 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.953237 (+ 30542us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b54003bae54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:11.953308 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.953482 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":84169,"prepare.queue_time_us":339227,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":94124,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":38007,"spinlock_wait_cycles":222623872,"thread_start_us":350901,"threads_started":2}
W20260809 23:20:11.973465 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.007s	sys 0.027s
W20260809 23:20:12.004117 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.008s	sys 0.027s
W20260809 23:20:12.004436 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 578324us
I20260809 23:20:11.959792 24425 rpcz_store.cc:276] 0809 23:20:10.501386 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.335533 (+834147us) service_pool.cc:227] Handling call
0809 23:20:11.335804 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.336005 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:11.550328 (+214323us) log.cc:842] Serialized 14 byte log entry
0809 23:20:11.954539 (+404211us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:11.954595 (+    56us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:11.954774 (+   179us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:11.954869 (+    95us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:11.955000 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65340,"prepare.queue_time_us":351586,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":70787,"thread_start_us":454876,"threads_started":2}
I20260809 23:20:11.887255 31762 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260809 23:20:11.913328 31038 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
W20260809 23:20:11.924220 31779 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.540s	user 0.010s	sys 0.000s
W20260809 23:20:11.999101 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.011s	sys 0.047s
W20260809 23:20:12.021250 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 144370us
I20260809 23:20:12.003624 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:12.021756 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 213070us
W20260809 23:20:12.027855 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.012s	sys 0.048s
I20260809 23:20:12.028617 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:12.019805 31300 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.031129 24427 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:11.925819 31778 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:11.925264 31579 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:11.998315 31781 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.385s	user 0.001s	sys 0.000s
W20260809 23:20:12.018239 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 206043us
I20260809 23:20:12.032910 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:12.030381 31260 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.907403 31765 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:11.960604 31500 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:12.067946 31760 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:12.112717 31760 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.112948 31500 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:11.998108 31315 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260809 23:20:12.051921 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 243249us
W20260809 23:20:12.051676 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 341034us
I20260809 23:20:12.064728 24429 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:20:12.137763 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:12.026525 31780 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260809 23:20:12.011874 31026 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20260809 23:20:11.955863 31825 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.206s
I20260809 23:20:12.018219 31021 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260809 23:20:12.103964 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:12.039779 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 141618us
I20260809 23:20:11.884984 31777 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:11.956156 31615 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
W20260809 23:20:12.076665 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48168.4us
W20260809 23:20:12.110402 31604 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.910s	user 0.008s	sys 0.002s
W20260809 23:20:12.104939 31688 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.825s	user 0.009s	sys 0.001s
I20260809 23:20:12.115751 31591 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.219995 24316 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:20:12.168442 31510 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.025s	sys 0.044s
I20260809 23:20:12.112040 31763 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.067756 24544 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:12.025641 31813 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:12.176141 24425 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:12.176748 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 356768us
I20260809 23:20:12.225276 24498 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:12.226650 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:12.123988 31764 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.739s	user 0.011s	sys 0.001s
I20260809 23:20:12.187080 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:12.044479 31793 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:12.167732 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122100us
I20260809 23:20:12.228677 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:12.229862 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:12.158900 31542 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.966s	user 0.008s	sys 0.004s
W20260809 23:20:12.152665 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 139648us
I20260809 23:20:12.221205 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:12.052205 31520 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:12.043572 31038 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:12.170197 31300 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:12.223770 31688 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:12.234133 31817 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:12.228320 31793 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:12.235052 31764 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.235842 31604 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:12.236452 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.371687 24539 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:12.122380 31537 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.099s
W20260809 23:20:12.249497 24311 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.671s	user 0.006s	sys 0.006s
I20260809 23:20:12.375211 24311 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:12.250561 24378 leader_election.cc:398] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:12.375738 31822 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:12.376037 24378 leader_election.cc:401] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260809 23:20:12.376159 31857 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:12.376185 31822 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:12.376765 31857 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:12.376812 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 126373us
I20260809 23:20:12.377358 31857 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:12.377291 31822 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.378175 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:12.378711 31822 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:12.379148 31270 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:12.379196 31038 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.185338 31778 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:12.190915 31760 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.271991 24430 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:12.034065 31779 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:12.301576 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 123336us
I20260809 23:20:12.240986 31662 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:12.263471 30919 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.938s	user 0.013s	sys 0.000s
I20260809 23:20:12.252341 31805 raft_consensus.cc:2741] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:20:12.246093 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:12.353019 31592 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.955s	user 0.013s	sys 0.000s
I20260809 23:20:12.379983 31270 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260809 23:20:12.226084 31510 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.025s	sys 0.044s
I20260809 23:20:12.038710 31783 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:12.411471 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 184878us
I20260809 23:20:12.494134 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:12.495870 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 249886us
I20260809 23:20:12.496999 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:12.461671 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50934.8us
I20260809 23:20:12.498063 24378 leader_election.cc:398] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:12.498344 24378 leader_election.cc:401] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260809 23:20:12.499747 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:12.500494 24534 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:12.462391 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51037us
I20260809 23:20:12.249644 31542 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.289247 31688 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:12.381667 31787 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:12.500782 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:12.501799 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.463901 31604 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:12.504038 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.504674 24498 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:12.411607 31842 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:12.519927 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:12.521688 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:12.521905 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:12.530593 24311 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:12.526063 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.530659 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:12.532125 24539 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:12.532464 24314 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:12.539189 24534 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:12.552611 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.561913 24422 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:12.561775 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:12.562543 24314 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:12.563916 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:12.564567 24541 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:12.565176 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:12.483503 31270 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:12.565846 24544 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:12.611654 31270 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:12.479449 31862 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:12.569638 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:12.566598 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:12.490563 31662 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:12.392062 31808 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.172s
I20260809 23:20:12.517935 31503 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:12.503208 31701 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:20:12.619407 31825 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:12.615495 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:12.640811 31727 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.427s	user 0.015s	sys 0.001s
I20260809 23:20:12.563432 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:12.614434 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:12.622057 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 382994us
W20260809 23:20:12.615103 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 363130us
I20260809 23:20:12.350205 31813 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.231s
I20260809 23:20:12.410454 31804 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.132s
I20260809 23:20:12.462915 30919 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.483348 31592 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:12.640982 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 503269us
I20260809 23:20:12.671337 31832 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:12.621609 31604 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:12.515565 31026 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:12.526170 31688 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.614564 31842 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:12.717659 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:12.578773 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.022s	sys 0.056s
W20260809 23:20:12.687568 31175 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.098s	user 0.009s	sys 0.004s
I20260809 23:20:12.469362 31793 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.678119 31765 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.685760 31825 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:12.662721 31777 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260809 23:20:12.711755 31503 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:12.692407 31840 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:12.800643 31765 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:12.802233 31866 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:12.803735 31026 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:12.768280 31684 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:12.801523 31361 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.530s	user 0.011s	sys 0.002s
I20260809 23:20:12.622128 31778 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.818523 31592 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:12.818885 31546 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.344s	user 0.008s	sys 0.004s
I20260809 23:20:12.815274 31026 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.940196 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:12.940650 30919 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.941826 31026 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.942597 31026 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:12.943136 30919 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:12.944687 30919 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.943857 31026 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.947263 30919 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.778432 31175 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.952013 31175 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.952843 25365 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260809 23:20:12.837471 31681 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.226s	user 0.011s	sys 0.000s
I20260809 23:20:12.954370 31681 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.956537 25369 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:20:12.826033 31503 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:12.957731 31681 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.836632 31542 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:12.779480 31648 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.236s	user 0.013s	sys 0.000s
I20260809 23:20:12.960433 31648 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:12.962456 24317 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260809 23:20:12.835358 31527 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.098s	user 0.011s	sys 0.000s
I20260809 23:20:12.962884 31681 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:12.964046 31527 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.965138 31503 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.810883 31765 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:12.821476 30629 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.553s	user 0.013s	sys 0.006s
I20260809 23:20:12.970781 31765 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:12.971889 30629 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:12.973869 31760 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:12.974478 31760 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:12.671953 24246 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.008s	sys 0.011s
W20260809 23:20:12.975481 24246 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.009s	sys 0.011s
I20260809 23:20:12.828650 31779 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.699543 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:12.824926 31817 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.006343 24545 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:13.007580 31817 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:12.751734 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.027s	sys 0.060s
I20260809 23:20:13.014210 24313 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:12.912451 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1085) took 1166 ms. Trace:
I20260809 23:20:13.015758 31579 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:12.999352 31760 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.016701 31579 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.017496 31760 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:13.015533 24540 rpcz_store.cc:276] 0809 23:20:11.745728 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.745888 (+   160us) service_pool.cc:227] Handling call
0809 23:20:11.746185 (+   297us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:11.804888 (+ 58703us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:11.804918 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:11.805379 (+   461us) log.cc:842] Serialized 26 byte log entry
0809 23:20:12.044053 (+238674us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:12.044094 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:12.044169 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:12.911903 (+867734us) raft_consensus.cc:1671] finished
0809 23:20:12.911941 (+    38us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:12.912216 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58292,"raft.queue_time_us":168109,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7155,"thread_start_us":238387,"threads_started":1,"wal-append.queue_time_us":996808}
I20260809 23:20:13.022548 31648 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.023172 31648 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.929205 24544 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:12.751679 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:13.034665 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 282982us
I20260809 23:20:12.937309 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:13.036041 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 99226.5us
I20260809 23:20:12.937969 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:13.039413 31765 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:13.039491 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:13.040045 31765 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:12.833035 31546 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.040585 31765 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.040973 31688 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:13.043174 31625 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.438s	user 0.014s	sys 0.004s
I20260809 23:20:13.043462 31546 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:12.836261 31569 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.111s	user 0.009s	sys 0.005s
I20260809 23:20:12.688917 24425 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.043785 31625 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.964736 25373 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
W20260809 23:20:12.901108 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 351346us
I20260809 23:20:12.697715 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:12.906948 24427 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:12.755838 24431 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:12.849699 31785 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.318s	user 0.010s	sys 0.000s
I20260809 23:20:13.086786 24314 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:13.086853 31625 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:12.697032 24311 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:13.089427 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:12.712433 24323 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:12.810596 31764 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.983342 30629 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.910535 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:12.835114 31821 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260809 23:20:12.806281 31825 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:13.092548 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 182465us
I20260809 23:20:13.093554 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.094161 31783 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:12.756692 31832 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:13.093554 31569 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:13.095481 31569 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.095649 31896 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:13.095980 31569 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.096225 31896 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.096634 31790 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:13.096356 31832 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.097086 31790 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.098098 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2904) took 1231 ms. Trace:
W20260809 23:20:12.833247 31841 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260809 23:20:12.844944 31842 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.098507 24430 rpcz_store.cc:276] 0809 23:20:11.866916 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.886646 (+ 19730us) service_pool.cc:227] Handling call
0809 23:20:11.887019 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:12.439208 (+552189us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.097505 (+658297us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:13.097873 (+   368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":884874624}
I20260809 23:20:13.094141 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:12.833338 31848 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
I20260809 23:20:13.101202 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:12.848109 31857 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.103808 24311 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:13.104789 31857 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:12.769526 31858 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20260809 23:20:13.106181 31858 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:12.823629 31860 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:13.107247 31860 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:13.107693 31860 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.108696 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:12.814414 31866 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.118518 31893 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:13.120150 31866 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.121527 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:12.835776 31872 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260809 23:20:12.749877 31878 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.134s
I20260809 23:20:13.098457 31790 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.124768 31790 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:12.993129 24312 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:12.996901 31661 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.128196 31661 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:13.129472 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2853) took 2062 ms (client timeout 2453 ms). Trace:
W20260809 23:20:13.130012 24312 rpcz_store.cc:269] 0809 23:20:11.067525 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.067689 (+   164us) service_pool.cc:227] Handling call
0809 23:20:13.129138 (+2061449us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.129375 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4028385536}
I20260809 23:20:12.970671 31537 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:13.131301 31537 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:13.132139 31871 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.404s	user 0.002s	sys 0.003s
I20260809 23:20:13.133960 31537 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:12.979916 31527 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.045416 24424 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:12.893545 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1189) took 1189 ms. Trace:
W20260809 23:20:13.135874 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 950) took 1598 ms (client timeout 1714 ms). Trace:
I20260809 23:20:13.135051 24309 rpcz_store.cc:276] 0809 23:20:11.703500 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:12.313987 (+610487us) service_pool.cc:227] Handling call
0809 23:20:12.374765 (+ 60778us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:12.382697 (+  7932us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:12.382744 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:12.438184 (+ 55440us) log.cc:842] Serialized 26 byte log entry
0809 23:20:12.618840 (+180656us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:12.624138 (+  5298us) log.cc:842] Serialized 14 byte log entry
0809 23:20:12.624347 (+   209us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:12.624422 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:12.892259 (+267837us) raft_consensus.cc:1671] finished
0809 23:20:12.892356 (+    97us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:12.893015 (+   659us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25899,"prepare.queue_time_us":7957,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":187920,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":77015,"thread_start_us":180290,"threads_started":1,"wal-append.queue_time_us":365174}
W20260809 23:20:13.136260 24424 rpcz_store.cc:269] 0809 23:20:11.537719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.578792 (+ 41073us) service_pool.cc:227] Handling call
0809 23:20:13.135626 (+1556834us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003eb2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.135799 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3369287936}
I20260809 23:20:13.137646 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.138454 24424 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:13.047178 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.047726 31579 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.048535 31300 raft_consensus.cc:2741] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260809 23:20:13.048761 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 349179us
I20260809 23:20:13.050666 24543 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:13.061723 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.070271 24535 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260809 23:20:13.073257 31270 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:12.976249 31026 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:13.076133 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 545568us
W20260809 23:20:13.076406 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2844) took 2058 ms (client timeout 1699 ms). Trace:
I20260809 23:20:13.086524 31785 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.161746 31527 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:13.163473 31793 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.164117 24321 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:13.164505 31527 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:13.165102 31527 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.165581 31527 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.133984 31774 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:13.159832 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51241.9us
I20260809 23:20:13.163718 31785 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:13.167847 31774 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.168123 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.090281 24425 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:13.096788 31896 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:13.167949 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.099522 31783 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.082731 31886 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:13.144232 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.151937 24423 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 13. Ops: []
I20260809 23:20:13.152609 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:13.153620 31810 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.050s	user 0.003s	sys 0.000s
I20260809 23:20:13.153700 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1180) took 1538 ms. Trace:
I20260809 23:20:13.154517 31764 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:13.154870 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.159781 31813 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:13.160790 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66743us
I20260809 23:20:12.896811 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:13.147068 24316 rpcz_store.cc:269] 0809 23:20:11.018301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.020136 (+  1835us) service_pool.cc:227] Handling call
0809 23:20:13.075907 (+2055771us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540019d2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.076148 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2354343808}
W20260809 23:20:12.836261 31835 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.454s	user 0.001s	sys 0.000s
I20260809 23:20:13.190856 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.191855 24535 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:13.194272 24429 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.194387 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:13.196465 31021 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:13.197045 31021 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.199326 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.200119 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2426) took 2697 ms. Trace:
I20260809 23:20:13.200609 31783 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.200484 24431 rpcz_store.cc:276] 0809 23:20:10.502621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.366451 (+863830us) service_pool.cc:227] Handling call
0809 23:20:11.366868 (+   417us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.085877 (+1719009us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.199720 (+113843us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:13.199950 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3193841024}
I20260809 23:20:13.201983 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.203135 31845 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:20:13.203661 31845 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:13.204909 24430 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:13.204779 31470 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:13.206274 31470 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:13.179241 24323 rpcz_store.cc:276] 0809 23:20:11.615167 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.615526 (+   359us) service_pool.cc:227] Handling call
0809 23:20:13.153279 (+1537753us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.153490 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2409905920}
I20260809 23:20:13.182371 31270 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.183141 31579 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:13.126415 31817 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.183880 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.188405 24312 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.174212 31886 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.206563 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.207002 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2860) took 1783 ms. Trace:
I20260809 23:20:13.208031 31902 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:13.208832 31898 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:13.209549 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2907) took 1146 ms. Trace:
I20260809 23:20:13.210129 31760 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:13.210506 31805 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.187167 31813 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.250594 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.210883 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.251618 24311 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:13.252678 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.253408 24311 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:20:13.254364 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:13.211748 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2906) took 1344 ms (client timeout 1556 ms). Trace:
I20260809 23:20:13.244792 24427 rpcz_store.cc:276] 0809 23:20:12.063173 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:12.065178 (+  2005us) service_pool.cc:227] Handling call
0809 23:20:12.065438 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.085217 (+1019779us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.209295 (+124078us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:13.209494 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1933376896}
W20260809 23:20:13.211973 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1187) took 1517 ms (client timeout 2007 ms). Trace:
I20260809 23:20:13.212148 31808 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:13.212548 31820 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:13.181053 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.213289 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.214118 31762 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:13.214960 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.215216 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1084) took 1829 ms. Trace:
I20260809 23:20:13.216182 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1030) took 2429 ms. Trace:
W20260809 23:20:13.198647 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2842) took 1747 ms (client timeout 2074 ms). Trace:
I20260809 23:20:13.202608 24431 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:20:13.208848 31413 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:13.225306 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.225845 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:13.226959 25373 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:13.227535 24435 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:13.262993 24534 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:13.188014 31764 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:13.198428 31765 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:13.273368 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70537.4us
I20260809 23:20:13.259661 24542 rpcz_store.cc:276] 0809 23:20:11.385270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.400587 (+ 15317us) service_pool.cc:227] Handling call
0809 23:20:11.400986 (+   399us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:11.401324 (+   338us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:11.401360 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:11.401846 (+   486us) log.cc:842] Serialized 26 byte log entry
0809 23:20:12.029371 (+627525us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:12.029430 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:12.029553 (+   123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:13.214750 (+1185197us) raft_consensus.cc:1671] finished
0809 23:20:13.214803 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:13.215031 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1235,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":599752,"threads_started":1,"wal-append.queue_time_us":826233}
I20260809 23:20:13.198232 31021 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.243243 31789 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.243656 31339 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.235792 24321 rpcz_store.cc:276] 0809 23:20:11.423598 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.426486 (+  2888us) service_pool.cc:227] Handling call
0809 23:20:11.426755 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.164679 (+1737924us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.206663 (+ 41984us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:13.206910 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3992312704}
I20260809 23:20:13.248692 24314 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:13.221506 31470 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:13.255657 24313 rpcz_store.cc:269] 0809 23:20:11.702646 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.702805 (+   159us) service_pool.cc:227] Handling call
0809 23:20:13.211747 (+1508942us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.211915 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3005965056}
I20260809 23:20:13.210043 31569 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.281944 31878 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:13.283270 24539 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:13.283098 31537 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:13.284029 31569 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.259804 24428 rpcz_store.cc:276] 0809 23:20:10.786754 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.810227 (+1023473us) service_pool.cc:227] Handling call
0809 23:20:13.215146 (+1404919us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.215963 (+   817us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225124608}
I20260809 23:20:13.284616 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.285547 24321 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:13.224403 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.286895 31862 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Stepping down as leader of term 8
I20260809 23:20:13.287451 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:20:13.255048 24422 rpcz_store.cc:269] 0809 23:20:11.867643 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.928378 (+ 60735us) service_pool.cc:227] Handling call
0809 23:20:13.211356 (+1282978us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400061654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.211577 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1360960256}
I20260809 23:20:13.290156 31765 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.291620 24544 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:13.238806 31886 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.292685 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:13.292871 31789 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.294440 24323 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.295408 31789 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.295825 31795 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:13.296254 31795 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:13.296854 31470 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.297267 31795 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.235953 31579 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.299165 31579 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.300189 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:13.300879 25373 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:13.302000 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.250463 31805 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.263422 31026 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.303889 31026 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.304363 25366 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:13.304639 31805 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.302024 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:13.306841 25365 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.266688 31884 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260809 23:20:13.310019 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.256805 31808 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.311089 24428 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:13.311331 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1183) took 1618 ms. Trace:
I20260809 23:20:13.260052 31820 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:13.312058 24317 rpcz_store.cc:276] 0809 23:20:11.692548 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.692708 (+   160us) service_pool.cc:227] Handling call
0809 23:20:11.693184 (+   476us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:12.985407 (+1292223us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.311036 (+325629us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000dc5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:13.311260 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2917756800}
I20260809 23:20:13.312919 31820 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.313921 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:13.267599 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 206158us
I20260809 23:20:13.264046 31902 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.265741 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.265343 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.281687 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.327998 31902 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:13.328456 24431 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:13.329047 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:13.329710 24542 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:13.281867 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.287354 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.331476 24534 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.331147 24317 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:13.332074 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2433) took 2132 ms. Trace:
I20260809 23:20:13.332525 24435 rpcz_store.cc:276] 0809 23:20:11.200036 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.742323 (+542287us) service_pool.cc:227] Handling call
0809 23:20:13.227132 (+1484809us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.273707 (+ 46575us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.331870 (+ 58163us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54003eb2d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.332016 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3392300800}
I20260809 23:20:13.268654 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:13.333165 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.334048 24435 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:13.259872 24543 rpcz_store.cc:269] 0809 23:20:11.451243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.504199 (+ 52956us) service_pool.cc:227] Handling call
0809 23:20:13.198424 (+1694225us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.198576 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3312969600}
I20260809 23:20:13.330636 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.335278 31690 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:13.336373 24542 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.310046 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.315657 24422 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:20:13.339031 24427 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:13.315728 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 928) took 2515 ms. Trace:
W20260809 23:20:13.317776 31834 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.058s	user 0.003s	sys 0.002s
W20260809 23:20:13.319478 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2835) took 1961 ms (client timeout 2236 ms). Trace:
I20260809 23:20:13.322202 31648 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.302706 25371 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:13.319502 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.227905 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.265708 24311 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.325789 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:13.330703 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.331727 24425 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:13.334558 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65948.4us
I20260809 23:20:13.357601 24539 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:13.298705 31470 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.363709 31825 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.311772 31808 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.367942 31808 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.370334 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.274312 31764 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:13.371594 31300 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:13.372026 31300 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:13.372366 31300 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.373155 31787 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:13.374449 31851 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:13.270691 24429 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.374820 31851 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.375573 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.376461 31537 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:13.376837 31537 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:13.377219 31537 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.335208 24317 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:13.267473 31903 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.094s	user 0.000s	sys 0.002s
I20260809 23:20:13.286834 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:13.347873 24545 rpcz_store.cc:269] 0809 23:20:11.358061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.360432 (+  2371us) service_pool.cc:227] Handling call
0809 23:20:13.319238 (+1958806us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400146c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.319408 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3225696128}
I20260809 23:20:13.359858 31795 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.380892 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:13.381763 24545 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:13.266458 31813 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.360579 24435 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260809 23:20:13.361169 31798 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:13.287525 31862 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.259106 31845 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.375968 31851 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.379978 31820 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.379781 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.344688 31896 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.340734 24423 rpcz_store.cc:276] 0809 23:20:10.800823 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.658887 (+858064us) service_pool.cc:227] Handling call
0809 23:20:13.151534 (+1492647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.217173 (+ 65639us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.315406 (+ 98233us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54003eb2d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.315651 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3403145984}
I20260809 23:20:13.412286 24546 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:20:13.306761 31026 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:13.418701 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74534.8us
I20260809 23:20:13.321188 31789 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
W20260809 23:20:13.413648 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88003.2us
I20260809 23:20:13.445714 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.448469 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.452667 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.445077 24321 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.472955 24431 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:13.477813 24544 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260809 23:20:13.383904 31300 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.510584 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.510867 31851 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.511976 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.512760 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.513950 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.464707 31760 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:13.527410 24323 raft_consensus.cc:2354] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:13.546557 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.556794 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.554790 24425 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:20:13.554117 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 152562us
I20260809 23:20:13.561553 25369 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:13.419247 31860 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.521234 31817 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:13.569707 24312 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:13.587298 24535 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:13.536317 31825 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.608757 31860 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:13.612844 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 116916us
I20260809 23:20:13.613294 31549 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.141s
I20260809 23:20:13.619037 31842 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.619722 31762 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:13.620380 31884 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.574186 31789 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:13.679840 31789 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:13.680383 31789 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:13.679912 31884 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.623523 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.624152 31862 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.681792 31862 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:13.625527 31763 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:13.682269 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.682729 31763 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:13.683198 24321 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.683713 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2406) took 1097 ms. Trace:
I20260809 23:20:13.684059 24321 rpcz_store.cc:276] 0809 23:20:12.586626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.681946 (+1095320us) service_pool.cc:227] Handling call
0809 23:20:13.683658 (+  1712us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:13.683792 31763 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.685017 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.631225 31813 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.592324 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:13.686007 24539 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:13.686327 25366 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:20:13.634883 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86819us
I20260809 23:20:13.453975 31832 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:13.687506 31832 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.626171 31527 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.601173 31760 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:13.644572 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.691803 31760 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:13.693439 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 93076.1us
I20260809 23:20:13.694310 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.618227 31537 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.696483 31537 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.645501 24542 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:13.697520 31413 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:13.698204 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.699569 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:13.700300 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:13.700532 31470 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:13.700778 24542 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:13.701264 31470 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.702102 31787 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:13.702627 31787 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.692034 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.692092 31527 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:13.703920 31787 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.705281 31300 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.706700 31762 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:13.707854 31793 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.544s	user 0.017s	sys 0.002s
I20260809 23:20:13.713115 31832 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.714344 31860 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:13.714944 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.715739 25375 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.703919 24321 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:13.640287 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2404) took 1075 ms. Trace:
I20260809 23:20:13.661165 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.553767 31648 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.717561 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:13.665899 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 210497us
W20260809 23:20:13.667987 31520 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.409s	user 0.008s	sys 0.021s
I20260809 23:20:13.667394 31886 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.734671 24311 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:13.670697 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:13.671371 31904 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.396s	user 0.000s	sys 0.001s
I20260809 23:20:13.671624 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.673775 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:13.675179 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53824.6us
I20260809 23:20:13.675822 31690 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:13.683344 31579 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.685297 31763 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.718281 24312 rpcz_store.cc:276] 0809 23:20:12.564637 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.167506 (+602869us) service_pool.cc:227] Handling call
0809 23:20:13.640205 (+472699us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:13.656370 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.718083 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.719297 24546 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:13.665637 31896 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.632781 31765 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:13.670482 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.755988 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.732700 25365 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:13.736590 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2421) took 1059 ms. Trace:
I20260809 23:20:13.735821 24427 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:13.740425 24539 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:13.740967 24534 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:20:13.762351 31906 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260809 23:20:13.740240 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2912) took 1152 ms. Trace:
I20260809 23:20:13.703253 31817 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:13.744392 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.637882 31026 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.710347 31842 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.747457 25373 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:13.743826 31648 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.685316 31798 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260809 23:20:13.740240 24542 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:13.764618 24543 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:13.763583 24311 rpcz_store.cc:276] 0809 23:20:12.677479 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.254048 (+576569us) service_pool.cc:227] Handling call
0809 23:20:13.736535 (+482487us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:13.702244 31470 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.743993 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:13.765447 31300 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:13.768569 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.861662 24546 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:13.771906 24378 thread.cc:636] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.004s	sys 0.019s
I20260809 23:20:13.779584 24321 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:13.779856 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.863242 24435 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 12
I20260809 23:20:13.785774 24535 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:13.797581 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.799814 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.768520 24428 rpcz_store.cc:276] 0809 23:20:12.565478 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.312084 (+746606us) service_pool.cc:227] Handling call
0809 23:20:13.312544 (+   460us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:13.313127 (+   583us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.313156 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:13.313733 (+   577us) log.cc:842] Serialized 52 byte log entry
0809 23:20:13.517761 (+204028us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:13.517808 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.517885 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:13.717931 (+200046us) raft_consensus.cc:1671] finished
0809 23:20:13.717951 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:13.718071 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3126,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":148,"thread_start_us":203640,"threads_started":1,"wal-append.queue_time_us":401257}
I20260809 23:20:13.810348 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:20:13.862769 24378 thread.cc:602] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.004s	sys 0.020s
I20260809 23:20:13.810854 30919 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:13.811595 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:13.811919 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.813372 31845 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 8 FOLLOWER]: Advancing to term 9
I20260809 23:20:13.813503 31798 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:13.814150 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.882835 24311 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:13.814812 24431 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:13.816319 31026 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.817181 31783 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:13.887790 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:13.817410 31648 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:13.817832 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1185) took 1851 ms. Trace:
I20260809 23:20:13.818133 31727 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:13.901645 24321 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:13.818993 31842 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:13.766505 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.045s	sys 0.075s
W20260809 23:20:13.903126 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.045s	sys 0.075s
I20260809 23:20:13.860376 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2438) took 1336 ms. Trace:
I20260809 23:20:13.860502 31886 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:13.803292 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.880725 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:13.796116 31661 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.667s	user 0.011s	sys 0.011s
I20260809 23:20:13.907995 24428 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:13.888353 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:13.888453 24323 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:13.888940 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 77125.5us
W20260809 23:20:13.877105 31906 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260809 23:20:13.887578 31832 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.889209 31300 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.883329 24314 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:13.911484 31688 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.911531 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2405) took 1343 ms. Trace:
I20260809 23:20:13.914361 24425 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:13.904173 24424 rpcz_store.cc:276] 0809 23:20:12.524214 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.288674 (+764460us) service_pool.cc:227] Handling call
0809 23:20:13.289328 (+   654us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:13.289829 (+   501us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.289859 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:13.306087 (+ 16228us) log.cc:842] Serialized 26 byte log entry
0809 23:20:13.486638 (+180551us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:13.486692 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.486783 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:13.860125 (+373342us) raft_consensus.cc:1671] finished
0809 23:20:13.860146 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:13.860286 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15543,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"thread_start_us":169432,"threads_started":1,"wal-append.queue_time_us":514722}
I20260809 23:20:13.931598 24321 rpcz_store.cc:276] 0809 23:20:12.565001 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.876829 (+1311828us) service_pool.cc:227] Handling call
0809 23:20:13.893035 (+ 16206us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:13.902146 (+  9111us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.907996 (+  5850us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:13.805317 31785 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260809 23:20:13.882400 31798 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.887607 31783 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:13.920496 31520 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:13.913865 31764 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.000s
I20260809 23:20:13.924156 31727 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:13.945255 31842 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:13.907032 31851 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
W20260809 23:20:13.926563 31808 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.558s	user 0.013s	sys 0.006s
I20260809 23:20:13.927135 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:13.927999 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:13.929735 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:13.930613 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:13.927135 31681 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.963s	user 0.004s	sys 0.012s
W20260809 23:20:14.005069 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 85198.8us
I20260809 23:20:13.916440 31789 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:14.011293 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81729.3us
I20260809 23:20:14.002626 31681 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:13.897814 31339 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:13.920640 31832 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.896822 24320 rpcz_store.cc:276] 0809 23:20:11.694152 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.694319 (+   167us) service_pool.cc:227] Handling call
0809 23:20:11.694669 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:11.694856 (+   187us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:11.909480 (+214624us) log.cc:842] Serialized 14 byte log entry
0809 23:20:13.183987 (+1274507us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b18004d6da0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:13.409513 (+225526us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:13.469118 (+ 59605us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:13.505100 (+ 35982us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:13.505191 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:13.545618 (+ 40427us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":191506,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":2020,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":194,"spinlock_wait_cycles":8903936,"thread_start_us":302746,"threads_started":2,"wal-append.queue_time_us":1757166}
I20260809 23:20:13.946838 31764 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.939584 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.027884 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:13.941627 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 169779us
W20260809 23:20:13.958254 31790 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.833s	user 0.011s	sys 0.007s
W20260809 23:20:13.959345 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1291) took 2216 ms (client timeout 1977 ms). Trace:
I20260809 23:20:13.960336 25368 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:20:13.961503 31579 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:13.963439 31783 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.941601 31688 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.964484 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:13.964974 31300 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:13.967867 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.041679 25373 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:13.935834 31817 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:13.969990 31798 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:13.972275 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.044615 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:13.973373 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.027055 31339 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:14.053930 25365 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:13.983150 31795 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:13.983539 25370 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:13.984229 24313 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:13.988257 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:13.990409 31820 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.537s	user 0.005s	sys 0.000s
I20260809 23:20:14.057670 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:14.002417 30919 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.029438 24544 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:14.004343 31842 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:13.918644 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2401) took 1395 ms. Trace:
I20260809 23:20:14.017095 31661 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:14.031590 25372 rpcz_store.cc:269] 0809 23:20:11.743340 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.743541 (+   201us) service_pool.cc:227] Handling call
0809 23:20:13.959092 (+2215551us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540073f5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:13.959237 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4285474176}
I20260809 23:20:14.060460 24314 rpcz_store.cc:276] 0809 23:20:12.522747 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.357731 (+834984us) service_pool.cc:227] Handling call
0809 23:20:13.918510 (+560779us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:14.039070 24321 raft_consensus.cc:2362] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11. Re-sending same reply.
I20260809 23:20:14.035717 31885 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.044239 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:13.975085 31884 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:13.959360 31026 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:14.070106 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.071502 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.050283 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:14.036500 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:14.061619 31795 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.067656 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.075769 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.068060 25372 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:14.049036 31681 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.073654 30919 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.071442 31885 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.093319 30919 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.093505 31885 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:14.074368 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.076808 25366 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
W20260809 23:20:14.085216 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62231.5us
I20260809 23:20:14.056516 31783 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:14.056757 31339 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:14.085520 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:14.084798 31795 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:14.090304 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:14.091291 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:14.101874 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.103410 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:14.104326 24320 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:14.108902 25375 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:14.110749 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.084333 24311 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:14.113837 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:14.100198 24321 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:14.100348 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.044s	sys 0.054s
I20260809 23:20:14.114744 24311 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260809 23:20:14.114985 31855 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260809 23:20:14.115274 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:14.115520 24321 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:14.114974 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.044s	sys 0.055s
I20260809 23:20:14.100749 24546 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:14.112762 25369 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 8. Current term is 9. Ops: []
I20260809 23:20:14.116215 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2411) took 1494 ms. Trace:
I20260809 23:20:14.116535 24321 rpcz_store.cc:276] 0809 23:20:12.621809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.115007 (+1493198us) service_pool.cc:227] Handling call
0809 23:20:14.115338 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:14.115971 (+   633us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.116120 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:14.117506 24321 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:14.118067 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2417) took 1489 ms. Trace:
I20260809 23:20:14.118360 24321 rpcz_store.cc:276] 0809 23:20:12.628469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.117028 (+1488559us) service_pool.cc:227] Handling call
0809 23:20:14.117352 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:14.117864 (+   512us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.118028 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:14.091032 31681 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.121532 24314 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:14.127873 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:14.095397 31798 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:14.135406 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:14.136148 24425 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:14.136703 24425 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:14.138002 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87856.8us
I20260809 23:20:14.139088 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:14.152554 25365 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:14.154809 24314 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260809 23:20:14.156410 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:14.121649 31579 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:14.181915 25366 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:14.073968 31842 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.158136 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1293) took 1206 ms. Trace:
I20260809 23:20:14.155947 31021 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260809 23:20:14.164840 31832 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.189299 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:14.196699 25372 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260809 23:20:14.178575 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.037s	sys 0.047s
I20260809 23:20:14.178730 24425 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:14.100759 31817 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.144361 31795 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:14.210287 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.037s	sys 0.047s
I20260809 23:20:14.145053 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2408) took 1524 ms. Trace:
W20260809 23:20:14.213068 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 706181us
I20260809 23:20:14.213099 24313 rpcz_store.cc:276] 0809 23:20:12.621029 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.281569 (+660540us) service_pool.cc:227] Handling call
0809 23:20:14.144864 (+863295us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:14.144990 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":968056576}
I20260809 23:20:14.215384 24313 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:20:14.181501 25369 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:14.252120 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:14.190086 24428 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:14.187628 25368 rpcz_store.cc:276] 0809 23:20:12.934978 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:12.954551 (+ 19573us) service_pool.cc:227] Handling call
0809 23:20:13.959862 (+1005311us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:14.031684 (+ 71822us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.136082 (+104398us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540073f5d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:14.140966 (+  4884us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2310611840}
I20260809 23:20:14.268851 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:14.279853 31813 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.002s
I20260809 23:20:14.283746 31817 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:14.285776 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.896s (SENT)
I20260809 23:20:14.286424 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:14.287408 31820 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:14.258854 31886 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:14.294003 24312 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:14.299209 31902 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:14.299486 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108423us
I20260809 23:20:14.300006 31902 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.300515 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:14.307382 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 126089us
I20260809 23:20:14.180755 31765 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:14.309666 24321 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260809 23:20:14.186972 31579 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:14.187603 31681 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.324852 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:14.329821 24312 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
W20260809 23:20:14.333052 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1287) took 2722 ms (client timeout 1830 ms). Trace:
I20260809 23:20:14.208066 31789 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:14.345489 25370 rpcz_store.cc:269] 0809 23:20:11.610757 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.611031 (+   274us) service_pool.cc:227] Handling call
0809 23:20:14.332793 (+2721762us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:14.332955 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4660001664}
I20260809 23:20:14.302829 31902 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.387492 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:14.397401 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.402647 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:14.330854 31688 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.345489 31579 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:14.412338 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.356483 31764 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:14.371315 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.989s (SENT) [suppressed 19 similar messages]
I20260809 23:20:14.436612 24543 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:14.423110 31795 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:14.444293 24540 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:14.361752 31851 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.452788 31820 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:14.469274 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.475195 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:14.478502 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 107814us
I20260809 23:20:14.421241 31789 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:14.480685 24546 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260809 23:20:14.482318 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:14.473219 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.034s	sys 0.051s
W20260809 23:20:14.486112 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.035s	sys 0.051s
W20260809 23:20:14.486706 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 567332us
W20260809 23:20:14.180819 31907 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.017s	sys 0.037s
I20260809 23:20:14.373088 31832 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260809 23:20:14.494195 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:14.492794 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.455475 31795 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:14.454736 31470 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.039s	sys 0.063s
W20260809 23:20:14.509433 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 487625us
W20260809 23:20:14.504845 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.010s	sys 0.086s
W20260809 23:20:14.517903 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.011s	sys 0.086s
I20260809 23:20:14.489428 31688 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:14.489684 31789 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:14.522475 24431 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:20:14.492805 31907 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.017s	sys 0.038s
W20260809 23:20:14.520982 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 220608us
I20260809 23:20:14.486248 31661 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:14.522977 24431 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:14.529671 24427 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
W20260809 23:20:14.508610 31470 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.039s	sys 0.064s
I20260809 23:20:14.462776 31851 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.505632 31795 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:14.519057 31817 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:14.540033 31845 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.004s
I20260809 23:20:14.526859 31789 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:14.539263 31787 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.577823 31787 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.580052 31787 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.580578 31787 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.581079 31787 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.583607 31661 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.584815 24543 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:20:14.585914 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2451) took 1217 ms. Trace:
I20260809 23:20:14.586279 24430 rpcz_store.cc:276] 0809 23:20:13.368069 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.412816 (+ 44747us) service_pool.cc:227] Handling call
0809 23:20:13.451764 (+ 38948us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:13.638134 (+186370us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.638164 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:13.745278 (+107114us) log.cc:842] Serialized 26 byte log entry
0809 23:20:14.250967 (+505689us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:14.251033 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.251160 (+   127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:14.585705 (+334545us) raft_consensus.cc:1671] finished
0809 23:20:14.585725 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:14.585855 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6510,"raft.queue_time_us":45423,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":125,"thread_start_us":456968,"threads_started":1,"wal-append.queue_time_us":788763}
I20260809 23:20:14.588490 31763 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.588542 31785 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:14.589183 31763 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.590435 31787 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.556799 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:14.567214 31935 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.213s	user 0.002s	sys 0.000s
W20260809 23:20:14.567540 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.015s	sys 0.025s
I20260809 23:20:14.571130 31817 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.590339 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.560258 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:14.556878 24431 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:14.593714 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76722.6us
W20260809 23:20:14.595193 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.016s	sys 0.025s
W20260809 23:20:14.596791 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72689.9us
I20260809 23:20:14.595660 24542 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:14.606599 24543 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:14.608269 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:14.612112 24309 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260809 23:20:14.613052 24309 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:14.562196 31820 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:14.594242 31845 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:14.618852 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.619926 24431 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:14.594889 31857 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.912s	user 0.064s	sys 0.117s
W20260809 23:20:14.649304 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43811.7us
W20260809 23:20:14.653321 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47903.7us
I20260809 23:20:14.656737 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.605226 31884 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:14.618449 31470 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:14.594879 31764 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260809 23:20:14.671201 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.607967 31805 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 5, Time since last communication: 0.057s
W20260809 23:20:14.616987 31862 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.935s	user 0.012s	sys 0.003s
W20260809 23:20:14.651976 31857 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.064s	sys 0.118s
I20260809 23:20:14.645799 31787 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260809 23:20:14.661047 31884 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:14.666400 31470 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:14.680116 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.646507 31820 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.674703 31845 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:14.694387 31727 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:14.731110 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.684027 31774 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:14.736451 24542 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:14.684343 31805 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.692838 31470 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:14.693560 31884 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:14.778715 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.740324 31955 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:14.787539 31938 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.225s	user 0.000s	sys 0.000s
I20260809 23:20:14.787616 31955 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:14.788686 31339 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.668s	user 0.012s	sys 0.000s
I20260809 23:20:14.789168 31339 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.790475 31778 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.096s
I20260809 23:20:14.791916 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:14.792771 24546 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 14
W20260809 23:20:14.792254 31943 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260809 23:20:14.795046 31928 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
W20260809 23:20:14.795957 31950 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260809 23:20:14.796557 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2917) took 1631 ms. Trace:
I20260809 23:20:14.796936 24422 rpcz_store.cc:276] 0809 23:20:13.164949 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.338813 (+173864us) service_pool.cc:227] Handling call
0809 23:20:13.339190 (+   377us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:13.339632 (+   442us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.339659 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:13.340145 (+   486us) log.cc:842] Serialized 26 byte log entry
0809 23:20:14.116126 (+775981us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:14.116171 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.116239 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:14.796317 (+680078us) raft_consensus.cc:1671] finished
0809 23:20:14.796357 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:14.796497 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62246,"raft.queue_time_us":7979,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":8210,"thread_start_us":775548,"threads_started":1,"wal-append.queue_time_us":1124133}
W20260809 23:20:14.806028 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2413) took 2179 ms (client timeout 1830 ms). Trace:
W20260809 23:20:14.807255 31924 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.256s	user 0.001s	sys 0.001s
I20260809 23:20:14.808909 31805 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.809688 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1300) took 1300 ms. Trace:
I20260809 23:20:14.779349 31944 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:14.810603 31944 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.811077 31805 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.812899 31944 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:14.815584 31937 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.264s	user 0.003s	sys 0.001s
I20260809 23:20:14.730705 31727 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:14.816905 31727 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.818099 31727 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:14.820087 31820 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.821867 31884 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:14.822182 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.732350 31845 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:14.840442 31845 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:14.842164 24311 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:14.843026 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2415) took 2215 ms. Trace:
I20260809 23:20:14.843148 31851 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.843484 24311 rpcz_store.cc:276] 0809 23:20:12.627579 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.115436 (+1487857us) service_pool.cc:227] Handling call
0809 23:20:14.115786 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:14.841959 (+726173us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54001b8b20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:14.842698 (+   739us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.842961 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1324772736}
W20260809 23:20:14.844614 31783 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.742s	user 0.012s	sys 0.000s
I20260809 23:20:14.810207 25371 rpcz_store.cc:276] 0809 23:20:13.509662 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.509828 (+   166us) service_pool.cc:227] Handling call
0809 23:20:13.548250 (+ 38422us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:13.610453 (+ 62203us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.610498 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:13.611290 (+   792us) log.cc:842] Serialized 52 byte log entry
0809 23:20:14.050158 (+438868us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:14.056387 (+  6229us) log.cc:842] Serialized 14 byte log entry
0809 23:20:14.092735 (+ 36348us) spinlock_profiling.cc:243] Waited 927 us on lock 0x7b180007ca20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:14.092928 (+   193us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.093000 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:14.809321 (+716321us) raft_consensus.cc:1671] finished
0809 23:20:14.809457 (+   136us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:14.809615 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4383,"prepare.queue_time_us":5909,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":1589,"raft.queue_time_us":102758,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"spinlock_wait_cycles":2132736,"thread_start_us":437663,"threads_started":1,"wal-append.queue_time_us":868964}
I20260809 23:20:14.853828 31789 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.855115 24544 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:14.859609 31805 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:14.860015 31805 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:14.861578 31805 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:14.863242 31805 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:14.815732 31945 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260809 23:20:14.806473 24309 rpcz_store.cc:269] 0809 23:20:12.626849 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.152153 (+525304us) service_pool.cc:227] Handling call
0809 23:20:14.805806 (+1653653us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:14.805942 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954135936}
I20260809 23:20:14.868604 24323 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:14.868889 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:14.739862 31774 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:14.893923 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66340.4us
I20260809 23:20:14.790541 31955 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:14.869341 31808 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.008s	sys 0.025s
I20260809 23:20:14.912178 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1105) took 1161 ms. Trace:
I20260809 23:20:14.918217 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.931524 24546 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:14.932569 24541 pending_rounds.cc:76] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 9
I20260809 23:20:14.932989 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:14.915171 31021 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:14.950977 24541 pending_rounds.cc:97] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260809 23:20:14.951376 24541 raft_consensus.cc:2866] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:14.956898 31817 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.962850 31763 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:14.963452 31763 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:14.963771 31763 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:14.964116 31763 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:14.966301 31339 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:14.929911 24534 rpcz_store.cc:276] 0809 23:20:13.750688 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.771108 (+ 20420us) service_pool.cc:227] Handling call
0809 23:20:13.771460 (+   352us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:13.809184 (+ 37724us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.809229 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:13.940060 (+130831us) log.cc:842] Serialized 26 byte log entry
0809 23:20:14.518336 (+578276us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:14.518375 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.518444 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:14.911952 (+393508us) raft_consensus.cc:1671] finished
0809 23:20:14.911977 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:14.912117 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":39091,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":562414,"threads_started":1,"wal-append.queue_time_us":713783}
I20260809 23:20:14.971712 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:14.972520 24546 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:14.982340 24316 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
W20260809 23:20:14.983718 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2863) took 1669 ms (client timeout 2180 ms). Trace:
W20260809 23:20:14.984069 24316 rpcz_store.cc:269] 0809 23:20:13.314302 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.592440 (+278138us) service_pool.cc:227] Handling call
0809 23:20:14.983563 (+1391123us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:14.983664 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2339409152}
I20260809 23:20:14.986449 24431 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:20:14.890731 31820 raft_consensus.cc:2741] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260809 23:20:15.003146 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.005s	sys 0.006s
W20260809 23:20:14.941121 31808 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.009s	sys 0.026s
W20260809 23:20:15.003691 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.005s	sys 0.006s
W20260809 23:20:15.046108 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 428694us
I20260809 23:20:15.047259 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:14.990065 31944 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:15.053503 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:15.058360 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:15.045106 31789 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:15.078696 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 390601us
I20260809 23:20:14.988365 31795 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.056205 31860 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:15.065936 31902 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:15.060436 31955 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.066076 31817 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:15.053553 31944 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:15.055344 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.017s	sys 0.014s
W20260809 23:20:15.106395 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.017s	sys 0.014s
W20260809 23:20:15.085747 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.002s	sys 0.017s
I20260809 23:20:15.096081 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:15.108327 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.002s	sys 0.017s
I20260809 23:20:15.086500 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1181) took 1105 ms. Trace:
I20260809 23:20:15.089900 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.087607 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:15.089505 31789 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:15.088873 31688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.018s	sys 0.001s
I20260809 23:20:15.098866 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.076639 31884 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:15.100698 31760 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.110814 24429 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 9.
I20260809 23:20:15.118523 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:15.055892 31300 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.033s
I20260809 23:20:15.109946 25363 rpcz_store.cc:276] 0809 23:20:13.981581 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:13.992502 (+ 10921us) service_pool.cc:227] Handling call
0809 23:20:13.992935 (+   433us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:13.993443 (+   508us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:13.993477 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:13.994068 (+   591us) log.cc:842] Serialized 52 byte log entry
0809 23:20:14.598697 (+604629us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:15.085926 (+487229us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.086163 (+   237us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.086242 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:15.086286 (+    44us) raft_consensus.cc:1671] finished
0809 23:20:15.086298 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:15.086445 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":470,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"thread_start_us":1075946,"threads_started":2,"wal-append.queue_time_us":567254}
I20260809 23:20:15.121544 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:15.132810 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74549.4us
I20260809 23:20:15.133792 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:15.082029 31832 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.001s	sys 0.019s
I20260809 23:20:15.091938 31727 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:15.140038 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.068687 31470 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.144047 31820 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:15.115783 31902 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:15.147487 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:15.119611 31688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.018s	sys 0.001s
I20260809 23:20:15.156504 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:15.117401 31944 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:15.162375 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:15.146375 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:15.121044 31681 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
W20260809 23:20:15.157174 31832 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.001s	sys 0.019s
I20260809 23:20:15.180871 24422 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:15.188050 24427 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:15.191349 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.194139 24431 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:15.197011 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:15.078277 31774 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.214007 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.208009 24429 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:15.193396 24427 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:15.158851 31790 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:15.155375 31817 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:15.215340 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.115363 31955 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.106029 31860 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.220278 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.223448 24423 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260809 23:20:15.230759 31927 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 23:20:13.254010 (+     0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.21.255.2:42099)
0809 23:20:14.116293 (+862283us) server_negotiation.cc:206] Beginning negotiation
0809 23:20:14.116348 (+    55us) server_negotiation.cc:399] Waiting for connection header
0809 23:20:14.116443 (+    95us) server_negotiation.cc:407] Connection header received
0809 23:20:14.116718 (+   275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 23:20:14.116746 (+    28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 23:20:14.117057 (+   311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 23:20:14.117331 (+   274us) server_negotiation.cc:226] Negotiated authn=SASL
0809 23:20:14.119061 (+  1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 23:20:14.325354 (+206293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 23:20:15.229228 (+903874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45890: BlockingRecv error: recv got EOF from 127.0.0.1:45890 (error 108)
Metrics: {"mutex_wait_us":947,"server-negotiator.queue_time_us":799803,"thread_start_us":610440,"threads_started":1}
I20260809 23:20:15.180712 31688 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:15.253211 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:15.254910 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:15.146348 31845 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.207345 31727 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:15.137014 31760 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.178376 31851 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260809 23:20:15.213506 31931 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:15.224700 31790 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:15.255656 31681 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:15.176748 31820 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.262670 24427 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.249343 31527 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
W20260809 23:20:15.266459 30919 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.012s	sys 0.015s
I20260809 23:20:15.225035 31975 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:15.304397 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:15.268683 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:15.321081 24546 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260809 23:20:15.270747 31787 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:15.367300 31798 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.231s	user 0.010s	sys 0.004s
I20260809 23:20:15.368238 24317 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260809 23:20:15.277480 31817 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:15.360090 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.311455 31931 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:15.362514 31978 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260809 23:20:15.353602 30919 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.012s	sys 0.015s
I20260809 23:20:15.316076 31974 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:15.312736 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:15.317065 31790 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:15.367161 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 687143us
I20260809 23:20:15.388832 31884 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:15.389114 31955 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:15.390044 31845 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:15.393406 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:15.380712 31795 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:15.396221 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2874) took 1427 ms. Trace:
W20260809 23:20:15.256323 31793 consensus_queue.cc:1159] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 12 status { last_received { term: 12 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 12 index: 10 } }
I20260809 23:20:15.335522 31727 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.376786 24430 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:15.355638 31975 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:15.438076 24546 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:15.438081 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:15.395668 31962 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
W20260809 23:20:15.407223 31661 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.011s	sys 0.032s
I20260809 23:20:15.448949 24317 rpcz_store.cc:276] 0809 23:20:13.968415 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.101607 (+133192us) service_pool.cc:227] Handling call
0809 23:20:15.396022 (+1294415us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:15.396160 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2910115584}
I20260809 23:20:15.400369 31970 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:15.434028 31950 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260809 23:20:15.384160 31931 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:15.392407 31763 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.399194 31300 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.113s
I20260809 23:20:15.539566 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:15.541234 24423 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:20:15.434542 30919 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:15.557051 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 252751us
I20260809 23:20:15.498021 31845 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:15.561645 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:15.563292 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:15.563315 31965 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.564029 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:15.564815 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:15.503335 31884 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:15.502069 31795 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:15.501659 31975 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:15.565189 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:15.576550 24316 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260809 23:20:15.566200 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:15.567283 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 295551us
I20260809 23:20:15.565518 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:15.569566 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:15.557636 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.006s	sys 0.017s
I20260809 23:20:15.498986 31470 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:15.492766 31851 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:15.575131 24312 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:15.577319 24323 raft_consensus.cc:2362] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12. Re-sending same reply.
W20260809 23:20:15.594089 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.043s	user 0.007s	sys 0.018s
I20260809 23:20:15.634567 24309 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:15.542505 31661 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.013s	sys 0.035s
I20260809 23:20:15.626664 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.627774 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:15.627172 31965 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.629395 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:15.631237 31963 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260809 23:20:15.429971 31974 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.404006 31902 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:15.578657 31958 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260809 23:20:15.652427 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:15.653460 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:15.576532 31957 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:15.500270 31727 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.657832 24314 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:15.670601 24534 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:15.680135 24316 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:15.679208 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.007s	sys 0.022s
W20260809 23:20:15.683604 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.007s	sys 0.022s
I20260809 23:20:15.677210 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:15.651434 31978 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260809 23:20:15.691641 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:15.644990 31961 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:15.693449 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:15.574769 31778 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.087s
W20260809 23:20:15.695758 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.074s	sys 0.098s
W20260809 23:20:15.719720 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.074s	sys 0.098s
I20260809 23:20:15.712805 24544 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:15.434772 31688 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:15.720237 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 665586us
W20260809 23:20:15.679263 31857 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.004s	sys 0.025s
I20260809 23:20:15.632236 31805 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
W20260809 23:20:15.697747 31930 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.008s	sys 0.008s
I20260809 23:20:15.737591 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.682214 31527 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.544359 31970 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.742759 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2865) took 1346 ms. Trace:
I20260809 23:20:15.713930 31860 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:15.632931 31795 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.694214 31980 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:15.746592 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 294917us
I20260809 23:20:15.639674 31845 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.631069 31681 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.751518 24427 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:15.672374 24317 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 9: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:15.739555 31857 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.004s	sys 0.025s
W20260809 23:20:15.739879 31930 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.009s	sys 0.009s
I20260809 23:20:15.672374 31851 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:15.752833 31579 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.284s	user 0.002s	sys 0.013s
I20260809 23:20:15.724377 31798 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:15.755103 24312 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:15.725487 31760 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:15.821895 31887 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 23:20:12.115552 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.21.255.2:42099 (local address 127.0.0.1:45890)
0809 23:20:12.834059 (+718507us) negotiation.cc:107] Waiting for socket to connect
0809 23:20:12.834126 (+    67us) client_negotiation.cc:174] Beginning negotiation
0809 23:20:12.834466 (+   340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 23:20:14.117474 (+1283008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 23:20:14.117543 (+    69us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 23:20:14.117664 (+   121us) client_negotiation.cc:189] Negotiated authn=SASL
0809 23:20:14.118529 (+   865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 23:20:14.118630 (+   101us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 23:20:14.424182 (+305552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 23:20:14.463696 (+ 39514us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 23:20:15.115246 (+651550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 23:20:15.132065 (+ 16819us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 23:20:15.312065 (+180000us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.21.255.2:42099: sent 0 of 16 requested bytes
Metrics: {"client-negotiator.queue_time_us":663638,"mutex_wait_us":6,"thread_start_us":860127,"threads_started":1}
I20260809 23:20:15.673427 31989 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260809 23:20:15.764699 24546 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:15.719676 31957 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.766010 24498 leader_election.cc:398] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:15.765908 31967 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:20:15.749333 24539 rpcz_store.cc:276] 0809 23:20:14.396260 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.396415 (+   155us) service_pool.cc:227] Handling call
0809 23:20:14.434431 (+ 38016us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:14.434884 (+   453us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.434922 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:14.435491 (+   569us) log.cc:842] Serialized 26 byte log entry
0809 23:20:14.898200 (+462709us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:14.898343 (+   143us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:14.898408 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:15.742558 (+844150us) raft_consensus.cc:1671] finished
0809 23:20:15.742581 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:15.742699 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53,"raft.queue_time_us":10525,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":12099,"thread_start_us":443560,"threads_started":1,"wal-append.queue_time_us":677808}
I20260809 23:20:15.777586 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2926) took 1667 ms. Trace:
I20260809 23:20:15.794807 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.863150 24435 rpcz_store.cc:276] 0809 23:20:14.110387 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.127629 (+ 17242us) service_pool.cc:227] Handling call
0809 23:20:14.128596 (+   967us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:14.185742 (+ 57146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.185787 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:14.263748 (+ 77961us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.107379 (+843631us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:15.107445 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.107563 (+   118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:15.769054 (+661491us) raft_consensus.cc:1671] finished
0809 23:20:15.777337 (+  8283us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:15.777512 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":242840,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":16711,"thread_start_us":797580,"threads_started":1,"wal-append.queue_time_us":1144522}
I20260809 23:20:15.836024 24498 leader_election.cc:401] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260809 23:20:15.865978 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.867126 31955 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:15.867571 31982 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:15.867545 31955 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:15.869053 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.870095 31955 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.871757 31955 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:15.833104 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 186197us
I20260809 23:20:15.759491 31820 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:15.867157 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 101365us
I20260809 23:20:15.866011 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:15.817756 31026 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.004s	sys 0.012s
W20260809 23:20:15.848006 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 79351.8us
W20260809 23:20:15.863785 24320 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.011s	sys 0.015s
I20260809 23:20:15.767330 31980 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.886512 24423 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:15.749792 31527 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:15.887508 24320 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.011s	sys 0.015s
I20260809 23:20:15.762857 31860 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.117s
I20260809 23:20:15.875277 31760 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260809 23:20:15.919431 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:15.921008 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:15.900619 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:15.886868 31026 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.006s	sys 0.014s
I20260809 23:20:15.923485 24312 raft_consensus.cc:2362] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12. Re-sending same reply.
I20260809 23:20:15.924598 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2869) took 1957 ms. Trace:
I20260809 23:20:15.930081 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:15.886269 31789 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.764s	user 0.012s	sys 0.000s
I20260809 23:20:15.929826 31727 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:15.930696 31470 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.940622 31470 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:15.931192 24313 rpcz_store.cc:276] 0809 23:20:13.966829 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.257477 (+290648us) service_pool.cc:227] Handling call
0809 23:20:14.328638 (+ 71161us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:20:14.432697 (+104059us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.432741 (+    44us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:20:14.647883 (+215142us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:14.652478 (+  4595us) log.cc:842] Serialized 78 byte log entry
0809 23:20:15.087033 (+434555us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:15.515106 (+428073us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.557933 (+ 42827us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.878321 (+320388us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.920612 (+ 42291us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.924297 (+  3685us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:15.924379 (+    82us) raft_consensus.cc:1671] finished
0809 23:20:15.924390 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:15.924540 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":163925,"prepare.queue_time_us":179162,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":126,"raft.queue_time_us":242613,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":11051,"spinlock_wait_cycles":29983488,"thread_start_us":1136499,"threads_started":3,"wal-append.queue_time_us":604098}
W20260809 23:20:15.922529 31964 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.421s	user 0.001s	sys 0.000s
I20260809 23:20:15.819825 31579 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.899341 31967 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:15.767954 31884 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.859151 31798 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:15.950829 24422 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.739s	user 0.009s	sys 0.000s
I20260809 23:20:16.048454 24422 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:20:16.040934 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 140824us
W20260809 23:20:16.011694 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94218.9us
I20260809 23:20:15.916174 31980 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:15.912247 31966 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.360s
I20260809 23:20:15.931890 31783 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.070729 24422 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260809 23:20:16.071439 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 150020us
I20260809 23:20:15.925841 31763 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.058578 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:15.925348 31996 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:16.023864 31991 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.437s	user 0.000s	sys 0.002s
W20260809 23:20:16.069295 31977 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.421s	user 0.000s	sys 0.001s
I20260809 23:20:16.035444 31774 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:15.926165 30919 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.163s
I20260809 23:20:16.107059 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:15.924763 31805 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.164s
I20260809 23:20:16.124853 30919 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:16.125427 30919 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:16.125638 24422 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:16.127665 24422 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:20:16.058300 31727 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
W20260809 23:20:16.111940 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 377555us
I20260809 23:20:16.133615 24546 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:16.113098 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59691.3us
I20260809 23:20:16.088986 31783 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.115880 31982 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:16.116765 31851 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.120065 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2728) took 1528 ms. Trace:
I20260809 23:20:16.163827 24545 rpcz_store.cc:276] 0809 23:20:14.592041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.592229 (+   188us) service_pool.cc:227] Handling call
0809 23:20:14.593750 (+  1521us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:14.594061 (+   311us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.594090 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:14.594541 (+   451us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.061917 (+467376us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:15.061971 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.062057 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.119850 (+1057793us) raft_consensus.cc:1671] finished
0809 23:20:16.119873 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.120005 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":56704,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":10660,"thread_start_us":416851,"threads_started":1,"wal-append.queue_time_us":847859}
I20260809 23:20:16.120628 31760 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:16.122138 31860 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:16.123155 31832 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.091450 31965 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.137431 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.140275 24422 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:16.036479 32038 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:15.971619 31789 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:16.089625 31974 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.215564 24546 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:16.212512 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75165.1us
I20260809 23:20:16.111128 31857 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.134404 32034 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:16.218264 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108197us
I20260809 23:20:16.111689 31996 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.221297 32032 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:16.221776 31579 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.221481 31967 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.222901 32008 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.329s	user 0.001s	sys 0.000s
W20260809 23:20:16.223482 32011 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.395s	user 0.000s	sys 0.001s
I20260809 23:20:16.135468 32017 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.201042 31820 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
W20260809 23:20:16.225476 31990 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.401s	user 0.000s	sys 0.001s
I20260809 23:20:16.225399 31470 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.225904 31832 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.201455 31760 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:16.211576 31845 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.234670 31860 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:16.295360 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 188673us
I20260809 23:20:16.219375 31795 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:16.294798 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:16.224453 31798 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.215993 31970 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.298054 31965 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:16.320330 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.026s	sys 0.023s
I20260809 23:20:16.126818 30919 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:16.324982 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90212.6us
I20260809 23:20:16.326393 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:16.323730 24424 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.001s	sys 0.005s
I20260809 23:20:16.234262 31764 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.331409 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.237449 31857 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.227255 32038 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.290030 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 426283us
I20260809 23:20:16.237847 32032 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:16.294829 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:16.242123 32034 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.316562 31727 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:16.465713 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2471) took 1178 ms. Trace:
I20260809 23:20:16.471860 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:16.391880 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89793.2us
I20260809 23:20:16.450870 32039 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260809 23:20:16.470505 32032 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:16.445977 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2460) took 1956 ms. Trace:
W20260809 23:20:16.424053 24427 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.667s	user 0.011s	sys 0.001s
I20260809 23:20:16.487118 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:16.476629 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.488343 24423 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:20:16.488906 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.006s	sys 0.001s
W20260809 23:20:16.477979 24424 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.004s	sys 0.008s
I20260809 23:20:16.489782 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.340739 24540 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:16.233160 31982 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:16.481413 24546 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:16.470942 24431 rpcz_store.cc:276] 0809 23:20:15.287604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.287802 (+   198us) service_pool.cc:227] Handling call
0809 23:20:15.320147 (+ 32345us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:15.818379 (+498232us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:15.818410 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:15.819032 (+   622us) log.cc:842] Serialized 52 byte log entry
0809 23:20:16.155723 (+336691us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:16.174311 (+ 18588us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.178145 (+  3834us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.180513 (+  2368us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:16.181864 (+  1351us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.465477 (+283613us) raft_consensus.cc:1671] finished
0809 23:20:16.465501 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.465649 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16060,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":88,"raft.queue_time_us":154079,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":6047,"spinlock_wait_cycles":7100800,"thread_start_us":739422,"threads_started":2,"wal-append.queue_time_us":531592}
W20260809 23:20:16.489291 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.006s	sys 0.002s
W20260809 23:20:16.418834 31964 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260809 23:20:16.507493 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2464) took 1340 ms. Trace:
W20260809 23:20:16.493109 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.039s	sys 0.035s
I20260809 23:20:16.485913 24428 rpcz_store.cc:276] 0809 23:20:14.489912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.490040 (+   128us) service_pool.cc:227] Handling call
0809 23:20:14.514369 (+ 24329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:15.014494 (+500125us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:15.014540 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:15.052615 (+ 38075us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.514672 (+462057us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:15.514727 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.514821 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.445738 (+930917us) raft_consensus.cc:1671] finished
0809 23:20:16.445769 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.445912 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":335245,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":897372,"threads_started":2,"wal-append.queue_time_us":678904}
W20260809 23:20:16.450771 32035 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
I20260809 23:20:16.507969 24427 rpcz_store.cc:276] 0809 23:20:15.167089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.168204 (+  1115us) service_pool.cc:227] Handling call
0809 23:20:15.169008 (+   804us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:15.641986 (+472978us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b58006044b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:20:16.507144 (+865158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:16.507177 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:16.507215 (+    38us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:16.507252 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:16.507320 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.507438 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2209,"spinlock_wait_cycles":76036480}
I20260809 23:20:16.433734 31851 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:16.449570 31944 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.208s	user 0.013s	sys 0.000s
W20260809 23:20:16.428565 32028 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260809 23:20:16.429479 31760 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.458976 31783 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.531631 31783 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.532109 31783 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.510175 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1194) took 2698 ms. Trace:
I20260809 23:20:16.498426 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:16.514168 31817 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.140s	user 0.011s	sys 0.000s
I20260809 23:20:16.534310 31817 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:16.535101 32036 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:16.535622 31944 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.535719 32036 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.516441 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.537429 24546 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:16.537873 31944 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.539152 32057 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260809 23:20:16.516803 31975 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.866s	user 0.009s	sys 0.001s
I20260809 23:20:16.535729 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2473) took 1026 ms. Trace:
I20260809 23:20:16.540670 31975 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.541221 32038 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.542356 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.540684 24430 rpcz_store.cc:276] 0809 23:20:15.508886 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.509002 (+   116us) service_pool.cc:227] Handling call
0809 23:20:15.553166 (+ 44164us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:15.567582 (+ 14416us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:15.567614 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:15.568168 (+   554us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.829887 (+261719us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:15.829949 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.830032 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.535474 (+705442us) raft_consensus.cc:1671] finished
0809 23:20:16.535495 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.535652 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13959,"raft.queue_time_us":19522,"spinlock_wait_cycles":108321024,"thread_start_us":260741,"threads_started":1,"wal-append.queue_time_us":466407}
I20260809 23:20:16.462697 31820 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:16.466887 32056 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260809 23:20:16.517203 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:16.460067 31832 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.455180 31980 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.469149 31996 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.347644 31470 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.183s
I20260809 23:20:16.532970 24320 rpcz_store.cc:276] 0809 23:20:13.807336 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.124362 (+317026us) service_pool.cc:227] Handling call
0809 23:20:14.143708 (+ 19346us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:14.352400 (+208692us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.352447 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:14.422021 (+ 69574us) log.cc:842] Serialized 52 byte log entry
0809 23:20:15.109212 (+687191us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:15.915843 (+806631us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.502793 (+586950us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.505620 (+  2827us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:16.505719 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.505764 (+    45us) raft_consensus.cc:1671] finished
0809 23:20:16.505780 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.505918 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":204143,"prepare.queue_time_us":227019,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":86,"raft.queue_time_us":209042,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":61251,"thread_start_us":1929455,"threads_started":3,"wal-append.queue_time_us":878199}
W20260809 23:20:16.511056 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.003s	sys 0.016s
I20260809 23:20:16.521471 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.522046 32052 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:16.486526 31965 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:16.529775 31790 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.090s	user 0.010s	sys 0.001s
I20260809 23:20:16.505725 31688 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.025s
I20260809 23:20:16.488205 31764 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.548583 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1040) took 1011 ms. Trace:
I20260809 23:20:16.556624 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.557853 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.601859 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:16.602258 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.003s	sys 0.016s
I20260809 23:20:16.596803 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.488495 31860 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.471244 31727 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.602835 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 113176us
I20260809 23:20:16.525002 31902 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.600282 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:16.610373 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:16.620141 24546 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:20:16.533721 32061 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:16.316907 32017 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.622520 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 124186us
I20260809 23:20:16.610191 31975 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.628069 24545 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:16.626488 24540 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:16.628247 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63108.7us
W20260809 23:20:16.630555 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 738483us
I20260809 23:20:16.619589 31774 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.613299 31805 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:16.642068 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.624099 24427 rpcz_store.cc:276] 0809 23:20:15.537169 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:16.546434 (+1009265us) service_pool.cc:227] Handling call
0809 23:20:16.548509 (+  2075us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:16.664988 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.525156 31851 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.637816 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:16.659955 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63278.5us
I20260809 23:20:16.674993 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.631273 32049 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:16.644362 31961 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.895s	user 0.010s	sys 0.041s
I20260809 23:20:16.632355 32039 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:16.644711 31996 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:16.660182 32038 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.692346 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.630152 31790 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.698916 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 188528us
I20260809 23:20:16.620707 31820 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.692881 24430 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:16.644074 32061 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:16.701889 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.703557 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.704216 32049 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:16.706759 32036 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.700075 32039 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:16.689352 32034 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:16.708052 32031 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260809 23:20:16.755160 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 249724us
I20260809 23:20:16.755961 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:16.756668 31955 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.874s	user 0.013s	sys 0.003s
I20260809 23:20:16.757141 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.757742 31955 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.758098 31984 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:16.688233 32048 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:16.760722 32048 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:16.761147 32032 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.761679 32048 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.719523 32038 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.764221 31795 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:16.764662 31795 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:16.764868 32038 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.765466 31795 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.767124 24321 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:16.767781 31993 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.934s	user 0.000s	sys 0.003s
I20260809 23:20:16.708475 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.769028 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2731) took 1947 ms. Trace:
I20260809 23:20:16.769084 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2470) took 1598 ms. Trace:
I20260809 23:20:16.769508 24541 rpcz_store.cc:276] 0809 23:20:14.821456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.879423 (+ 57967us) service_pool.cc:227] Handling call
0809 23:20:14.931979 (+ 52556us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:14.955062 (+ 23083us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.955099 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:14.955559 (+   460us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.437823 (+482264us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:15.437863 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.437928 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.767097 (+1329169us) spinlock_profiling.cc:243] Waited 9.39 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:16.768705 (+  1608us) raft_consensus.cc:1671] finished
0809 23:20:16.768730 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.768965 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21593472,"thread_start_us":481894,"threads_started":1,"wal-append.queue_time_us":769323}
I20260809 23:20:16.769610 24429 rpcz_store.cc:276] 0809 23:20:15.171125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.215403 (+ 44278us) service_pool.cc:227] Handling call
0809 23:20:15.246760 (+ 31357us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:15.259672 (+ 12912us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:15.259715 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:15.533272 (+273557us) spinlock_profiling.cc:243] Waited 5.51 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:15.566756 (+ 33484us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.825647 (+258891us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:15.825692 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.825784 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:16.768765 (+942981us) raft_consensus.cc:1671] finished
0809 23:20:16.768787 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.769026 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12413,"raft.queue_time_us":137970,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":12680448,"thread_start_us":258497,"threads_started":1,"wal-append.queue_time_us":1060346}
I20260809 23:20:16.771662 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.666690 31975 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:16.772251 31975 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:16.772673 31975 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:16.718158 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.664125 31805 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:16.774914 32039 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.775890 31805 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:16.776171 32039 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:16.716600 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:16.715879 24540 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:16.725533 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.772688 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.708050 31967 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:16.778347 24541 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
W20260809 23:20:16.698401 31961 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.011s	sys 0.041s
I20260809 23:20:16.627497 32052 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.749606 32057 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:16.751531 31790 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.752686 32017 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:16.742031 31857 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.771791 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1212) took 1199 ms. Trace:
I20260809 23:20:16.815261 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:16.815419 24316 rpcz_store.cc:276] 0809 23:20:15.572644 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.627435 (+ 54791us) service_pool.cc:227] Handling call
0809 23:20:16.771609 (+1144174us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:16.771735 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90952064}
I20260809 23:20:16.775547 24427 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260809 23:20:16.723991 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:16.758296 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.784325 24544 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:16.782248 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2735) took 1794 ms. Trace:
W20260809 23:20:16.798152 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 156191us
I20260809 23:20:16.820554 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.766782 31795 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:16.668021 31774 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.758211 31955 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:16.831646 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 158486us
I20260809 23:20:16.832844 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:16.833369 24269 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.542s (SENT)
I20260809 23:20:16.833438 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:16.644470 31902 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.822206 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.727237 32061 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:16.775077 31984 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.838881 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.755172 31851 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:16.834573 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 109150us
I20260809 23:20:16.845743 24541 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:16.817754 24542 rpcz_store.cc:276] 0809 23:20:14.987561 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.987699 (+   138us) service_pool.cc:227] Handling call
0809 23:20:15.006979 (+ 19280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:15.007223 (+   244us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:15.437809 (+430586us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.741516 (+1303707us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b18004e3640. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:20:16.741876 (+   360us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.781802 (+ 39926us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:16.781906 (+   104us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:16.781956 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:16.782039 (+    83us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:16.782180 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1007957,"prepare.queue_time_us":739072,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":391,"raft.queue_time_us":190845,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":45556,"spinlock_wait_cycles":10720512,"thread_start_us":684127,"threads_started":2,"wal-append.queue_time_us":43515}
I20260809 23:20:16.801038 32057 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:16.843751 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131909us
I20260809 23:20:16.800716 31808 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.797369 31961 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:16.795518 31957 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.857904 24541 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:16.857952 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:16.863330 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.663110 31982 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:16.873510 32075 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260809 23:20:16.800841 32036 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:16.875702 31808 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:16.877846 31808 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.879103 31820 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.880616 31832 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Leader election won for term 9
I20260809 23:20:16.880357 31961 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.883828 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.885596 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:16.886902 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.887531 24431 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:16.888336 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.865900 31793 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:16.867614 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:16.894909 24423 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:16.868347 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:16.895327 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.871495 32068 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:16.845175 32034 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.877625 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.830462 31763 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.865279 24542 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 9. Current term is 10. Ops: []
I20260809 23:20:16.895620 24422 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:16.896636 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:16.875573 32036 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:16.899492 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66182.4us
I20260809 23:20:16.898057 31765 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:16.901590 32040 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:16.902894 32071 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.904426 31955 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:16.904863 32051 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:20:16.905447 31884 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:20:16.906194 31774 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.894496 31793 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:16.907637 32086 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:16.909039 31527 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:16.909051 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:16.911938 32024 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:16.849341 32038 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:16.916409 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:16.860707 31980 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:16.863512 32057 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:16.923460 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57848.3us
I20260809 23:20:16.849195 32084 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:16.895147 31958 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:20:16.864076 32032 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.993927 31851 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:16.994542 31965 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:17.025225 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 183070us
I20260809 23:20:16.962383 32068 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260809 23:20:16.926882 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49361.7us
I20260809 23:20:16.926728 31688 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:17.055025 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2474) took 1545 ms. Trace:
I20260809 23:20:17.136433 24435 rpcz_store.cc:276] 0809 23:20:15.509213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.870717 (+361504us) service_pool.cc:227] Handling call
0809 23:20:15.871283 (+   566us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:15.871671 (+   388us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:15.898556 (+ 26885us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.776441 (+877885us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b18004d3e00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:17.044205 (+267764us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:17.044293 (+    88us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:17.044415 (+   122us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.044546 (+   131us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.054965 (+ 10419us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56846,"prepare.queue_time_us":22446,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":46099,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":10634,"spinlock_wait_cycles":352193408,"thread_start_us":267410,"threads_started":1}
W20260809 23:20:16.963467 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 205199us
I20260809 23:20:16.996970 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.138223 24423 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:16.897352 31783 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:16.955333 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58796.3us
I20260809 23:20:17.112324 32038 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.023394 31765 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:16.917495 31931 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.341s	user 0.013s	sys 0.003s
I20260809 23:20:16.863138 31984 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:16.924733 32034 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:17.035151 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112935us
I20260809 23:20:16.982862 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:16.907675 31764 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:16.918440 31832 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:17.025383 32072 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:16.926201 31763 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.035987 31860 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.096589 32047 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:17.098043 32088 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260809 23:20:17.104971 24544 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:17.148288 32038 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.100601 31793 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:17.139253 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 222944us
I20260809 23:20:17.199242 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.141381 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:17.001544 32091 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.101s	user 0.000s	sys 0.003s
I20260809 23:20:17.143889 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.151051 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.155722 24534 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:17.206058 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55105.3us
I20260809 23:20:17.166797 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.145758 31955 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:17.211652 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.084908 32071 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.082320 31470 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.144539 31851 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:17.194118 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57845.9us
I20260809 23:20:17.155103 31688 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.156661 31931 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.159602 32068 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:17.207784 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66512.5us
I20260809 23:20:17.201782 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.208770 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:17.210371 24541 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260809 23:20:17.169692 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 214615us
I20260809 23:20:17.233632 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:17.193572 32032 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260809 23:20:17.219069 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80614.3us
I20260809 23:20:17.237484 25365 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:17.237730 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:17.218889 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.239655 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.239712 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.115248 32024 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.235473 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.216342 31783 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:17.216832 32036 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:17.241858 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 135077us
I20260809 23:20:17.219766 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:17.245344 25370 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260809 23:20:17.218895 31860 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:17.250176 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.251185 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.231513 31982 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:17.249392 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.252388 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:17.248006 25370 pending_rounds.cc:76] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 9
I20260809 23:20:17.264210 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:17.267459 25370 pending_rounds.cc:97] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260809 23:20:17.268321 25370 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:17.269932 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.270787 25365 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:17.271056 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.238979 32049 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:17.189544 32072 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.245626 32078 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:17.272212 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.280249 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.277555 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.282853 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.237529 32032 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.291728 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.229102 31688 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.311182 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.311592 32061 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.313190 31688 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.313623 32061 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.245438 31961 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:17.314828 31961 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.315225 31961 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.316098 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:17.316762 31862 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:17.317210 31862 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:17.317682 24544 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:17.318562 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.319161 24544 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:17.285122 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.288275 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.247435 31783 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.253489 31965 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:17.289331 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44272.2us
W20260809 23:20:17.272385 32079 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260809 23:20:17.294029 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.294309 32107 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:17.294667 32094 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.139s
I20260809 23:20:17.295300 32095 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
W20260809 23:20:17.296271 32089 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:20:17.336465 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.291666 31808 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:17.298600 32071 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.292476 31765 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.299039 32098 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:17.300181 31763 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:17.240959 32068 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.354460 31808 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.356617 32068 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:17.356772 31808 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:17.301785 32108 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260809 23:20:17.351145 32098 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.358217 32098 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.362802 25373 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:17.335294 32107 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:17.364331 32107 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.365653 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.365885 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.372116 32040 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:17.372551 32040 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:17.373477 32040 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:17.376893 32093 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.278s	user 0.000s	sys 0.005s
I20260809 23:20:17.318245 31862 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.377804 31970 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:17.377926 31862 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.245395 31832 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.230801 31975 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.303347 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.303781 31948 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:17.242621 31764 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260809 23:20:17.306854 32112 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:17.307253 32032 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.274329 32048 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:17.308317 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1211) took 1745 ms. Trace:
I20260809 23:20:17.309406 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.309736 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:17.309758 31981 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.576s	user 0.000s	sys 0.002s
I20260809 23:20:17.311143 32072 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.257333 31944 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.278496 24542 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:17.426030 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.250778 31902 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.294679 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.321516 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.317292 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.431391 24539 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260809 23:20:17.292574 31982 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:17.322907 24316 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:17.431602 24534 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:17.230319 31470 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.325850 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.327628 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.408610 31763 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:17.331434 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.336258 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:17.297068 32099 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260809 23:20:17.437451 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:17.308367 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.289456 31884 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:17.285192 31727 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:17.340937 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.440845 31944 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.441509 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.441797 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.440858 31727 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:17.442574 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:17.443492 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.443296 31727 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.347955 32057 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.351341 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.446216 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.351188 31765 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.447990 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2729) took 2831 ms. Trace:
I20260809 23:20:17.448333 24543 rpcz_store.cc:276] 0809 23:20:14.616602 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.616791 (+   189us) service_pool.cc:227] Handling call
0809 23:20:14.639243 (+ 22452us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:14.639527 (+   284us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:14.639567 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:14.657294 (+ 17727us) log.cc:842] Serialized 26 byte log entry
0809 23:20:15.154361 (+497067us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:15.154399 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.155535 (+  1136us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:17.308999 (+2153464us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:17.447651 (+138652us) raft_consensus.cc:1671] finished
0809 23:20:17.447674 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.447912 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10968320,"thread_start_us":474224,"threads_started":1,"wal-append.queue_time_us":965178}
I20260809 23:20:17.449323 31765 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.378260 31970 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:17.450381 31970 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.451771 31970 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.452975 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.354509 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.454876 24545 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:17.455166 32111 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260809 23:20:17.324188 31783 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.323065 31774 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:17.456480 32109 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:20:17.456430 31774 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.321297 31967 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260809 23:20:17.459221 31967 consensus_queue.cc:1225] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:17.459606 31931 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.409324 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:17.460861 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51622.3us
I20260809 23:20:17.461880 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.462049 31470 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.416602 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.413311 31845 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260809 23:20:17.463346 24428 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:17.464205 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.465190 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.466269 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.418505 24540 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:17.417996 24309 rpcz_store.cc:276] 0809 23:20:15.562807 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.692952 (+130145us) service_pool.cc:227] Handling call
0809 23:20:15.744627 (+ 51675us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:15.862555 (+117928us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:15.862595 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:15.880453 (+ 17858us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:15.880536 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.308081 (+1427545us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54003c23d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:17.308138 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.308258 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265910656}
I20260809 23:20:17.458779 31774 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.418056 25366 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:17.391608 32040 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.424067 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:17.468616 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.469236 25369 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.470010 31931 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:17.469252 25366 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:17.471905 31763 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:17.474617 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2733) took 2557 ms. Trace:
W20260809 23:20:17.404665 32080 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:20:17.431488 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.431092 25363 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:17.405735 31948 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:17.477859 24542 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:17.293720 32078 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.439111 31361 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.439414 24534 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:20:17.503938 24429 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:17.439236 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:17.439795 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 211567us
W20260809 23:20:17.440088 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112578us
I20260809 23:20:17.440467 31982 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.441915 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.329267 31795 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.421136 31300 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260809 23:20:17.344937 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.346652 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:17.447542 32057 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.302367 31968 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:17.459708 31967 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:17.408399 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.475595 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.527319 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.475728 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.528111 24312 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.528824 24423 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:17.428778 32039 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.652s	user 0.009s	sys 0.003s
I20260809 23:20:17.479403 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:17.480590 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.535881 24316 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:17.536911 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.537598 24316 raft_consensus.cc:2362] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13. Re-sending same reply.
I20260809 23:20:17.481094 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:17.538376 25366 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.538841 25373 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:17.478760 25372 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:17.502892 24317 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:17.507975 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.478746 24535 rpcz_store.cc:276] 0809 23:20:14.917556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.917700 (+   144us) service_pool.cc:227] Handling call
0809 23:20:14.961359 (+ 43659us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:14.961569 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:15.326199 (+364630us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.748495 (+1422296us) log.cc:842] Serialized 14 byte log entry
0809 23:20:17.376318 (+627823us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:17.473983 (+ 97665us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b18004f2700. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:20:17.474189 (+   206us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:17.474248 (+    59us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:17.474316 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.474416 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.474551 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1105922,"prepare.queue_time_us":1754792,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":97,"raft.queue_time_us":250014,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":66224,"spinlock_wait_cycles":7510016,"thread_start_us":821768,"threads_started":3,"wal-append.queue_time_us":349657}
I20260809 23:20:17.510442 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.431700 32034 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.519266 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.569005 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.550882 25366 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:17.569789 24317 raft_consensus.cc:2362] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11. Re-sending same reply.
I20260809 23:20:17.436316 31984 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.523736 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.576081 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:17.577134 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.578220 24423 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:17.477276 31851 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.579222 25371 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:17.508193 32078 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.462785 31470 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.522856 24544 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:20:17.533166 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 186605us
I20260809 23:20:17.533433 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:17.544996 24323 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:17.436465 32048 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:17.511459 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.570868 24429 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:17.573088 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.536352 31967 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:17.573992 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66094.6us
I20260809 23:20:17.474193 31763 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:17.574749 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55562.7us
I20260809 23:20:17.583923 32072 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:17.584120 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.584340 32034 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:17.704661 24317 raft_consensus.cc:2362] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13. Re-sending same reply.
I20260809 23:20:17.596900 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.600374 31948 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:17.540266 24498 connection.cc:232] RPC call timeout handler was delayed by -3.1080632179509848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
I20260809 23:20:17.510036 31361 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.580049 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:17.704265 32031 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260809 23:20:17.518111 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.643924 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.648775 24312 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:20:17.719015 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75227.8us
W20260809 23:20:17.719318 24498 connection.cc:232] RPC call timeout handler was delayed by -2.2782737261150032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
W20260809 23:20:17.719939 24498 connection.cc:232] RPC call timeout handler was delayed by -1.108273726189509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32420000000000004s for a response
I20260809 23:20:17.720651 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:17.689180 25369 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:17.522454 31884 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.535969 31968 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:17.714409 25368 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:17.694789 24314 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:17.487938 31579 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:17.700054 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 177663us
I20260809 23:20:17.697834 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:17.729396 24544 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:17.701908 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.731572 31968 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.732869 31968 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:17.737797 32020 consensus_queue.cc:1159] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 11 status { last_received { term: 8 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:20:17.739076 31361 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.740985 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.741210 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.702605 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.578308 32112 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:17.703567 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.743211 24316 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:17.517946 31931 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.172s
I20260809 23:20:17.705751 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.745868 24323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.707844 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.601574 31975 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.526909 31795 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.602777 31980 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:17.710729 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.717505 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.760617 24317 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.719959 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:17.697786 32119 raft_consensus.cc:2741] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:20:17.771253 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.726547 31774 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.772006 24312 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:17.749818 31727 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:17.772753 31727 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:17.773413 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:17.773702 31727 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.776317 32057 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.779024 31765 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:17.779618 31765 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.780092 31765 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.783231 32122 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:17.740610 31984 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:17.783783 32122 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.784379 31984 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.731609 32048 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:17.793344 32038 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.585s	user 0.006s	sys 0.004s
I20260809 23:20:17.794062 30919 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.793839 32038 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.795504 32098 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:17.796005 31805 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.020s	user 0.013s	sys 0.000s
I20260809 23:20:17.796479 31805 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:17.798084 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2472) took 2289 ms. Trace:
I20260809 23:20:17.745312 32112 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.798902 32112 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.798460 24424 rpcz_store.cc:276] 0809 23:20:15.508512 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.508677 (+   165us) service_pool.cc:227] Handling call
0809 23:20:15.555237 (+ 46560us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:15.721829 (+166592us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:16.489775 (+767946us) log.cc:842] Serialized 14 byte log entry
0809 23:20:16.627916 (+138141us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:16.627971 (+    55us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:16.628016 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.797817 (+1169801us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:17.797879 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.798002 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126830,"prepare.queue_time_us":24084,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":120920,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":7712,"spinlock_wait_cycles":382396800,"thread_start_us":768615,"threads_started":2,"wal-append.queue_time_us":171020}
I20260809 23:20:17.773473 24535 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260809 23:20:17.801924 24424 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:17.740368 31884 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.803541 24314 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:17.740659 31948 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.755748 32072 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:17.805415 32121 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.456s	user 0.003s	sys 0.004s
I20260809 23:20:17.807049 31795 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.747159 31783 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.809065 31783 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.809808 24313 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:17.812847 31793 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.003s
I20260809 23:20:17.756062 24323 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:17.770396 32123 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.783401 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1206) took 2222 ms. Trace:
I20260809 23:20:17.840054 24321 rpcz_store.cc:276] 0809 23:20:15.561426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:15.563661 (+  2235us) service_pool.cc:227] Handling call
0809 23:20:15.563959 (+   298us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:16.767561 (+1203602us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.783139 (+1015578us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:17.783331 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2766589824}
I20260809 23:20:17.803184 24309 raft_consensus.cc:2354] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:17.751494 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.817080 24435 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:17.817548 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.842232 24314 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:20:17.755280 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47531.4us
I20260809 23:20:17.842916 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:17.819933 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.606801 31961 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.845862 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.843986 24316 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
W20260809 23:20:17.763511 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43661.8us
I20260809 23:20:17.846740 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:17.847481 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:17.745435 32034 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:17.832577 24544 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:17.837414 24422 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:17.838464 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.749370 31845 raft_consensus.cc:2741] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260809 23:20:17.846549 24312 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:20:17.838591 31955 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.000s	sys 0.025s
I20260809 23:20:17.848176 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.862632 31021 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:17.838315 31793 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.856508 24317 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:20:17.849006 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97577.9us
I20260809 23:20:17.912060 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.859709 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:17.848575 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.002s	sys 0.014s
I20260809 23:20:17.826951 32098 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.839190 31764 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:17.866365 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:17.898111 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50065.3us
I20260809 23:20:17.920965 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.921377 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.822855 24429 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260809 23:20:17.896152 32107 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.531s	user 0.006s	sys 0.004s
I20260809 23:20:17.846307 31783 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.943174 31783 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.944866 32057 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:17.945305 31783 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.945698 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:17.948055 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:17.948829 24317 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:17.917616 25370 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:17.917665 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:17.845880 31795 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:17.950268 24427 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:17.916409 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.002s	sys 0.014s
I20260809 23:20:17.919287 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2884) took 1059 ms. Trace:
I20260809 23:20:17.953027 24541 rpcz_store.cc:276] 0809 23:20:16.860058 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:16.884827 (+ 24769us) service_pool.cc:227] Handling call
0809 23:20:16.885105 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:17.232810 (+347705us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.910338 (+677528us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:17.919220 (+  8882us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":746847488}
I20260809 23:20:17.857864 32034 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.898182 31021 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.951887 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:17.923961 24266 leader_election.cc:398] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:17.968216 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1123) took 1483 ms. Trace:
I20260809 23:20:17.968530 24266 leader_election.cc:401] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:17.924685 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:17.926020 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66403us
W20260809 23:20:17.969367 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45598.9us
I20260809 23:20:17.969753 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:17.921599 24542 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:17.970376 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:17.927811 31774 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:17.927614 31902 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260809 23:20:17.929342 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62963.9us
I20260809 23:20:17.933014 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.920346 32065 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:17.917053 31361 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:17.939210 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:17.920289 32119 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:17.983925 24545 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:17.980722 24534 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:17.941742 31793 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:17.942338 25375 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:17.900298 31955 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.000s	sys 0.028s
W20260809 23:20:17.900421 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55696.9us
I20260809 23:20:17.926868 31798 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:17.971762 24542 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:17.895679 31884 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.961566 31974 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:18.007232 32122 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:17.840869 32032 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.009327 32032 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:17.975452 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.015851 24311 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:18.017575 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.980031 31795 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.967674 31021 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:18.024232 31795 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:17.972891 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48299.2us
I20260809 23:20:17.968580 24546 rpcz_store.cc:276] 0809 23:20:16.484943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:16.641948 (+157005us) service_pool.cc:227] Handling call
0809 23:20:16.667454 (+ 25506us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:16.778328 (+110874us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:16.778360 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:16.778403 (+    43us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:16.778438 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:17.968000 (+1189562us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:17.968050 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:17.968165 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254530816}
I20260809 23:20:17.914091 32068 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:18.027556 32068 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.027938 32068 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.917795 31851 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:18.028573 31851 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:17.981256 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.933665 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:18.031378 24539 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:17.980626 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:18.032553 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52018.8us
I20260809 23:20:18.033757 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.034461 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.035921 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1125) took 1221 ms. Trace:
I20260809 23:20:18.036139 24546 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:18.036310 24540 rpcz_store.cc:276] 0809 23:20:16.814506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:16.859047 (+ 44541us) service_pool.cc:227] Handling call
0809 23:20:16.913572 (+ 54525us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:17.467398 (+553826us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:18.035730 (+568332us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b5400147654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:18.035868 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1160617216}
W20260809 23:20:18.041957 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66617us
I20260809 23:20:18.042801 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:18.043183 32057 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:18.043644 24542 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 10: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:18.044946 31760 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:18.046650 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.047348 25365 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:18.047634 31851 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.047693 32138 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:18.048483 32138 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:18.034873 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.051288 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.051887 24311 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:18.052366 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.054373 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:18.055020 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.055361 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.056635 24546 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:18.057581 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.993731 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:18.058624 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:18.059832 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66202.4us
I20260809 23:20:18.061002 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:17.972828 32034 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:17.987113 31361 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:17.989290 31955 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:18.027854 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.976938 31783 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.990348 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:17.980211 32095 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:17.977246 31862 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:17.976773 31661 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:18.109232 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47270.1us
I20260809 23:20:17.989082 31984 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.062999 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.064729 24316 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:18.065233 24323 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11.
I20260809 23:20:18.057008 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:17.977166 32098 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:18.030572 24541 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:18.071722 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.072261 31973 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:18.075380 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:20:18.075685 31361 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:18.070451 31787 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260809 23:20:18.065678 31805 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:18.077839 32086 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.962s	user 0.009s	sys 0.002s
I20260809 23:20:18.079546 31955 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.080242 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.068176 31765 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.081601 31974 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:18.081619 25373 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:18.082108 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54321.4us
I20260809 23:20:18.171031 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:18.172320 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.082619 31681 raft_consensus.cc:2741] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260809 23:20:18.084221 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:18.084424 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:18.085179 32007 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.191s	user 0.009s	sys 0.004s
I20260809 23:20:18.084939 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.174337 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:18.176652 24546 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:18.085389 31817 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:18.178887 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:18.085186 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:18.182868 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:18.085772 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:18.087649 31764 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:18.089373 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.089849 31021 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.079481 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.095281 31774 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:18.097313 24545 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260809 23:20:17.983237 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 262688us
I20260809 23:20:18.101426 24544 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:18.001437 24317 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:18.104635 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.111771 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.114889 25371 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:18.195653 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:18.195868 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:18.121740 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.067257 31884 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:17.926941 32115 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.224s
I20260809 23:20:18.198939 25371 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.13-12.13]
W20260809 23:20:18.199440 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87764.8us
I20260809 23:20:18.127733 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.126585 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:18.101696 32072 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:18.204002 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:18.077260 24309 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:18.112412 31970 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.660s	user 0.010s	sys 0.002s
I20260809 23:20:18.132016 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.141731 32024 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:17.951102 32107 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.114737 31968 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:18.155918 32047 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:18.157006 31973 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:18.159519 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.154570 31798 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:18.164956 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.103113 32095 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:18.191740 25373 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:18.187882 25370 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:18.165684 32086 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:18.193038 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:17.846392 32123 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.205053 24313 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:18.190296 31931 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:18.208330 24314 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:18.236184 25365 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:18.269903 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76937.1us
I20260809 23:20:18.142868 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:18.200428 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.255084 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2934) took 3769 ms. Trace:
I20260809 23:20:18.204198 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.215252 32115 consensus_queue.cc:1225] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:18.215641 32119 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:18.178701 31817 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.179266 32098 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:18.193243 32133 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260809 23:20:18.302094 31817 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:18.191359 31764 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:18.295750 31681 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:18.220080 25369 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:18.217538 24323 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:18.308528 32098 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.308992 32098 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:18.308627 31967 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.310083 31967 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:18.206297 32056 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:18.272573 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.312119 24541 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:18.250458 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:18.271791 25366 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:18.273669 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.266132 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:18.297438 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97105.3us
I20260809 23:20:18.314620 24378 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11.
I20260809 23:20:18.315040 24378 leader_election.cc:401] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260809 23:20:18.271625 24425 rpcz_store.cc:276] 0809 23:20:14.485452 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:14.485605 (+   153us) service_pool.cc:227] Handling call
0809 23:20:14.507013 (+ 21408us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:20:15.005301 (+498288us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:15.005335 (+    34us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:20:15.053845 (+ 48510us) log.cc:842] Serialized 78 byte log entry
0809 23:20:15.451221 (+397376us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:15.542130 (+ 90909us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.568984 (+ 26854us) log.cc:842] Serialized 14 byte log entry
0809 23:20:15.573508 (+  4524us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:15.573589 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:17.769334 (+2195745us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b1800511300. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:18.084462 (+315128us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:18.254774 (+170312us) raft_consensus.cc:1671] finished
0809 23:20:18.254798 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:18.255018 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95014,"prepare.queue_time_us":204227,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":7324,"raft.queue_time_us":362120,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":134182,"spinlock_wait_cycles":80914048,"thread_start_us":848513,"threads_started":2,"wal-append.queue_time_us":686064}
I20260809 23:20:18.201749 31783 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:18.318956 25363 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:20:18.321736 24546 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:18.244393 31021 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:18.268834 32078 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.000s
I20260809 23:20:18.334653 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.335016 24545 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260809 23:20:18.218961 32072 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.324918 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:18.332913 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:18.338095 24312 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:18.311535 32112 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.512s	user 0.006s	sys 0.004s
I20260809 23:20:18.248679 32052 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.271883 32095 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:18.250406 32039 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:18.270979 32047 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:18.346711 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 364855us
I20260809 23:20:18.339058 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.324395 24313 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260809 23:20:18.318572 31842 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.009s	sys 0.036s
I20260809 23:20:18.259766 31862 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:18.263983 31765 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.269747 31931 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.218307 31961 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.254634 32038 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.222743 31805 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.115s
I20260809 23:20:18.344280 32072 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:18.236114 32024 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.109s
I20260809 23:20:18.335647 31681 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:18.323330 32056 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:18.366750 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.341157 32119 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:18.294658 31968 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.341089 31688 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:18.344835 31783 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:18.350811 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84755.6us
W20260809 23:20:18.365428 31842 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.009s	sys 0.037s
I20260809 23:20:18.347239 32118 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:18.349954 31579 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:18.384431 31961 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:18.387799 24316 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:18.251262 32068 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.392613 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:18.358209 32039 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.365620 32052 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:18.372759 31974 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.033s
I20260809 23:20:18.337081 32078 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.345191 32112 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:18.394896 31948 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.590s	user 0.003s	sys 0.007s
I20260809 23:20:18.264353 32034 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.412407 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.410156 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.418658 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.369397 31931 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:18.395784 31765 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260809 23:20:18.369478 31339 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.029s
I20260809 23:20:18.401754 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:18.502163 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100512us
I20260809 23:20:18.387714 31862 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260809 23:20:18.427217 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.503638 25369 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.504312 25369 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:18.415330 31361 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.428275 31688 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:18.267465 24167 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 7 to 9, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 9 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:18.434981 32123 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.435524 32056 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.438970 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:18.439739 31968 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.381330 31973 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.402736 32115 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:18.444639 31817 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.447410 31783 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.416270 31793 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260809 23:20:18.559533 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:18.373113 31661 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260809 23:20:18.455366 31851 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:18.456399 31579 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:18.387948 31798 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260809 23:20:18.457424 32039 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:18.420331 31961 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.459110 32118 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:18.459065 32068 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.482901 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.484411 24320 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.485424 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.488569 24422 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260809 23:20:18.489905 32138 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:18.492720 32052 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
W20260809 23:20:18.492942 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74378.5us
I20260809 23:20:18.572484 31851 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:18.493165 24170 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 239) took 35387 ms (client timeout 15000 ms). Trace:
I20260809 23:20:18.493515 31948 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:18.494397 32098 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.495028 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.406584 24312 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:18.431605 32107 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.506788 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.505729 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.508842 24323 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:18.441951 31681 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.516377 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.464612 32020 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260809 23:20:18.530128 25363 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:18.531256 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:18.555733 25369 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 12. (term mismatch)
W20260809 23:20:18.519593 32065 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.000s
I20260809 23:20:18.554893 24425 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260809 23:20:18.638413 24428 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:18.505421 32024 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:18.523576 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:18.560230 24430 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
W20260809 23:20:18.521134 31470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.013s	sys 0.012s
I20260809 23:20:18.525000 32115 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:18.588311 24314 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:18.590899 25373 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:18.591853 25366 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:18.596259 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:18.644646 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48476.5us
I20260809 23:20:18.597987 24320 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:18.600458 25365 raft_consensus.cc:2354] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:18.600734 24317 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:18.615141 25369 pending_rounds.cc:76] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 11
I20260809 23:20:18.615197 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.646732 25369 pending_rounds.cc:97] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 11.12
I20260809 23:20:18.647167 25369 raft_consensus.cc:2866] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:18.604449 25370 raft_consensus.cc:2362] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14. Re-sending same reply.
I20260809 23:20:18.620227 24541 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:18.649549 25363 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260809 23:20:18.652583 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:18.655316 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.660135 24378 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:18.593060 31361 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:18.659230 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 135770us
I20260809 23:20:18.659606 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.713389 25363 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:18.714087 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.706907 24378 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:18.665048 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.598639 31948 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.649281 32115 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:18.727478 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.659109 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:18.709988 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69691.4us
I20260809 23:20:18.710892 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:18.642880 31470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.013s	sys 0.013s
I20260809 23:20:18.598145 31851 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.717427 25366 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:18.717983 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 215029us
W20260809 23:20:18.719892 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59920.1us
I20260809 23:20:18.600062 32098 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.715916 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.595206 32052 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:18.722946 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:18.734958 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.016s	sys 0.015s
I20260809 23:20:18.735561 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:18.757699 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.016s	sys 0.015s
I20260809 23:20:18.757893 25365 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:18.758035 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 539510us
I20260809 23:20:18.748660 25373 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 9.
I20260809 23:20:18.656018 31661 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:18.649467 32065 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:18.657315 31842 raft_consensus.cc:2741] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260809 23:20:18.745538 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:18.785321 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.729481 31765 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:18.747555 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.747633 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.715852 32107 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:18.659197 32024 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:18.753057 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.596907 31764 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.763842 25363 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:18.810217 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:18.767550 32126 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:18.593261 24167 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 8 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:18.777904 32049 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:18.778745 32149 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:18.823207 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.723336 31339 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260809 23:20:18.731673 31948 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:18.834031 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:18.789407 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.789896 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2432) took 1365 ms. Trace:
I20260809 23:20:18.789915 24316 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:18.659112 31688 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:18.732445 31805 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 13
I20260809 23:20:18.596686 32072 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.854378 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.855039 24544 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:18.855343 32024 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:18.856333 32072 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.794426 25369 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:18.859732 31764 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:18.860642 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.861573 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:18.863157 31817 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.863663 32052 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.822705 32049 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:18.800406 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.014s	sys 0.029s
I20260809 23:20:18.864923 32049 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:18.865097 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.014s	sys 0.029s
W20260809 23:20:18.865641 32146 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:20:18.864725 32052 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:18.803293 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.804575 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:18.716171 32048 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
W20260809 23:20:18.818168 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58311.8us
I20260809 23:20:18.778913 31793 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.020s
W20260809 23:20:18.824189 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.027s	sys 0.034s
I20260809 23:20:18.817843 32126 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:18.830075 32078 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.756023 32112 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.793733 32082 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:18.838016 24320 rpcz_store.cc:276] 0809 23:20:17.424932 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:17.425081 (+   149us) service_pool.cc:227] Handling call
0809 23:20:18.789717 (+1364636us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:18.789843 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2369965312}
I20260809 23:20:18.799046 31986 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:18.846804 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:18.850548 31961 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:18.800793 32154 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:18.807068 32107 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:18.851996 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:19.013638 24317 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:18.868294 24321 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:18.876737 24544 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:18.802089 25368 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:18.884925 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:18.826321 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:18.885213 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.028s	sys 0.034s
I20260809 23:20:18.886472 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:18.878518 25370 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:19.017417 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:18.877475 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:18.891151 25366 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:20:18.744858 31579 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:18.806073 31727 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.017s	sys 0.044s
I20260809 23:20:18.867767 31688 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:18.935963 32048 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.003994 24545 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:18.827775 32084 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:18.896167 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:18.927309 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:19.004666 32082 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:19.016312 32107 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:19.006016 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45607.9us
I20260809 23:20:18.974007 24311 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:19.019003 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:18.845060 32034 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:18.823105 32149 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:19.018141 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 133323us
I20260809 23:20:19.067787 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2881) took 1015 ms. Trace:
I20260809 23:20:18.867673 32083 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:19.071005 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:18.902308 31968 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:19.024576 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.075888 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:19.025192 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:19.031167 31688 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:19.003752 31986 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:19.044664 32020 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:19.079943 31790 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:19.075052 32119 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.668s	user 0.011s	sys 0.000s
W20260809 23:20:19.092353 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.008s	sys 0.064s
W20260809 23:20:19.092780 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.008s	sys 0.064s
I20260809 23:20:19.075225 32084 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:18.936489 31832 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:18.898517 31970 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:19.111946 31970 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.114481 31970 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:19.049226 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 153143us
I20260809 23:20:18.835050 32047 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.115788 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:19.116415 32047 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.053366 25366 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260809 23:20:18.598294 24170 rpcz_store.cc:269] 0809 23:19:43.134135 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:43.134344 (+   209us) service_pool.cc:227] Handling call
0809 23:20:18.492843 (+35358499us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:20:12.228046 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:20:12.234046 (+  6000us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:20:12.234086 (+    40us) write_op.cc:435] Acquired schema lock
0809 23:20:12.234102 (+    16us) tablet.cc:660] Decoding operations
0809 23:20:12.235119 (+  1017us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:20:12.235182 (+    63us) write_op.cc:641] Partition lock acquired for write op
0809 23:20:12.235214 (+    32us) tablet.cc:683] Acquiring locks for 10 operations
0809 23:20:12.236012 (+   798us) tablet.cc:699] Row locks acquired
0809 23:20:12.236033 (+    21us) write_op.cc:260] PREPARE: finished
0809 23:20:12.236174 (+   141us) write_op.cc:270] Start()
0809 23:20:12.236285 (+   111us) write_op.cc:276] Timestamp: P: 1786317612236124 usec, L: 0
0809 23:20:12.236304 (+    19us) op_driver.cc:331] REPLICATION: starting
0809 23:20:12.236914 (+   610us) log.cc:842] Serialized 3950 byte log entry
0809 23:20:13.109456 (+872542us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:13.675423 (+565967us) op_driver.cc:444] REPLICATION: finished
0809 23:20:14.437596 (+762173us) write_op.cc:301] APPLY: starting
0809 23:20:14.437741 (+   145us) tablet.cc:1365] starting BulkCheckPresence
0809 23:20:14.847359 (+409618us) tablet.cc:1368] finished BulkCheckPresence
0809 23:20:14.847403 (+    44us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:20:14.969321 (+121918us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:20:14.969961 (+   640us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0809 23:20:14.970016 (+    55us) write_op.cc:312] APPLY: finished
0809 23:20:15.145027 (+175011us) log.cc:842] Serialized 97 byte log entry
0809 23:20:15.831987 (+686960us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:15.914482 (+ 82495us) write_op.cc:454] Released schema lock
0809 23:20:15.931617 (+ 17135us) write_op.cc:341] FINISH: Updating metrics
0809 23:20:16.058826 (+127209us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:16.069218 (+ 10392us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":38190,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":761802,"apply.run_cpu_time_us":39685,"apply.run_wall_time_us":1658423,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45887,"num_ops":10,"prepare.queue_time_us":484506,"prepare.run_cpu_time_us":13836,"prepare.run_wall_time_us":326118,"raft.queue_time_us":541371,"raft.run_cpu_time_us":56028,"raft.run_wall_time_us":916222,"replication_time_us":1438995,"spinlock_wait_cycles":8868736,"thread_start_us":1961199,"threads_started":5,"wal-append.queue_time_us":1348126,"wal-append.run_cpu_time_us":16171,"wal-append.run_wall_time_us":2558249}]]}
I20260809 23:20:18.795524 32118 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:19.024201 31727 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.017s	sys 0.045s
I20260809 23:20:19.019541 31984 raft_consensus.cc:2741] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260809 23:20:19.068169 24311 rpcz_store.cc:276] 0809 23:20:18.052589 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.052812 (+   223us) service_pool.cc:227] Handling call
0809 23:20:18.074777 (+ 21965us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:18.938616 (+863839us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54001ee220. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:19.067478 (+128862us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.067730 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1525726464}
I20260809 23:20:19.034212 24317 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:19.025401 31021 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.008s	sys 0.031s
I20260809 23:20:19.027434 32078 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:18.897562 24312 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260809 23:20:19.076072 24544 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:18.900602 31948 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.016062 32154 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:18.937254 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.156629 24540 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:19.103884 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.009s	sys 0.036s
W20260809 23:20:19.157080 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.009s	sys 0.036s
W20260809 23:20:19.157433 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 764352us
I20260809 23:20:19.074461 31968 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:19.116559 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:19.083778 31986 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:19.116884 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.118798 31832 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:19.083487 31661 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:19.167131 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 280714us
W20260809 23:20:19.168251 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51771.6us
I20260809 23:20:19.168344 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:19.169207 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:19.088928 32112 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:19.110450 32145 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.454s	user 0.000s	sys 0.001s
I20260809 23:20:19.092533 32119 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11.
W20260809 23:20:19.110263 32095 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.738s	user 0.007s	sys 0.003s
I20260809 23:20:19.154695 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:19.193229 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:19.194130 25373 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:19.062335 32048 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.163867 24545 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:19.073211 32126 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.085798 31790 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:19.121428 31793 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:19.152709 32147 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.397s	user 0.000s	sys 0.001s
I20260809 23:20:19.086719 32082 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.163214 32072 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.203234 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:19.148427 31021 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.008s	sys 0.032s
I20260809 23:20:19.148775 31300 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:19.179873 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:19.089756 32020 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.148329 32138 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.154243 31970 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.190970 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.149209 31817 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260809 23:20:19.194072 25366 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:19.159799 32032 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:19.192456 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.121615 32149 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.198726 32123 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
I20260809 23:20:19.379418 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:19.192350 31842 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.188038 31984 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:19.305886 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:19.188282 32078 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:19.383469 24544 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:19.388391 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 208619us
I20260809 23:20:19.309011 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:19.394723 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:19.396247 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.022s	sys 0.025s
W20260809 23:20:19.396821 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.023s	sys 0.025s
I20260809 23:20:19.397284 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:19.398079 24321 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:19.394227 24425 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:19.258978 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2954) took 1467 ms. Trace:
I20260809 23:20:19.402585 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:19.403343 25366 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:19.402187 24424 rpcz_store.cc:276] 0809 23:20:17.791301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:17.802661 (+ 11360us) service_pool.cc:227] Handling call
0809 23:20:17.820333 (+ 17672us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:17.820659 (+   326us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:17.820688 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.101513 (+280825us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:18.113520 (+ 12007us) log.cc:842] Serialized 26 byte log entry
0809 23:20:18.865646 (+752126us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:18.865684 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:18.865748 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.258571 (+392823us) raft_consensus.cc:1671] finished
0809 23:20:19.258592 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.258856 (+   264us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13786,"raft.queue_time_us":81,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":15954,"spinlock_wait_cycles":7335168,"thread_start_us":751709,"threads_started":1,"wal-append.queue_time_us":1050057}
I20260809 23:20:19.174110 32152 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:19.381336 32161 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.326s	user 0.001s	sys 0.000s
W20260809 23:20:19.238617 32056 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.005s	sys 0.042s
W20260809 23:20:19.202584 31973 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.038s	sys 0.046s
I20260809 23:20:19.406841 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:19.408072 32049 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.543s	user 0.010s	sys 0.001s
I20260809 23:20:19.417376 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:19.213907 32153 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:19.390548 32147 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260809 23:20:19.429862 31955 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.009s	sys 0.019s
I20260809 23:20:19.406649 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:19.428618 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:19.429536 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:19.396077 31783 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.020s	sys 0.016s
I20260809 23:20:19.430547 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.434520 25373 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:19.408980 31339 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.010s	sys 0.011s
I20260809 23:20:19.485031 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 10.
I20260809 23:20:19.489161 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:19.243005 31026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.005s	sys 0.034s
I20260809 23:20:19.390264 31984 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:19.435935 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.000s	sys 0.026s
I20260809 23:20:19.199506 32112 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.188628 31661 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.490123 31948 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.235841 32084 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.518522 32107 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.520287 25370 raft_consensus.cc:2354] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
W20260809 23:20:19.494246 31026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.009s	sys 0.043s
I20260809 23:20:19.389773 32034 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:20:19.433435 32056 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.005s	sys 0.054s
I20260809 23:20:19.407918 31300 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:19.199049 24167 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 8 to 9, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 9 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:19.531524 31300 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.532480 24167 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 6 to 8, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:19.535118 25372 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:19.537036 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1261) took 1476 ms. Trace:
I20260809 23:20:19.537726 31986 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.538584 32048 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.476224 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:19.540850 32159 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20260809 23:20:19.542609 32162 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
W20260809 23:20:19.434043 31973 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.044s	sys 0.046s
I20260809 23:20:19.485749 24435 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:19.486704 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:19.555773 24424 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:19.436400 32024 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.013s	sys 0.018s
I20260809 23:20:19.496202 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:19.559096 25365 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:19.497139 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:19.237465 32072 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:19.470667 31857 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.004s	sys 0.033s
I20260809 23:20:19.201550 32095 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:19.476128 31931 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.975s	user 0.009s	sys 0.004s
W20260809 23:20:19.562984 31857 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.005s	sys 0.033s
W20260809 23:20:19.558709 32024 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.013s	sys 0.027s
I20260809 23:20:19.575372 31931 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.576522 24435 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:19.577643 31931 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:19.500622 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.001s	sys 0.026s
I20260809 23:20:19.579150 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2901) took 1283 ms. Trace:
I20260809 23:20:19.579541 24539 rpcz_store.cc:276] 0809 23:20:18.295404 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.295592 (+   188us) service_pool.cc:227] Handling call
0809 23:20:18.305340 (+  9748us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.305697 (+   357us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.305726 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.306280 (+   554us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.016486 (+710206us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:19.016524 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.016595 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.578950 (+562355us) raft_consensus.cc:1671] finished
0809 23:20:19.578972 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.579095 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2358,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":709822,"threads_started":1,"wal-append.queue_time_us":908060}
I20260809 23:20:19.581466 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.567680 31973 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.583695 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2905) took 1121 ms. Trace:
I20260809 23:20:19.583588 31973 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.584052 24542 rpcz_store.cc:276] 0809 23:20:18.461980 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.462263 (+   283us) service_pool.cc:227] Handling call
0809 23:20:18.462681 (+   418us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.463293 (+   612us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.463332 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.485229 (+ 21897us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.106122 (+620893us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:19.106193 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.106291 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.583485 (+477194us) raft_consensus.cc:1671] finished
0809 23:20:19.583505 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.583632 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":532,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10422,"thread_start_us":584767,"threads_started":1,"wal-append.queue_time_us":981178}
I20260809 23:20:19.586160 31973 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:19.489260 31339 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.010s	sys 0.011s
W20260809 23:20:19.484143 31783 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.021s	sys 0.016s
W20260809 23:20:19.478205 31955 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.009s	sys 0.019s
I20260809 23:20:19.587710 31955 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.587931 24309 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:19.588064 25373 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:19.467903 32153 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:19.588671 32153 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:19.588222 31955 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.392086 32138 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.590833 31955 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.591480 32020 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.592607 32138 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:19.592288 31783 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:19.499853 32098 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.593674 32098 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.427120 25366 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:19.394573 31970 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.554483 32154 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.554669 32082 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:19.597317 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2763) took 1684 ms. Trace:
I20260809 23:20:19.597661 31970 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.436779 31884 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260809 23:20:19.523284 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:19.376672 31832 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.600214 32154 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:19.600589 32154 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:19.525385 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.601444 32154 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.290889 31968 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.395545 31842 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.554782 32163 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:19.603628 24544 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
W20260809 23:20:19.477115 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.020s	sys 0.020s
W20260809 23:20:19.612251 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.020s	sys 0.020s
I20260809 23:20:19.537381 25372 rpcz_store.cc:276] 0809 23:20:18.060794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.066682 (+  5888us) service_pool.cc:227] Handling call
0809 23:20:18.221430 (+154748us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.535543 (+1314113us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.536059 (+   516us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400717854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:19.536850 (+   791us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400717850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:19.536982 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3367593216}
W20260809 23:20:19.539924 32170 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260809 23:20:19.616011 24534 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:19.565229 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.013s	sys 0.026s
I20260809 23:20:19.565223 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:19.631193 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.014s	sys 0.026s
I20260809 23:20:19.631804 25370 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:19.569212 32172 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260809 23:20:19.476061 32049 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.569475 32107 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.642946 31842 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.643932 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1224) took 1126 ms. Trace:
I20260809 23:20:19.644011 32098 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.582680 31688 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.644373 25375 rpcz_store.cc:276] 0809 23:20:18.517268 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.517411 (+   143us) service_pool.cc:227] Handling call
0809 23:20:18.552142 (+ 34731us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.557293 (+  5151us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.557321 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.557722 (+   401us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.002395 (+444673us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:19.002453 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.002547 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.643727 (+641180us) raft_consensus.cc:1671] finished
0809 23:20:19.643747 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.643874 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15770,"raft.queue_time_us":3016,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":20003,"thread_start_us":435672,"threads_started":1,"wal-append.queue_time_us":695768}
I20260809 23:20:19.645848 24422 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:19.646040 32095 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.571714 31986 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.623056 32163 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.550830 24167 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 6 to 10, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:19.649147 32163 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.648975 24167 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 5 to 7, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 7 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:20:19.573743 32115 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.830s	user 0.007s	sys 0.004s
I20260809 23:20:19.651460 32115 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.651360 32049 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:19.652769 31805 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.780s	user 0.009s	sys 0.001s
I20260809 23:20:19.653183 31805 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:19.653631 32115 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.558202 32078 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.219010 31793 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:19.483698 31765 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.682s	user 0.009s	sys 0.001s
I20260809 23:20:19.589195 25373 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:19.428221 32068 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.594588 32112 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.442826 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:19.597679 24543 rpcz_store.cc:276] 0809 23:20:17.913130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:17.914587 (+  1457us) service_pool.cc:227] Handling call
0809 23:20:17.934330 (+ 19743us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:17.934690 (+   360us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:17.935020 (+   330us) log.cc:842] Serialized 14 byte log entry
0809 23:20:18.414649 (+479629us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.475582 (+ 60933us) log.cc:842] Serialized 26 byte log entry
0809 23:20:18.475748 (+   166us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:18.475782 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:18.476740 (+   958us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.597102 (+1120362us) raft_consensus.cc:1671] finished
0809 23:20:19.597129 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.597261 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":231,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":16302,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":39474,"thread_start_us":479208,"threads_started":1,"wal-append.queue_time_us":708920}
I20260809 23:20:19.606248 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:19.391484 31790 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.608297 24542 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260809 23:20:19.662546 32052 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.000s
I20260809 23:20:19.641142 31661 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.616564 31470 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260809 23:20:19.647907 32072 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.654691 24427 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:19.662755 25365 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
W20260809 23:20:19.675676 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.000s	sys 0.054s
I20260809 23:20:19.677333 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:19.682090 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76004.9us
W20260809 23:20:19.699062 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.000s	sys 0.054s
I20260809 23:20:19.702638 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:19.680753 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.030s	sys 0.042s
I20260809 23:20:19.503563 31984 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:19.681366 31765 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:19.686053 32163 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:19.719733 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:19.704088 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.031s	sys 0.042s
W20260809 23:20:19.761831 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48156.1us
I20260809 23:20:19.769419 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:19.690905 31688 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:19.717895 31681 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.024s	sys 0.025s
I20260809 23:20:19.776401 24430 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:19.685268 32068 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.790920 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2891) took 1713 ms. Trace:
I20260809 23:20:19.791554 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:19.791378 24309 rpcz_store.cc:276] 0809 23:20:18.077095 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.213829 (+136734us) service_pool.cc:227] Handling call
0809 23:20:18.214132 (+   303us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.588367 (+1374235us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.790641 (+202274us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:19.790849 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3158951168}
I20260809 23:20:19.792815 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:19.771869 32032 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:19.809420 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.000s	sys 0.037s
W20260809 23:20:19.783500 31681 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.024s	sys 0.025s
I20260809 23:20:19.717316 31783 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260809 23:20:19.782557 31931 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:19.816247 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.000s	sys 0.038s
I20260809 23:20:19.824676 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.776381 32163 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:19.826912 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:19.816632 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.838469 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.840775 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.841151 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:19.796963 32049 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260809 23:20:19.845417 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.007s	sys 0.012s
I20260809 23:20:19.814491 31765 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:19.849799 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.007s	sys 0.012s
I20260809 23:20:19.847934 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:19.851869 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.022s	sys 0.017s
I20260809 23:20:19.853845 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2961) took 1392 ms. Trace:
I20260809 23:20:19.857327 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:19.859612 24427 rpcz_store.cc:276] 0809 23:20:18.461444 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.467686 (+  6242us) service_pool.cc:227] Handling call
0809 23:20:18.468041 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.689586 (+1221545us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.853618 (+164032us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:19.853790 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2728405248}
I20260809 23:20:19.854257 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1138) took 1845 ms. Trace:
I20260809 23:20:19.860843 24534 rpcz_store.cc:276] 0809 23:20:18.008467 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.017291 (+  8824us) service_pool.cc:227] Handling call
0809 23:20:19.854100 (+1836809us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:19.854201 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3674259328}
I20260809 23:20:19.817438 31817 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:19.862120 31817 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.691088 24167 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 12 to 13, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 13 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:19.863606 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2779) took 1029 ms. Trace:
I20260809 23:20:19.863966 24546 rpcz_store.cc:276] 0809 23:20:18.834270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.847678 (+ 13408us) service_pool.cc:227] Handling call
0809 23:20:18.895689 (+ 48011us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.896059 (+   370us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.896092 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.896547 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.631759 (+735212us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:19.631818 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.631912 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.863410 (+231498us) raft_consensus.cc:1671] finished
0809 23:20:19.863430 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.863552 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9932,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":734795,"threads_started":1,"wal-append.queue_time_us":936521}
I20260809 23:20:19.865634 24424 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:19.866237 31765 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:19.867509 31765 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:19.868907 32068 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:19.868507 31765 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.863982 24167 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 7 to 12, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:19.877012 31817 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.873742 24167 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 12, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 12 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:19.860208 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:19.878803 32174 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260809 23:20:19.881374 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.022s	sys 0.018s
I20260809 23:20:19.881687 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2900) took 1092 ms. Trace:
I20260809 23:20:19.882638 24313 rpcz_store.cc:276] 0809 23:20:18.789616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.790748 (+  1132us) service_pool.cc:227] Handling call
0809 23:20:18.816144 (+ 25396us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.841862 (+ 25718us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.841909 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.842475 (+   566us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.612726 (+770251us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:19.612781 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.612868 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.881461 (+268593us) raft_consensus.cc:1671] finished
0809 23:20:19.881488 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.881620 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2224,"raft.queue_time_us":51220,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":24041,"thread_start_us":727850,"threads_started":1,"wal-append.queue_time_us":903423}
I20260809 23:20:19.880594 24167 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 3 to 11. New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:20:19.881953 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:19.888690 24167 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 6 to 10, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:19.896178 31793 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:19.897450 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.833601 32032 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.906190 32032 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:19.907018 32163 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:19.912113 31842 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:19.903539 24167 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 10, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:19.912626 31842 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:19.913101 31026 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:19.915202 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1229) took 1393 ms. Trace:
W20260809 23:20:19.915138 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55018.9us
I20260809 23:20:19.916071 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:19.915678 25369 rpcz_store.cc:276] 0809 23:20:18.521880 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.861813 (+339933us) service_pool.cc:227] Handling call
0809 23:20:18.862208 (+   395us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:18.862624 (+   416us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:18.862662 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:18.870285 (+  7623us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.578742 (+708457us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:19.578779 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.578841 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:19.915000 (+336159us) raft_consensus.cc:1671] finished
0809 23:20:19.915019 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:19.915143 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":704,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":678225,"threads_started":1,"wal-append.queue_time_us":817747}
I20260809 23:20:19.915431 24167 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 5 to 8, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 8 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:19.917013 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:19.918382 24498 leader_election.cc:398] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:19.918910 24498 leader_election.cc:401] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260809 23:20:19.920408 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:19.921823 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:19.924247 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:19.924448 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76603.2us
I20260809 23:20:19.924901 25373 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:19.927899 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:19.845896 31681 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:19.947444 25368 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:19.934549 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.005s	sys 0.010s
W20260809 23:20:19.949548 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.006s	sys 0.010s
W20260809 23:20:19.949888 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 540095us
W20260809 23:20:19.952013 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.012s	sys 0.016s
I20260809 23:20:19.951828 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.952621 24534 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
W20260809 23:20:19.953737 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.013s	sys 0.016s
I20260809 23:20:19.953608 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:19.914760 31842 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:19.954999 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1317) took 1235 ms (client timeout 1551 ms). Trace:
I20260809 23:20:19.955404 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.956158 24546 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:19.955458 25371 rpcz_store.cc:269] 0809 23:20:18.712091 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:18.727157 (+ 15066us) service_pool.cc:227] Handling call
0809 23:20:19.946869 (+1219712us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:19.947050 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2377575680}
W20260809 23:20:19.957957 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.014s	sys 0.010s
W20260809 23:20:19.958264 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.014s	sys 0.010s
I20260809 23:20:19.962713 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:19.963775 24546 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260809 23:20:19.966235 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:19.943007 31021 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.008s	sys 0.015s
I20260809 23:20:19.966953 24546 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:19.955932 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:19.995760 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:20.003535 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:20.004024 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:20.001850 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:20.004613 24316 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:19.997984 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1275) took 1210 ms. Trace:
I20260809 23:20:20.005213 25368 rpcz_store.cc:276] 0809 23:20:18.787311 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.029669 (+242358us) service_pool.cc:227] Handling call
0809 23:20:19.947173 (+917504us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.951507 (+  4334us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:19.997734 (+ 46227us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b5400442ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:19.997916 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2023353728}
I20260809 23:20:19.971160 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:20.012861 24427 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:19.953024 31727 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:20.018257 24424 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:20.018368 24311 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260809 23:20:20.069391 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.011s	sys 0.019s
I20260809 23:20:20.019023 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:20.092851 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.011s	sys 0.019s
I20260809 23:20:20.020126 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:20.093076 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.007s	sys 0.013s
I20260809 23:20:20.003255 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:20.028127 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:20.093715 24312 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:20.051621 24423 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.009s	sys 0.020s
I20260809 23:20:19.957512 32032 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:20.026146 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:20.096307 24423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.009s	sys 0.020s
I20260809 23:20:20.065887 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:20.096895 24323 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:20.067358 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:20.067880 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:20.093042 24534 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:20.099327 24428 raft_consensus.cc:2354] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:20.063050 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:20.099898 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:20.100440 24427 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:20.067204 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:20.087639 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:20.102766 24313 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:20.003325 31842 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:20.088289 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:20.105413 24435 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:20.105556 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:19.953184 31681 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:20.091612 24542 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:20.093596 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 686971us
W20260809 23:20:20.065404 31579 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.030s	sys 0.041s
I20260809 23:20:20.020020 31021 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:20.248195 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.023s	sys 0.051s
W20260809 23:20:20.248867 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.023s	sys 0.051s
W20260809 23:20:19.967901 32123 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.012s	sys 0.005s
I20260809 23:20:20.110658 24422 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:20.263394 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:20.265213 24422 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:19.967311 31798 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:20.285670 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:20.188772 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81574.5us
I20260809 23:20:20.190779 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:20.206879 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:20.287740 24428 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:20.211504 24541 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.005s	sys 0.013s
W20260809 23:20:20.290987 24541 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.006s	sys 0.013s
W20260809 23:20:20.093768 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.007s	sys 0.013s
I20260809 23:20:20.095729 24316 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:20.259835 24323 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:20.292896 24539 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.011s	sys 0.016s
I20260809 23:20:20.269309 24320 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:20.285140 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:20.296262 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:20.255888 31902 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:20.318810 31902 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.091939 32192 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:20.202911 31579 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.031s	sys 0.042s
W20260809 23:20:20.293488 24539 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.011s	sys 0.016s
I20260809 23:20:20.186316 32197 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
W20260809 23:20:20.255604 31300 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.011s	sys 0.007s
I20260809 23:20:20.334438 32166 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:20.098981 32032 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:20.335310 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 145507us
I20260809 23:20:20.408632 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:20.335623 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 271222us
I20260809 23:20:20.406517 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:20.414057 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:20.433048 25370 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.012s	sys 0.021s
W20260809 23:20:20.269793 32118 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.055s	sys 0.086s
W20260809 23:20:20.245708 32153 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.656s	user 0.011s	sys 0.000s
W20260809 23:20:20.262400 32123 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.013s	sys 0.006s
I20260809 23:20:20.331863 24422 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:20.284652 31967 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:20:20.391222 25363 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.013s	sys 0.013s
W20260809 23:20:20.450672 25363 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.013s	sys 0.013s
I20260809 23:20:20.342829 32193 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:20.381284 31790 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.012s	sys 0.014s
I20260809 23:20:20.368412 31793 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:20.345999 24312 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:20.415366 31851 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.014s	sys 0.013s
W20260809 23:20:20.403882 31300 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.011s	sys 0.008s
I20260809 23:20:20.322333 31339 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:20.422039 24424 raft_consensus.cc:2396] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260809 23:20:20.293474 31021 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:20.415633 31931 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.002s	sys 0.027s
W20260809 23:20:20.434921 25370 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.012s	sys 0.021s
I20260809 23:20:20.346886 31661 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260809 23:20:20.449936 31967 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:20.482358 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:20.449631 32118 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.055s	sys 0.086s
W20260809 23:20:20.453348 32183 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.613s	user 0.001s	sys 0.001s
I20260809 23:20:20.432807 32149 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:20.337224 31798 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.407035 32166 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:20.478111 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.008s	sys 0.029s
I20260809 23:20:20.434875 31579 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:20.418546 32032 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:20.452185 32078 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.010s	sys 0.024s
W20260809 23:20:20.465138 31790 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.012s	sys 0.014s
W20260809 23:20:20.483248 31851 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.014s	sys 0.013s
W20260809 23:20:20.477655 32163 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.002s	sys 0.013s
W20260809 23:20:20.505738 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.024s	sys 0.058s
W20260809 23:20:20.611032 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s	user 0.024s	sys 0.059s
I20260809 23:20:20.092464 24167 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 2 to 10. New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:20.511018 24431 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:20.366811 32201 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.143s
I20260809 23:20:20.490345 32126 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:20.507071 31765 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.010s	sys 0.030s
W20260809 23:20:20.505664 31931 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.002s	sys 0.027s
I20260809 23:20:20.559136 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:20:20.565577 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.009s	sys 0.030s
W20260809 23:20:20.780963 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 859148us
I20260809 23:20:20.781697 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:20.479899 32048 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
W20260809 23:20:20.558571 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.006s	sys 0.041s
W20260809 23:20:20.783330 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.007s	sys 0.041s
I20260809 23:20:20.709235 31688 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:20.451001 32197 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.125s
I20260809 23:20:20.455039 32153 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.559711 32175 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:20.587044 32078 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.010s	sys 0.025s
I20260809 23:20:20.494330 31793 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:20.622982 32170 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260809 23:20:20.568408 31470 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:20.688779 32217 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260809 23:20:20.631758 31765 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.011s	sys 0.031s
I20260809 23:20:20.699679 32107 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:20.609833 32163 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.003s	sys 0.013s
I20260809 23:20:20.596529 32118 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:20.690040 31984 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.929s	user 0.015s	sys 0.000s
W20260809 23:20:20.749248 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 681980us
W20260809 23:20:20.775046 24321 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.028s	sys 0.026s
W20260809 23:20:20.887743 24321 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.028s	sys 0.026s
W20260809 23:20:20.699528 32218 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260809 23:20:20.703287 31764 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:20.783715 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 801693us
I20260809 23:20:20.487310 32123 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260809 23:20:20.678886 31931 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:20.800329 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:20.811714 32153 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:20.820235 32204 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.357s	user 0.000s	sys 0.001s
I20260809 23:20:20.823200 31798 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:20.823961 32154 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:20.788862 32190 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.699s	user 0.001s	sys 0.001s
I20260809 23:20:20.769485 31681 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:20.842955 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2911) took 1467 ms. Trace:
W20260809 23:20:20.847116 24428 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.559s	user 0.009s	sys 0.006s
W20260809 23:20:20.838460 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 402305us
W20260809 23:20:20.805328 32082 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.208s	user 0.014s	sys 0.000s
I20260809 23:20:20.962217 24428 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:20.896859 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1326) took 1703 ms. Trace:
I20260809 23:20:20.888412 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:20.841816 31470 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:20.841585 32203 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
W20260809 23:20:20.906378 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.005s	sys 0.037s
W20260809 23:20:20.965914 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.006s	sys 0.037s
W20260809 23:20:20.903223 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 492272us
I20260809 23:20:20.836549 32240 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:20.800474 31688 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.882608 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:20.888479 31765 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:20.887583 32192 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:20.969471 25363 rpcz_store.cc:276] 0809 23:20:19.193461 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.200698 (+  7237us) service_pool.cc:227] Handling call
0809 23:20:19.201061 (+   363us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:19.201463 (+   402us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:19.201501 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:19.201956 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:20:19.885296 (+683340us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:20.451122 (+565826us) log.cc:842] Serialized 14 byte log entry
0809 23:20:20.896424 (+445302us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:20.896531 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:20.896593 (+    62us) raft_consensus.cc:1671] finished
0809 23:20:20.896613 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:20.896781 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16178,"prepare.queue_time_us":708397,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":76102,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":39742,"thread_start_us":1595818,"threads_started":3,"wal-append.queue_time_us":1438890}
W20260809 23:20:20.963574 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 75242.4us
I20260809 23:20:20.977425 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:20.881371 32098 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:20.783459 32166 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.961647 24541 rpcz_store.cc:276] 0809 23:20:19.375795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.642423 (+266628us) service_pool.cc:227] Handling call
0809 23:20:19.642718 (+   295us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.643060 (+   342us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:20.291509 (+648449us) log.cc:842] Serialized 14 byte log entry
0809 23:20:20.842565 (+551056us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:20.842621 (+    56us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:20.842691 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:20.842762 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:20.842879 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9827,"prepare.queue_time_us":1026501,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":10459,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":27034,"thread_start_us":1079296,"threads_started":2}
W20260809 23:20:20.967952 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 167568us
I20260809 23:20:20.968314 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:20.881930 32048 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:20.981070 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:20.804139 32175 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.908327 32118 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:20.993028 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:20.907325 31764 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:20.883952 32107 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.000979 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1240) took 1456 ms. Trace:
I20260809 23:20:20.961835 32231 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:20.971657 32153 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:20.998203 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:20.970149 32203 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:21.003811 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121320us
I20260809 23:20:20.991825 32082 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.015802 25372 rpcz_store.cc:276] 0809 23:20:19.544640 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.613395 (+ 68755us) service_pool.cc:227] Handling call
0809 23:20:19.624564 (+ 11169us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:19.625003 (+   439us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:19.625039 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:19.625586 (+   547us) log.cc:842] Serialized 26 byte log entry
0809 23:20:20.088150 (+462564us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:20.088206 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:20.088445 (+   239us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:21.000694 (+912249us) raft_consensus.cc:1671] finished
0809 23:20:21.000718 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.000895 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9435,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":10207,"thread_start_us":462174,"threads_started":1,"wal-append.queue_time_us":872725}
I20260809 23:20:21.023326 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:21.005530 32213 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.476s	user 0.001s	sys 0.001s
W20260809 23:20:21.025614 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 301135us
I20260809 23:20:20.907172 31984 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:20.994204 31021 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:20.995200 32098 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.034607 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.015988 31902 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:21.040071 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:21.022763 32072 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:21.044888 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2449) took 1089 ms. Trace:
I20260809 23:20:21.005096 32048 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:20.995308 31765 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.031934 31764 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.026216 31931 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.024178 32231 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.024420 32082 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.030036 31339 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:21.171578 24544 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:21.183761 24541 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:21.174577 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:21.184744 24546 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:21.007223 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:21.201841 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 195163us
I20260809 23:20:21.202776 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.183107 24309 rpcz_store.cc:276] 0809 23:20:19.955349 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.955486 (+   137us) service_pool.cc:227] Handling call
0809 23:20:19.955786 (+   300us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:19.967428 (+ 11642us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:19.967457 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:20.019142 (+ 51685us) log.cc:842] Serialized 26 byte log entry
0809 23:20:20.509847 (+490705us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:20.985889 (+476042us) log.cc:842] Serialized 14 byte log entry
0809 23:20:21.018189 (+ 32300us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:21.018331 (+   142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:21.018394 (+    63us) raft_consensus.cc:1671] finished
0809 23:20:21.018405 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.044815 (+ 26410us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":85537,"prepare.queue_time_us":309438,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":47993,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":21851,"thread_start_us":845739,"threads_started":2,"wal-append.queue_time_us":751177}
W20260809 23:20:21.164746 31842 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.962s	user 0.012s	sys 0.000s
W20260809 23:20:21.174212 32223 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:20:21.058898 31884 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:21.091413 31857 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.007s	sys 0.011s
I20260809 23:20:21.163374 32259 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:20:21.221203 32126 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.551s	user 0.007s	sys 0.007s
I20260809 23:20:21.175464 32209 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:21.166256 32153 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:21.168808 31688 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:21.172367 32098 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:21.166515 31984 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:21.187160 32072 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:21.236505 31579 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.657s	user 0.008s	sys 0.004s
I20260809 23:20:21.175855 32261 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:21.176247 32247 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:21.172901 32175 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:21.238626 32214 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.489s	user 0.001s	sys 0.000s
I20260809 23:20:21.059402 32240 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:21.164255 32107 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:21.246554 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.250s	user 0.013s	sys 0.042s
W20260809 23:20:21.313923 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.318s	user 0.014s	sys 0.042s
I20260809 23:20:21.293171 32262 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:21.216578 31902 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.175022 32203 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.239289 31764 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:21.259255 25366 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260809 23:20:21.329348 25366 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 7
I20260809 23:20:21.242610 32192 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:21.258312 31026 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.029s	sys 0.004s
I20260809 23:20:21.174681 32260 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260809 23:20:21.244820 31842 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:21.349407 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:21.322196 32262 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:21.245388 31857 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.010s	sys 0.017s
I20260809 23:20:21.215786 31339 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.247169 32259 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:21.352219 32259 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.353250 31339 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:21.353593 32259 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:21.281836 31851 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.004s	sys 0.041s
I20260809 23:20:21.254391 32209 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:21.305559 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.209s	user 0.005s	sys 0.009s
I20260809 23:20:21.310854 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:21.300047 32032 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.000s
I20260809 23:20:21.292665 32126 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:21.173339 32118 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.248010 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2520) took 1769 ms. Trace:
W20260809 23:20:21.369292 24435 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.071s	user 0.017s	sys 0.058s
W20260809 23:20:21.369848 24435 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.072s	user 0.017s	sys 0.058s
I20260809 23:20:21.299192 32072 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:21.300432 31579 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:21.249993 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:21.397639 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 147769us
I20260809 23:20:21.300613 32261 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:21.303504 32247 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:21.400913 32246 raft_consensus.cc:2741] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260809 23:20:21.404233 32247 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:21.303164 31984 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:21.272176 24541 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:21.342572 24546 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:21.329689 25366 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260809 23:20:21.317497 32123 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:21.324610 32233 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.359s	user 0.000s	sys 0.001s
W20260809 23:20:21.347688 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 379404us
I20260809 23:20:21.392987 32126 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:21.435173 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.006s	sys 0.040s
I20260809 23:20:21.350479 32262 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:21.443141 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2784) took 2185 ms. Trace:
I20260809 23:20:21.356424 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:21.357494 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 389675us
I20260809 23:20:21.357429 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1242) took 1752 ms. Trace:
W20260809 23:20:21.360059 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.263s	user 0.005s	sys 0.009s
W20260809 23:20:21.347795 31026 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.029s	sys 0.004s
I20260809 23:20:21.448776 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:21.346871 31793 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:21.368669 24429 rpcz_store.cc:276] 0809 23:20:19.478713 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.541256 (+ 62543us) service_pool.cc:227] Handling call
0809 23:20:19.541631 (+   375us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:19.541988 (+   357us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:19.542027 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:19.542535 (+   508us) log.cc:842] Serialized 26 byte log entry
0809 23:20:20.292509 (+749974us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:20.292545 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:20.292621 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:21.247719 (+955098us) raft_consensus.cc:1671] finished
0809 23:20:21.247771 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.247949 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":48803,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":125,"thread_start_us":749587,"threads_started":1,"wal-append.queue_time_us":892487}
I20260809 23:20:21.405930 32082 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:21.369925 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 335478us
I20260809 23:20:21.395440 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:21.392983 32275 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.360229 32203 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:21.400751 32261 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:21.327307 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:21.434128 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.362198 32209 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:21.507836 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73795.6us
I20260809 23:20:21.434759 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:21.511272 24541 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:21.512063 32212 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:21.438537 31339 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:21.408167 25366 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:21.330350 31902 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.348155 25368 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260809 23:20:21.442749 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.006s	sys 0.040s
I20260809 23:20:21.328735 32240 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:21.448680 25375 rpcz_store.cc:276] 0809 23:20:19.605089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.645813 (+ 40724us) service_pool.cc:227] Handling call
0809 23:20:19.646163 (+   350us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:19.646549 (+   386us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:19.646579 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:19.676083 (+ 29504us) log.cc:842] Serialized 52 byte log entry
0809 23:20:20.249242 (+573159us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:20.249287 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:20.249356 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:21.357118 (+1107762us) raft_consensus.cc:1671] finished
0809 23:20:21.357148 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.357339 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2140,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":536317,"threads_started":1,"wal-append.queue_time_us":1322558}
W20260809 23:20:21.357445 31851 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.004s	sys 0.042s
I20260809 23:20:21.461756 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:21.462116 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 259399us
I20260809 23:20:21.541659 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.481637 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2516) took 2049 ms. Trace:
I20260809 23:20:21.483671 24314 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260809 23:20:21.544606 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.447072 24539 rpcz_store.cc:276] 0809 23:20:19.258104 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.641429 (+383325us) service_pool.cc:227] Handling call
0809 23:20:19.641741 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.641961 (+   220us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:20.381400 (+739439us) log.cc:842] Serialized 14 byte log entry
0809 23:20:21.442710 (+1061310us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:21.442785 (+    75us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:21.442856 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:21.442947 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.443067 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":922138,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":271935,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":7568,"thread_start_us":1799939,"threads_started":3,"wal-append.queue_time_us":835413}
I20260809 23:20:21.478551 24535 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:21.546103 24314 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:21.489670 31974 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:21.370517 32032 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.433396 31884 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:21.491608 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:21.494226 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:21.438417 32123 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:21.495052 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.495517 32298 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:21.496661 32274 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260809 23:20:21.497707 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.498448 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:21.498888 32267 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.499593 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.500773 32279 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.498456 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.499617 24534 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
W20260809 23:20:21.503288 32263 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260809 23:20:21.442307 31765 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:21.503873 32257 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260809 23:20:21.504364 32231 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:21.506645 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.849s (SENT) [suppressed 4 similar messages]
I20260809 23:20:21.554433 31974 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:21.519724 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120760us
I20260809 23:20:21.447038 32048 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:21.523725 24378 connection.cc:232] RPC call timeout handler was delayed by -1.341256600106135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
I20260809 23:20:21.551909 31026 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:21.566613 32123 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:21.605036 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.440749 31984 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.497745 32245 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:21.556442 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:21.558055 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66551.6us
I20260809 23:20:21.519490 32084 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.517479 31793 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:21.520299 31339 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.578397 24316 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:21.572075 24311 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:21.611285 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:21.606431 24317 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:21.613297 24317 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:21.613744 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:21.582712 24496 leader_election.cc:398] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:21.484606 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.551067 32107 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:21.596246 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:21.595551 32279 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.601125 32175 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:21.553815 32240 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:21.574280 32305 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:21.649156 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53019.2us
I20260809 23:20:21.650811 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.577993 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:21.491744 31967 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.575811 32068 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:21.580451 32304 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.615841 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:21.614787 24496 leader_election.cc:401] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260809 23:20:21.506186 32275 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.652031 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:21.594949 32301 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:21.598484 31974 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:21.566397 31884 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.665280 24312 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260809 23:20:21.665349 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58755.6us
I20260809 23:20:21.605724 31832 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:21.669230 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92180.1us
I20260809 23:20:21.595081 32309 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:21.669746 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:21.631963 32231 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:21.669838 24311 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:21.671445 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:21.947042 25370 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:21.893100 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 386462us
I20260809 23:20:21.630899 32175 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:21.918593 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 248916us
I20260809 23:20:21.662860 31793 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:22.003736 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.005771 24543 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 12 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 12.
I20260809 23:20:21.896664 31967 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:21.917101 32231 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:21.607795 31026 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.573746 32298 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.631420 32281 raft_consensus.cc:2741] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260809 23:20:21.584892 32126 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.950963 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:21.583439 32267 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.846827 32278 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:21.893024 31851 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:21.968611 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 363605us
I20260809 23:20:21.963231 32272 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.137s
I20260809 23:20:22.051594 24498 leader_election.cc:398] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:22.052644 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:22.024367 32231 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:21.982465 32152 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:21.983245 32321 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.038s
I20260809 23:20:21.986845 31884 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:21.987155 32240 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.988667 31902 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:21.994022 32193 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.506s	user 0.008s	sys 0.005s
I20260809 23:20:22.092523 32193 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:21.945787 32301 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.093148 32197 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:22.093562 32197 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.094630 32301 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.094411 32197 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:21.995504 32247 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.554s	user 0.008s	sys 0.000s
I20260809 23:20:22.095752 32197 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:22.095829 32247 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.098065 32247 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.008836 24539 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260809 23:20:22.100080 32319 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260809 23:20:21.658083 32084 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.103664 32300 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.434s	user 0.000s	sys 0.000s
I20260809 23:20:22.104143 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1149) took 3084 ms. Trace:
I20260809 23:20:22.105046 32275 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.104609 24544 rpcz_store.cc:276] 0809 23:20:19.019511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.615346 (+595835us) service_pool.cc:227] Handling call
0809 23:20:19.615734 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:21.184000 (+1568266us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:22.103833 (+919833us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:22.104066 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3577623936}
I20260809 23:20:22.106137 31984 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.106325 32084 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.107685 31793 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.647706 32107 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.117008 32107 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:21.964143 32049 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:22.117764 32049 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:22.118144 32049 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:22.020669 24311 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:21.631692 32280 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.399s
I20260809 23:20:22.123588 32280 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:22.123908 32280 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.124637 32280 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.126036 32280 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:21.668444 32068 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.618032 32245 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.007934 32175 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:21.849192 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 171155us
I20260809 23:20:21.658818 32302 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.407s
I20260809 23:20:22.180889 32302 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:22.181296 32302 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.182101 32302 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.183777 32302 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:22.045243 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95659.6us
I20260809 23:20:22.187862 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:22.044037 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 392566us
I20260809 23:20:22.049124 32048 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260809 23:20:22.049436 32313 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
W20260809 23:20:22.050196 32149 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.493s	user 0.011s	sys 0.001s
I20260809 23:20:22.050520 32281 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:22.211544 32281 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:22.212014 32281 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:21.974992 31765 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.215461 25369 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:22.036931 32268 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.587s	user 0.001s	sys 0.001s
I20260809 23:20:22.083719 32152 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.665126 32305 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:21.675801 32304 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:21.968372 31968 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:22.053056 32024 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:22.084407 32309 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
W20260809 23:20:22.227989 32082 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.003s	sys 0.006s
I20260809 23:20:22.130848 32267 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.111016 32095 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.133041 31339 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.136158 32298 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.144677 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.299773 25375 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:22.144989 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.145511 25363 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:22.145893 31884 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:22.146994 31764 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.800s	user 0.007s	sys 0.003s
I20260809 23:20:22.046236 32177 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:22.147491 25371 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260809 23:20:22.147987 31470 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.157s	user 0.010s	sys 0.002s
I20260809 23:20:22.078634 32032 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:22.287814 24246 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.005s	sys 0.009s
I20260809 23:20:22.107162 32279 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:22.125506 32259 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.771s	user 0.008s	sys 0.003s
I20260809 23:20:21.996521 31974 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.191653 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:22.196132 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1157) took 1153 ms. Trace:
I20260809 23:20:22.176527 31984 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.204492 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.343089 24429 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
W20260809 23:20:22.343283 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 151742us
W20260809 23:20:22.073575 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8524198897648603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
I20260809 23:20:22.342096 24542 rpcz_store.cc:276] 0809 23:20:21.042754 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.042950 (+   196us) service_pool.cc:227] Handling call
0809 23:20:21.045915 (+  2965us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:21.323208 (+277293us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:21.323250 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:21.323303 (+    53us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:21.323345 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:22.195801 (+872456us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:22.195866 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:22.196059 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637147904}
I20260809 23:20:22.216606 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1239) took 2704 ms. Trace:
I20260809 23:20:22.224171 24430 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:22.227027 24428 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260809 23:20:22.346065 24428 pending_rounds.cc:76] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:20:22.346302 24428 pending_rounds.cc:97] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:20:22.346549 24428 raft_consensus.cc:2866] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:20:22.235562 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 619743us
I20260809 23:20:22.344697 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:22.246393 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.203533 32048 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:22.227803 32118 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.004s	sys 0.007s
I20260809 23:20:22.215902 32192 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:22.221511 32153 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.077682 31579 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Leader election won for term 10
I20260809 23:20:22.371852 31579 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.373109 31579 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.373971 32304 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.375533 32068 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.292824 32326 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.076s
I20260809 23:20:22.376427 31974 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:22.377460 32326 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:22.378099 32326 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:22.381554 32072 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.975s	user 0.009s	sys 0.001s
I20260809 23:20:22.382100 32072 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.282253 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:22.272174 32154 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.307s	user 0.008s	sys 0.011s
I20260809 23:20:22.204063 32193 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260809 23:20:22.267359 32149 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.300532 25368 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:22.442588 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1246) took 2519 ms (client timeout 2175 ms). Trace:
W20260809 23:20:22.443181 25369 rpcz_store.cc:269] 0809 23:20:19.923511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.931018 (+  7507us) service_pool.cc:227] Handling call
0809 23:20:22.442330 (+2511312us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:22.442496 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5199731072}
I20260809 23:20:22.215391 31765 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.285202 32313 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260809 23:20:22.304647 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.272073 32024 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.307704 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.456847 24312 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:22.324922 31470 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.148578 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.469667 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.458081 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
W20260809 23:20:22.359036 32118 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.004s	sys 0.007s
I20260809 23:20:22.481652 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:22.297101 32082 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.007s	sys 0.007s
I20260809 23:20:22.327770 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.345839 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:22.199693 32261 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.680s	user 0.009s	sys 0.000s
W20260809 23:20:22.216387 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.002s	sys 0.006s
W20260809 23:20:22.333022 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 409299us
I20260809 23:20:22.319146 32247 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.352770 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:22.353255 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.369206 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:22.496429 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 127331us
I20260809 23:20:22.350785 24425 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:22.342671 32298 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260809 23:20:22.349987 32279 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:22.277132 24435 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.002s	sys 0.012s
I20260809 23:20:22.461483 32329 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.315526 32095 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.359522 32048 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:22.316076 31884 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.367326 32192 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.455833 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.454828 31974 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.484675 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.336326 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.486708 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.490367 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:22.492650 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.003s	sys 0.007s
I20260809 23:20:22.594121 25371 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:22.593334 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:22.594781 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.13302e+06us
I20260809 23:20:22.344560 25366 rpcz_store.cc:276] 0809 23:20:19.495641 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.595340 (+ 99699us) service_pool.cc:227] Handling call
0809 23:20:19.595627 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:21.518075 (+1922448us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:22.199479 (+681404us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54006004d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:22.199695 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3824811264}
I20260809 23:20:22.596297 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.476557 32325 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
W20260809 23:20:22.500346 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 154724us
I20260809 23:20:22.661432 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.491261 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:22.481487 32203 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.967s	user 0.011s	sys 0.000s
W20260809 23:20:22.567028 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69617.8us
I20260809 23:20:22.566483 24316 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:22.577770 24435 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.961s	user 0.002s	sys 0.012s
I20260809 23:20:22.388926 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:22.390197 32153 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.350137 32320 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.146s
W20260809 23:20:22.499544 32308 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.004s	sys 0.039s
W20260809 23:20:22.639981 32262 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.152s	user 0.009s	sys 0.002s
I20260809 23:20:22.589674 25369 raft_consensus.cc:2354] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:22.458401 32149 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.591776 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 239081us
I20260809 23:20:22.359629 31984 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.587150 32049 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.579458 32240 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:22.566571 32245 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:22.619192 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.028s	sys 0.031s
I20260809 23:20:22.599004 32325 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:22.623999 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.566282 31727 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:22.793994 25373 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:22.664331 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.665412 25366 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:22.706310 32278 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.000s
W20260809 23:20:22.597903 31967 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.576s	user 0.005s	sys 0.005s
I20260809 23:20:22.675740 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:22.677481 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 333362us
I20260809 23:20:22.368093 32259 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.211s
I20260809 23:20:22.838578 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:22.839375 24534 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:22.672801 25371 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:20:22.741318 31832 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.792s	user 0.014s	sys 0.000s
I20260809 23:20:22.684100 24545 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:22.682392 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:22.587604 32082 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:22.681584 32308 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.005s	sys 0.039s
I20260809 23:20:22.646842 32068 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.628943 31974 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.665243 32336 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:22.499542 32261 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.622757 31884 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.593809 32305 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:22.650584 32152 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.698114 32320 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:22.478240 32326 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.591712 32279 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.109s
I20260809 23:20:22.585813 32329 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:22.650061 32197 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.554s	user 0.011s	sys 0.000s
W20260809 23:20:22.636274 32123 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.988s	user 0.002s	sys 0.010s
I20260809 23:20:22.871536 24544 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:22.724656 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:22.729138 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.872689 25369 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:22.872956 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 148492us
I20260809 23:20:22.873847 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.874037 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:22.730584 24539 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260809 23:20:22.875028 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 212071us
I20260809 23:20:22.629914 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:22.590710 32247 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.702870 32258 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.467s	user 0.004s	sys 0.009s
I20260809 23:20:22.675226 32331 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:22.708340 32280 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.545s	user 0.001s	sys 0.009s
W20260809 23:20:22.714917 32209 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.203s	user 0.014s	sys 0.000s
I20260809 23:20:22.798467 32325 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.715183 32262 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.802129 32175 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.612s	user 0.007s	sys 0.000s
I20260809 23:20:22.795849 32340 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.667591 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2526) took 2471 ms. Trace:
I20260809 23:20:22.803800 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:22.804173 32346 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.146s	user 0.000s	sys 0.002s
I20260809 23:20:22.709110 31765 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.805544 25366 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:22.812381 32149 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:22.814139 32231 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.727s	user 0.004s	sys 0.007s
I20260809 23:20:22.818645 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.820186 32072 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:22.820667 32334 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.823239 31984 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:22.824610 25363 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:22.825796 32049 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.726065 32177 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.740891 32240 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:22.838557 24498 connection.cc:232] RPC call timeout handler was delayed by -2.6661198898218572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21960000000000002s for a response
I20260809 23:20:22.840672 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.591971 32192 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:22.842218 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 166600us
I20260809 23:20:22.794440 32126 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.706854 31470 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:22.845263 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:22.944877 24313 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:22.726015 32338 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:22.894914 31967 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:22.842698 32281 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.003s
I20260809 23:20:22.741144 32118 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.706425 32203 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.136s
I20260809 23:20:22.844143 31727 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:22.847855 31948 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.846653 32287 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:22.844447 32082 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:22.844285 32304 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.869081 24534 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:22.853837 32320 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:22.729524 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.841269 32278 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
W20260809 23:20:22.791803 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.029s	sys 0.032s
I20260809 23:20:22.882815 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:22.974507 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 91779.5us
I20260809 23:20:22.975549 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:22.976316 31688 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:22.976428 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:22.883966 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 254168us
I20260809 23:20:22.977257 31688 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:22.855171 32302 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.977679 31688 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:22.885790 31832 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:22.875752 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:22.977964 32302 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.977981 32259 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:22.890813 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:22.978804 31832 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:22.895350 25373 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:22.979295 25369 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:22.846118 32068 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:22.895824 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2917) took 2916 ms (client timeout 2040 ms). Trace:
I20260809 23:20:22.897725 32329 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:22.895143 32209 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.907910 32247 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.910127 32261 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.912262 31974 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:22.992138 31968 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.716s	user 0.007s	sys 0.006s
I20260809 23:20:22.928627 24541 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
W20260809 23:20:22.936700 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1250) took 1625 ms (client timeout 1733 ms). Trace:
W20260809 23:20:23.011869 24380 connection.cc:232] RPC call timeout handler was delayed by -1.1896630940027535s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20400000000000001s for a response
W20260809 23:20:23.016165 25370 rpcz_store.cc:269] 0809 23:20:21.311760 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.311990 (+   230us) service_pool.cc:227] Handling call
0809 23:20:22.936514 (+1624524us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:22.936641 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1007985920}
I20260809 23:20:22.890365 32336 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:22.888335 32331 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:22.942499 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 345051us
I20260809 23:20:22.984025 31832 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
W20260809 23:20:22.943015 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 179332us
W20260809 23:20:22.840592 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 243972us
I20260809 23:20:22.950994 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.935534 32325 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:22.876384 32123 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.954715 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.978122 31688 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:22.978837 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 103164us
I20260809 23:20:22.894662 24435 rpcz_store.cc:276] 0809 23:20:20.195746 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:20.297089 (+101343us) service_pool.cc:227] Handling call
0809 23:20:20.297491 (+   402us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:20.297749 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:21.433728 (+1135979us) log.cc:842] Serialized 14 byte log entry
0809 23:20:22.667241 (+1233513us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:22.667302 (+    61us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:22.667354 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:22.667421 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:22.667538 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":881415,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":432221,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":127,"thread_start_us":2368647,"threads_started":3,"wal-append.queue_time_us":604071}
W20260809 23:20:22.981293 24540 rpcz_store.cc:269] 0809 23:20:19.971789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.971964 (+   175us) service_pool.cc:227] Handling call
0809 23:20:22.887959 (+2915995us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:22.888087 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3427034368}
I20260809 23:20:22.919092 32175 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.941627 32334 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.924132 32340 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.955678 32281 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.987531 25368 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 16. Ops: []
I20260809 23:20:22.991211 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:22.980155 32302 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:22.997493 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:22.970438 32118 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:22.982890 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:23.050282 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 67452.2us
I20260809 23:20:23.025427 31832 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:22.979008 32259 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:23.000435 24539 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:23.001622 24429 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:22.954262 32299 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:22.944414 32177 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.920883 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
W20260809 23:20:22.942914 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4843198899179697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2398s for a response
I20260809 23:20:23.019445 24427 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:23.024112 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:23.071816 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9861294499132782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
I20260809 23:20:22.987378 32359 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:23.001591 32279 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.029309 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.033854 25370 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:22.953573 32126 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.034818 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.078364 24435 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 13
I20260809 23:20:23.035637 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:22.972944 31948 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:22.855044 32341 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.061s
I20260809 23:20:23.072106 32207 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.087468 32341 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:23.075103 32359 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.088452 32359 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:23.088244 32192 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.079871 32279 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.039213 32360 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260809 23:20:22.953917 32338 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.091439 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:23.037235 32320 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.044456 32262 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.047165 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.093041 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:22.997824 25373 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.025972 32331 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:23.051304 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.094810 25370 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:22.947039 32240 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:23.053985 24542 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:23.055171 31968 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:23.096481 31968 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:23.096851 31968 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:23.031163 32368 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:23.056526 32175 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:21.542896 24423 rpcz_store.cc:276] 0809 23:20:19.432254 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.432475 (+   221us) service_pool.cc:227] Handling call
0809 23:20:19.432817 (+   342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:19.433075 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:20.096700 (+663625us) log.cc:842] Serialized 14 byte log entry
0809 23:20:21.481165 (+1384465us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:21.481239 (+    74us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:21.481311 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:21.481419 (+   108us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:21.481562 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41807,"prepare.queue_time_us":1608892,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":64779,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4494,"thread_start_us":1975187,"threads_started":2,"wal-append.queue_time_us":1303921}
I20260809 23:20:23.057341 32068 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:23.057672 32361 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:20:22.975152 32287 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.058771 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:23.060767 32247 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.063527 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.068073 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:22.931201 32231 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:23.072235 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9506198898889124s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29910000000000003s for a response
W20260809 23:20:23.242116 24498 connection.cc:232] RPC call timeout handler was delayed by -1.8826294499449432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
W20260809 23:20:23.242439 24498 connection.cc:232] RPC call timeout handler was delayed by -1.8520294500049204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
W20260809 23:20:23.242830 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0969997190404683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
I20260809 23:20:23.108269 32262 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.243587 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:23.072330 24541 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.073405 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:23.076140 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.024689 32082 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.085575 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.086364 32126 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.064028 32261 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.087936 32341 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:23.093269 24540 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:22.974989 32304 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.094715 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.055766 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.101007 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:23.040755 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:23.086625 32177 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.111783 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.115149 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:23.121407 24430 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:23.049842 31300 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260809 23:20:23.227447 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:23.232656 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 173930us
I20260809 23:20:23.245642 24539 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:23.251016 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 177661us
I20260809 23:20:23.255136 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.078027 32048 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.256417 24498 leader_election.cc:398] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:23.107146 31832 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.256560 24429 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:23.260406 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.260828 24422 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:23.080276 32260 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.165s
I20260809 23:20:23.264921 24423 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:23.267295 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.267638 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.270190 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:23.275180 24543 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:23.256909 24313 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:23.263288 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 162378us
I20260809 23:20:23.261510 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.268824 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:23.270467 24498 leader_election.cc:401] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260809 23:20:23.279165 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.266789 24425 raft_consensus.cc:2362] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11. Re-sending same reply.
I20260809 23:20:23.281054 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.281934 24535 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:23.281836 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.229519 32154 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.241151 32068 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:23.282734 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.283578 25370 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.283929 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.295576 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:23.286329 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:23.296895 24535 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:23.295877 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:23.292240 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 266312us
I20260809 23:20:23.287547 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:23.298375 25370 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:23.303758 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.308625 24431 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:23.309010 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:23.313381 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.086395 32259 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.325934 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.279389 31974 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:23.260103 32196 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:23.087536 32207 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.304589 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:23.330384 24422 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:23.338239 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1278) took 1194 ms. Trace:
I20260809 23:20:23.344674 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:23.345170 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:23.346129 32371 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260809 23:20:23.347752 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.384378 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:23.352109 32377 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.233s	user 0.000s	sys 0.000s
W20260809 23:20:23.355268 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98973.9us
I20260809 23:20:23.391839 24498 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:23.393024 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:23.303314 32154 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.357873 32334 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.268028 32336 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260809 23:20:23.075460 32299 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.284439 32360 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20260809 23:20:23.300127 32203 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:23.361598 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:23.427932 32203 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.428422 32203 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.430297 32203 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.432192 32024 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.387593 32260 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:23.371073 25365 rpcz_store.cc:276] 0809 23:20:22.144040 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.144201 (+   161us) service_pool.cc:227] Handling call
0809 23:20:22.144601 (+   400us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:22.191164 (+ 46563us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:22.191193 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:22.191232 (+    39us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:22.191260 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.337982 (+1146722us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:23.338035 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:23.338178 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":106625280}
I20260809 23:20:23.299544 32098 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:23.380724 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:23.442165 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61518.4us
I20260809 23:20:23.302274 32209 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.255242 32247 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.401063 24309 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:23.326695 31984 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:23.403227 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.448861 24425 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:23.453958 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:23.448455 31661 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:23.440862 32281 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.406467 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.325259 32068 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.421674 32280 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.030s
I20260809 23:20:23.327068 31579 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.408675 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:23.470606 24541 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:23.331851 32341 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.332412 32279 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.257431 32262 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.346550 31688 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.332021 32338 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.331635 32123 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.423830 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.425189 32395 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:23.388124 31974 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:23.406250 32344 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:23.435923 32192 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.463533 31661 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.465968 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.259825 32261 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.117741 32368 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.448433 31984 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.473981 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:23.504768 25365 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:23.473162 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1285) took 1324 ms. Trace:
I20260809 23:20:23.505597 25371 rpcz_store.cc:276] 0809 23:20:22.148943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.952258 (+803315us) service_pool.cc:227] Handling call
0809 23:20:22.952669 (+   411us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:22.952962 (+   293us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:22.952997 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:22.953505 (+   508us) log.cc:842] Serialized 26 byte log entry
0809 23:20:23.027354 (+ 73849us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:23.028635 (+  1281us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.028734 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:23.472912 (+444178us) raft_consensus.cc:1671] finished
0809 23:20:23.472942 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:23.473092 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":26740,"threads_started":1,"wal-append.queue_time_us":82948}
I20260809 23:20:23.507577 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:23.508313 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:23.451927 32098 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:23.426035 32370 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260809 23:20:23.487087 24541 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.497742 32395 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:23.501328 32231 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.499857 31661 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:23.471616 32068 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.510670 31984 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:23.513674 32304 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.521610 32338 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:23.521776 32395 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:23.527067 32048 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.540717 32368 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.540547 32240 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.541194 32068 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.541431 32247 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.542590 32240 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.543000 32126 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:23.542820 32098 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.543511 32126 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:23.544044 32126 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:23.534596 32338 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:23.549546 32261 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.549552 32262 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.553999 32331 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.555310 32359 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.559742 32359 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.562947 32395 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.565056 32395 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:23.475559 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 198936us
W20260809 23:20:23.604097 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.007s	sys 0.002s
W20260809 23:20:23.604923 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.007s	sys 0.003s
I20260809 23:20:23.375689 24313 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:23.444180 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:23.417501 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.611418 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.613238 24422 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:23.614192 24425 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:23.405047 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:23.615645 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.617851 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.496820 24429 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:23.618587 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:23.618671 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 213828us
I20260809 23:20:23.620154 24425 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:23.621627 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.627118 24425 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:20:23.628192 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.630092 24425 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:23.609719 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.633808 24539 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:23.635077 24540 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260809 23:20:23.635113 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.636345 24423 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:23.637355 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.637385 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:23.640120 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.641458 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.624688 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:23.638705 24313 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.642922 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:23.645201 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 209233us
I20260809 23:20:23.646026 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.653889 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:23.655797 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:23.645252 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 315912us
I20260809 23:20:23.656924 25365 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:23.657392 24423 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:23.658334 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:23.656464 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.661592 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.661970 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:23.666802 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:23.669636 25365 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:23.475975 31579 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.685577 31688 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.689648 32331 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.536882 31974 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.693222 32334 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.694442 32334 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.696342 32240 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.696581 32196 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:23.697256 32359 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:23.697383 32240 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.698132 32359 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:23.557044 32338 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.695384 32368 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.694264 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:23.697701 32196 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:23.703291 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49681.3us
I20260809 23:20:23.710193 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.686717 31579 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:23.718051 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1255) took 1482 ms (client timeout 1817 ms). Trace:
W20260809 23:20:23.718603 24312 rpcz_store.cc:269] 0809 23:20:22.235309 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.281995 (+ 46686us) service_pool.cc:227] Handling call
0809 23:20:23.717527 (+1435532us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:23.717824 (+   297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156420992}
I20260809 23:20:23.720395 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.720677 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1251) took 2043 ms. Trace:
I20260809 23:20:23.721149 24317 rpcz_store.cc:276] 0809 23:20:21.677157 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.969737 (+292580us) service_pool.cc:227] Handling call
0809 23:20:21.970154 (+   417us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:22.200117 (+229963us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:22.200161 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:22.200216 (+    55us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:22.200260 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.720165 (+1519905us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:23.720230 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:23.720467 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":528321408}
I20260809 23:20:23.723549 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2920) took 2110 ms. Trace:
I20260809 23:20:23.723790 32302 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:23.724026 24316 rpcz_store.cc:276] 0809 23:20:21.612922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.617641 (+  4719us) service_pool.cc:227] Handling call
0809 23:20:21.617918 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:22.672882 (+1054964us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.722905 (+1050023us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:23.723321 (+   416us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2042818816}
I20260809 23:20:23.726179 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.726603 32302 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:23.696476 32262 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:23.536078 32154 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.709749 32401 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:23.743212 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.743511 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.732733 32302 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:23.747991 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54155.1us
I20260809 23:20:23.757233 32240 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:23.718632 32409 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
W20260809 23:20:23.763432 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 138966us
I20260809 23:20:23.766443 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:23.762377 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:23.773537 24313 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:23.774008 32406 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.775112 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.775195 32406 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.777068 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.778339 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1248) took 2468 ms. Trace:
I20260809 23:20:23.778825 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:23.780977 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1294) took 1600 ms (client timeout 1557 ms). Trace:
I20260809 23:20:23.781272 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:23.784799 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 340505us
W20260809 23:20:23.776088 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1256) took 1509 ms (client timeout 1967 ms). Trace:
I20260809 23:20:23.778568 32406 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.791808 24423 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:23.759586 31688 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:23.787397 25375 rpcz_store.cc:269] 0809 23:20:22.179803 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.306838 (+127035us) service_pool.cc:227] Handling call
0809 23:20:23.780233 (+1473395us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:23.780580 (+   347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":985881984}
I20260809 23:20:23.797725 24316 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:23.788055 25366 rpcz_store.cc:269] 0809 23:20:22.266704 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.596046 (+329342us) service_pool.cc:227] Handling call
0809 23:20:23.775615 (+1179569us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:23.775847 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":157527808}
I20260809 23:20:23.717773 32407 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260809 23:20:23.812618 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.778855 32048 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:23.699931 32359 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.815307 25365 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:23.818127 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:23.819195 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1288) took 1613 ms (client timeout 2028 ms). Trace:
W20260809 23:20:23.819697 25363 rpcz_store.cc:269] 0809 23:20:22.161458 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.490135 (+328677us) service_pool.cc:227] Handling call
0809 23:20:23.774543 (+1284408us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:23.774844 (+   301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":528127872}
I20260809 23:20:23.786725 25372 rpcz_store.cc:276] 0809 23:20:21.309943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.310099 (+   156us) service_pool.cc:227] Handling call
0809 23:20:21.324256 (+ 14157us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:22.427513 (+1103257us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:22.850584 (+423071us) log.cc:842] Serialized 14 byte log entry
0809 23:20:22.952968 (+102384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:22.953016 (+    48us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:22.953064 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.777373 (+824309us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400631354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:23.777447 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:23.778008 (+   561us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":470502,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":1967,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":31720,"spinlock_wait_cycles":2536202880,"thread_start_us":486887,"threads_started":2,"wal-append.queue_time_us":190450}
I20260809 23:20:23.822618 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1258) took 1221 ms. Trace:
I20260809 23:20:23.823171 25368 rpcz_store.cc:276] 0809 23:20:22.591084 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.039724 (+448640us) service_pool.cc:227] Handling call
0809 23:20:23.076495 (+ 36771us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:23.076907 (+   412us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:23.076924 (+    17us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:23.097954 (+ 21030us) log.cc:842] Serialized 52 byte log entry
0809 23:20:23.451002 (+353048us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:23.451999 (+   997us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.452068 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:23.656151 (+204083us) raft_consensus.cc:1671] finished
0809 23:20:23.656194 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:23.786498 (+130304us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b540071dc50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:23.811921 (+ 25423us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1013,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":8820,"spinlock_wait_cycles":50412288,"thread_start_us":333921,"threads_started":1,"wal-append.queue_time_us":406190}
I20260809 23:20:23.824805 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:23.826337 32304 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.827041 32304 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.829854 32240 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:23.830262 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:23.808357 32401 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:23.831590 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:23.837894 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76032.6us
I20260809 23:20:23.816270 32368 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:23.844748 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63757.4us
I20260809 23:20:23.827057 24312 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:23.835896 31021 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:23.859510 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:23.863732 32418 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.186s	user 0.000s	sys 0.002s
I20260809 23:20:23.865440 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.882648 32401 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.859187 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:23.879096 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79811.1us
I20260809 23:20:23.867447 32304 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.895670 32196 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.895951 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:23.896667 31661 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:23.898387 32368 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.868166 32240 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:23.917191 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.917055 24422 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:23.882952 31948 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:23.930855 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.943308 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.915067 32409 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:23.942803 32280 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:23.920998 31688 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:23.955729 24429 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:23.960263 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:23.960737 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:23.951835 31661 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:23.979916 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:23.982808 24422 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:23.982905 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.012698 24317 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:24.012998 31688 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.022907 24316 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.023326 24313 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:24.025475 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:24.027809 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107466us
I20260809 23:20:24.028343 31470 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:24.029354 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.031452 32240 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.032752 32166 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:24.020483 24534 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:24.019081 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.041246 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:24.042332 24422 raft_consensus.cc:2396] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260809 23:20:24.043022 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.043218 24422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:20:24.044082 24422 raft_consensus.cc:595] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:24.044582 24422 raft_consensus.cc:603] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:24.045136 24313 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:24.047065 32231 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:24.046802 32152 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:24.047886 32231 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:24.049098 31968 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.952s	user 0.011s	sys 0.013s
I20260809 23:20:24.049664 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1260) took 1426 ms. Trace:
I20260809 23:20:24.048705 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2459) took 1072 ms. Trace:
I20260809 23:20:24.051244 24312 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:24.036978 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76102.5us
I20260809 23:20:24.051200 24313 rpcz_store.cc:276] 0809 23:20:22.976103 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.044208 (+1068105us) service_pool.cc:227] Handling call
0809 23:20:24.044774 (+   566us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.045627 (+   853us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.048620 (+  2993us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.049731 31968 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.059798 24425 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:24.054769 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:24.070001 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:24.022531 32334 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.071651 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 998) took 1043 ms. Trace:
I20260809 23:20:24.076042 24422 rpcz_store.cc:276] 0809 23:20:23.016788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.045512 (+1028724us) service_pool.cc:227] Handling call
0809 23:20:24.046017 (+   505us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.046457 (+   440us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.046480 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.060159 (+ 13679us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:24.060217 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.060303 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.060442 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:24.071969 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 156073us
I20260809 23:20:24.077746 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:24.085597 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1007) took 1049 ms. Trace:
I20260809 23:20:24.050460 25373 rpcz_store.cc:276] 0809 23:20:22.623163 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.094067 (+470904us) service_pool.cc:227] Handling call
0809 23:20:23.094379 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:23.519788 (+425409us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:23.818268 (+298480us) log.cc:842] Serialized 14 byte log entry
0809 23:20:24.012392 (+194124us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.012515 (+   123us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:24.012662 (+   147us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.048384 (+ 35722us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b18001865e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:20:24.049151 (+   767us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:24.049228 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.049435 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77087,"prepare.queue_time_us":296447,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":9570,"raft.queue_time_us":7284,"spinlock_wait_cycles":1071011712,"thread_start_us":124389,"threads_started":2}
I20260809 23:20:24.085976 24429 rpcz_store.cc:276] 0809 23:20:23.036094 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.986667 (+950573us) service_pool.cc:227] Handling call
0809 23:20:24.085528 (+ 98861us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.091132 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:24.071606 31661 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:24.095198 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:24.096220 24535 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:24.097296 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:24.099874 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 754548us
W20260809 23:20:24.102797 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73852.7us
I20260809 23:20:24.103201 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1298) took 1051 ms. Trace:
I20260809 23:20:24.104182 24378 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:24.104570 24378 leader_election.cc:401] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:24.107692 24313 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260809 23:20:24.108492 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2921) took 1115 ms. Trace:
I20260809 23:20:24.108824 24313 rpcz_store.cc:276] 0809 23:20:22.992919 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.070576 (+1077657us) service_pool.cc:227] Handling call
0809 23:20:24.107161 (+ 36585us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.108165 (+  1004us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.108416 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.110623 24313 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260809 23:20:24.112210 32231 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:24.113353 32297 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.438s	user 0.006s	sys 0.006s
I20260809 23:20:24.113518 32409 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.069459 32417 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:24.115644 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:24.115778 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.116462 31968 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.117183 32211 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13.
I20260809 23:20:24.118474 32422 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:24.119601 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1296) took 1768 ms. Trace:
I20260809 23:20:24.121024 25375 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:24.123407 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.124322 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:24.115696 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:24.080983 32359 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:24.127033 24428 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:24.129420 32302 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:24.131263 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 203805us
I20260809 23:20:24.133065 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:24.133720 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:24.070319 32280 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.124624 25372 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:24.147336 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2922) took 1144 ms. Trace:
I20260809 23:20:24.159711 24313 rpcz_store.cc:276] 0809 23:20:22.993230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.109713 (+1116483us) service_pool.cc:227] Handling call
0809 23:20:24.110302 (+   589us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.137279 (+ 26977us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.137555 (+   276us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:24.124814 32312 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.277s	user 0.002s	sys 0.001s
I20260809 23:20:24.148947 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:24.155284 25373 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:24.154448 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:24.148372 25365 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:24.157936 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:24.166157 24316 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:24.172840 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.174114 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.103637 25370 rpcz_store.cc:276] 0809 23:20:23.051486 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.316159 (+264673us) service_pool.cc:227] Handling call
0809 23:20:23.347577 (+ 31418us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:23.512356 (+164779us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:23.512401 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:23.518872 (+  6471us) log.cc:842] Serialized 26 byte log entry
0809 23:20:23.759091 (+240219us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:23.809660 (+ 50569us) log.cc:842] Serialized 14 byte log entry
0809 23:20:23.930226 (+120566us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:23.943703 (+ 13477us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:24.091485 (+147782us) raft_consensus.cc:1671] finished
0809 23:20:24.091506 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.103012 (+ 11506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10967,"prepare.queue_time_us":210982,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":68,"raft.queue_time_us":289925,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":54308,"thread_start_us":277560,"threads_started":4,"wal-append.queue_time_us":157791}
I20260809 23:20:24.174906 24429 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:24.175158 24314 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:24.175547 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:24.176833 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:24.177410 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1170) took 1745 ms (client timeout 2070 ms). Trace:
I20260809 23:20:24.177660 25370 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:24.177809 24542 rpcz_store.cc:269] 0809 23:20:22.431972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.432093 (+   121us) service_pool.cc:227] Handling call
0809 23:20:24.176977 (+1744884us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:24.177182 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1322113408}
I20260809 23:20:24.180608 31661 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:24.182384 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58166.3us
I20260809 23:20:24.183753 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:24.184178 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1168) took 1848 ms (client timeout 1708 ms). Trace:
W20260809 23:20:24.184634 24545 rpcz_store.cc:269] 0809 23:20:22.335872 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.336008 (+   136us) service_pool.cc:227] Handling call
0809 23:20:24.183745 (+1847737us) spinlock_profiling.cc:243] Waited 89.9 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:24.183972 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206747904}
I20260809 23:20:24.185369 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:24.185842 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:24.186220 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70792.2us
I20260809 23:20:24.186581 32239 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260809 23:20:24.175709 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2918) took 2673 ms (client timeout 1713 ms). Trace:
I20260809 23:20:24.188194 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.186866 24545 raft_consensus.cc:2396] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260809 23:20:24.207228 24535 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:24.208395 25368 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:24.207935 24534 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:24.213297 25366 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:24.175638 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 999) took 1158 ms. Trace:
I20260809 23:20:24.222767 24429 rpcz_store.cc:276] 0809 23:20:23.017054 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.174072 (+1157018us) service_pool.cc:227] Handling call
0809 23:20:24.174468 (+   396us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.175393 (+   925us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.175568 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.188220 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2972) took 1995 ms. Trace:
I20260809 23:20:24.201057 24422 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:24.229066 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.168325 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:24.220237 24314 rpcz_store.cc:269] 0809 23:20:21.502664 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.556124 (+ 53460us) service_pool.cc:227] Handling call
0809 23:20:24.173748 (+2617624us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800001898. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:20:24.175596 (+  1848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2329984}
W20260809 23:20:24.189448 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1166) took 1854 ms (client timeout 2071 ms). Trace:
I20260809 23:20:24.238620 24312 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:24.232916 24431 rpcz_store.cc:276] 0809 23:20:22.192476 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.193078 (+   602us) service_pool.cc:227] Handling call
0809 23:20:22.215303 (+ 22225us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:23.320566 (+1105263us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.187750 (+867184us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400061654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:24.188010 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2500139392}
I20260809 23:20:24.241778 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.186220 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.238297 24317 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:24.194042 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2538) took 1190 ms. Trace:
I20260809 23:20:24.245368 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:24.245688 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.159591 32359 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:24.261377 24425 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:20:24.261722 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:24.180622 31984 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.004s
W20260809 23:20:24.239888 24544 rpcz_store.cc:269] 0809 23:20:22.334784 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.334960 (+   176us) service_pool.cc:227] Handling call
0809 23:20:24.188918 (+1853958us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54003dca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:24.189263 (+   345us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":864252672}
I20260809 23:20:24.247695 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:24.271615 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:24.271893 24312 raft_consensus.cc:2354] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:24.188704 32202 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:24.187135 32417 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:24.183749 31764 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:24.208940 32302 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:24.227725 32304 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:24.274636 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 125674us
I20260809 23:20:24.272687 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:24.207443 32262 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:24.281993 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.203733 32231 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:24.153651 25369 rpcz_store.cc:276] 0809 23:20:22.351026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.023464 (+672438us) service_pool.cc:227] Handling call
0809 23:20:23.024866 (+  1402us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:23.025297 (+   431us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:23.025326 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:23.031680 (+  6354us) log.cc:842] Serialized 52 byte log entry
0809 23:20:23.517133 (+485453us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:23.979230 (+462097us) log.cc:842] Serialized 14 byte log entry
0809 23:20:24.082298 (+103068us) log.cc:842] Serialized 14 byte log entry
0809 23:20:24.118952 (+ 36654us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.119082 (+   130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:24.119160 (+    78us) raft_consensus.cc:1671] finished
0809 23:20:24.119180 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.119385 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2292,"prepare.queue_time_us":399240,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":5345,"raft.queue_time_us":39622,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":20784,"spinlock_wait_cycles":41296896,"thread_start_us":680224,"threads_started":2,"wal-append.queue_time_us":508772}
I20260809 23:20:24.212627 32280 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.246214 32331 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:24.291366 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.291774 32331 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:24.292119 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105982us
I20260809 23:20:24.292346 32331 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.292723 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.293601 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.294075 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.299949 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2544) took 1230 ms. Trace:
I20260809 23:20:24.300379 24431 rpcz_store.cc:276] 0809 23:20:23.069814 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.262831 (+1193017us) service_pool.cc:227] Handling call
0809 23:20:24.299280 (+ 36449us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.299590 (+   310us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.299617 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.299676 (+    59us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:24.299706 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.299767 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.299875 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.302425 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2979) took 1231 ms. Trace:
I20260809 23:20:24.303335 32331 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.306897 24498 leader_election.cc:398] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14.
I20260809 23:20:24.307265 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.308041 32331 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.308569 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.309132 24312 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:24.310657 24312 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:24.288199 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.289219 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.261193 31661 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:24.277592 31764 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:24.279161 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:24.310819 24313 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:24.311256 25370 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260809 23:20:24.302764 24431 rpcz_store.cc:276] 0809 23:20:23.071217 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.301425 (+1230208us) service_pool.cc:227] Handling call
0809 23:20:24.301780 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.302081 (+   301us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.302104 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.302164 (+    60us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:24.302196 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.302254 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.302362 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.271275 32359 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:24.312093 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 65349.1us
I20260809 23:20:24.313014 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:24.345534 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 100252us
I20260809 23:20:24.352664 24316 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14. Re-sending same reply.
I20260809 23:20:24.329826 25373 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:24.244956 24424 rpcz_store.cc:276] 0809 23:20:23.003713 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.003923 (+   210us) service_pool.cc:227] Handling call
0809 23:20:23.024242 (+ 20319us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:23.080326 (+ 56084us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:23.250745 (+170419us) log.cc:842] Serialized 14 byte log entry
0809 23:20:23.465193 (+214448us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:23.470854 (+  5661us) log.cc:842] Serialized 26 byte log entry
0809 23:20:23.471025 (+   171us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:23.471289 (+   264us) log.cc:842] Serialized 14 byte log entry
0809 23:20:23.471551 (+   262us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.182524 (+710973us) spinlock_profiling.cc:243] Waited 55.8 ms on lock 0x7b540018cc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:24.182601 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:24.182759 (+   158us) raft_consensus.cc:1671] finished
0809 23:20:24.182798 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.193830 (+ 11032us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":268667,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":77,"raft.queue_time_us":2293,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":11256,"spinlock_wait_cycles":169745408,"thread_start_us":297387,"threads_started":2,"wal-append.queue_time_us":269371}
I20260809 23:20:24.351454 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.207175 32422 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.360550 25365 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:24.363137 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.289039 31579 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:24.373762 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.374864 24312 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:24.364107 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 176016us
I20260809 23:20:24.333922 31764 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:24.381946 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.311755 32432 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260809 23:20:24.381471 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 208868us
I20260809 23:20:24.382843 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.353765 32417 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:24.382284 25368 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:24.389756 24545 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:24.333724 32302 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:24.392118 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98711.8us
I20260809 23:20:24.392949 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.396067 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.393894 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.397197 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:24.284813 31984 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.397665 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.353734 32240 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:24.399226 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.334180 32304 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.400480 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:24.387854 32432 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:24.377740 32231 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.415334 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:24.423122 24323 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:24.423857 32261 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:24.424299 32245 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:24.424679 24544 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:24.424950 32245 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.397466 32262 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:24.427239 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48081.7us
I20260809 23:20:24.416029 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:24.429251 24545 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.429314 32432 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.429862 31931 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:24.417995 24320 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260809 23:20:24.431797 24431 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260809 23:20:24.432488 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2980) took 1361 ms. Trace:
I20260809 23:20:24.432881 24431 rpcz_store.cc:276] 0809 23:20:23.071415 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.346694 (+1275279us) service_pool.cc:227] Handling call
0809 23:20:24.381832 (+ 35138us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.432206 (+ 50374us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.432417 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:24.433354 32259 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.009s	sys 0.017s
W20260809 23:20:24.433786 32259 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.009s	sys 0.017s
I20260809 23:20:24.434063 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:24.437364 32126 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.786s	user 0.005s	sys 0.011s
I20260809 23:20:24.437892 32126 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.416232 25369 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:24.439457 24424 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:24.440590 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2982) took 1366 ms. Trace:
I20260809 23:20:24.441071 24424 rpcz_store.cc:276] 0809 23:20:23.074471 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.373197 (+1298726us) service_pool.cc:227] Handling call
0809 23:20:24.439160 (+ 65963us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.440318 (+  1158us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.440525 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.441792 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.443432 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2983) took 1368 ms. Trace:
I20260809 23:20:24.443796 24424 rpcz_store.cc:276] 0809 23:20:23.074792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.442400 (+1367608us) service_pool.cc:227] Handling call
0809 23:20:24.442756 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.443059 (+   303us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.443107 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.443162 (+    55us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:24.443214 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.443276 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.443381 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.424327 32261 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:24.445477 32261 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.419625 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.432350 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.441395 24544 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:24.419204 32402 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:24.467864 24542 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:24.486016 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.488212 25363 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:24.497400 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:24.500517 24543 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:24.503804 32422 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.440624 32302 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.505739 31984 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:24.508453 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:24.509049 32443 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260809 23:20:24.509181 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1259) took 1161 ms. Trace:
I20260809 23:20:24.486137 32259 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:24.509873 24314 rpcz_store.cc:276] 0809 23:20:23.347418 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.302944 (+955526us) service_pool.cc:227] Handling call
0809 23:20:24.303335 (+   391us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.303588 (+   253us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.303613 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:24.304190 (+   577us) log.cc:842] Serialized 26 byte log entry
0809 23:20:24.386290 (+ 82100us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:24.386351 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.386439 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:24.508755 (+122316us) raft_consensus.cc:1671] finished
0809 23:20:24.508790 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.509026 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":81590,"threads_started":1,"wal-append.queue_time_us":202377}
I20260809 23:20:24.510908 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.516819 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:24.518002 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98459.6us
I20260809 23:20:24.519016 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.520020 31661 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:24.520569 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.506699 31984 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260809 23:20:24.522102 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.522320 31661 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:24.523113 31688 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:24.523461 32336 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:24.523887 32336 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:24.524326 32336 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:24.524495 24424 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:20:24.525936 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2984) took 1448 ms. Trace:
I20260809 23:20:24.526481 24424 rpcz_store.cc:276] 0809 23:20:23.077943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.470461 (+1392518us) service_pool.cc:227] Handling call
0809 23:20:24.524044 (+ 53583us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.525486 (+  1442us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.525871 (+   385us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.510278 32259 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.529369 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2543) took 1477 ms. Trace:
I20260809 23:20:24.529482 32240 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.529798 24422 rpcz_store.cc:276] 0809 23:20:23.051779 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.261321 (+1209542us) service_pool.cc:227] Handling call
0809 23:20:24.296710 (+ 35389us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.297194 (+   484us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.297225 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:24.297796 (+   571us) log.cc:842] Serialized 26 byte log entry
0809 23:20:24.398308 (+100512us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:24.398354 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.398443 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:24.529098 (+130655us) raft_consensus.cc:1671] finished
0809 23:20:24.529128 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.529270 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":92823,"threads_started":1,"wal-append.queue_time_us":147968}
I20260809 23:20:24.530783 32259 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.531940 24422 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:24.535650 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1014) took 1415 ms. Trace:
I20260809 23:20:24.535989 24422 rpcz_store.cc:276] 0809 23:20:23.119795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.531165 (+1411370us) service_pool.cc:227] Handling call
0809 23:20:24.531586 (+   421us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.535326 (+  3740us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.535544 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.537294 24422 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:24.537638 24422 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:24.546670 32299 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.532472 32240 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.534839 32432 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.547472 32299 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:24.547993 32299 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.549248 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1264) took 1226 ms. Trace:
I20260809 23:20:24.551257 32126 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.532472 32302 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:24.552219 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49156.6us
I20260809 23:20:24.532886 32331 raft_consensus.cc:2741] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260809 23:20:24.553782 32432 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:24.554694 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85227us
I20260809 23:20:24.555547 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:24.556537 32207 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.556941 32318 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:24.557260 32357 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:24.557936 32318 consensus_queue.cc:1225] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:24.558506 32276 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:24.559043 32239 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:24.564836 24535 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:24.538559 24422 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.559543 32239 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:24.578390 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56378.3us
W20260809 23:20:24.580753 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 197988us
I20260809 23:20:24.583112 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:24.580221 32207 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:24.584447 24422 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:24.552917 32166 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20260809 23:20:24.549702 25371 rpcz_store.cc:276] 0809 23:20:23.322500 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.508982 (+186482us) service_pool.cc:227] Handling call
0809 23:20:23.511815 (+  2833us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:23.655567 (+143752us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:23.853028 (+197461us) log.cc:842] Serialized 14 byte log entry
0809 23:20:24.069525 (+216497us) log.cc:842] Serialized 14 byte log entry
0809 23:20:24.082374 (+ 12849us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:24.082440 (+    66us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:24.092570 (+ 10130us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.545816 (+453246us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x7b1800225700. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:24.547260 (+  1444us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b1800225700. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:20:24.548795 (+  1535us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:24.548855 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:24.549037 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78573,"prepare.queue_time_us":282064,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":1900,"raft.queue_time_us":47277,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7310,"spinlock_wait_cycles":590289792,"thread_start_us":125634,"threads_started":2,"wal-append.queue_time_us":179547}
I20260809 23:20:24.559134 32276 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:24.590214 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:24.512367 32455 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
W20260809 23:20:24.598340 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 200750us
I20260809 23:20:24.576472 31579 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:24.607836 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 276993us
W20260809 23:20:24.610893 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121863us
I20260809 23:20:24.613938 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:24.615618 31688 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:24.617398 32262 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:24.616946 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:24.621743 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:24.623330 32034 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:24.625901 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.627394 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.599642 32302 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:24.595567 32331 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:24.635777 24545 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:24.631753 32034 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:24.638001 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 123389us
I20260809 23:20:24.638113 32207 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.604933 32126 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:24.600853 32231 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:24.642731 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:24.686064 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.648974 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:24.687122 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:24.650012 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:24.635460 31968 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:24.710462 32225 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:24.711040 32467 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:24.617112 32239 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.694918 32262 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:24.720167 32469 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
W20260809 23:20:24.808661 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 100899us
I20260809 23:20:24.833753 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:24.837141 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 190461us
I20260809 23:20:24.838135 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:24.839373 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:24.840569 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 149779us
I20260809 23:20:24.685629 32211 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.161s
I20260809 23:20:24.719321 32126 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:24.732741 32331 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:24.720943 32391 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:24.721953 32262 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:24.742584 31968 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:24.718891 32207 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:24.843822 32417 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:24.846668 32231 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:24.901571 32340 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.730913 32034 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.902225 32283 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260809 23:20:24.903499 32299 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.907907 32456 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:24.906800 32202 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.621s	user 0.004s	sys 0.005s
W20260809 23:20:24.915548 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61442.7us
I20260809 23:20:24.730644 32336 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.864601 31968 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:24.949467 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111458us
W20260809 23:20:24.951364 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 247236us
I20260809 23:20:24.952529 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:24.911749 24545 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:24.915014 32231 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:24.943331 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87046.3us
I20260809 23:20:24.915627 32391 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:24.958065 32202 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:24.911239 31764 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.926787 31579 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:24.964615 32118 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:24.935580 32456 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:24.976503 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:24.978019 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:24.957754 32231 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:24.849490 32276 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:24.984941 32456 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:24.992996 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 376586us
I20260809 23:20:25.004640 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:25.018318 24422 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260809 23:20:25.043716 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 72688.9us
W20260809 23:20:25.040767 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 401872us
I20260809 23:20:24.989934 32469 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:25.097385 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.097844 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2547) took 1747 ms. Trace:
I20260809 23:20:25.100030 24422 rpcz_store.cc:276] 0809 23:20:23.341234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.536712 (+1195478us) service_pool.cc:227] Handling call
0809 23:20:24.537130 (+   418us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.088649 (+551519us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.088902 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:24.991926 32202 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:25.105844 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1246) took 3382 ms (client timeout 2151 ms). Trace:
W20260809 23:20:25.115302 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122785us
I20260809 23:20:25.116451 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:25.106335 24311 rpcz_store.cc:269] 0809 23:20:21.610522 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.611011 (+   489us) service_pool.cc:227] Handling call
0809 23:20:24.981168 (+3370157us) spinlock_profiling.cc:243] Waited 58 ms on lock 0x7b540043bcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:24.993308 (+ 12140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":133339904}
I20260809 23:20:25.119448 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.041203 32471 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260809 23:20:25.121052 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:25.122079 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2823) took 1054 ms. Trace:
I20260809 23:20:25.123204 32475 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:25.123823 32475 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:25.124408 32118 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:25.124881 32118 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:25.125331 32118 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:25.125427 32359 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.751s	user 0.004s	sys 0.008s
I20260809 23:20:25.127148 32207 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260809 23:20:25.128289 32314 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 3.162s	user 0.007s	sys 0.013s
I20260809 23:20:25.130050 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1159) took 3728 ms. Trace:
I20260809 23:20:25.095402 32211 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:25.131218 32211 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.132428 32287 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.133262 24422 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:25.133525 32211 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:25.099323 32262 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.102119 32469 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:25.122459 24544 rpcz_store.cc:276] 0809 23:20:24.067975 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.509503 (+441528us) service_pool.cc:227] Handling call
0809 23:20:24.509888 (+   385us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.510337 (+   449us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.514040 (+  3703us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:24.592125 (+ 78085us) log.cc:842] Serialized 26 byte log entry
0809 23:20:24.989263 (+397138us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:24.989314 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:24.989402 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:25.121826 (+132424us) raft_consensus.cc:1671] finished
0809 23:20:25.121847 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.122008 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":38055,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":374986,"threads_started":1,"wal-append.queue_time_us":514832}
I20260809 23:20:25.139250 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.135728 32469 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:25.124946 32475 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.147742 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:25.130401 24546 rpcz_store.cc:276] 0809 23:20:21.401841 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.438961 (+ 37120us) service_pool.cc:227] Handling call
0809 23:20:21.486564 (+ 47603us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:21.486829 (+   265us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:21.486873 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:21.487394 (+   521us) log.cc:842] Serialized 26 byte log entry
0809 23:20:21.609485 (+122091us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:21.609530 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:21.609616 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:25.127222 (+3517606us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:25.129545 (+  2323us) raft_consensus.cc:1671] finished
0809 23:20:25.129573 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.129864 (+   291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16964224,"thread_start_us":113349,"threads_started":1,"wal-append.queue_time_us":349774}
I20260809 23:20:25.151331 25372 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:25.148936 32211 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:25.157547 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.034032 32126 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.178036 31842 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:25.178351 32465 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.195s
I20260809 23:20:25.179036 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.179293 32465 consensus_queue.cc:1225] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:25.179996 32465 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:25.180433 32465 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:25.180473 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.181270 24546 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:25.181713 32465 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.179168 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:25.183598 25368 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:25.178112 32336 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.184826 32336 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:25.185349 32336 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.178704 31842 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:25.187326 31842 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:25.188040 32336 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.191165 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:25.191785 25371 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:25.156514 32211 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:25.195622 32211 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:25.182211 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:25.200980 24546 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:25.202292 24545 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:25.204234 25375 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:25.149629 32472 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:25.126222 32359 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.225947 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.161677 32302 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:25.232661 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96815.5us
I20260809 23:20:25.232555 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:25.235646 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 134627us
W20260809 23:20:25.220490 32261 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.743s	user 0.010s	sys 0.003s
I20260809 23:20:25.200830 31931 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:25.250977 31931 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:25.251698 24430 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2534) took 2297 ms (client timeout 1733 ms). Trace:
W20260809 23:20:25.252193 24430 rpcz_store.cc:269] 0809 23:20:22.954300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.954470 (+   170us) service_pool.cc:227] Handling call
0809 23:20:25.251268 (+2296798us) spinlock_profiling.cc:243] Waited 87.3 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:25.251486 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200873344}
I20260809 23:20:25.253608 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.254279 24430 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:25.255302 32207 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:25.256034 32466 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.306s	user 0.001s	sys 0.000s
I20260809 23:20:25.255846 32207 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.256574 32245 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.258098 32207 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:25.238761 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.260617 32095 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.240420 24542 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:25.177371 24422 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
W20260809 23:20:25.244963 32177 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260809 23:20:25.249001 32153 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:25.249665 32261 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:25.238950 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.006s	sys 0.006s
W20260809 23:20:25.270124 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.006s	sys 0.006s
I20260809 23:20:25.269724 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2990) took 1637 ms. Trace:
I20260809 23:20:25.278499 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.275908 24545 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:25.279781 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:25.275629 24423 rpcz_store.cc:276] 0809 23:20:23.624735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.893066 (+268331us) service_pool.cc:227] Handling call
0809 23:20:25.250831 (+1357765us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:25.261915 (+ 11084us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":815140608}
I20260809 23:20:25.290482 32326 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:25.290601 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:25.291205 32326 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.291630 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:25.292176 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:25.292723 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:25.293859 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:25.294756 32481 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:20:25.295130 31884 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:25.297196 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2812) took 1579 ms. Trace:
I20260809 23:20:25.296881 31968 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.300032 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.306116 31931 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.307598 32359 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:25.308997 32304 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:25.310155 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2914) took 3810 ms. Trace:
I20260809 23:20:25.165213 32391 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.311789 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.312462 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.313138 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.315732 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.293859 24312 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:25.260490 31764 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:25.324450 24314 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:25.370375 25373 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:25.429029 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:25.370293 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2546) took 2080 ms. Trace:
I20260809 23:20:25.430083 24424 rpcz_store.cc:276] 0809 23:20:23.290604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.535505 (+1244901us) service_pool.cc:227] Handling call
0809 23:20:24.536839 (+  1334us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.537652 (+   813us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:24.992686 (+455034us) log.cc:842] Serialized 14 byte log entry
0809 23:20:25.322439 (+329753us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.322783 (+   344us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:25.369877 (+ 47094us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.369999 (+   122us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.370209 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43588,"prepare.queue_time_us":709876,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":92862,"thread_start_us":620759,"threads_started":2}
I20260809 23:20:25.411928 24320 rpcz_store.cc:276] 0809 23:20:21.499994 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.565672 (+ 65678us) service_pool.cc:227] Handling call
0809 23:20:21.571745 (+  6073us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.430778 (+2859033us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.309806 (+879028us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:25.310079 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6416826240}
W20260809 23:20:25.384166 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2908) took 3900 ms (client timeout 1992 ms). Trace:
I20260809 23:20:25.427938 24423 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:25.432818 24323 rpcz_store.cc:269] 0809 23:20:21.483975 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.484107 (+   132us) service_pool.cc:227] Handling call
0809 23:20:25.311439 (+3827332us) spinlock_profiling.cc:243] Waited 2.74 s on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:25.384070 (+ 72631us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6305006080}
W20260809 23:20:25.474442 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54109.7us
W20260809 23:20:25.476076 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 198977us
W20260809 23:20:25.483449 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94922.7us
I20260809 23:20:25.485558 24545 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:25.486871 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.483139 31884 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:25.492679 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:25.493134 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:25.493535 24320 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:25.494311 24312 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:25.483518 24546 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:25.332796 24534 rpcz_store.cc:276] 0809 23:20:23.717749 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.233902 (+516153us) service_pool.cc:227] Handling call
0809 23:20:24.287980 (+ 54078us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:24.586720 (+298740us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:24.586760 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:24.691905 (+105145us) log.cc:842] Serialized 26 byte log entry
0809 23:20:25.089821 (+397916us) raft_consensus.cc:1631] Marking committed up to 5
0809 23:20:25.089879 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.089968 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:25.296970 (+207002us) raft_consensus.cc:1671] finished
0809 23:20:25.296991 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.297120 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":632,"raft.queue_time_us":145497,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":80,"thread_start_us":368341,"threads_started":2,"wal-append.queue_time_us":463746}
W20260809 23:20:25.522837 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 246450us
I20260809 23:20:25.284135 32302 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.527592 24422 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:25.529037 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 215992us
I20260809 23:20:25.529105 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.532472 24423 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.432278 32493 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:25.532720 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.533437 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.474984 32304 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:25.323076 32471 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:25.530179 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.538903 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.535965 24423 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260809 23:20:25.542670 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 242796us
W20260809 23:20:25.546212 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47572.6us
W20260809 23:20:25.543388 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 502352us
I20260809 23:20:25.539851 24424 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:25.557163 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:25.557929 24423 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:25.541757 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.558797 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.557101 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:25.564234 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1184) took 1408 ms. Trace:
I20260809 23:20:25.564739 24534 rpcz_store.cc:276] 0809 23:20:24.156151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.523082 (+1366931us) service_pool.cc:227] Handling call
0809 23:20:25.544961 (+ 21879us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.553013 (+  8052us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.553056 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.563903 (+ 10847us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:25.563942 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.564009 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.564152 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.573606 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.535458 32493 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:25.546444 32202 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:25.582362 32049 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
W20260809 23:20:25.583407 32494 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260809 23:20:25.528177 31931 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:25.589524 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 2988) took 2246 ms. Trace:
I20260809 23:20:25.590667 24430 rpcz_store.cc:276] 0809 23:20:23.343392 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.535650 (+2192258us) service_pool.cc:227] Handling call
0809 23:20:25.570588 (+ 34938us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.589162 (+ 18574us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.589201 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.589252 (+    51us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:25.589289 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.589352 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.589458 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.592437 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.592868 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.593748 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:25.556128 31339 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:25.586056 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49255.2us
I20260809 23:20:25.570137 32254 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:25.601143 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.602811 24534 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:25.615510 24423 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:25.623188 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66160.2us
I20260809 23:20:25.623386 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:25.626075 31681 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:25.626597 32488 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260809 23:20:25.626709 31681 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:25.629150 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105353us
W20260809 23:20:25.631829 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 101729us
I20260809 23:20:25.632200 24422 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:25.628083 31968 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:25.589277 31764 raft_consensus.cc:2741] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:25.675078 31681 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.675688 31931 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.675551 32493 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.677251 32493 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:25.678087 24320 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:25.678473 31931 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:25.679064 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 202075us
I20260809 23:20:25.684176 32315 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:25.680804 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2550) took 2230 ms. Trace:
I20260809 23:20:25.680073 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1185) took 1481 ms. Trace:
I20260809 23:20:25.680158 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.685523 24422 rpcz_store.cc:276] 0809 23:20:23.450635 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.679820 (+2229185us) service_pool.cc:227] Handling call
0809 23:20:25.680161 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.680417 (+   256us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.680458 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.680532 (+    74us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:25.680584 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.680652 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.680750 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.685972 24545 rpcz_store.cc:276] 0809 23:20:24.198886 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.536113 (+1337227us) service_pool.cc:227] Handling call
0809 23:20:25.536429 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.628663 (+ 92234us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.628703 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.679782 (+ 51079us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:25.679853 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.679914 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.680021 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.545310 32304 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:25.680841 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66497.1us
I20260809 23:20:25.616664 32478 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260809 23:20:25.681675 24423 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:25.682086 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 108588us
I20260809 23:20:25.682643 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:25.623822 32254 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:25.681003 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80009.8us
I20260809 23:20:25.683351 31300 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:25.681054 24424 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260809 23:20:25.726801 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:25.803063 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 270403us
I20260809 23:20:25.568544 32471 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:25.728299 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1338) took 1509 ms (client timeout 1773 ms). Trace:
I20260809 23:20:25.684808 32505 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:25.801990 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.840657 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2552) took 1760 ms. Trace:
I20260809 23:20:25.843052 24423 rpcz_store.cc:276] 0809 23:20:24.075283 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.681175 (+1605892us) service_pool.cc:227] Handling call
0809 23:20:25.681463 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.825576 (+144113us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.825807 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.779428 31968 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:25.684609 32315 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:25.759337 25370 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:25.614159 31579 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:25.561559 32034 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260809 23:20:25.780248 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.633349 32177 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:25.684540 32153 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:25.738458 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.685530 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:25.642494 31842 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.793052 32261 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:25.839859 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2549) took 2388 ms. Trace:
I20260809 23:20:25.842298 24534 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:25.920698 24424 rpcz_store.cc:276] 0809 23:20:23.447647 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.616123 (+2168476us) service_pool.cc:227] Handling call
0809 23:20:25.680789 (+ 64666us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:25.834160 (+153371us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.835904 (+  1744us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:25.842314 25375 rpcz_store.cc:269] 0809 23:20:24.218786 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.218958 (+   172us) service_pool.cc:227] Handling call
0809 23:20:25.728055 (+1509097us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:25.728227 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1340882816}
W20260809 23:20:25.858417 32492 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
W20260809 23:20:25.858788 32491 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260809 23:20:25.825469 31300 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:25.859175 32496 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:25.863497 24423 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
W20260809 23:20:25.803855 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 177947us
I20260809 23:20:25.946025 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:25.836000 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:25.902108 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 219558us
I20260809 23:20:25.825477 32254 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:25.903441 32471 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:25.907105 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66023.6us
I20260809 23:20:25.913937 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:25.916339 32078 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:25.917958 32304 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.951426 32487 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:25.952087 32487 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:25.900128 32505 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:25.952678 32505 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:25.953141 32487 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.938580 24424 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:25.955449 32049 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:25.955498 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2555) took 1851 ms. Trace:
I20260809 23:20:25.956264 32261 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:25.956434 32049 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
W20260809 23:20:25.939107 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 158951us
I20260809 23:20:25.956449 24424 rpcz_store.cc:276] 0809 23:20:24.104531 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.921504 (+1816973us) service_pool.cc:227] Handling call
0809 23:20:25.938353 (+ 16849us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.955229 (+ 16876us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.955435 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.957615 31842 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.958210 32476 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:25.959087 32500 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:25.958878 32476 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.901049 32034 consensus_queue.cc:1225] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:25.958294 24424 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:25.859012 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2553) took 1755 ms. Trace:
I20260809 23:20:25.961762 25375 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:25.961714 24422 rpcz_store.cc:276] 0809 23:20:24.103751 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.857982 (+1754231us) service_pool.cc:227] Handling call
0809 23:20:25.858352 (+   370us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.858579 (+   227us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.858602 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:25.858671 (+    69us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:25.858710 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.858781 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.858913 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:25.942077 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 256716us
I20260809 23:20:25.943475 32486 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:25.964756 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:25.947827 32496 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:25.965883 24316 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:25.966070 32496 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:25.964429 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 137296us
I20260809 23:20:25.948112 24498 leader_election.cc:398] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 10 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11.
I20260809 23:20:25.948446 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2452) took 5838 ms. Trace:
I20260809 23:20:25.943714 31300 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:25.971460 31727 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:25.949329 32302 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:25.972819 31727 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:25.973356 31727 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:25.919379 31968 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.964947 32486 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:25.977443 32486 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:25.978296 31968 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:25.981025 31931 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:25.981638 24422 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:25.982443 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2559) took 1876 ms. Trace:
I20260809 23:20:25.982769 31931 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:25.901224 32315 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:25.948927 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.950591 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2554) took 1846 ms. Trace:
I20260809 23:20:25.985837 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:25.950225 24314 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:25.986388 24534 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:25.949698 32118 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.824s	user 0.011s	sys 0.000s
W20260809 23:20:25.951563 32484 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260809 23:20:25.963593 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.989169 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:25.769958 31339 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.958545 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:25.956892 32261 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:25.971383 32078 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:25.983266 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:25.982771 24422 rpcz_store.cc:276] 0809 23:20:24.105922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.962832 (+1856910us) service_pool.cc:227] Handling call
0809 23:20:25.963162 (+   330us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.982200 (+ 19038us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.982386 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.984436 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2558) took 1856 ms. Trace:
W20260809 23:20:25.944582 32469 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.799s	user 0.011s	sys 0.001s
I20260809 23:20:25.986102 24423 rpcz_store.cc:276] 0809 23:20:24.104205 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.862882 (+1758677us) service_pool.cc:227] Handling call
0809 23:20:25.863265 (+   383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.944013 (+ 80748us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.950527 (+  6514us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.942572 32478 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:25.990490 24424 rpcz_store.cc:276] 0809 23:20:24.105514 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.957689 (+1852175us) service_pool.cc:227] Handling call
0809 23:20:25.958120 (+   431us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.961355 (+  3235us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.961527 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.996042 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:25.996655 24423 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:25.996898 24422 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:25.997514 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2560) took 1800 ms. Trace:
I20260809 23:20:25.997979 24423 rpcz_store.cc:276] 0809 23:20:24.197493 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.990812 (+1793319us) service_pool.cc:227] Handling call
0809 23:20:25.991109 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.997209 (+  6100us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.997447 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:26.004192 24423 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:20:25.974386 31727 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:25.995491 32118 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.005172 32202 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260809 23:20:26.005661 31727 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:25.964989 32212 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:26.006995 32118 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:26.007561 32118 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:26.008275 25368 pending_rounds.cc:76] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:20:26.008836 25368 pending_rounds.cc:97] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260809 23:20:26.009222 25368 raft_consensus.cc:2866] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:26.010406 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:26.007423 32212 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:26.012099 32212 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.012521 32212 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:26.008982 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:25.966194 32476 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:26.024886 24422 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:26.025880 24424 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260809 23:20:26.014446 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:26.030959 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195051us
I20260809 23:20:26.034910 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3003) took 1001 ms. Trace:
I20260809 23:20:26.035604 24424 rpcz_store.cc:276] 0809 23:20:25.033235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.993981 (+960746us) service_pool.cc:227] Handling call
0809 23:20:26.025452 (+ 31471us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.026381 (+   929us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.034805 (+  8424us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:25.983180 31931 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:26.013877 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2808) took 2926 ms (client timeout 1973 ms). Trace:
W20260809 23:20:26.021955 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54719.5us
W20260809 23:20:26.038689 24540 rpcz_store.cc:269] 0809 23:20:23.070414 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.256140 (+185726us) service_pool.cc:227] Handling call
0809 23:20:25.986610 (+2730470us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:25.996374 (+  9764us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":527358720}
I20260809 23:20:26.039156 31931 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.040469 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:25.967985 24321 rpcz_store.cc:276] 0809 23:20:19.957310 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:19.957425 (+   115us) service_pool.cc:227] Handling call
0809 23:20:19.966688 (+  9263us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:19.969135 (+  2447us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:20.888197 (+919062us) log.cc:842] Serialized 14 byte log entry
0809 23:20:21.449770 (+561573us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:24.560740 (+3110970us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:24.561348 (+   608us) log.cc:842] Serialized 26 byte log entry
0809 23:20:24.996009 (+434661us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:25.081158 (+ 85149us) log.cc:842] Serialized 14 byte log entry
0809 23:20:25.222265 (+141107us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:25.242508 (+ 20243us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:25.789699 (+547191us) raft_consensus.cc:1671] finished
0809 23:20:25.789732 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:25.795733 (+  6001us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104918,"prepare.queue_time_us":896653,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":4569,"raft.queue_time_us":203125,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":25796,"spinlock_wait_cycles":25196800,"thread_start_us":1837305,"threads_started":4,"wal-append.queue_time_us":1240600,"wal-append.run_cpu_time_us":2356,"wal-append.run_wall_time_us":1521795}
I20260809 23:20:26.041628 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.049057 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:25.991815 32078 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:26.012933 32304 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:26.047355 24540 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:26.003240 32500 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:26.008347 32118 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.058827 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.062132 24540 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260809 23:20:25.944223 31579 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:26.062587 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82343.1us
I20260809 23:20:26.040469 31931 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.043119 32476 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.077837 32476 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.078719 31579 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.079541 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.080669 32476 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.003651 32469 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.082284 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:26.083735 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.084337 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:26.084942 31884 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:26.085171 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.086908 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:26.087446 24544 raft_consensus.cc:2354] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15.
I20260809 23:20:26.087937 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.088761 25365 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260809 23:20:26.090178 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:26.064927 32401 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260809 23:20:26.067030 31681 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260809 23:20:26.076596 32078 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.097681 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:26.075639 32456 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.968s	user 0.008s	sys 0.003s
I20260809 23:20:26.091379 32513 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:26.097936 32245 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260809 23:20:26.104755 32513 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:26.105233 32513 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:26.105285 32078 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:26.012112 32261 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.068898 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.106719 24534 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260809 23:20:26.066012 32118 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:26.080739 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:26.096765 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.115417 24424 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:26.100893 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.116230 32126 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:26.116756 32126 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:26.117062 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.117257 32126 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:26.115793 24424 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:26.122568 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 164073us
I20260809 23:20:26.123747 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.115131 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:26.101317 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.101873 32401 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:26.103463 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.084003 31339 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:26.102161 32500 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.147140 32309 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:26.149307 31884 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:26.150851 31842 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:26.151306 31579 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:26.155689 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:26.157321 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56105.8us
I20260809 23:20:26.176283 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:26.170101 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73456.1us
I20260809 23:20:26.170403 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1086) took 1538 ms. Trace:
I20260809 23:20:26.015791 32478 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.138557 32304 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:26.173677 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70329.8us
I20260809 23:20:26.182192 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:26.183013 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.136082 32476 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.113871 32469 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.146188 32401 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:26.184012 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:26.184856 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 104180us
I20260809 23:20:26.177569 24423 rpcz_store.cc:276] 0809 23:20:24.681234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.122350 (+1441116us) service_pool.cc:227] Handling call
0809 23:20:26.170282 (+ 47932us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:26.173830 24424 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.112917 32034 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
I20260809 23:20:26.152864 32509 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260809 23:20:26.174224 31470 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:26.203230 24534 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:26.205798 24424 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:26.182389 32245 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:26.185695 31681 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:26.219587 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102620us
I20260809 23:20:26.106654 32456 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.223119 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:26.224857 24423 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:26.172117 32471 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.044s
I20260809 23:20:26.185917 32304 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:26.226050 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:26.204525 31339 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:26.248926 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:26.204355 31842 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:26.255380 31339 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:26.265239 32254 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.247112 32304 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.265419 32509 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:26.281167 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:26.210101 31579 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:26.275271 32401 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.246322 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:26.287834 25363 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:26.206629 32476 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.306035 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2824) took 2098 ms. Trace:
I20260809 23:20:26.246081 31021 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:26.305388 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 260005us
I20260809 23:20:26.308797 24534 rpcz_store.cc:276] 0809 23:20:24.199346 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.228295 (+2028949us) service_pool.cc:227] Handling call
0809 23:20:26.281092 (+ 52797us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.293011 (+ 11919us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:26.293038 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:26.293083 (+    45us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:26.297709 (+  4626us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.297774 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:26.297921 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:26.287047 31339 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.320567 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:26.300941 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 118002us
I20260809 23:20:26.280985 32469 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:20:26.297955 32231 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.304s	user 0.013s	sys 0.001s
W20260809 23:20:26.312769 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84037.7us
I20260809 23:20:26.318102 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:26.303984 32523 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.086s	user 0.002s	sys 0.000s
W20260809 23:20:26.293650 32211 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.096s	user 0.010s	sys 0.005s
I20260809 23:20:26.326112 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:26.335950 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88386.4us
I20260809 23:20:26.367851 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:26.368171 25365 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:26.371582 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:26.372481 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44109.2us
I20260809 23:20:26.325868 31339 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.326797 32509 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:26.295619 31300 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.400771 25371 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260809 23:20:26.378549 24534 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260809 23:20:26.401695 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2825) took 2202 ms. Trace:
I20260809 23:20:26.327792 32469 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:26.372246 24423 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:26.401928 24534 rpcz_store.cc:276] 0809 23:20:24.199718 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.327093 (+2127375us) service_pool.cc:227] Handling call
0809 23:20:26.372797 (+ 45704us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:26.401411 (+ 28614us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.401646 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:26.388971 32509 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.329169 32231 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.338254 32401 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:26.409363 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.409870 25365 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260809 23:20:26.407918 31579 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260809 23:20:26.408200 32509 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:26.416342 32259 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:26.420821 24316 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:26.420109 32153 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:26.433355 32539 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.436846 32254 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.437181 31300 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.336549 32211 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.437813 31579 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:26.440776 32211 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.444545 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:26.445382 25365 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:26.446053 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:26.447700 32433 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:26.448180 32433 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:26.433873 32118 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.467545 24534 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260809 23:20:26.475474 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 288935us
W20260809 23:20:26.489874 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80596.6us
I20260809 23:20:26.455134 32211 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.519469 25371 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:20:26.533206 24422 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
W20260809 23:20:26.533793 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 198969us
I20260809 23:20:26.533967 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:26.500034 32527 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260809 23:20:26.534760 24321 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:26.483031 32456 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:26.536403 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:26.419904 32259 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.429526 32357 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.072s
I20260809 23:20:26.486255 32525 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:26.546105 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:26.553020 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:26.417584 31842 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.434204 32539 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.449611 32433 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.544574 32478 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:26.554075 32476 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:26.569227 32212 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.556s	user 0.008s	sys 0.008s
I20260809 23:20:26.538765 32034 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260809 23:20:26.486418 32469 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.587780 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:26.588768 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1334) took 2457 ms. Trace:
I20260809 23:20:26.624828 24321 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:26.543741 32211 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:26.614418 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0888561711180955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1773s for a response
I20260809 23:20:26.454641 32530 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.162s
I20260809 23:20:26.557344 32486 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260809 23:20:26.624603 25366 rpcz_store.cc:276] 0809 23:20:24.131437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.133724 (+  2287us) service_pool.cc:227] Handling call
0809 23:20:24.170047 (+ 36323us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.228186 (+ 58139us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.588526 (+2360340us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b5400442ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:26.588690 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82638592}
I20260809 23:20:26.608126 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:26.629738 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:26.565623 32525 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.549536 32537 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.035s
I20260809 23:20:26.598547 32548 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:26.584220 32433 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.640115 24424 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260809 23:20:26.656528 24424 pending_rounds.cc:76] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:20:26.656842 24424 pending_rounds.cc:97] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260809 23:20:26.657145 24424 raft_consensus.cc:2866] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:26.655798 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:26.650424 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:26.624733 32471 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:26.668713 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:26.673374 24424 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.645071 32525 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:26.659308 32469 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:26.664363 32548 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:26.661266 32231 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:26.676077 32177 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.004s	sys 0.004s
I20260809 23:20:26.625880 32212 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:26.639662 31579 consensus_queue.cc:1159] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 14 status { last_received { term: 14 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 14 index: 10 } }
I20260809 23:20:26.671895 32471 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:26.695794 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 328059us
W20260809 23:20:26.886665 32177 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.011s	sys 0.029s
I20260809 23:20:26.888268 32177 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:26.676630 32551 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:26.724540 25369 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:26.887195 32471 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:26.819231 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2561) took 1549 ms. Trace:
I20260809 23:20:26.892571 24422 rpcz_store.cc:276] 0809 23:20:25.269996 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.026452 (+756456us) service_pool.cc:227] Handling call
0809 23:20:26.026793 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.538541 (+511748us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.818928 (+280387us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400332cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:26.819160 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1164103040}
I20260809 23:20:26.736157 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2827) took 2270 ms. Trace:
I20260809 23:20:26.894629 24534 rpcz_store.cc:276] 0809 23:20:24.465648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.411215 (+1945567us) service_pool.cc:227] Handling call
0809 23:20:26.428578 (+ 17363us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:26.485860 (+ 57282us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.735833 (+249973us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b54000b5c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:26.736074 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":71713536}
I20260809 23:20:26.900727 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:26.903817 32082 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:26.904518 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:26.905356 24422 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:26.695887 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:26.809609 32524 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260809 23:20:26.755973 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.893s (SENT) [suppressed 25 similar messages]
I20260809 23:20:26.909222 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:26.737685 32340 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
W20260809 23:20:26.914528 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 158593us
I20260809 23:20:26.913197 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:26.915917 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:26.916626 24424 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:26.842306 32537 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.914953 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:26.696658 32118 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.778774 32211 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:26.836141 31967 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:26.917452 24422 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.780529 32259 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:26.926307 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1342) took 2678 ms (client timeout 1915 ms). Trace:
W20260809 23:20:26.931931 25373 rpcz_store.cc:269] 0809 23:20:24.239899 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.362835 (+122936us) service_pool.cc:227] Handling call
0809 23:20:26.918585 (+2555750us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:26.918716 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2314270976}
I20260809 23:20:26.929463 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:26.930378 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:26.781219 31026 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:26.694401 32548 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:26.937503 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:26.906603 32354 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:26.906808 32539 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:26.945375 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.907647 32551 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:26.947584 24541 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:26.944638 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2929) took 3939 ms (client timeout 2189 ms). Trace:
W20260809 23:20:26.951319 24539 rpcz_store.cc:269] 0809 23:20:23.005460 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.279918 (+274458us) service_pool.cc:227] Handling call
0809 23:20:26.920679 (+3640761us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:26.944558 (+ 23879us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":769883904}
I20260809 23:20:26.953091 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.954892 24422 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:26.945264 32357 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:26.956900 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:26.961390 32304 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.006s
I20260809 23:20:26.961802 32304 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:26.942131 32354 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:26.963531 32465 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:20:26.964644 32354 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.965773 32354 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:26.954267 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:26.744102 32456 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.182s
I20260809 23:20:26.961236 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:26.947276 25325 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 1.708s (SENT)
W20260809 23:20:26.931324 32505 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.978s	user 0.004s	sys 0.012s
I20260809 23:20:26.820977 31339 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.973061 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.942428 31967 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:26.970510 32259 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:26.967471 32315 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.960s	user 0.004s	sys 0.017s
I20260809 23:20:27.005264 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:26.969398 24534 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:26.903676 32546 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260809 23:20:27.010416 31967 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.010653 32546 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:26.992269 32259 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:27.011296 32212 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:26.968829 32126 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.851s	user 0.004s	sys 0.013s
I20260809 23:20:26.746335 32509 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.011468 32259 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:27.008947 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:26.991991 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.005940 24314 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:27.011132 32546 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:26.990139 32456 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260809 23:20:27.004762 32401 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.005s
I20260809 23:20:27.016039 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:26.928184 32537 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:27.006973 31931 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.901s	user 0.008s	sys 0.012s
I20260809 23:20:27.045452 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:27.021260 32496 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.053s	user 0.003s	sys 0.014s
I20260809 23:20:27.031154 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.032313 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.033185 24539 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:26.990298 32505 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:27.041765 32501 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.360s	user 0.002s	sys 0.005s
I20260809 23:20:27.057369 24314 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260809 23:20:27.043435 32497 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.359s	user 0.008s	sys 0.016s
I20260809 23:20:27.043550 32175 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Stepping down as leader of term 9
I20260809 23:20:27.020480 24424 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:27.052248 24534 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:27.043187 32095 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:27.061831 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:27.063222 24424 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:27.006433 32315 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.057090 32496 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.063117 32095 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:27.066356 24323 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:27.070586 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55915us
I20260809 23:20:27.073030 24540 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:27.061081 32175 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:27.074630 32500 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:27.075361 32553 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:20:27.075486 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.076253 24323 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:27.075390 32500 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.077139 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.077683 32326 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.077422 32095 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.078522 32500 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.078763 32095 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:27.079247 32212 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:27.079324 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.092180 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:27.080435 32175 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.083377 31832 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:27.084923 32315 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.085048 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1266) took 2706 ms. Trace:
I20260809 23:20:27.072745 32546 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.093200 24534 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:27.093420 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:27.024435 32126 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.102102 25370 rpcz_store.cc:276] 0809 23:20:24.378462 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.394504 (+ 16042us) service_pool.cc:227] Handling call
0809 23:20:25.728855 (+1334351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.919719 (+190864us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.084840 (+1165121us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54007d05d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.084985 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3044163456}
I20260809 23:20:27.105216 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.066649 32568 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:27.105453 32304 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.105993 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:27.103101 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:27.108578 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:27.112836 32262 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:27.120523 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:27.113395 32500 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:27.119369 32530 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260809 23:20:27.122841 32259 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.126492 32509 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.127607 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1276) took 1135 ms. Trace:
I20260809 23:20:27.127760 32259 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.128001 25369 rpcz_store.cc:276] 0809 23:20:25.992697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.992816 (+   119us) service_pool.cc:227] Handling call
0809 23:20:26.007463 (+ 14647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.892160 (+884697us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.127381 (+235221us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:27.127547 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1648522496}
I20260809 23:20:27.132102 32505 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.136586 32126 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.099464 32212 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.139317 25366 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:27.139539 32212 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.105676 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2809) took 3389 ms. Trace:
I20260809 23:20:27.083385 31339 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.143075 24541 rpcz_store.cc:276] 0809 23:20:23.716194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.716819 (+   625us) service_pool.cc:227] Handling call
0809 23:20:26.941677 (+3224858us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.947986 (+  6309us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.105484 (+157498us) spinlock_profiling.cc:243] Waited 3.21 s on lock 0x7b54000eecd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.105618 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7390038912}
I20260809 23:20:27.144886 24321 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:27.116852 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:27.014081 32177 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:27.147531 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44501us
I20260809 23:20:27.121560 32032 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260809 23:20:27.122891 32496 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.117411 32175 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 9 FOLLOWER]: Advancing to term 10
I20260809 23:20:27.146747 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.147607 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:27.155531 24541 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:27.154142 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2939) took 2007 ms (client timeout 1568 ms). Trace:
W20260809 23:20:27.156730 24311 rpcz_store.cc:269] 0809 23:20:25.147018 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.153943 (+2006925us) service_pool.cc:214] Skipping call since client already timed out
0809 23:20:27.154070 (+   127us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:20:27.145488 32304 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:27.154961 32276 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.163761 32193 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:27.164332 32193 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:27.164119 32276 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.164769 32193 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:27.151170 31026 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:27.173301 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.173966 24323 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:27.172415 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.094533 32551 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.175391 24311 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:27.168514 32530 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.159749 32032 consensus_queue.cc:1225] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:27.166209 32505 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:27.177680 32279 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:27.182320 32038 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.183295 32505 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.183121 32038 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.167275 32304 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:27.197530 32276 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:27.212599 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1186) took 2608 ms (client timeout 1708 ms). Trace:
I20260809 23:20:27.214412 32240 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:27.214778 32279 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:27.222146 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 274892us
I20260809 23:20:27.223658 32456 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:27.224612 32521 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.152s	user 0.002s	sys 0.000s
I20260809 23:20:27.226045 32038 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.217713 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.227139 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:27.245841 24544 rpcz_store.cc:269] 0809 23:20:24.605419 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.145263 (+539844us) service_pool.cc:227] Handling call
0809 23:20:27.212338 (+2067075us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.212498 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2135697280}
W20260809 23:20:27.181844 32513 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.076s	user 0.012s	sys 0.007s
W20260809 23:20:27.238622 32469 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.549s	user 0.014s	sys 0.000s
I20260809 23:20:27.267712 24435 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:27.265712 32279 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.268858 31661 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.269668 31661 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.272797 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1290) took 1263 ms. Trace:
I20260809 23:20:27.273247 24312 rpcz_store.cc:276] 0809 23:20:26.009026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.009233 (+   207us) service_pool.cc:227] Handling call
0809 23:20:26.009690 (+   457us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:26.010150 (+   460us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:26.010183 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:26.010766 (+   583us) log.cc:842] Serialized 26 byte log entry
0809 23:20:26.042496 (+ 31730us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:26.042951 (+   455us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.043425 (+   474us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:27.223450 (+1180025us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:27.272390 (+ 48940us) raft_consensus.cc:1671] finished
0809 23:20:27.272413 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:27.272710 (+   297us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1668,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":118,"spinlock_wait_cycles":91776384,"thread_start_us":31253,"threads_started":1,"wal-append.queue_time_us":32437}
I20260809 23:20:27.244237 32401 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:27.238720 32539 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:27.225528 32530 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.239817 32395 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.218762 32304 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:27.278514 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.246068 32240 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:27.263643 32212 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:27.288421 32318 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:27.290109 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.259725 31339 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:27.292412 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.294884 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.298861 25370 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:27.300454 31688 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.265748 32534 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:27.303831 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.307512 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.308300 32537 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.310123 24541 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:27.373952 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:27.373924 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.313642 24312 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:27.371800 24311 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:27.376891 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:27.376799 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.378795 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:27.303877 32469 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.257982 32471 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:27.381080 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 376384us
I20260809 23:20:27.309754 32505 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:27.376840 32525 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.002s
I20260809 23:20:27.389109 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:27.386763 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.375113 32534 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:27.393752 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.302253 32401 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.396906 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.396638 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.392795 32525 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:27.390923 32433 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.729s	user 0.008s	sys 0.004s
I20260809 23:20:27.389428 32551 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:27.384298 32175 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:27.392795 32231 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.404575 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.396171 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:27.370471 32318 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.405685 32240 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.431628 32240 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:27.405679 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.320451 32395 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.406833 32534 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:27.436326 32572 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260809 23:20:27.436498 32534 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:27.438071 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.438601 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.439267 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.439842 25371 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:27.440011 25370 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:20:27.441387 25366 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260809 23:20:27.260040 32513 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.413163 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.417651 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:27.424909 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:27.428754 24424 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14.
W20260809 23:20:27.398247 32517 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.341s	user 0.004s	sys 0.000s
I20260809 23:20:27.377290 31688 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.424840 32175 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:27.414345 32082 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.005s
W20260809 23:20:27.418382 32518 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.318s	user 0.002s	sys 0.003s
W20260809 23:20:27.420331 32571 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.259s	user 0.000s	sys 0.002s
I20260809 23:20:27.447096 24541 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:27.448657 25370 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260809 23:20:27.385900 32469 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.444128 24312 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:27.453629 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.452243 32530 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.461830 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.463909 32341 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:27.464396 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2941) took 2949 ms. Trace:
I20260809 23:20:27.464784 24534 rpcz_store.cc:276] 0809 23:20:24.539661 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.073597 (+2533936us) service_pool.cc:227] Handling call
0809 23:20:27.086603 (+ 13006us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:27.108209 (+ 21606us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.464159 (+355950us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:27.464327 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11608960}
I20260809 23:20:27.466094 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.467224 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.467859 24311 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:27.428699 32551 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:27.427162 32193 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.478627 32513 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.478999 32193 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.481452 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1183) took 3328 ms. Trace:
I20260809 23:20:27.481588 32038 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:27.481869 24546 rpcz_store.cc:276] 0809 23:20:24.152659 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.521045 (+1368386us) service_pool.cc:227] Handling call
0809 23:20:25.530750 (+  9705us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:25.538443 (+  7693us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.538473 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:26.968043 (+1429570us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:26.983053 (+ 15010us) log.cc:842] Serialized 26 byte log entry
0809 23:20:27.045029 (+ 61976us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:27.045063 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.045135 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:27.481164 (+436029us) raft_consensus.cc:1671] finished
0809 23:20:27.481192 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:27.481394 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95996,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":86561,"spinlock_wait_cycles":3844480,"thread_start_us":35173,"threads_started":1,"wal-append.queue_time_us":100725}
W20260809 23:20:27.484769 32354 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.518s	user 0.005s	sys 0.008s
I20260809 23:20:27.486016 24312 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:27.487197 24429 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260809 23:20:27.490506 24534 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:27.453637 31764 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:27.497120 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1286) took 1551 ms. Trace:
I20260809 23:20:27.493832 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:27.494074 24431 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:27.498157 24541 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:27.505927 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:27.497526 24320 rpcz_store.cc:276] 0809 23:20:25.945722 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.945877 (+   155us) service_pool.cc:227] Handling call
0809 23:20:25.946246 (+   369us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:25.967635 (+ 21389us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.967668 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:25.970854 (+  3186us) log.cc:842] Serialized 26 byte log entry
0809 23:20:26.042545 (+ 71691us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:26.042584 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.042651 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:27.397644 (+1354993us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:27.485966 (+ 88322us) raft_consensus.cc:1671] finished
0809 23:20:27.485992 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:27.497052 (+ 11060us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22687,"raft.queue_time_us":21198,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":103,"spinlock_wait_cycles":68700032,"thread_start_us":70872,"threads_started":1,"wal-append.queue_time_us":71280}
I20260809 23:20:27.435163 32231 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:27.505384 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1008) took 4456 ms (client timeout 1593 ms). Trace:
I20260809 23:20:27.520074 24311 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:27.498462 32336 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.546298 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:27.525920 24428 rpcz_store.cc:269] 0809 23:20:23.037234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.048929 (+ 11695us) service_pool.cc:227] Handling call
0809 23:20:27.468918 (+4419989us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54003ebcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.493862 (+ 24944us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1935963392}
I20260809 23:20:27.547210 24311 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:27.539812 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1191) took 2736 ms. Trace:
I20260809 23:20:27.558808 24544 rpcz_store.cc:276] 0809 23:20:24.803760 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.488709 (+2684949us) service_pool.cc:227] Handling call
0809 23:20:27.539153 (+ 50444us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:27.539390 (+   237us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:27.539443 (+    53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:27.539500 (+    57us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:27.539543 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.539609 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:27.539741 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:27.561327 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.478422 32175 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.460804 32328 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.068s
I20260809 23:20:27.564587 24320 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:27.571666 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.572041 24423 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: [8.9-10.10]
I20260809 23:20:27.572600 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:27.660043 25370 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:27.601089 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.744446 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:27.540508 32401 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.765339 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.768064 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:27.773684 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:27.787122 24428 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:27.789381 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:27.571673 31688 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.607596 32328 consensus_queue.cc:1225] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:20:27.743947 32279 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.529549 32513 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.223s
I20260809 23:20:27.573221 32395 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.704279 31764 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.572439 32578 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:27.575920 32469 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.571861 32318 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:27.797152 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2811) took 4080 ms. Trace:
I20260809 23:20:27.585068 32471 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.810429 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.815840 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.846282 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.848328 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.819893 32321 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.781116 32177 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.822310 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.852115 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.852726 24311 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:27.559665 32336 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.834134 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:27.782979 32082 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.797063 32505 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.770998 32577 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:27.843139 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.862254 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:27.873503 31884 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.837878 32328 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:27.850451 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.906862 24320 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:27.879591 32471 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:27.860719 32359 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.890738 32469 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.891235 24425 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:27.891403 32505 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.863575 32525 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:27.903033 32212 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:27.878479 32082 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.836309 32279 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.844152 24543 rpcz_store.cc:276] 0809 23:20:23.717268 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.723245 (+  5977us) service_pool.cc:227] Handling call
0809 23:20:23.723744 (+   499us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.500987 (+777243us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.795254 (+3294267us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:27.797093 (+  1839us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1733665024}
I20260809 23:20:27.938024 24498 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:27.918619 31832 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:27.901870 24311 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:27.912269 31884 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.913218 32493 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:27.864898 32578 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:27.913172 32328 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:27.884290 25370 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:27.864725 32095 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.771s	user 0.006s	sys 0.008s
I20260809 23:20:27.871845 32422 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:27.934505 32577 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:27.851519 32177 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.951020 32193 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.954406 32240 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:27.836536 31764 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.851460 32321 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.799083 32551 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.949892 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:27.954483 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.913391 32513 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.953186 31832 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:27.957561 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 996) took 7726 ms (client timeout 2899 ms). Trace:
W20260809 23:20:27.867615 32548 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.896s	user 0.013s	sys 0.001s
I20260809 23:20:27.937165 24541 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:27.918675 32278 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:27.971510 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:27.944761 32212 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:27.935875 24425 raft_consensus.cc:603] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:27.983966 24320 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:27.938208 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.990049 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2821) took 3949 ms. Trace:
I20260809 23:20:28.082820 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.082620 24542 rpcz_store.cc:276] 0809 23:20:24.040718 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.482318 (+441600us) service_pool.cc:227] Handling call
0809 23:20:24.539437 (+ 57119us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:25.262114 (+722677us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.989782 (+2727668us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54001fd954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:27.989990 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1611476736}
I20260809 23:20:28.083936 24320 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:27.990736 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1289) took 1999 ms. Trace:
I20260809 23:20:28.036248 24428 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:28.084609 24316 rpcz_store.cc:276] 0809 23:20:25.991081 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.991200 (+   119us) service_pool.cc:227] Handling call
0809 23:20:26.013893 (+ 22693us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.421376 (+407483us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:27.990443 (+1569067us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54003868d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:27.990661 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":935317376}
I20260809 23:20:28.088678 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:28.086957 32401 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:28.092814 32276 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:28.091765 32401 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.106894 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:27.992976 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2490) took 1311 ms. Trace:
I20260809 23:20:28.145828 24314 rpcz_store.cc:276] 0809 23:20:26.681669 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.024753 (+343084us) service_pool.cc:227] Handling call
0809 23:20:27.989111 (+964358us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.992881 (+  3770us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11322624}
I20260809 23:20:28.147128 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:28.148175 32175 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.113s	sys 0.171s
I20260809 23:20:28.148371 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:27.933715 32471 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.934717 32469 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:27.918655 32262 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260809 23:20:28.150909 32469 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.980365 32328 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:28.151556 32095 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:27.950659 32359 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.152100 32095 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:28.153198 32095 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.153844 32359 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.154553 32095 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:28.078399 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:28.032516 32279 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.122969 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.029867 32177 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.157754 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.159050 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.031176 32082 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.028170 32505 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:28.084750 24427 rpcz_store.cc:269] 0809 23:20:20.090965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:20.190386 (+ 99421us) service_pool.cc:227] Handling call
0809 23:20:27.771881 (+7581495us) spinlock_profiling.cc:243] Waited 2.73 s on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:27.817290 (+ 45409us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6282833664}
I20260809 23:20:27.994849 32422 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.124945 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:28.129480 32581 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260809 23:20:28.130252 32212 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.131294 31764 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.132606 31851 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Stepping down as leader of term 10
W20260809 23:20:28.133095 32341 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.669s	user 0.009s	sys 0.003s
I20260809 23:20:28.123761 31688 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:28.133935 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2953) took 2110 ms. Trace:
W20260809 23:20:28.191470 32583 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
I20260809 23:20:28.191628 24540 rpcz_store.cc:276] 0809 23:20:26.023849 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.087685 (+ 63836us) service_pool.cc:227] Handling call
0809 23:20:28.133664 (+2045979us) spinlock_profiling.cc:243] Waited 983 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.133857 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2261043584}
I20260809 23:20:28.193050 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.193980 24540 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:28.024757 32321 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.135298 31884 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.137674 32401 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.196552 31817 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.197137 31817 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:27.994484 32193 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.093472 32276 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.191483 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2548) took 4810 ms. Trace:
I20260809 23:20:28.191635 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.198439 24430 rpcz_store.cc:276] 0809 23:20:23.381094 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.623610 (+2242516us) service_pool.cc:227] Handling call
0809 23:20:25.679569 (+ 55959us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:26.053025 (+373456us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:26.053075 (+    50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:26.053133 (+    58us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:26.053177 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.191212 (+2138035us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:28.191255 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:28.191402 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":841399168}
I20260809 23:20:28.199294 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:28.199698 32276 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.199868 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:28.200435 24430 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:28.161923 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 267235us
I20260809 23:20:28.200480 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.201108 32359 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:28.201390 25363 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:28.201625 32359 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.202185 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.155567 32471 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:28.125801 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.204001 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.204859 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:28.162825 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 212903us
I20260809 23:20:28.194041 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:28.206410 25373 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:28.206364 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.164433 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:28.169806 24316 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:28.208367 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:28.161430 24314 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15. Re-sending same reply.
I20260809 23:20:28.188591 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.187206 31851 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:28.199381 24323 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:28.148605 32175 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.114s	sys 0.172s
I20260809 23:20:28.203720 32359 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:27.991549 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.191485 31688 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:28.168859 32505 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:28.215649 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:28.024170 32513 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.081671 32548 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.210367 24311 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:28.210068 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:28.211104 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 402933us
W20260809 23:20:28.177539 32534 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.741s	user 0.007s	sys 0.005s
I20260809 23:20:28.159740 24542 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:28.175621 32318 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.208997 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.218133 24540 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260809 23:20:28.218886 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.219934 24427 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:28.220979 24312 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:28.220863 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.222016 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:28.222278 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.222752 25363 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:28.224185 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.227427 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.227344 31851 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.228689 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:28.229671 25370 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:28.228905 31851 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
W20260809 23:20:28.235301 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2482) took 1603 ms (client timeout 1871 ms). Trace:
W20260809 23:20:28.235749 24321 rpcz_store.cc:269] 0809 23:20:26.632397 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.632669 (+   272us) service_pool.cc:227] Handling call
0809 23:20:28.235109 (+1602440us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.235243 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":840696192}
I20260809 23:20:28.237159 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.237776 24321 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14. Re-sending same reply.
I20260809 23:20:28.239861 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.160977 32328 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.241480 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:28.095759 32278 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.243829 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.211455 32276 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:28.249066 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.254086 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.254693 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.229012 32193 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:28.257889 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.218912 31884 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.259150 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.251271 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.268527 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.027546 32577 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.023598 32578 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.240819 32505 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.256707 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.274470 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.246949 32406 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:28.278149 24540 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:28.284667 24311 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:28.256690 32359 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:28.256383 32262 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:28.290058 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.235368 32318 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.276188 25370 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:28.271780 32534 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.292585 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.296868 24542 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:28.297287 24321 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:28.297698 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2830) took 3349 ms. Trace:
I20260809 23:20:28.298161 24542 rpcz_store.cc:276] 0809 23:20:24.948240 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.279068 (+3330828us) service_pool.cc:227] Handling call
0809 23:20:28.292196 (+ 13128us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.297404 (+  5208us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.297633 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.305599 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.306665 24311 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:28.288367 32175 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.291831 32193 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:28.315266 25373 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:28.336331 31832 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.320261 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:28.337414 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 82878.3us
W20260809 23:20:28.338043 32506 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.663s	user 0.016s	sys 0.012s
I20260809 23:20:28.338533 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.321426 24542 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:28.322706 32078 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:28.322782 32548 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.339363 32078 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:28.339820 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.340161 32422 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.340795 32493 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:28.340353 32078 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.341480 32493 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:28.341866 32078 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:28.324659 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.281414 31884 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.322808 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2904) took 6801 ms. Trace:
I20260809 23:20:28.349527 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.348637 24309 rpcz_store.cc:276] 0809 23:20:21.482621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:21.482769 (+   148us) service_pool.cc:227] Handling call
0809 23:20:21.483097 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:23.446347 (+1963250us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.246774 (+4800427us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:28.284343 (+ 37569us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4410278272}
I20260809 23:20:28.335781 32328 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.302292 32341 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.334095 32278 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.345719 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2831) took 2796 ms. Trace:
I20260809 23:20:28.353626 24542 rpcz_store.cc:276] 0809 23:20:25.548940 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.299334 (+2750394us) service_pool.cc:227] Handling call
0809 23:20:28.321189 (+ 21855us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.345386 (+ 24197us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.345649 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.354367 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.289318 32505 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.355489 25373 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260809 23:20:28.355762 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.322404 32247 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.338385 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.258586 32276 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.358301 25373 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:28.351341 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.332166 32175 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.359301 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.359903 24314 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:28.360742 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.361821 32175 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.373203 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2483) took 1736 ms. Trace:
I20260809 23:20:28.363775 32505 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.357908 32247 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.365443 25363 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:28.366106 32422 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:28.302776 32359 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:28.355508 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:28.356809 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.362591 25373 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:28.347913 32401 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260809 23:20:28.373754 24314 rpcz_store.cc:276] 0809 23:20:26.636941 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.360498 (+1723557us) service_pool.cc:227] Handling call
0809 23:20:28.373144 (+ 12646us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.389670 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:28.379256 24320 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
W20260809 23:20:28.372066 32304 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.006s	user 0.006s	sys 0.006s
I20260809 23:20:28.394253 32304 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.395206 32328 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:28.395587 32328 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:28.395702 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40954 (request call id 1) took 4259 ms. Trace:
I20260809 23:20:28.396411 32304 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:28.397078 24543 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:28.300246 32534 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.398845 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:28.400377 24543 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:28.400446 32534 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.401010 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2832) took 2851 ms. Trace:
I20260809 23:20:28.396471 32328 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.380353 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:28.376379 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:28.378736 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.403805 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:28.404243 32231 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:28.404712 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.405678 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:28.367182 32341 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:28.406888 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1335) took 4275 ms (client timeout 1893 ms). Trace:
W20260809 23:20:28.407328 25372 rpcz_store.cc:269] 0809 23:20:24.131877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.214491 (+ 82614us) service_pool.cc:227] Handling call
0809 23:20:28.406684 (+4192193us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.406794 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1627080064}
I20260809 23:20:28.408228 32341 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.390987 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:28.409042 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2942) took 1147 ms. Trace:
I20260809 23:20:28.409394 24323 rpcz_store.cc:276] 0809 23:20:27.261366 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.380885 (+119519us) service_pool.cc:227] Handling call
0809 23:20:28.408898 (+1028013us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.408990 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1841859840}
I20260809 23:20:28.409910 24314 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260809 23:20:28.410230 24314 pending_rounds.cc:76] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 8
I20260809 23:20:28.410425 24314 pending_rounds.cc:97] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260809 23:20:28.410526 24323 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:28.410652 24314 raft_consensus.cc:2866] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:28.411209 25368 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:28.411514 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2492) took 1697 ms. Trace:
I20260809 23:20:28.416018 32513 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.411192 32594 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.420946 32594 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:28.421586 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1000) took 5400 ms (client timeout 2091 ms). Trace:
I20260809 23:20:28.393522 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3007) took 2234 ms. Trace:
W20260809 23:20:28.422062 24435 rpcz_store.cc:269] 0809 23:20:23.020809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:23.261264 (+240455us) service_pool.cc:227] Handling call
0809 23:20:28.421336 (+5160072us) spinlock_profiling.cc:243] Waited 3.99 s on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.421518 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9174521728}
I20260809 23:20:28.422204 24423 rpcz_store.cc:276] 0809 23:20:26.159479 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.402501 (+243022us) service_pool.cc:227] Handling call
0809 23:20:26.402842 (+   341us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:27.659614 (+1256772us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.393255 (+733641us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540018c254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:28.393460 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2688216448}
I20260809 23:20:28.423306 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:28.424352 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3025) took 1258 ms. Trace:
I20260809 23:20:28.424736 24435 rpcz_store.cc:276] 0809 23:20:27.166232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.423051 (+1256819us) service_pool.cc:227] Handling call
0809 23:20:28.424288 (+  1237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.425774 32594 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.425969 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:28.401711 32534 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260809 23:20:28.426798 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3023) took 1277 ms. Trace:
I20260809 23:20:28.427146 32600 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:28.428383 32600 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:28.428757 32600 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:28.429792 32600 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.427150 24435 rpcz_store.cc:276] 0809 23:20:27.149816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.425728 (+1275912us) service_pool.cc:227] Handling call
0809 23:20:28.426747 (+  1019us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.405994 24309 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:28.435518 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2488) took 1768 ms. Trace:
I20260809 23:20:28.435842 24309 rpcz_store.cc:276] 0809 23:20:26.667304 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.405605 (+1738301us) service_pool.cc:227] Handling call
0809 23:20:28.405831 (+   226us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.435262 (+ 29431us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.435450 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.436192 32318 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.290733 31688 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.436924 32429 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:28.406122 24544 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260809 23:20:28.438486 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2945) took 2854 ms. Trace:
I20260809 23:20:28.438898 24544 rpcz_store.cc:276] 0809 23:20:25.583964 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.405521 (+2821557us) service_pool.cc:227] Handling call
0809 23:20:28.405831 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.438232 (+ 32401us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.438433 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.440135 24309 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:20:28.423396 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:28.441200 24423 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:28.441288 32401 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:28.441772 32401 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:28.442516 24542 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:28.443710 24542 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:28.443575 32359 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.444339 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2948) took 2818 ms. Trace:
I20260809 23:20:28.444697 24542 rpcz_store.cc:276] 0809 23:20:25.626362 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.443278 (+2816916us) service_pool.cc:227] Handling call
0809 23:20:28.443552 (+   274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.444130 (+   578us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.444291 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.445789 24542 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 10. Current term is 11. Ops: []
I20260809 23:20:28.446046 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:28.446327 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2834) took 2676 ms. Trace:
I20260809 23:20:28.431366 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2493) took 1672 ms. Trace:
W20260809 23:20:28.446822 25321 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.087s (SENT)
I20260809 23:20:28.441545 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.447621 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:28.447990 24435 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:28.447465 24323 rpcz_store.cc:276] 0809 23:20:26.759198 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.410073 (+1650875us) service_pool.cc:227] Handling call
0809 23:20:28.410375 (+   302us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.410914 (+   539us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.431309 (+ 20395us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.449268 24323 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260809 23:20:28.449898 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2497) took 1416 ms. Trace:
I20260809 23:20:28.450002 25363 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260809 23:20:28.450209 24323 rpcz_store.cc:276] 0809 23:20:27.033055 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.448720 (+1415665us) service_pool.cc:227] Handling call
0809 23:20:28.449032 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:28.449678 (+   646us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.449840 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.396140 24425 rpcz_store.cc:276] 0809 23:20:24.147447 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.227814 (+ 80367us) service_pool.cc:227] Handling call
0809 23:20:28.395492 (+4167678us) spinlock_profiling.cc:243] Waited 3.63 s on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.395612 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8346218752}
I20260809 23:20:28.452255 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.452402 32240 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:28.452939 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:28.453132 32240 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:28.453645 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3029) took 1002 ms. Trace:
I20260809 23:20:28.404017 32304 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:28.369753 24430 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:28.458853 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1038) took 1071 ms. Trace:
I20260809 23:20:28.377595 32278 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:28.434091 24541 raft_consensus.cc:2396] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260809 23:20:28.452884 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.402871 32328 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:28.440714 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1298) took 1408 ms. Trace:
I20260809 23:20:28.432080 32231 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:28.457113 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 77251.1us
I20260809 23:20:28.706383 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:28.401343 24543 rpcz_store.cc:276] 0809 23:20:25.549269 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.399927 (+2850658us) service_pool.cc:227] Handling call
0809 23:20:28.400200 (+   273us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.400775 (+   575us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.400960 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.454084 24425 rpcz_store.cc:276] 0809 23:20:27.450920 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.452020 (+1001100us) service_pool.cc:227] Handling call
0809 23:20:28.453579 (+  1559us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.920928 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.921753 24425 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260809 23:20:28.922170 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1030) took 2444 ms (client timeout 3160 ms). Trace:
W20260809 23:20:28.922539 24425 rpcz_store.cc:269] 0809 23:20:26.478336 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.920610 (+2442274us) service_pool.cc:227] Handling call
0809 23:20:28.922121 (+  1511us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.459386 24430 rpcz_store.cc:276] 0809 23:20:27.387527 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.199628 (+812101us) service_pool.cc:227] Handling call
0809 23:20:28.458796 (+259168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.152866 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.448954 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:28.581885 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 2998) took 4150 ms (client timeout 1859 ms). Trace:
W20260809 23:20:29.172123 24431 rpcz_store.cc:269] 0809 23:20:24.431448 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.433780 (+  2332us) service_pool.cc:227] Handling call
0809 23:20:28.581676 (+4147896us) spinlock_profiling.cc:243] Waited 3.02 s on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.581804 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6953371392}
W20260809 23:20:29.172457 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3031) took 1657 ms (client timeout 1967 ms). Trace:
I20260809 23:20:29.173273 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.173911 24431 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:28.583552 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3021) took 1851 ms (client timeout 2210 ms). Trace:
W20260809 23:20:29.175117 24428 rpcz_store.cc:269] 0809 23:20:26.732541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.788079 (+1055538us) service_pool.cc:227] Handling call
0809 23:20:28.583375 (+795296us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b540018d154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.583487 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405077760}
I20260809 23:20:29.175941 32212 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:29.176112 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.177300 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.177775 24428 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.178218 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2585) took 1104 ms. Trace:
W20260809 23:20:28.649305 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1029) took 2171 ms (client timeout 2087 ms). Trace:
W20260809 23:20:29.179333 24424 rpcz_store.cc:269] 0809 23:20:26.477733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.096162 (+618429us) service_pool.cc:227] Handling call
0809 23:20:28.649075 (+1552913us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.649227 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":712240896}
I20260809 23:20:29.180385 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.181533 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.182327 24424 raft_consensus.cc:2396] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260809 23:20:29.182610 32574 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.873s	user 0.004s	sys 0.010s
I20260809 23:20:28.411967 24314 rpcz_store.cc:276] 0809 23:20:26.713973 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.409339 (+1695366us) service_pool.cc:227] Handling call
0809 23:20:28.409605 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.411144 (+  1539us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.411341 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.183725 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:28.680048 24323 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:29.185206 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1200) took 1279 ms (client timeout 1650 ms). Trace:
I20260809 23:20:29.185575 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2499) took 2102 ms. Trace:
I20260809 23:20:29.185947 24323 rpcz_store.cc:276] 0809 23:20:27.083413 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.450854 (+1367441us) service_pool.cc:227] Handling call
0809 23:20:28.679828 (+228974us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.185288 (+505460us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.185518 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.187338 24323 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:29.187861 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2500) took 2104 ms. Trace:
I20260809 23:20:29.188144 24323 rpcz_store.cc:276] 0809 23:20:27.083845 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.186670 (+2102825us) service_pool.cc:227] Handling call
0809 23:20:29.187011 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.187650 (+   639us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.187815 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.189379 24323 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:29.189863 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2502) took 2105 ms. Trace:
I20260809 23:20:29.190167 24323 rpcz_store.cc:276] 0809 23:20:27.084569 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.188801 (+2104232us) service_pool.cc:227] Handling call
0809 23:20:29.189065 (+   264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.189662 (+   597us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.189820 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.695674 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.191250 24311 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:28.446651 24542 rpcz_store.cc:276] 0809 23:20:25.769924 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.445378 (+2675454us) service_pool.cc:227] Handling call
0809 23:20:28.445630 (+   252us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.446107 (+   477us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.446262 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.383266 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:28.640591 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2829) took 4016 ms. Trace:
I20260809 23:20:28.639808 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1182) took 4491 ms. Trace:
I20260809 23:20:28.638460 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2942) took 3097 ms. Trace:
I20260809 23:20:28.713259 24543 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20260809 23:20:28.728332 24309 rpcz_store.cc:276] 0809 23:20:27.032421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.436628 (+1404207us) service_pool.cc:227] Handling call
0809 23:20:28.439831 (+  3203us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.440494 (+   663us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.440673 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:28.475417 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.594610 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.473568 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:28.533905 31817 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.392472 31884 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.595996 32034 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:28.484524 32247 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:28.472103 32278 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:28.467145 32304 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:28.442566 32401 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:28.639758 32471 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:28.462457 32508 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.780s	user 0.020s	sys 0.033s
W20260809 23:20:28.653522 32391 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.681s	user 0.009s	sys 0.005s
I20260809 23:20:28.672991 32338 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:28.643980 32566 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.542s	user 0.004s	sys 0.010s
I20260809 23:20:28.713860 32577 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:28.513067 32567 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.498s	user 0.014s	sys 0.013s
I20260809 23:20:28.458779 32587 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:28.464926 32600 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:28.848140 25366 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260809 23:20:28.858686 24423 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:28.923449 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:28.936132 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 2960) took 1488 ms. Trace:
I20260809 23:20:28.952847 32276 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:28.953771 32270 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:28.958772 32354 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:28.972913 32202 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:28.978909 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2959) took 2335 ms (client timeout 2600 ms). Trace:
I20260809 23:20:28.979548 24546 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:28.985635 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:28.541658 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:29.001147 32475 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.052s	user 0.011s	sys 0.007s
I20260809 23:20:29.011332 31857 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:29.019961 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1317) took 2794 ms. Trace:
I20260809 23:20:29.037120 32261 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:20:29.041953 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 963101us
W20260809 23:20:29.049575 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.038s	sys 0.054s
W20260809 23:20:29.051788 24380 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:34762 dropped due to backpressure. The service queue is full; it has 50 items.
I20260809 23:20:29.053704 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.069008 32557 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.072074 32336 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.083184 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2542) took 6049 ms. Trace:
I20260809 23:20:29.101389 32546 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:29.102460 32024 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:29.107713 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1328) took 1738 ms (client timeout 2077 ms). Trace:
I20260809 23:20:29.108338 25371 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:29.125379 31579 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.131598 31661 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.128656 24376 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:34778 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3033)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3024)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3041)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3040)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3043)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3035)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3036)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3042)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2583)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3039)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3032)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1041)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1043)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3045)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3038)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3037)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1042)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3034)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40954 (request call id 2)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3008)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3009)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2568)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2569)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1031)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2570)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2571)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2572)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3016)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1034)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1035)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3018)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1036)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3019)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3022)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2574)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2575)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3026)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3028)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2577)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1039)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3030)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2586)
I20260809 23:20:29.139842 32320 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:29.146545 31790 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:29.148953 32380 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:29.149892 32512 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 3.054s	user 0.007s	sys 0.020s
I20260809 23:20:29.152662 32038 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:28.519409 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:29.157908 24164 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 3 to 7, term changed from 3 to 12, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to <none>, VOTER 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) evicted. New cstate: current_term: 12 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.153486 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:29.168957 32239 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.169072 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.178546 24428 rpcz_store.cc:276] 0809 23:20:28.073482 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.177130 (+1103648us) service_pool.cc:227] Handling call
0809 23:20:29.178171 (+  1041us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.276665 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.276971 32469 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:29.277489 32469 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.277842 32469 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:29.278086 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.278432 31579 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.279189 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.279883 32261 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:29.280248 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:29.185611 24541 rpcz_store.cc:269] 0809 23:20:27.905763 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.078036 (+172273us) service_pool.cc:227] Handling call
0809 23:20:29.185004 (+1106968us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400147654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.185142 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":640145792}
I20260809 23:20:29.280786 32261 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.282122 32261 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:29.282297 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:29.280524 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 738868us
W20260809 23:20:29.283035 32475 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.334s	user 0.011s	sys 0.009s
W20260809 23:20:29.283530 32391 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.311s	user 0.011s	sys 0.005s
I20260809 23:20:29.283257 24535 rpcz_store.cc:276] 0809 23:20:24.148007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.595567 (+447560us) service_pool.cc:227] Handling call
0809 23:20:24.631623 (+ 36056us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:24.637295 (+  5672us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:24.986678 (+349383us) log.cc:842] Serialized 14 byte log entry
0809 23:20:28.639203 (+3652525us) spinlock_profiling.cc:243] Waited 8 ms on lock 0x7b18004d22a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:20:28.639445 (+   242us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:28.639498 (+    53us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:28.639552 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.639642 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:28.639752 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":507460,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":1673,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"spinlock_wait_cycles":18404736,"thread_start_us":512542,"threads_started":2,"wal-append.queue_time_us":239819,"wal-append.run_cpu_time_us":2433,"wal-append.run_wall_time_us":1406836}
W20260809 23:20:29.285169 25321 leader_election.cc:334] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 3.160s (SENT)
I20260809 23:20:29.285504 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.285588 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:29.286793 32034 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:29.288345 24309 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [10.7-12.8]
I20260809 23:20:29.320026 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2944) took 2037 ms. Trace:
I20260809 23:20:29.319685 24164 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 7 to 9, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 9 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:29.320497 24309 rpcz_store.cc:276] 0809 23:20:27.282593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.287744 (+2005151us) service_pool.cc:227] Handling call
0809 23:20:29.288113 (+   369us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:29.319736 (+ 31623us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.319973 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.322184 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.323117 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:29.288932 24543 pending_rounds.cc:76] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 10
I20260809 23:20:29.324265 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.323781 24543 pending_rounds.cc:97] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20260809 23:20:29.325736 24543 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:29.326496 31579 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:29.326687 32550 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.278092 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1296) took 1889 ms. Trace:
W20260809 23:20:29.327636 24378 connection.cc:232] RPC call timeout handler was delayed by -1.1667557989712805s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20260809 23:20:29.327643 32550 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.330590 32550 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.332612 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2837) took 3487 ms. Trace:
I20260809 23:20:29.333153 24543 rpcz_store.cc:276] 0809 23:20:25.845303 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.712265 (+2866962us) service_pool.cc:227] Handling call
0809 23:20:28.712751 (+   486us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.326506 (+613755us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.332555 (+  6049us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.182715 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2580) took 1789 ms. Trace:
I20260809 23:20:29.266247 32225 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:29.335268 24424 rpcz_store.cc:276] 0809 23:20:27.393405 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.181271 (+1787866us) service_pool.cc:227] Handling call
0809 23:20:29.182668 (+  1397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.335475 32225 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.336305 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.336838 24424 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.337644 32225 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.288857 32380 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.339593 32380 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.290133 24425 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
W20260809 23:20:29.340269 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2587) took 1163 ms (client timeout 1503 ms). Trace:
W20260809 23:20:29.340598 24425 rpcz_store.cc:269] 0809 23:20:28.176993 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.923191 (+746198us) service_pool.cc:227] Handling call
0809 23:20:29.340212 (+417021us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.322067 24164 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 11, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.341652 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.342254 24425 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
W20260809 23:20:29.173069 24435 rpcz_store.cc:269] 0809 23:20:27.514686 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.448720 (+934034us) service_pool.cc:227] Handling call
0809 23:20:29.172376 (+723656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.289618 24540 rpcz_store.cc:276] 0809 23:20:25.541277 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.286906 (+2745629us) service_pool.cc:227] Handling call
0809 23:20:28.287252 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.292618 (+  5366us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:28.292651 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:28.292691 (+    40us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:28.292728 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.638178 (+345450us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b54003dca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:28.638232 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:28.638390 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4516,"raft.queue_time_us":1421,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6840,"spinlock_wait_cycles":9129856}
I20260809 23:20:29.343469 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.344283 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3036) took 1102 ms. Trace:
I20260809 23:20:29.344651 24435 rpcz_store.cc:276] 0809 23:20:28.242117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.343229 (+1101112us) service_pool.cc:227] Handling call
0809 23:20:29.344232 (+  1003us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:29.286258 25369 rpcz_store.cc:269] 0809 23:20:27.369759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.387259 (+ 17500us) service_pool.cc:227] Handling call
0809 23:20:29.107462 (+1720203us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.107646 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2457714560}
I20260809 23:20:29.345969 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:29.346727 25369 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:29.347570 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.292415 32577 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.348161 25369 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:29.348308 32577 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:29.349094 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.349680 25369 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:29.350467 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:29.291965 31817 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:29.293444 32068 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:29.351260 32068 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:29.293756 24539 rpcz_store.cc:269] 0809 23:20:26.643060 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.954010 (+310950us) service_pool.cc:227] Handling call
0809 23:20:28.978675 (+2024665us) spinlock_profiling.cc:243] Waited 63.7 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.978834 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146569472}
I20260809 23:20:29.293968 32391 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:29.353291 32068 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.353050 24313 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:29.353888 32032 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.354255 24539 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260809 23:20:29.354362 32032 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.295203 31884 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260809 23:20:29.355027 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.294678 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3033) took 1510 ms. Trace:
I20260809 23:20:29.355930 24435 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:29.289176 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.296984 32354 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:29.355888 24430 rpcz_store.cc:276] 0809 23:20:27.784172 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.152587 (+1368415us) service_pool.cc:227] Handling call
0809 23:20:29.294617 (+142030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.357031 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.356976 32354 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.358065 32354 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.238001 32417 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.342339 24164 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 11, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.360528 24164 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 6 to 13, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 13 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.301086 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.300886 25368 rpcz_store.cc:276] 0809 23:20:26.225584 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.298422 (+ 72838us) service_pool.cc:227] Handling call
0809 23:20:28.411006 (+2112584us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:28.411792 (+   786us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.019607 (+607815us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b540076c5d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.019892 (+   285us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4830889856}
I20260809 23:20:29.283049 24541 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:29.363093 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.363225 24164 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:29.365152 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.365072 24164 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 4 to 13, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:29.369958 24431 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:29.370067 32032 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:29.370646 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3024) took 2212 ms (client timeout 2678 ms). Trace:
W20260809 23:20:29.371031 24431 rpcz_store.cc:269] 0809 23:20:27.158414 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.172995 (+2014581us) service_pool.cc:227] Handling call
0809 23:20:29.370459 (+197464us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.371893 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.372501 31681 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.373147 31681 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.374447 24316 raft_consensus.cc:2396] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260809 23:20:29.375473 32287 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:29.375818 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2496) took 2346 ms (client timeout 2839 ms). Trace:
I20260809 23:20:29.375931 32287 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:29.376256 24316 rpcz_store.cc:269] 0809 23:20:27.029005 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.245488 (+1216483us) service_pool.cc:227] Handling call
0809 23:20:29.375633 (+1130145us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.375767 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2512652160}
I20260809 23:20:29.376906 31681 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.377655 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.378400 24316 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:29.378310 32287 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.302443 32304 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.379002 24422 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
W20260809 23:20:29.379981 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1032) took 2860 ms (client timeout 1852 ms). Trace:
W20260809 23:20:29.380296 24422 rpcz_store.cc:269] 0809 23:20:26.519577 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.956486 (+436909us) service_pool.cc:227] Handling call
0809 23:20:29.379824 (+2423338us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.379922 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5567817984}
I20260809 23:20:29.381268 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.301656 32279 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260809 23:20:29.303459 32270 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.304062 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.304318 31790 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.382598 25373 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:29.383742 31790 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.383950 32270 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.384053 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.384471 25363 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
I20260809 23:20:29.385329 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.385831 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.386554 24435 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:29.299639 24320 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:29.298816 32276 raft_consensus.cc:2741] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:29.387017 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3032) took 1823 ms. Trace:
W20260809 23:20:29.286656 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.040s	sys 0.056s
I20260809 23:20:29.387461 24435 rpcz_store.cc:276] 0809 23:20:27.564990 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.385602 (+1820612us) service_pool.cc:227] Handling call
0809 23:20:29.386952 (+  1350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.300204 32557 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:29.305897 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 136902us
W20260809 23:20:29.306612 32508 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.624s	user 0.021s	sys 0.034s
I20260809 23:20:29.402693 32201 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.423727 32201 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.424762 32557 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.425413 24430 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.425839 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.426193 32201 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:20:29.307531 32278 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:29.307711 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 322101us
I20260809 23:20:29.426899 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.427820 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.311019 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1286) took 2850 ms. Trace:
W20260809 23:20:29.311019 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 791644us
W20260809 23:20:29.311401 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 928285us
I20260809 23:20:29.312415 24312 rpcz_store.cc:276] 0809 23:20:27.448235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.904407 (+456172us) service_pool.cc:227] Handling call
0809 23:20:28.935940 (+1031533us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:28.936066 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":318365440}
I20260809 23:20:29.401471 25369 raft_consensus.cc:2362] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15. Re-sending same reply.
I20260809 23:20:29.403575 24314 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:29.436735 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1303) took 2141 ms. Trace:
I20260809 23:20:29.437072 24314 rpcz_store.cc:276] 0809 23:20:27.294984 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.435965 (+2140981us) service_pool.cc:227] Handling call
0809 23:20:29.436274 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.436449 (+   175us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.436474 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.436512 (+    38us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.436542 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.436587 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.436683 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.439492 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1302) took 2145 ms. Trace:
I20260809 23:20:29.286376 24535 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260809 23:20:29.288872 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3027) took 2106 ms (client timeout 2000 ms). Trace:
I20260809 23:20:29.440063 24424 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:20:29.440440 24423 rpcz_store.cc:269] 0809 23:20:27.182059 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.423173 (+1241114us) service_pool.cc:227] Handling call
0809 23:20:29.288790 (+865617us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.440781 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2588) took 1090 ms. Trace:
I20260809 23:20:29.441139 24424 rpcz_store.cc:276] 0809 23:20:28.350744 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.336062 (+985318us) service_pool.cc:227] Handling call
0809 23:20:29.440728 (+104666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.441380 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:29.441532 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2477) took 4153 ms (client timeout 1949 ms). Trace:
I20260809 23:20:29.315137 32401 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:29.439121 32320 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.448172 24423 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260809 23:20:29.342742 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3035) took 1129 ms. Trace:
I20260809 23:20:29.448978 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.428418 25365 rpcz_store.cc:276] 0809 23:20:26.460579 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.619857 (+159278us) service_pool.cc:227] Handling call
0809 23:20:26.620458 (+   601us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:26.620895 (+   437us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:26.620950 (+    55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:26.622056 (+  1106us) log.cc:842] Serialized 26 byte log entry
0809 23:20:26.775397 (+153341us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:26.775454 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:26.775555 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:28.512406 (+1736851us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:29.310723 (+798317us) raft_consensus.cc:1671] finished
0809 23:20:29.310744 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.310957 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57515,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":2067,"spinlock_wait_cycles":7334656,"thread_start_us":150948,"threads_started":1,"wal-append.queue_time_us":301701}
I20260809 23:20:29.287992 32034 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.355057 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2952) took 3331 ms. Trace:
I20260809 23:20:29.450213 24539 rpcz_store.cc:276] 0809 23:20:26.023471 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.353625 (+3330154us) service_pool.cc:227] Handling call
0809 23:20:29.354013 (+   388us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.354785 (+   772us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.355001 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.292863 32202 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.357043 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.297998 31857 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.359072 32417 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.299160 24380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1087) took 2935 ms. Trace:
I20260809 23:20:29.275614 24323 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:29.376178 24545 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:29.381894 24422 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 10.
I20260809 23:20:29.386338 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:29.286967 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1365) took 1070 ms. Trace:
I20260809 23:20:29.388231 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1279) took 4113 ms. Trace:
I20260809 23:20:29.388571 32254 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:29.286012 24429 rpcz_store.cc:276] 0809 23:20:23.033542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:24.247231 (+1213689us) service_pool.cc:227] Handling call
0809 23:20:27.486868 (+3239637us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:27.494784 (+  7916us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.082571 (+1587787us) spinlock_profiling.cc:243] Waited 3.15 s on lock 0x7b54003ebcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.082724 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7255931264}
I20260809 23:20:29.391383 24544 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:29.370641 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:29.392346 32336 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260809 23:20:29.393257 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62876.4us
I20260809 23:20:29.486126 32278 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:29.486508 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:29.365590 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:29.487205 31984 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.487703 31984 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.488276 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2508) took 2242 ms. Trace:
I20260809 23:20:29.488646 24323 rpcz_store.cc:276] 0809 23:20:27.242149 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.190872 (+1948723us) service_pool.cc:227] Handling call
0809 23:20:29.274450 (+ 83578us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.484011 (+209561us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.484205 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.489056 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.489656 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.403921 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2840) took 3383 ms. Trace:
W20260809 23:20:29.491051 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 125427us
I20260809 23:20:29.490276 24540 rpcz_store.cc:276] 0809 23:20:26.020469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.343539 (+3323070us) service_pool.cc:227] Handling call
0809 23:20:29.343826 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.344034 (+   208us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:29.392730 (+ 48696us) log.cc:842] Serialized 14 byte log entry
0809 23:20:29.403530 (+ 10800us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.403587 (+    57us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.403644 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.403702 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.403832 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":44268,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":602,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":105,"thread_start_us":58841,"threads_started":2,"wal-append.queue_time_us":2416}
I20260809 23:20:29.492512 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2838) took 3511 ms. Trace:
I20260809 23:20:29.365306 24541 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:29.299444 32471 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260809 23:20:29.409780 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1301) took 2128 ms. Trace:
W20260809 23:20:29.495299 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2950) took 3868 ms (client timeout 1701 ms). Trace:
I20260809 23:20:29.422226 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.427914 25375 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.309406 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1192) took 4505 ms. Trace:
I20260809 23:20:29.287249 24534 rpcz_store.cc:276] 0809 23:20:24.624000 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.479162 (+2855162us) service_pool.cc:227] Handling call
0809 23:20:27.479503 (+   341us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:27.491600 (+ 12097us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:28.640350 (+1148750us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b54000b4d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:28.640534 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24721664}
I20260809 23:20:29.483000 25366 rpcz_store.cc:276] 0809 23:20:28.216440 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.367222 (+150782us) service_pool.cc:227] Handling call
0809 23:20:28.847455 (+480233us) spinlock_profiling.cc:243] Waited 68.3 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:28.847706 (+   251us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.286669 (+438963us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.286894 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":157156096}
I20260809 23:20:29.427510 24430 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:29.320744 32231 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.377669 32550 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:29.483043 24380 rpcz_store.cc:276] 0809 23:20:26.364537 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.299109 (+2934572us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260809 23:20:29.327342 25371 rpcz_store.cc:276] 0809 23:20:27.388922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.442818 (+ 53896us) service_pool.cc:227] Handling call
0809 23:20:29.108133 (+1665315us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.276297 (+168164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.277919 (+  1622us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400550850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.278037 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3825499904}
I20260809 23:20:29.443801 24321 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:29.439924 24312 rpcz_store.cc:276] 0809 23:20:27.294504 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.434251 (+2139747us) service_pool.cc:227] Handling call
0809 23:20:29.439005 (+  4754us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.439196 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.439220 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.439261 (+    41us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.439295 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.439345 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.439446 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.445678 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:29.505676 24323 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 9. Current term is 10. Ops: []
W20260809 23:20:29.505995 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60432.2us
I20260809 23:20:29.498896 24164 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 14. New cstate: current_term: 14 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.508031 24427 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260809 23:20:29.291407 25372 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.492861 24543 rpcz_store.cc:276] 0809 23:20:25.981011 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.333977 (+3352966us) service_pool.cc:227] Handling call
0809 23:20:29.334263 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.334470 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:20:29.384928 (+ 50458us) log.cc:842] Serialized 14 byte log entry
0809 23:20:29.492131 (+107203us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.492182 (+    51us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:29.492238 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.492301 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.492462 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":47198,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":157030,"threads_started":2}
I20260809 23:20:29.448355 32320 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.449119 31727 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:29.458077 24423 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:29.461071 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.543927 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.483153 24428 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:29.545342 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2583) took 2045 ms. Trace:
I20260809 23:20:29.511222 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2950) took 2111 ms. Trace:
I20260809 23:20:29.545869 24428 rpcz_store.cc:276] 0809 23:20:27.500088 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.356732 (+1856644us) service_pool.cc:227] Handling call
0809 23:20:29.545246 (+188514us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37888}
I20260809 23:20:29.546224 24323 rpcz_store.cc:276] 0809 23:20:27.399978 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.491185 (+2091207us) service_pool.cc:227] Handling call
0809 23:20:29.504455 (+ 13270us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.510298 (+  5843us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.510978 (+   680us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.455776 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1197) took 3089 ms. Trace:
I20260809 23:20:29.484390 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:29.547153 24539 rpcz_store.cc:276] 0809 23:20:26.366397 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.455017 (+3088620us) service_pool.cc:227] Handling call
0809 23:20:29.455314 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.455493 (+   179us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.455519 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.455555 (+    36us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:29.455584 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.455631 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.455724 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:29.548023 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63738.4us
I20260809 23:20:29.548843 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:29.549608 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 128027us
I20260809 23:20:29.510407 25370 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:29.550127 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.464296 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.551278 24424 raft_consensus.cc:2362] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15. Re-sending same reply.
I20260809 23:20:29.455003 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.482580 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.552425 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.553354 24435 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:29.553468 25369 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:29.483776 25363 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:29.544756 25365 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 9.
I20260809 23:20:29.486670 32278 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:29.488050 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2947) took 3865 ms. Trace:
I20260809 23:20:29.555449 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2485) took 2913 ms. Trace:
I20260809 23:20:29.489720 24429 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:29.561895 24321 rpcz_store.cc:276] 0809 23:20:26.641978 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.361797 (+1719819us) service_pool.cc:227] Handling call
0809 23:20:29.443556 (+1081759us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.505157 (+ 61601us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.555248 (+ 50091us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000dbbd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.555392 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2487069440}
I20260809 23:20:29.416795 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.493165 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2961) took 2849 ms. Trace:
I20260809 23:20:29.419924 24542 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.563769 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.495127 24311 rpcz_store.cc:276] 0809 23:20:27.281453 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.191871 (+1910418us) service_pool.cc:227] Handling call
0809 23:20:29.192204 (+   333us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.192440 (+   236us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.192466 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:29.192886 (+   420us) log.cc:842] Serialized 26 byte log entry
0809 23:20:29.409366 (+216480us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.409421 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.409526 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:29.409588 (+    62us) raft_consensus.cc:1671] finished
0809 23:20:29.409601 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.409725 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":216089,"threads_started":1,"wal-append.queue_time_us":214275}
I20260809 23:20:29.497275 24546 rpcz_store.cc:276] 0809 23:20:24.804122 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:27.483623 (+2679501us) service_pool.cc:227] Handling call
0809 23:20:28.979341 (+1495718us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.308651 (+329310us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.309214 (+   563us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b540029d450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.309341 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3438967168}
W20260809 23:20:29.495769 24545 rpcz_store.cc:269] 0809 23:20:25.627056 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.738109 (+111053us) service_pool.cc:227] Handling call
0809 23:20:29.495127 (+3757018us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b5400080a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.495232 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7948602240}
I20260809 23:20:29.565605 24312 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
W20260809 23:20:29.485837 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 115269us
I20260809 23:20:29.563841 24540 rpcz_store.cc:276] 0809 23:20:26.643882 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.492308 (+2848426us) service_pool.cc:227] Handling call
0809 23:20:29.492590 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.492867 (+   277us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:29.492898 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.492935 (+    37us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:29.492971 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.493024 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.493117 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11207,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4670}
I20260809 23:20:29.496456 24431 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:29.557804 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.567474 24543 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:29.567704 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:29.568755 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:29.503428 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.568833 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:29.569754 24320 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 10.
I20260809 23:20:29.569267 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:29.564634 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:29.565871 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2842) took 2669 ms. Trace:
I20260809 23:20:29.503410 32270 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:29.443329 24535 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:29.485339 24317 rpcz_store.cc:276] 0809 23:20:25.274866 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.280934 (+  6068us) service_pool.cc:227] Handling call
0809 23:20:25.285052 (+  4118us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:25.330035 (+ 44983us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:25.330076 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:28.231565 (+2901489us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b18000488e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:20:28.232062 (+   497us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:28.232752 (+   690us) log.cc:842] Serialized 52 byte log entry
0809 23:20:29.387820 (+1155068us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:29.387871 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.387946 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:29.387993 (+    47us) raft_consensus.cc:1671] finished
0809 23:20:29.388007 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.388176 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61278,"raft.queue_time_us":44645,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":48737,"spinlock_wait_cycles":214023808,"thread_start_us":1154575,"threads_started":1,"wal-append.queue_time_us":253601}
I20260809 23:20:29.509099 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1309) took 1106 ms. Trace:
I20260809 23:20:29.563376 24309 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:29.572093 25372 rpcz_store.cc:276] 0809 23:20:28.403051 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.408166 (+  5115us) service_pool.cc:227] Handling call
0809 23:20:28.594426 (+186260us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:20:29.509040 (+914614us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98301824}
I20260809 23:20:29.449167 24425 rpcz_store.cc:276] 0809 23:20:28.213217 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.341390 (+1128173us) service_pool.cc:227] Handling call
0809 23:20:29.342693 (+  1303us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.573169 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:29.482333 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.571076 24545 rpcz_store.cc:276] 0809 23:20:26.896122 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.565134 (+2669012us) service_pool.cc:227] Handling call
0809 23:20:29.565412 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.565584 (+   172us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.565609 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.565648 (+    39us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:29.565681 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.565730 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.565820 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.573948 25371 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.567567 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.574688 24545 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:29.484258 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3039) took 1200 ms. Trace:
I20260809 23:20:29.454788 32276 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:29.454638 32201 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.103s
I20260809 23:20:29.558286 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.561717 24544 rpcz_store.cc:276] 0809 23:20:25.622737 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.439637 (+2816900us) service_pool.cc:227] Handling call
0809 23:20:28.439963 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.486130 (+1046167us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.487822 (+  1692us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:29.487993 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187354368}
I20260809 23:20:29.511531 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:29.442191 24313 rpcz_store.cc:269] 0809 23:20:25.288390 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:25.293270 (+  4880us) service_pool.cc:227] Handling call
0809 23:20:29.441321 (+4148051us) spinlock_profiling.cc:243] Waited 4.03 s on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.441457 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9259391744}
I20260809 23:20:29.564400 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.577768 24323 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260809 23:20:29.578138 24542 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260809 23:20:29.486840 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.562274 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:29.504040 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1280) took 3480 ms (client timeout 2072 ms). Trace:
I20260809 23:20:29.578989 24543 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260809 23:20:29.578802 25375 rpcz_store.cc:269] 0809 23:20:26.024033 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:26.024183 (+   150us) service_pool.cc:227] Handling call
0809 23:20:29.503836 (+3479653us) spinlock_profiling.cc:243] Waited 3.38 s on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.503965 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7762700032}
I20260809 23:20:29.579320 24323 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260809 23:20:29.571043 24320 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:29.580013 24313 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:20:29.581758 24323 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260809 23:20:29.582803 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2846) took 2393 ms. Trace:
I20260809 23:20:29.580109 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1305) took 1895 ms. Trace:
I20260809 23:20:29.499737 31984 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:29.454867 32034 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:29.581032 24430 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:29.576422 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.581221 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1195) took 3522 ms. Trace:
I20260809 23:20:29.582918 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:29.583856 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2579) took 2208 ms (client timeout 1697 ms). Trace:
I20260809 23:20:29.494715 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2954) took 3439 ms. Trace:
I20260809 23:20:29.583758 24320 rpcz_store.cc:276] 0809 23:20:27.685041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.570441 (+1885400us) service_pool.cc:227] Handling call
0809 23:20:29.570725 (+   284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.579866 (+  9141us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.580031 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.318790 32469 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.575538 24422 rpcz_store.cc:276] 0809 23:20:28.284562 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.381038 (+1096476us) service_pool.cc:227] Handling call
0809 23:20:29.484199 (+103161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.584926 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1353) took 2008 ms. Trace:
I20260809 23:20:29.663034 24317 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260809 23:20:29.663646 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2968) took 1687 ms. Trace:
I20260809 23:20:29.571864 32202 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.585829 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:29.668989 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 83239.5us
I20260809 23:20:29.669539 25363 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:29.669692 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:29.502650 32254 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.587505 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2951) took 2175 ms. Trace:
I20260809 23:20:29.672384 24312 rpcz_store.cc:276] 0809 23:20:27.400295 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.505636 (+2105341us) service_pool.cc:227] Handling call
0809 23:20:29.565225 (+ 59589us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.575654 (+ 10429us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.575863 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.505584 24380 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:34762 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3037)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1042)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3049)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3034)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40954 (request call id 2)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3008)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3009)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2568)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2569)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1031)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2570)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2571)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2572)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3016)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1034)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1035)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3018)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1036)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3019)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3022)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2574)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2575)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3026)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3028)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2577)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2578)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1039)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1040)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3030)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3047)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3048)
I20260809 23:20:29.580821 32270 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:29.582837 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 96066.3us
I20260809 23:20:29.545689 31681 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:29.590260 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2843) took 2684 ms. Trace:
I20260809 23:20:29.586817 32024 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:29.590627 24311 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:29.440528 32577 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.591426 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.680709 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 2956) took 2278 ms. Trace:
I20260809 23:20:29.592065 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.506242 31790 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.589582 24323 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:29.593418 24542 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260809 23:20:29.656541 24541 rpcz_store.cc:276] 0809 23:20:26.055529 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.364847 (+3309318us) service_pool.cc:227] Handling call
0809 23:20:29.365143 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.494383 (+129240us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.494578 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.558712 31817 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:29.595306 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.658253 24320 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:20:29.563717 32032 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:29.659236 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.660831 31984 raft_consensus.cc:2741] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260809 23:20:29.670207 32276 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:29.654804 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92621.6us
I20260809 23:20:29.663956 24317 rpcz_store.cc:276] 0809 23:20:27.976142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.571905 (+1595763us) service_pool.cc:227] Handling call
0809 23:20:29.660417 (+ 88512us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.663407 (+  2990us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.663593 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.719767 25368 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:29.720006 24164 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:29.662571 25370 rpcz_store.cc:276] 0809 23:20:27.576682 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.314190 (+737508us) service_pool.cc:227] Handling call
0809 23:20:29.503986 (+1189796us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.550359 (+ 46373us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.584750 (+ 34391us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54007053d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.584871 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2734048128}
I20260809 23:20:29.722748 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.588654 32068 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.099s
W20260809 23:20:29.664513 24427 rpcz_store.cc:269] 0809 23:20:27.375789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:28.239167 (+863378us) service_pool.cc:227] Handling call
0809 23:20:29.579058 (+1339891us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:29.583789 (+  4731us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2849708928}
I20260809 23:20:29.666666 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3038) took 1390 ms. Trace:
I20260809 23:20:29.727510 24430 rpcz_store.cc:276] 0809 23:20:28.277943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.496908 (+1218965us) service_pool.cc:227] Handling call
0809 23:20:29.666604 (+169696us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.729161 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.593910 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1198) took 2698 ms. Trace:
I20260809 23:20:29.731282 24546 rpcz_store.cc:276] 0809 23:20:26.895490 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.565034 (+2669544us) service_pool.cc:227] Handling call
0809 23:20:29.593306 (+ 28272us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.593532 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.593557 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.593612 (+    55us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:29.593643 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.593693 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.593811 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.733791 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1206) took 1018 ms. Trace:
I20260809 23:20:29.583657 24543 rpcz_store.cc:276] 0809 23:20:27.189125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.578413 (+2389288us) service_pool.cc:227] Handling call
0809 23:20:29.578670 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.579499 (+   829us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.582723 (+  3224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.735988 24543 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:29.593896 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1308) took 1216 ms. Trace:
I20260809 23:20:29.560257 32550 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:29.571431 32239 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:29.667629 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:29.580590 31727 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:29.662585 24535 rpcz_store.cc:276] 0809 23:20:26.058767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.442789 (+3384022us) service_pool.cc:227] Handling call
0809 23:20:29.443156 (+   367us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.580978 (+137822us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.581164 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.588411 25375 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [13.11-14.12]
I20260809 23:20:29.674571 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.684585 32270 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:29.750093 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1334) took 1181 ms. Trace:
I20260809 23:20:29.750466 25375 rpcz_store.cc:276] 0809 23:20:28.568627 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.587833 (+1019206us) service_pool.cc:227] Handling call
0809 23:20:29.588166 (+   333us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:29.749796 (+161630us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.750032 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.751495 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.581805 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:29.679593 24545 rpcz_store.cc:276] 0809 23:20:26.899798 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.574102 (+2674304us) service_pool.cc:227] Handling call
0809 23:20:29.574454 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.584268 (+  9814us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.584408 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.681171 24311 rpcz_store.cc:276] 0809 23:20:27.402072 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.564724 (+2162652us) service_pool.cc:227] Handling call
0809 23:20:29.590361 (+ 25637us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.680444 (+ 90083us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.680647 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.590536 32225 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:29.684131 24313 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:29.697202 32546 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.681038 24321 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:29.680029 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:29.590548 32212 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:29.588197 32391 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:29.689535 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:29.660707 24534 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260809 23:20:29.685518 24542 pending_rounds.cc:76] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 9
I20260809 23:20:29.505899 32231 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.577451 32401 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.106s
I20260809 23:20:29.769883 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.770689 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:29.594326 32508 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:29.778998 32508 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:29.779040 24150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 141457us
I20260809 23:20:29.779510 32508 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:29.780336 24539 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260809 23:20:29.780385 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:29.557981 32304 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Stepping down as leader of term 10
I20260809 23:20:29.781085 24427 raft_consensus.cc:2362] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14. Re-sending same reply.
I20260809 23:20:29.781172 32304 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:29.781976 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.782291 32276 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:29.726719 25363 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260809 23:20:29.737155 24316 rpcz_store.cc:276] 0809 23:20:28.377498 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.574438 (+1196940us) service_pool.cc:227] Handling call
0809 23:20:29.593294 (+ 18856us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.593499 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.593532 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.593586 (+    54us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.593621 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.593681 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.593814 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.684765 32299 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:29.690076 31817 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.744182 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1208) took 1027 ms. Trace:
I20260809 23:20:29.730589 24430 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:20:29.734217 24546 rpcz_store.cc:276] 0809 23:20:28.715687 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.732798 (+1017111us) service_pool.cc:227] Handling call
0809 23:20:29.733173 (+   375us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.733406 (+   233us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.733440 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.733499 (+    59us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:29.733543 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.733607 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.733728 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.746883 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.588920 32471 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.101s
I20260809 23:20:29.679538 31681 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.675495 25369 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260809 23:20:29.753165 24545 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260809 23:20:29.754741 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80277.8us
I20260809 23:20:29.686506 25365 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260809 23:20:29.752300 24428 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:29.758541 32336 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:29.762746 25366 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:29.765098 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.765374 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:29.811467 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46186.8us
I20260809 23:20:29.694895 32032 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:29.766898 24534 pending_rounds.cc:76] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:29.817766 24534 pending_rounds.cc:97] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260809 23:20:29.766307 24323 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:29.767208 24542 pending_rounds.cc:97] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260809 23:20:29.770315 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.810140 24424 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:29.818567 24542 raft_consensus.cc:2866] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:29.757869 32225 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:29.775698 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08001d36a0 after lost signal to thread 22536
W20260809 23:20:29.821765 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800329a40 after lost signal to thread 24220
I20260809 23:20:29.818879 24429 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:29.822235 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800320a60 after lost signal to thread 24244
I20260809 23:20:29.694169 31790 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.824751 24534 raft_consensus.cc:2866] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:29.824895 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:29.825436 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08001df220 after lost signal to thread 24355
I20260809 23:20:29.777794 32231 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:29.778191 32391 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.780862 24539 pending_rounds.cc:76] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:20:29.839562 24539 pending_rounds.cc:97] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 13.12
W20260809 23:20:29.783761 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 103802us
I20260809 23:20:29.839282 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2964) took 2834 ms. Trace:
I20260809 23:20:29.840276 24534 rpcz_store.cc:276] 0809 23:20:27.004662 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.575495 (+2570833us) service_pool.cc:227] Handling call
0809 23:20:29.660417 (+ 84922us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.829612 (+169195us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.839216 (+  9604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.790468 24427 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:29.839890 24539 raft_consensus.cc:2866] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:29.749075 31727 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.800334 32380 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:29.767669 32587 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:29.809180 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.802070 31817 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.808972 32401 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.816030 32212 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:29.817332 24428 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:20:29.877434 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3009) took 3717 ms. Trace:
I20260809 23:20:29.839429 31984 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:29.800276 24543 rpcz_store.cc:276] 0809 23:20:28.716284 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.735392 (+1019108us) service_pool.cc:227] Handling call
0809 23:20:29.735771 (+   379us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.743887 (+  8116us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.744114 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.776847 24544 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260809 23:20:29.790236 32299 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.827594 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.827620 24435 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:29.830478 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.829905 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2973) took 1944 ms. Trace:
I20260809 23:20:29.908344 24542 rpcz_store.cc:276] 0809 23:20:27.885631 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.592815 (+1707184us) service_pool.cc:227] Handling call
0809 23:20:29.593103 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.825319 (+232216us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.829843 (+  4524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.846155 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3008) took 3686 ms. Trace:
I20260809 23:20:29.909430 24427 rpcz_store.cc:276] 0809 23:20:26.159792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.782851 (+3623059us) service_pool.cc:227] Handling call
0809 23:20:29.784560 (+  1709us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.841245 (+ 56685us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.846092 (+  4847us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.848860 24424 pending_rounds.cc:76] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 9
I20260809 23:20:29.910792 24424 pending_rounds.cc:97] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260809 23:20:29.911340 24424 raft_consensus.cc:2866] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:29.782378 32304 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.776124 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:29.851058 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800167b60 after lost signal to thread 24469
I20260809 23:20:29.809935 31681 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.856415 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.823604 32032 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:29.864398 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:29.917241 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3034) took 1829 ms. Trace:
I20260809 23:20:29.857584 32380 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.825241 32225 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:29.757941 32618 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260809 23:20:29.785884 32407 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260809 23:20:29.874627 25372 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 13.
I20260809 23:20:29.748339 32278 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.840773 31579 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:29.877787 24428 rpcz_store.cc:276] 0809 23:20:26.159976 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.810350 (+3650374us) service_pool.cc:227] Handling call
0809 23:20:29.817078 (+  6728us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.877152 (+ 60074us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.877376 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.819476 24430 raft_consensus.cc:595] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:29.945895 24430 raft_consensus.cc:603] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:20:29.826743 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87010.8us
I20260809 23:20:29.947803 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:29.773700 32546 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.839272 32557 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:29.856549 31727 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:29.907495 24423 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 11.
W20260809 23:20:29.914486 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08003a7260 after lost signal to thread 24579
I20260809 23:20:29.916283 24422 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260809 23:20:29.917594 24425 rpcz_store.cc:276] 0809 23:20:28.087483 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.830952 (+1743469us) service_pool.cc:227] Handling call
0809 23:20:29.917170 (+ 86218us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:29.866811 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 283946us
W20260809 23:20:29.958328 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080039a1c0 after lost signal to thread 24580
I20260809 23:20:29.923677 32380 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:29.965713 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080032e8e0 after lost signal to thread 25410
I20260809 23:20:29.856909 32417 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:29.875986 32354 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.002s
I20260809 23:20:29.858322 32320 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.906524 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2977) took 1555 ms. Trace:
I20260809 23:20:29.863579 32336 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.064s
W20260809 23:20:29.919713 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 250092us
I20260809 23:20:29.976965 24541 rpcz_store.cc:276] 0809 23:20:28.351437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.686232 (+1334795us) service_pool.cc:227] Handling call
0809 23:20:29.740493 (+ 54261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.906049 (+165556us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.906098 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.906157 (+    59us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:29.906212 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.906294 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:29.906450 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":211135,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11552,"thread_start_us":163944,"threads_started":1}
I20260809 23:20:29.983386 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:29.880595 32024 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:29.866122 32587 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:29.931079 24427 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:29.931460 24431 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:29.930327 24435 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:29.985641 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2570) took 3500 ms. Trace:
I20260809 23:20:29.954581 32557 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:29.985975 24435 rpcz_store.cc:276] 0809 23:20:26.484785 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.907834 (+3423049us) service_pool.cc:227] Handling call
0809 23:20:29.925592 (+ 17758us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.985419 (+ 59827us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.985592 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:29.935598 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105168us
I20260809 23:20:29.988708 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.856127 32321 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260809 23:20:29.950071 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:29.956508 31661 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:29.954124 24535 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:29.955003 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3037) took 1699 ms. Trace:
I20260809 23:20:29.955446 32469 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:29.960425 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:30.000345 22539 debug-util.cc:398] Leaking SignalData structure 0x7b0800179dc0 after lost signal to thread 24498
I20260809 23:20:29.954803 32201 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:29.973549 24428 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:29.992961 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 152110us
I20260809 23:20:30.001389 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2572) took 3485 ms. Trace:
I20260809 23:20:29.979938 24422 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260809 23:20:30.002122 24422 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 9
I20260809 23:20:29.880525 32401 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.976038 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:29.959666 32475 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:29.953711 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2845) took 2838 ms. Trace:
I20260809 23:20:30.007098 24544 rpcz_store.cc:276] 0809 23:20:27.115389 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.576213 (+2460824us) service_pool.cc:227] Handling call
0809 23:20:29.659151 (+ 82938us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.905105 (+245954us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.953356 (+ 48251us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:29.953634 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269756672}
I20260809 23:20:29.962779 32577 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:29.990203 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.013875 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:29.982306 32471 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:29.874758 32550 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.000409 24542 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:29.847496 31857 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.003405 24543 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:29.993059 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2571) took 3477 ms. Trace:
I20260809 23:20:30.062077 24427 rpcz_store.cc:276] 0809 23:20:26.515387 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.910367 (+3394980us) service_pool.cc:227] Handling call
0809 23:20:29.928157 (+ 17790us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.992765 (+ 64608us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.992991 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.994781 24423 rpcz_store.cc:276] 0809 23:20:28.256152 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.827323 (+1571171us) service_pool.cc:227] Handling call
0809 23:20:29.954946 (+127623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.001724 24428 rpcz_store.cc:276] 0809 23:20:26.515884 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.945709 (+3429825us) service_pool.cc:227] Handling call
0809 23:20:29.973309 (+ 27600us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.001115 (+ 27806us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.001318 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.002416 24422 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260809 23:20:30.064183 24422 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:30.064960 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1034) took 3506 ms. Trace:
I20260809 23:20:30.065287 24422 rpcz_store.cc:276] 0809 23:20:26.558246 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.964377 (+3406131us) service_pool.cc:227] Handling call
0809 23:20:29.979472 (+ 15095us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.064697 (+ 85225us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.064899 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.993429 32557 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:29.992748 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1031) took 3514 ms. Trace:
I20260809 23:20:30.065779 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.066316 24431 rpcz_store.cc:276] 0809 23:20:26.478734 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.906978 (+3428244us) service_pool.cc:227] Handling call
0809 23:20:29.931242 (+ 24264us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.992427 (+ 61185us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:29.992679 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:29.877939 32276 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:29.995388 32287 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:30.004742 32201 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:29.960093 31884 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260809 23:20:29.959218 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:30.103098 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 144002us
I20260809 23:20:30.060601 32321 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
I20260809 23:20:30.006546 32475 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.026475 31984 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.028438 32299 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:30.126116 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.032877 24430 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:30.034394 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1366) took 1324 ms. Trace:
I20260809 23:20:30.126817 25372 rpcz_store.cc:276] 0809 23:20:28.710460 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.595119 (+884659us) service_pool.cc:227] Handling call
0809 23:20:30.034164 (+439045us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54004a5bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:30.034325 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306939648}
I20260809 23:20:30.047752 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1305) took 1676 ms. Trace:
I20260809 23:20:30.127758 25373 rpcz_store.cc:276] 0809 23:20:28.371248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.593977 (+1222729us) service_pool.cc:227] Handling call
0809 23:20:29.679869 (+ 85892us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.957348 (+277479us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.957379 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:29.957414 (+    35us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:29.957450 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.047449 (+ 89999us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54003dfed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:30.047515 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.047684 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":637645696}
I20260809 23:20:30.049074 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1209) took 1332 ms. Trace:
I20260809 23:20:30.051213 32639 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:30.052094 32254 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.056284 32338 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:30.056914 32640 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260809 23:20:30.003978 32625 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260809 23:20:29.975781 32068 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
W20260809 23:20:29.994309 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44340.7us
I20260809 23:20:30.016023 32634 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:30.071452 24422 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:20:30.046582 32546 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.079114 24431 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:30.014443 32618 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.005818 24435 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260809 23:20:30.240897 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2574) took 3153 ms. Trace:
I20260809 23:20:30.241097 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1035) took 3665 ms. Trace:
I20260809 23:20:30.019565 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.241328 24431 rpcz_store.cc:276] 0809 23:20:27.087688 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.071973 (+2984285us) service_pool.cc:227] Handling call
0809 23:20:30.078906 (+  6933us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.239887 (+160981us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.240841 (+   954us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.057556 32641 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.105513 24427 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cec68a6991264e72a23a8750a420795d"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:40954
W20260809 23:20:30.242274 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 227065us
I20260809 23:20:30.121757 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.063831 32471 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.152470 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3018) took 3566 ms. Trace:
I20260809 23:20:30.157657 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:30.257395 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 99863.7us
I20260809 23:20:30.258260 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.259245 24430 rpcz_store.cc:276] 0809 23:20:26.585709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.032256 (+3446547us) service_pool.cc:227] Handling call
0809 23:20:30.032636 (+   380us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.152126 (+119490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.152400 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.254148 24546 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:30.158262 32279 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:30.017026 32024 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.231293 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3022) took 3160 ms. Trace:
I20260809 23:20:30.231678 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:30.233415 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93007.4us
I20260809 23:20:30.129345 24535 rpcz_store.cc:276] 0809 23:20:28.716578 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.749285 (+1032707us) service_pool.cc:227] Handling call
0809 23:20:29.953655 (+204370us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.994713 (+ 41058us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.048840 (+ 54127us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400115650. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:30.049011 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":469270912}
I20260809 23:20:30.129657 32475 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:30.242616 24427 raft_consensus.cc:595] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.068410 32231 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.103s
I20260809 23:20:30.326910 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2575) took 3231 ms. Trace:
I20260809 23:20:30.328596 24431 rpcz_store.cc:276] 0809 23:20:27.095594 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.242214 (+3146620us) service_pool.cc:227] Handling call
0809 23:20:30.326295 (+ 84081us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.326528 (+   233us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:30.326564 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:30.326619 (+    55us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:30.326653 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.326708 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.326828 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.329566 24422 rpcz_store.cc:276] 0809 23:20:27.071117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.070455 (+2999338us) service_pool.cc:227] Handling call
0809 23:20:30.070975 (+   520us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.230998 (+160023us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.231236 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.254293 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:30.334542 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 331317us
I20260809 23:20:30.335378 32212 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
W20260809 23:20:30.335873 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 264504us
W20260809 23:20:30.336598 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82395.8us
I20260809 23:20:30.337926 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:30.261699 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:30.338001 25321 leader_election.cc:334] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.852s (SENT)
I20260809 23:20:30.259660 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:30.339488 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79930us
I20260809 23:20:30.340137 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:30.341058 32577 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:30.341456 32577 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:30.341825 32577 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:30.158252 32338 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:30.171607 32640 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.173125 31790 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:30.333832 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 339064us
I20260809 23:20:30.333915 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1307) took 2130 ms. Trace:
W20260809 23:20:30.168859 25371 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.594s	user 0.014s	sys 0.000s
I20260809 23:20:30.241425 24435 rpcz_store.cc:276] 0809 23:20:26.576079 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.987054 (+3410975us) service_pool.cc:227] Handling call
0809 23:20:30.005513 (+ 18459us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:30.240814 (+235301us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.241047 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.328179 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1304) took 3017 ms. Trace:
W20260809 23:20:30.343713 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.006s	sys 0.015s
I20260809 23:20:30.233865 32639 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:30.078858 32587 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.338732 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:30.389463 24427 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260809 23:20:30.423043 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.018s	sys 0.017s
I20260809 23:20:30.423633 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:30.424427 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3028) took 3173 ms. Trace:
I20260809 23:20:30.424830 24427 rpcz_store.cc:276] 0809 23:20:27.250850 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.329473 (+3078623us) service_pool.cc:227] Handling call
0809 23:20:30.388880 (+ 59407us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.423038 (+ 34158us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.424372 (+  1334us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.268576 24164 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 12, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:30.126549 31984 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:30.421720 25371 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:30.253885 32287 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:30.125990 32391 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.428531 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1335) took 1859 ms. Trace:
I20260809 23:20:30.422883 24314 rpcz_store.cc:276] 0809 23:20:27.311037 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.438000 (+2126963us) service_pool.cc:227] Handling call
0809 23:20:29.438245 (+   245us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.438406 (+   161us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:29.491604 (+ 53198us) log.cc:842] Serialized 14 byte log entry
0809 23:20:30.327846 (+836242us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:30.327899 (+    53us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:30.327948 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.328011 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.328116 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":149,"prepare.queue_time_us":53017,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":132464,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":11790,"thread_start_us":835703,"threads_started":2,"wal-append.queue_time_us":666585}
W20260809 23:20:30.338760 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 77140.9us
I20260809 23:20:30.345831 32038 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:30.245440 31661 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:20:30.431720 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 93772.5us
I20260809 23:20:30.343439 24543 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:30.432924 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92857.8us
I20260809 23:20:30.454527 31661 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:30.369516 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 138124us
I20260809 23:20:30.456948 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:30.445530 32417 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:30.446655 32202 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:30.451088 24427 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260809 23:20:30.458439 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:30.130045 32469 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.459127 32469 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.417394 24309 rpcz_store.cc:276] 0809 23:20:28.203740 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.573112 (+1369372us) service_pool.cc:227] Handling call
0809 23:20:29.596764 (+ 23652us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.597011 (+   247us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:29.672170 (+ 75159us) log.cc:842] Serialized 14 byte log entry
0809 23:20:30.333552 (+661382us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:30.333602 (+    50us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:30.333651 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.333728 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.333857 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7347,"prepare.queue_time_us":82119,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":134356,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":30190,"thread_start_us":648772,"threads_started":2,"wal-append.queue_time_us":383038}
I20260809 23:20:30.457902 32417 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.461738 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:30.457180 31661 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.464938 31661 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:30.463129 32417 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:30.466485 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1039) took 3075 ms. Trace:
I20260809 23:20:30.461580 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1040) took 3066 ms. Trace:
I20260809 23:20:30.420361 32239 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:30.419543 32279 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:30.260085 32546 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.336021 32212 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:30.398005 32231 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:30.384814 24422 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:30.480510 32279 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.440259 25371 rpcz_store.cc:276] 0809 23:20:28.568931 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.511001 (+942070us) service_pool.cc:227] Handling call
0809 23:20:30.428410 (+917409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.481062 32279 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:30.468271 24435 rpcz_store.cc:276] 0809 23:20:27.391383 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.422386 (+3031003us) service_pool.cc:227] Handling call
0809 23:20:30.441441 (+ 19055us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.465159 (+ 23718us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:30.465302 (+   143us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:30.465465 (+   163us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:30.465606 (+   141us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.465832 (+   226us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.466269 (+   437us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":9549,"threads_started":1}
I20260809 23:20:30.480216 32239 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.430966 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1204) took 1716 ms. Trace:
I20260809 23:20:30.480708 24427 rpcz_store.cc:276] 0809 23:20:27.394206 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.425613 (+3031407us) service_pool.cc:227] Handling call
0809 23:20:30.450167 (+ 24554us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.458922 (+  8755us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.460790 (+  1868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.487671 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2578) took 3181 ms. Trace:
I20260809 23:20:30.489534 32212 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.490793 24422 rpcz_store.cc:276] 0809 23:20:27.306567 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.383974 (+3077407us) service_pool.cc:227] Handling call
0809 23:20:30.384640 (+   666us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.487411 (+102771us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.487617 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.492269 24422 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:30.160959 32321 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:30.498062 24435 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260809 23:20:30.486208 32239 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:30.487119 32546 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.344123 32634 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:30.489221 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 152921us
I20260809 23:20:30.495148 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3030) took 3018 ms. Trace:
I20260809 23:20:30.510815 24422 rpcz_store.cc:276] 0809 23:20:27.476212 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.491621 (+3015409us) service_pool.cc:227] Handling call
0809 23:20:30.491941 (+   320us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.492609 (+   668us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.494625 (+  2016us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.486662 24540 rpcz_store.cc:276] 0809 23:20:28.715039 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.717168 (+1002129us) service_pool.cc:227] Handling call
0809 23:20:29.717709 (+   541us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:29.717996 (+   287us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:29.740221 (+ 22225us) log.cc:842] Serialized 14 byte log entry
0809 23:20:30.423519 (+683298us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:30.423573 (+    54us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:30.430608 (+  7035us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.430706 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.430866 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":29929,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":504,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4872,"thread_start_us":682957,"threads_started":1}
I20260809 23:20:30.248205 32354 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.185s
I20260809 23:20:30.398389 32640 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:30.457157 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:30.427924 31579 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.506968 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.440502 31790 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:30.507571 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2581) took 3084 ms. Trace:
I20260809 23:20:30.439127 32587 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:30.437583 32407 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:30.535951 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 110756us
I20260809 23:20:30.513180 24422 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d273006c869f4749852291bcee467158"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:20:30.490474 32231 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:30.518159 24546 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:30.524668 24435 rpcz_store.cc:276] 0809 23:20:27.423390 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.491140 (+3067750us) service_pool.cc:227] Handling call
0809 23:20:30.494229 (+  3089us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:30.499950 (+  5721us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.507517 (+  7567us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.527071 24427 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260809 23:20:30.503221 32633 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.520s	user 0.001s	sys 0.001s
I20260809 23:20:30.495306 32661 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:30.552115 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2586) took 2438 ms. Trace:
I20260809 23:20:30.254426 32618 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.544785 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:30.546016 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 208146us
I20260809 23:20:30.564019 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.508802 32401 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.523866 31579 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:30.542222 24422 raft_consensus.cc:595] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.557606 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:30.569325 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 99259.8us
I20260809 23:20:30.569700 24422 raft_consensus.cc:603] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:30.520668 32417 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:30.552825 24427 rpcz_store.cc:276] 0809 23:20:28.113249 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.492547 (+2379298us) service_pool.cc:227] Handling call
0809 23:20:30.522186 (+ 29639us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.544658 (+ 22472us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.551998 (+  7340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.272217 32641 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.553596 32254 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:30.548054 32304 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:30.547684 32649 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:30.529825 32674 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260809 23:20:30.568451 32034 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.906s	user 0.004s	sys 0.012s
I20260809 23:20:30.561121 32336 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:30.597241 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.595901 24542 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:30.599592 24427 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260809 23:20:30.559123 32661 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:30.600275 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3048) took 1275 ms. Trace:
I20260809 23:20:30.498586 32338 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:30.586570 31817 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
I20260809 23:20:30.602519 24427 rpcz_store.cc:276] 0809 23:20:29.324451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.574846 (+1250395us) service_pool.cc:227] Handling call
0809 23:20:30.596330 (+ 21484us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.600018 (+  3688us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.600210 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:30.580117 32417 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:30.517410 32644 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20260809 23:20:30.606756 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.606844 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.610070 24422 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:20:30.560644 31790 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
W20260809 23:20:30.601243 32508 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.818s	user 0.014s	sys 0.004s
I20260809 23:20:30.610913 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.597471 32254 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:30.594434 32649 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:30.572257 32068 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260809 23:20:30.594267 32600 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:30.598871 32401 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.593613 32674 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:30.626797 24427 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260809 23:20:30.629199 32652 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260809 23:20:30.630749 32338 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:30.608145 31817 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:30.631373 32032 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.008s
I20260809 23:20:30.608249 32201 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.633734 31793 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260809 23:20:30.635360 31790 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:30.692438 31790 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:30.693547 31790 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.698545 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2569) took 4512 ms. Trace:
I20260809 23:20:30.700057 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1317) took 1270 ms. Trace:
I20260809 23:20:30.701957 32641 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.638841 32657 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.703645 32657 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.706434 32657 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.639272 32034 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.707278 32034 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.710412 32034 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.639954 32618 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.635779 31884 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:30.608872 32231 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:30.627403 32254 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.611194 32202 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:30.729559 32417 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.730369 32202 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.730770 32202 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:30.618098 32645 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
I20260809 23:20:30.747457 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:30.600885 32321 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.668851 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.677984 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.686050 31817 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:30.687083 32642 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
I20260809 23:20:30.674396 32032 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.691442 32679 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:30.624003 32508 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.691497 32684 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:30.698973 24429 rpcz_store.cc:276] 0809 23:20:26.185904 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.822755 (+3636851us) service_pool.cc:227] Handling call
0809 23:20:29.865638 (+ 42883us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.876334 (+ 10696us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.876396 (+    62us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:29.895737 (+ 19341us) log.cc:842] Serialized 26 byte log entry
0809 23:20:30.273328 (+377591us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:30.273379 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.273468 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:30.698325 (+424857us) raft_consensus.cc:1671] finished
0809 23:20:30.698346 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.698483 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30765,"raft.queue_time_us":10297,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":28115,"thread_start_us":342298,"threads_started":1,"wal-append.queue_time_us":774980}
I20260809 23:20:30.745626 32254 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:30.636756 32644 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260809 23:20:30.640481 32270 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.868s	user 0.010s	sys 0.000s
I20260809 23:20:30.617197 32320 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:30.632376 32546 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.577286 32639 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:30.728981 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122300us
I20260809 23:20:30.776396 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.729328 31790 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:30.617247 32634 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:30.734440 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 277342us
I20260809 23:20:30.735893 24423 raft_consensus.cc:1203] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Deduplicated request from leader. Original: 10.10->[11.11-12.12]   Dedup: 12.12->[]
I20260809 23:20:30.658861 32649 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:30.616035 32471 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.661811 32674 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:30.788555 31681 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.863s	user 0.011s	sys 0.006s
W20260809 23:20:30.713282 32380 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.734s	user 0.008s	sys 0.004s
I20260809 23:20:30.789144 31681 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.789511 32380 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.790205 32202 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.604460 32661 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:30.753115 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:30.753613 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 214797us
I20260809 23:20:30.797452 24534 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260809 23:20:30.674011 32676 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:30.734525 32231 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.700578 25370 rpcz_store.cc:276] 0809 23:20:29.429326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.755641 (+326315us) service_pool.cc:227] Handling call
0809 23:20:29.756055 (+   414us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:29.831054 (+ 74999us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.831091 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:29.852312 (+ 21221us) log.cc:842] Serialized 26 byte log entry
0809 23:20:30.261725 (+409413us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:30.261769 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.261836 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:30.699774 (+437938us) raft_consensus.cc:1671] finished
0809 23:20:30.699802 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.699986 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74501,"raft.queue_time_us":84465,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":15680,"thread_start_us":345358,"threads_started":1,"wal-append.queue_time_us":631144}
I20260809 23:20:30.670428 32401 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.703673 32279 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.780084 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:30.857415 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77430.9us
I20260809 23:20:30.858352 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:30.762842 32225 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.828s	user 0.010s	sys 0.000s
I20260809 23:20:30.734723 31884 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.871194 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:30.871615 24435 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260809 23:20:30.785444 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:30.871963 24435 pending_rounds.cc:76] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 6
I20260809 23:20:30.872200 24435 pending_rounds.cc:97] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260809 23:20:30.872448 24435 raft_consensus.cc:2866] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:20:30.787118 31727 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.818s	user 0.004s	sys 0.012s
I20260809 23:20:30.787406 24423 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: [11.11-12.12]
I20260809 23:20:30.746588 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3016) took 4211 ms. Trace:
W20260809 23:20:30.791535 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.014s	sys 0.018s
I20260809 23:20:30.877169 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:30.877281 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.941s	user 0.014s	sys 0.018s
I20260809 23:20:30.878150 24323 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:30.804416 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.804877 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.806516 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:30.878252 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.1021e+06us
I20260809 23:20:30.765636 32320 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:30.808830 32380 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.809353 32231 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:30.813828 31984 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.823892 25372 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [13.13-13.13]
W20260809 23:20:30.827080 32557 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.724s	user 0.009s	sys 0.003s
I20260809 23:20:30.896979 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:30.834491 32660 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.286s	user 0.000s	sys 0.002s
I20260809 23:20:30.865841 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:30.783520 32684 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:30.867115 25321 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.666s (SENT)
W20260809 23:20:30.882612 32696 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260809 23:20:30.870028 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:30.872808 24435 pending_rounds.cc:97] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260809 23:20:30.906674 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:30.907125 24435 raft_consensus.cc:2866] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:30.877565 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.879876 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:30.808256 32202 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:30.952603 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46517.5us
W20260809 23:20:30.954339 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74571us
W20260809 23:20:30.872328 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 86989.3us
I20260809 23:20:30.876940 24425 rpcz_store.cc:276] 0809 23:20:26.534998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.960154 (+3425156us) service_pool.cc:227] Handling call
0809 23:20:29.981767 (+ 21613us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:20:29.985265 (+  3498us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.985294 (+    29us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:20:29.992451 (+  7157us) log.cc:842] Serialized 78 byte log entry
0809 23:20:30.395546 (+403095us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:30.395602 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.395693 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:30.735452 (+339759us) raft_consensus.cc:1671] finished
0809 23:20:30.735482 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:30.746506 (+ 11024us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5129,"raft.queue_time_us":3947,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":8404,"thread_start_us":368199,"threads_started":1,"wal-append.queue_time_us":682882}
I20260809 23:20:30.887419 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.891104 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.893980 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.956550 24320 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:30.957504 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.895344 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.896963 31984 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:30.898074 24313 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:30.897986 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:30.959254 24312 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:30.959659 24422 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:30.772382 32270 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:30.960208 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.960196 32695 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:30.867846 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.905799 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:30.961901 24427 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:30.962018 32270 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:30.903594 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:30.948534 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 142779us
I20260809 23:20:30.962042 25368 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260809 23:20:30.962699 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:30.873991 32338 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:30.963078 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96021.5us
I20260809 23:20:30.961094 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:30.963918 24427 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:30.963964 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:30.964457 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.965039 25368 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:30.965245 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:30.965611 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.967126 32641 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:30.967393 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.967221 31984 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:30.968040 25369 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:30.968497 32678 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:30.968863 31984 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:30.969347 32678 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:30.965258 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:30.969929 31984 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:30.970368 31984 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:30.968766 32687 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:30.971189 32644 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:30.971693 32687 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:30.972177 32687 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:30.972012 32678 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:20:30.960415 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:30.960438 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:30.976297 24422 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:30.969631 32694 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260809 23:20:30.971194 31984 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:30.893087 32320 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:30.999509 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:30.887900 32679 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:30.887513 32550 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:20:30.957105 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79656.6us
I20260809 23:20:31.000092 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:30.956281 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 90566.6us
I20260809 23:20:30.977327 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.011719 32212 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:31.012871 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.012979 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.000974 32469 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.014259 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.014820 24422 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:30.989759 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:30.975502 32278 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:30.997546 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3047) took 1673 ms. Trace:
I20260809 23:20:31.016203 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:31.035606 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.038494 24435 rpcz_store.cc:276] 0809 23:20:29.324168 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.543774 (+1219606us) service_pool.cc:227] Handling call
0809 23:20:30.871273 (+327499us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:30.997217 (+125944us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.997475 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:31.039004 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.035691 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.039480 24321 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:31.049649 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.050837 24425 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:30.876252 32225 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.061540 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:30.898975 32546 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.120s
I20260809 23:20:30.981266 32684 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:31.037551 32577 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.675s	user 0.009s	sys 0.004s
I20260809 23:20:31.071369 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:30.890228 31727 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.109736 24422 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [11.9-13.10]
I20260809 23:20:31.000554 32202 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.006898 32618 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:31.074966 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58885.2us
I20260809 23:20:30.979784 32694 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260809 23:20:31.075826 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.184432 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:31.010118 32679 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.184635 24539 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:31.061825 32669 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260809 23:20:30.922462 32557 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.050192 32695 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.060817 32657 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:31.186524 32655 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.003s
W20260809 23:20:31.115731 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 140909us
I20260809 23:20:31.118005 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.223784 24309 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:31.064787 31681 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.078042 32684 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.078763 31857 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.119050 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.007588 32338 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
W20260809 23:20:31.173223 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55849.7us
I20260809 23:20:31.249665 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:31.116767 32681 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260809 23:20:30.952586 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:31.010273 32550 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.194664 32679 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:31.061982 31661 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.107s
I20260809 23:20:31.147835 31793 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:31.268334 31793 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.268592 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.269202 31793 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:31.115592 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44333.8us
I20260809 23:20:31.271605 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:31.145573 32032 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:31.210606 32691 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260809 23:20:31.210891 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260809 23:20:31.211568 32701 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:20:31.077502 32686 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.141s
I20260809 23:20:31.274791 32686 consensus_queue.cc:1225] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:31.215372 32279 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.215986 31884 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.216799 32687 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.276050 32600 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:31.276661 32600 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:31.276448 32674 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.277189 32600 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:31.118863 32676 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.101s
I20260809 23:20:31.277733 24317 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:31.278034 32676 consensus_queue.cc:1225] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:31.220166 32508 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.057799 32278 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.280364 32278 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:31.281057 32702 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260809 23:20:31.282544 32705 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260809 23:20:31.283116 32338 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:31.266723 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.283524 24535 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:31.283573 32338 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:31.202792 32660 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260809 23:20:31.108634 32577 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.286796 32577 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.287120 32202 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:31.287525 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2568) took 5102 ms. Trace:
I20260809 23:20:31.287875 24424 rpcz_store.cc:276] 0809 23:20:26.185308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:29.819151 (+3633843us) service_pool.cc:227] Handling call
0809 23:20:29.848596 (+ 29445us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:29.954800 (+106204us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:29.954833 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:29.955424 (+   591us) log.cc:842] Serialized 52 byte log entry
0809 23:20:30.333564 (+378140us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:30.416766 (+ 83202us) log.cc:842] Serialized 14 byte log entry
0809 23:20:30.506706 (+ 89940us) log.cc:842] Serialized 14 byte log entry
0809 23:20:30.560268 (+ 53562us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:30.560365 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:30.697163 (+136798us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b18005108e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:30.833689 (+136526us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:31.287311 (+453622us) raft_consensus.cc:1671] finished
0809 23:20:31.287331 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:31.287469 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45140,"prepare.queue_time_us":284110,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":4940,"raft.queue_time_us":423721,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":27569,"spinlock_wait_cycles":24523904,"thread_start_us":482105,"threads_started":3,"wal-append.queue_time_us":468544}
I20260809 23:20:31.289933 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.291544 32674 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.292127 24539 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:31.292569 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.293118 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:31.293126 24535 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.297364 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.298499 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.299191 32202 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:31.299733 32202 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:31.300379 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.300973 24424 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.300611 32202 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.226692 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:31.168131 32239 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.003s
I20260809 23:20:31.302670 32239 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.304647 32239 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.222168 32657 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:31.262111 24150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 309566us
I20260809 23:20:31.262617 32550 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:31.306320 32657 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:31.181555 32471 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260809 23:20:31.262308 32694 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:31.245621 32684 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:31.265399 32682 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:31.314041 32682 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:31.314446 32682 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.075383 32024 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260809 23:20:31.271503 32697 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:20:31.315774 32697 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:31.316144 32697 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:31.219799 32380 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:31.317040 32380 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:31.317262 32697 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.317930 32380 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.318589 32697 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:31.319555 32225 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.319674 32471 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:31.320192 32471 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:31.284073 24422 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:31.320552 32471 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.313573 32278 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:31.062551 32699 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.149s
I20260809 23:20:31.316077 24535 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:31.322976 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96354.5us
I20260809 23:20:31.316365 32557 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:31.256151 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 137178us
I20260809 23:20:31.333714 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.324045 24544 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:31.327791 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:31.335103 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:31.335718 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 300098us
I20260809 23:20:31.336761 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.251384 31681 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.252588 32417 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:31.327314 32475 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.939s	user 0.010s	sys 0.001s
I20260809 23:20:31.277776 32687 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:31.258754 32068 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:31.326587 32278 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:31.295317 32338 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.343484 31884 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:31.257169 31857 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.344172 32674 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260809 23:20:31.342500 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 282987us
I20260809 23:20:31.348959 32508 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:31.369444 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.369781 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.367156 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.371342 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:31.387993 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47013.1us
I20260809 23:20:31.420164 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.424906 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:31.363782 31817 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.429798 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.363822 31727 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.346830 31681 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.368327 32674 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.437652 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.422369 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.370405 32618 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:31.349464 32417 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.363622 32687 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260809 23:20:31.440325 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:31.426869 32407 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.873s	user 0.005s	sys 0.005s
I20260809 23:20:31.438524 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.441766 32674 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:31.361714 32712 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.093s
I20260809 23:20:31.351531 32475 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.426393 32557 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:31.425526 31884 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:31.442016 32699 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:31.467795 32711 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.108s
I20260809 23:20:31.468766 31681 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.372692 32024 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260809 23:20:31.513832 32675 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:31.470517 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.514256 32675 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:31.515266 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.515110 32675 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.515867 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:31.516203 32675 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:31.516744 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.471240 32724 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.517369 24427 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 10.
I20260809 23:20:31.476312 31857 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:31.477844 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.538249 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.430418 32338 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:31.485159 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44913.4us
I20260809 23:20:31.359526 32068 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.486498 24424 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:31.446098 32618 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.492276 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.490024 32407 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.499461 32550 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:31.488854 32304 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.894s	user 0.013s	sys 0.001s
I20260809 23:20:31.501354 32475 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.502622 24431 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260809 23:20:31.484790 32469 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:31.578155 32469 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.580180 32068 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:31.580350 32469 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:31.505704 32417 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:31.581947 31579 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.994s	user 0.012s	sys 0.000s
I20260809 23:20:31.582340 31579 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.506858 32471 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:31.508353 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 519858us
W20260809 23:20:31.578701 31790 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.794s	user 0.009s	sys 0.002s
I20260809 23:20:31.530750 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:31.534731 24435 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260809 23:20:31.585466 24428 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 10. Current term is 11. Ops: []
W20260809 23:20:31.482355 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 112605us
W20260809 23:20:31.586293 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55614.2us
I20260809 23:20:31.587666 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:31.566488 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.590132 24424 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:31.442516 31817 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:31.540504 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 194275us
I20260809 23:20:31.490746 32712 consensus_queue.cc:1225] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:31.568106 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:31.492277 32662 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:31.496047 32663 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:31.594550 25369 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:31.565361 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.572592 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:31.573552 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:31.538894 32254 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.000s
I20260809 23:20:31.569649 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:31.534058 32724 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.511495 32657 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.606281 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:31.594340 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.570580 24546 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:31.598281 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:31.607724 24435 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:31.608223 25369 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:31.598381 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.598989 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.608667 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:31.609815 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.597280 24540 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:31.575178 32694 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.611310 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.538203 32699 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:31.610189 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:31.602274 25368 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:31.630300 25373 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:31.611161 24321 raft_consensus.cc:2354] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:31.629812 24546 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:31.610952 32321 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260809 23:20:31.633791 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.628685 32687 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:31.609946 32254 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.634433 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.635308 32663 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.630050 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:31.661904 25366 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.636111 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.662875 24316 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:31.596962 32032 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:31.640748 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.641362 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:31.644011 24546 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260809 23:20:31.641376 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.642386 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:31.645030 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46735.4us
I20260809 23:20:31.667023 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.667274 24435 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:31.646044 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72570.7us
I20260809 23:20:31.647805 32475 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.650427 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:31.634872 32320 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.000s
I20260809 23:20:31.667495 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.639070 24314 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:31.669692 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.670207 24546 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.14-14.14]
I20260809 23:20:31.667060 24321 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:31.641887 32321 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:31.714499 24314 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260809 23:20:31.651948 32279 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:31.723773 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.595407 32728 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:31.608492 31790 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:31.717317 25373 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:31.714082 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.584959 32679 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.731854 24316 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [13.13-13.13]
I20260809 23:20:31.723062 24546 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260809 23:20:31.668892 32032 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.671733 32336 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:31.734911 24321 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260809 23:20:31.735255 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.745493 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1036) took 5133 ms. Trace:
I20260809 23:20:31.783461 24435 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:31.783687 24423 rpcz_store.cc:276] 0809 23:20:26.612439 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.063338 (+3450899us) service_pool.cc:227] Handling call
0809 23:20:30.735706 (+672368us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:30.883395 (+147689us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:31.745220 (+861825us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54002f8fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:31.745413 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1545491712}
I20260809 23:20:31.785291 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.778661 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:31.788180 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:31.789069 25368 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 9 FOLLOWER]: Advancing to term 11
I20260809 23:20:31.795847 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.720747 32225 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.719849 32663 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
W20260809 23:20:31.746665 32587 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.208s	user 0.010s	sys 0.011s
I20260809 23:20:31.823799 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.733348 32728 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.721933 32321 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 10 FOLLOWER]: Advancing to term 11
I20260809 23:20:31.720474 32687 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:31.777804 32336 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:31.842351 24314 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:31.713054 32721 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:31.641973 32694 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.844753 24429 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:31.825111 24435 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:31.731041 32743 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.831295 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:31.860447 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:31.726272 32475 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.130s
I20260809 23:20:31.733682 32697 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:31.747844 32032 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:31.836730 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.718084 32508 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:31.610167 32724 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.844764 32657 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.617730 31884 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.14, Last appended by leader: 14, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.709434 32320 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.653245 32684 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.886338 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:31.720361 32726 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.134s
I20260809 23:20:31.890033 24316 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:31.722225 31793 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.867990 32391 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.895347 24424 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:31.898550 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.885653 32336 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:31.849462 32741 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:31.853566 31579 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:31.904695 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 236011us
I20260809 23:20:31.776432 32557 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.109s
I20260809 23:20:31.860997 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.933362 24423 raft_consensus.cc:2362] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16. Re-sending same reply.
I20260809 23:20:31.934222 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.934964 24423 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:31.886178 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:31.932732 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.943608 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:31.945799 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.946102 32407 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:31.946586 32407 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:31.946993 32407 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:31.947686 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:31.951467 32730 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
W20260809 23:20:31.952819 32732 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260809 23:20:31.953311 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.953903 24425 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:31.954618 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:31.958067 25365 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260809 23:20:31.958263 24425 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260809 23:20:31.850768 32225 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:31.979074 25368 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 11.
I20260809 23:20:31.982101 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:31.906620 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.868292 32728 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:31.925729 24314 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 11: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:31.929241 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 261799us
W20260809 23:20:31.927105 32640 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.379s	user 0.014s	sys 0.000s
I20260809 23:20:32.023962 25372 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:32.029138 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.934530 32550 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:32.036981 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 130477us
I20260809 23:20:32.077492 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:31.929775 32254 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:31.834098 32699 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.025305 32471 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:32.065999 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:32.035447 32737 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.389s	user 0.001s	sys 0.000s
I20260809 23:20:31.906693 32697 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:31.903810 32721 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.161s
W20260809 23:20:32.082006 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 356187us
I20260809 23:20:32.088387 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:31.930466 32391 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.089083 24423 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260809 23:20:32.069370 32634 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.010s	sys 0.013s
I20260809 23:20:32.029040 32759 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:32.038276 31817 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:32.078233 32682 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.762s	user 0.012s	sys 0.004s
I20260809 23:20:32.089577 32276 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:32.099825 24423 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:32.121411 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:32.123091 25371 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260809 23:20:32.123566 25371 pending_rounds.cc:76] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:20:32.123822 25371 pending_rounds.cc:97] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260809 23:20:32.124115 25371 raft_consensus.cc:2866] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:31.899529 32743 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:32.100191 32634 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.010s	sys 0.013s
I20260809 23:20:32.093437 32475 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:32.136330 32225 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:32.137202 32746 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260809 23:20:32.137650 32320 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:20:32.128634 32679 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.140089 32759 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:32.141052 32759 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.143551 32684 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.145193 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:32.145498 32697 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.146024 32697 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:32.147527 32761 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:32.148222 32761 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:32.148658 32761 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:32.149466 32761 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.031056 32640 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.139003 31793 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:32.136535 32742 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
I20260809 23:20:32.160835 25373 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260809 23:20:32.175107 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1229) took 1310 ms. Trace:
I20260809 23:20:32.177703 24534 rpcz_store.cc:276] 0809 23:20:30.864809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.876300 (+ 11491us) service_pool.cc:227] Handling call
0809 23:20:32.174724 (+1298424us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54001483a8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297d9434 00007fac287a2437 00007fac287a1bc8 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86
0809 23:20:32.175046 (+   322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29941120}
I20260809 23:20:32.178390 24546 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260809 23:20:32.181247 25366 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.510s	user 0.007s	sys 0.003s
W20260809 23:20:32.187121 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 162624us
I20260809 23:20:32.151542 32724 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:32.186791 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88892.2us
I20260809 23:20:32.182013 25366 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:32.188097 32721 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:32.139015 32749 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
W20260809 23:20:32.196352 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 103390us
I20260809 23:20:32.125723 32726 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:32.094965 32254 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260809 23:20:32.201226 24539 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:32.159771 32737 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:20:32.153885 32475 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:32.201391 32663 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:31.975050 32741 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.071787 32508 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.188616 32320 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:32.084227 32687 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:32.210645 32661 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.359s	user 0.011s	sys 0.000s
W20260809 23:20:32.238354 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83851.7us
I20260809 23:20:32.256791 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:32.258052 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:32.207933 32640 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:32.204381 32726 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:32.264878 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:32.266038 25370 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260809 23:20:32.120723 32682 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.205128 32752 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:20:32.276602 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71649.5us
I20260809 23:20:32.201839 32225 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260809 23:20:32.242647   302 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:32.280067 32649 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.489s	user 0.012s	sys 0.000s
I20260809 23:20:32.272439 32724 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:32.247442 32557 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260809 23:20:32.262729 32587 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
W20260809 23:20:32.290520 32287 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.840s	user 0.014s	sys 0.000s
W20260809 23:20:32.290621 32231 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.337s	user 0.014s	sys 0.001s
I20260809 23:20:32.243448 32721 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260809 23:20:32.245082 32697 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260809 23:20:32.246025 32663 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.315937   332 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.284801   323 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:32.239565   300 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.094s
I20260809 23:20:32.289778   326 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.356341   326 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.239936 32407 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:32.359452   331 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260809 23:20:32.360600 32407 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:32.360642 32391 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.365583 32741 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.283720 32475 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.372539 32475 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:32.324117 32287 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:32.300640   302 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.335788 32401 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.270998 32661 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.285069 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:32.312865 32600 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.035s	user 0.013s	sys 0.005s
W20260809 23:20:32.355793 32338 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.785s	user 0.010s	sys 0.001s
I20260809 23:20:32.312134 32557 consensus_queue.cc:1225] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:32.365720 32728 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.391758 24430 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:32.358731   326 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:32.412379 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132041us
I20260809 23:20:32.417660 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:32.420463 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:32.422175 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:32.413236 24423 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:32.301406 32649 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.318749 32231 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.444383 25363 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:32.424293 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:32.398972 32401 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.424950 32697 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:32.385643 32407 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:32.499542 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:32.508746 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:32.500028 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75815.9us
I20260809 23:20:32.423897   300 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:32.446776 24545 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:32.307673 32726 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.424068 32338 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.425153 32741 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:32.631112 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 122475us
W20260809 23:20:32.632473 24266 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.705s (SENT) [suppressed 29 similar messages]
I20260809 23:20:32.633428 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:32.633862 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:32.660303 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:32.660948 24314 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:32.634189 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.662153 25366 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:32.514017 32401 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.446309   326 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:32.507134 32721 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.670997 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.640321 24311 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
W20260809 23:20:32.520731 32032 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.605s	user 0.010s	sys 0.002s
I20260809 23:20:32.679010 25370 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:32.374383   332 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.514861 32587 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:32.484545   337 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:32.514683 32407 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:32.520833 32557 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:32.663055 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.635222 25368 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:32.640666 32682 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.674994 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.709632 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.675707 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:32.644034   300 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.683053 24314 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:32.728689 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:32.522398   334 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:32.731796 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:32.630816 32254 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:32.666829 32674 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.168s	user 0.008s	sys 0.004s
W20260809 23:20:32.692243 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59824.6us
W20260809 23:20:32.661500 32618 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.087s	user 0.020s	sys 0.008s
I20260809 23:20:32.704617 25366 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 11: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:32.709738 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:32.410876 32600 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.639786 32391 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260809 23:20:32.723061 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:32.728859 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:32.658913 32684 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader election won for term 11
W20260809 23:20:32.697257 32336 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.739s	user 0.008s	sys 0.003s
I20260809 23:20:32.770736 24314 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:32.841676 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:32.842422 24313 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:32.696708 32587 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:32.771704 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:32.697768 32726 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:32.787564 25375 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:32.791790 32401 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:32.793171   309 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.454s	user 0.000s	sys 0.001s
I20260809 23:20:32.798801 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:32.974998 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 246218us
I20260809 23:20:32.688695 32338 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.807123 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:32.804397 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1328) took 1770 ms. Trace:
I20260809 23:20:32.682693 32032 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:32.795166 32321 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.907s	user 0.022s	sys 0.000s
I20260809 23:20:32.837533 25368 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:32.682507 32721 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.836678 32270 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.845824 32336 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:32.957253   350 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.154s	user 0.002s	sys 0.000s
W20260809 23:20:32.958365 32475 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.574s	user 0.005s	sys 0.006s
I20260809 23:20:32.844349 32407 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.959857 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:32.745246   300 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.961872 32728 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.963670 32600 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:32.964113 32662 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.355s	user 0.013s	sys 0.005s
I20260809 23:20:32.863006 32587 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:32.802946 32674 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.803819   329 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.025120 24323 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:33.025283   329 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.028570 32336 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:32.964529   307 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.453s	user 0.000s	sys 0.003s
I20260809 23:20:33.032001   300 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:33.032852 32401 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:32.811890 32618 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.034519 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:32.992483 32682 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.035557 32682 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.037766 32682 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:32.965438   318 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.122s	user 0.000s	sys 0.002s
I20260809 23:20:33.011237 32270 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.778563   334 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:33.039739 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2980) took 1601 ms (client timeout 2063 ms). Trace:
W20260809 23:20:33.040123 24540 rpcz_store.cc:269] 0809 23:20:31.438046 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.438233 (+   187us) service_pool.cc:227] Handling call
0809 23:20:33.039553 (+1601320us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:33.039682 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248548096}
I20260809 23:20:33.040884 32270 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.041049   334 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.966913   348 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.041657 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3065) took 1717 ms. Trace:
I20260809 23:20:33.042502 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:32.966065 32684 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:33.043655 25370 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:33.043918 32684 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.017956 32694 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:32.968374   323 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:33.047207   323 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:33.048470 32739 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.415s	user 0.002s	sys 0.000s
I20260809 23:20:33.048115   323 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.049300   323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:33.049916 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3059) took 1940 ms. Trace:
W20260809 23:20:33.051662 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2601) took 2163 ms (client timeout 1705 ms). Trace:
W20260809 23:20:33.051882   317 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
I20260809 23:20:33.050324 24422 rpcz_store.cc:276] 0809 23:20:31.109252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.321032 (+211780us) service_pool.cc:227] Handling call
0809 23:20:31.321393 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.321729 (+   336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:31.321756 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:31.322341 (+   585us) log.cc:842] Serialized 26 byte log entry
0809 23:20:31.490649 (+168308us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:31.490685 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:31.490756 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:33.047704 (+1556948us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:33.049610 (+  1906us) raft_consensus.cc:1671] finished
0809 23:20:33.049637 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.049854 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":927,"spinlock_wait_cycles":43762304,"thread_start_us":164449,"threads_started":1,"wal-append.queue_time_us":306294}
W20260809 23:20:33.052101 24429 rpcz_store.cc:269] 0809 23:20:30.888334 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.955050 (+ 66716us) service_pool.cc:227] Handling call
0809 23:20:33.051433 (+2096383us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b54003b9554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:33.051588 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1941319552}
W20260809 23:20:32.970512 32724 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.663s	user 0.006s	sys 0.006s
I20260809 23:20:33.052858 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.053130 32724 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:33.053568 24422 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:32.846422 32661 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.053669 32724 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:32.958487 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:33.054153 24427 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:20:33.019573 32662 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:33.056457 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2533) took 1456 ms. Trace:
I20260809 23:20:32.710665   337 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.956102   302 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.976246 24320 rpcz_store.cc:276] 0809 23:20:31.033614 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.035841 (+  2227us) service_pool.cc:227] Handling call
0809 23:20:31.036220 (+   379us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.206161 (+169941us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:31.206196 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:31.220768 (+ 14572us) log.cc:842] Serialized 26 byte log entry
0809 23:20:31.368222 (+147454us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:31.368260 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.776454 (+1408194us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:32.776505 (+    51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:32.776549 (+    44us) raft_consensus.cc:1671] finished
0809 23:20:32.776561 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:32.804294 (+ 27733us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7785,"raft.queue_time_us":9085,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":35897,"spinlock_wait_cycles":350243456,"thread_start_us":147094,"threads_started":1,"wal-append.queue_time_us":250812,"wal-append.run_cpu_time_us":6600,"wal-append.run_wall_time_us":1010610}
W20260809 23:20:33.058430 32700 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.843s	user 0.006s	sys 0.015s
I20260809 23:20:33.059048   337 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.979539 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:20:32.983515 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1331) took 1618 ms. Trace:
I20260809 23:20:33.060437 24311 rpcz_store.cc:276] 0809 23:20:31.364894 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.365029 (+   135us) service_pool.cc:227] Handling call
0809 23:20:31.419640 (+ 54611us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.673768 (+1254128us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.983284 (+309516us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540019cdd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:32.983454 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2532801920}
I20260809 23:20:33.061395 32278 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.062409 32278 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:32.964571 32231 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:32.985101 32762 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:33.063470 32762 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.959771   332 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.991806 24316 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
W20260809 23:20:32.965322 32508 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.009s	sys 0.002s
I20260809 23:20:32.993685 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.065855 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:32.996686 32321 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:32.997509 24164 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 5 to 13. New cstate: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:32.999799 25368 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:33.031786 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 224728us
I20260809 23:20:32.999053 32338 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:32.973795   326 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:32.988299 32225 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:33.015959 24424 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:33.016587 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.028182   329 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:33.024369 32032 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.024514 32674 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:32.866605   325 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.042068   348 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:32.967025 32557 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:32.809939 32254 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.053390 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.089279 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1348) took 1311 ms. Trace:
I20260809 23:20:33.090044 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.090765 24429 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:33.090620 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2883) took 1534 ms (client timeout 1832 ms). Trace:
W20260809 23:20:33.091252 24539 rpcz_store.cc:269] 0809 23:20:31.556012 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.597832 (+ 41820us) service_pool.cc:227] Handling call
0809 23:20:33.090441 (+1492609us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:33.090560 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1217028992}
I20260809 23:20:33.091603 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:33.091766 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.090538 32254 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.042039 24425 rpcz_store.cc:276] 0809 23:20:31.324646 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.991193 (+666547us) service_pool.cc:227] Handling call
0809 23:20:32.031646 (+ 40453us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:32.031963 (+   317us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:32.031998 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:32.032472 (+   474us) log.cc:842] Serialized 26 byte log entry
0809 23:20:32.199801 (+167329us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:32.199855 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.199951 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:33.041451 (+841500us) raft_consensus.cc:1671] finished
0809 23:20:33.041472 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.041598 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16791,"raft.queue_time_us":89540,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":10376,"thread_start_us":119012,"threads_started":1,"wal-append.queue_time_us":657927}
I20260809 23:20:32.965252 24314 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:33.062687 24311 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:33.093840 24429 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:33.056835 24309 rpcz_store.cc:276] 0809 23:20:31.600483 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.670797 (+ 70314us) service_pool.cc:227] Handling call
0809 23:20:31.719160 (+ 48363us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.903337 (+184177us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:31.903378 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:31.903885 (+   507us) log.cc:842] Serialized 26 byte log entry
0809 23:20:32.240620 (+336735us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:32.247385 (+  6765us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.247458 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:33.050712 (+803254us) spinlock_profiling.cc:243] Waited 8.76 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:33.056180 (+  5468us) raft_consensus.cc:1671] finished
0809 23:20:33.056201 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.056392 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1190200,"raft.queue_time_us":28267,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":1188989,"spinlock_wait_cycles":20144384,"thread_start_us":348950,"threads_started":2,"wal-append.queue_time_us":750589}
I20260809 23:20:33.073788 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:33.074836 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.010s	sys 0.019s
W20260809 23:20:33.095865 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.010s	sys 0.020s
I20260809 23:20:33.075657 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:33.078203 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61719.3us
I20260809 23:20:33.099102 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:33.099789 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.099726 25366 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:33.101095 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.097657 25369 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:33.020459 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.026700 32721 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.109565 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.080386 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:33.068437 32508 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.010s	sys 0.002s
I20260809 23:20:33.081507 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:33.118906 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.16479e+06us
I20260809 23:20:33.123544 24422 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:33.123603 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.057380 32679 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:33.079107 32225 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:33.088750 32744 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.201s	user 0.001s	sys 0.000s
I20260809 23:20:33.089649 25373 rpcz_store.cc:276] 0809 23:20:31.777948 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.889401 (+111453us) service_pool.cc:227] Handling call
0809 23:20:31.925006 (+ 35605us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:32.177696 (+252690us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.089023 (+911327us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:33.089224 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":523741568}
I20260809 23:20:33.082110 32338 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:33.112936 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.081440 32726 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.113029 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.113997 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.112975 32546 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:33.115252 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.150552 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:33.150734 24311 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:33.151129 25368 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260809 23:20:33.115885 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:33.124512 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.126873 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.151575 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.152668 24429 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:33.153492 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.076231 32676 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:33.154165 24429 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:33.054171 32724 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.177875 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.141640 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.146345 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.146576 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.131587 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.179478 24540 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:33.154134 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.084371   326 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:33.183045   326 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:33.147426 32699 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.014s	sys 0.040s
I20260809 23:20:33.165107 25366 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:33.099992 32557 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.125793   302 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:33.188050 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:33.125612 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.132220 32270 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:33.191071 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.191246 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.191628 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:33.211846 24313 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:33.192481 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:33.188297 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61541us
I20260809 23:20:33.179555 24546 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 11.
I20260809 23:20:33.132639 32657 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260809 23:20:33.221839 24311 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:33.222096 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.196916 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.197050 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.199471 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.132367 32679 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:33.205202   353 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
W20260809 23:20:33.206238 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80780.4us
W20260809 23:20:33.188658 32687 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.004s	sys 0.019s
I20260809 23:20:33.214978 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.229991 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:33.230108 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 97230.6us
I20260809 23:20:33.195324 32640 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.215749 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.177196 32726 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.154604 32618 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.153687 32546 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:33.211104 24429 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:33.224154 24422 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 12: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:33.226647 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.19749e+06us
I20260809 23:20:33.246768 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.247411 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:33.199031 32699 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.830s	user 0.014s	sys 0.040s
I20260809 23:20:33.231719 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1240) took 1642 ms. Trace:
I20260809 23:20:33.210825 32270 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:33.219085 32336 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:33.239073 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.240571 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.201702 32694 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.199316 32338 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:33.243371 24311 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260809 23:20:33.209273 32391 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:33.228420 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:33.278705 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50362.7us
I20260809 23:20:33.245575 32657 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:33.254397 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130911us
I20260809 23:20:33.287648 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:33.288269 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:33.288763 32557 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.257797 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1324) took 2388 ms. Trace:
I20260809 23:20:33.155661   336 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
I20260809 23:20:33.289448 24317 rpcz_store.cc:276] 0809 23:20:30.869106 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.869258 (+   152us) service_pool.cc:227] Handling call
0809 23:20:30.895990 (+ 26732us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:31.278173 (+382183us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.257572 (+1979399us) spinlock_profiling.cc:243] Waited 382 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:33.257740 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":877506944}
I20260809 23:20:33.290761 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.291831 32741 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:33.292515 32741 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:33.191883 32676 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.221138   329 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:33.187196 32417 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260809 23:20:33.294907 32676 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.260628 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.295373 32417 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:33.295671   329 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:33.296223 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.296502   329 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:33.297034 24534 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:33.297902 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.252408 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.252673 24544 rpcz_store.cc:276] 0809 23:20:31.589114 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:32.110595 (+521481us) service_pool.cc:227] Handling call
0809 23:20:32.151231 (+ 40636us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.151431 (+   200us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:33.044862 (+893431us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b18002a5200. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:20:33.045019 (+   157us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.231135 (+186116us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:33.231184 (+    49us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:33.231226 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.231451 (+   225us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.231634 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37993,"prepare.queue_time_us":64597,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":51031,"spinlock_wait_cycles":37044608,"thread_start_us":247109,"threads_started":2,"wal-append.queue_time_us":175338}
I20260809 23:20:33.299885 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.300308 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.262940 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.301731 32724 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.302320 32724 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.302973 32724 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.229796   302 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:33.308173   302 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.237370 32024 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:33.100248   325 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.312911   325 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.312789   329 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.245780   353 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:33.215386   367 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.280555 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.244128 32475 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:33.279882 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.241753 32508 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260809 23:20:33.289920   336 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
W20260809 23:20:33.232478 32639 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.523s	user 0.008s	sys 0.009s
I20260809 23:20:33.295754 32417 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:33.300514 24422 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:33.357273 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.357344 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.262887 24546 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:33.270092 24425 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:33.259502 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:33.263996 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.257398 32212 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:20:33.317268 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58444.7us
I20260809 23:20:33.236698 32634 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:33.368139 32639 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.660s	user 0.008s	sys 0.010s
I20260809 23:20:33.371781 32634 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.372977 32304 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.354206   367 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.373883 32634 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:33.368386   376 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260809 23:20:33.374962   367 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.328121 32321 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.286042 32694 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.358534 32674 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:33.379616 32747 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:33.380245 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 116318us
I20260809 23:20:33.380633 32032 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:33.327538 32391 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:33.381283 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.266368 32712 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
I20260809 23:20:33.383177   339 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:33.383478 32712 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:33.383584   339 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:33.383982 32712 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:33.384222   339 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:33.385607   348 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.385150 32712 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.350672 32699 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:33.387388 32674 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:33.388381 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.319911 32724 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:33.342324 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79508.6us
I20260809 23:20:33.266435 32270 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:33.265070   365 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:33.346041 24429 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:33.386777 32712 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:33.351356 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71585.6us
I20260809 23:20:33.288488 24311 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12.
I20260809 23:20:33.353250 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.353242 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:33.245350 32640 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.314472 32763 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:33.285684 32679 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.391227   337 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:33.382184 32391 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.366331   329 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:33.359843 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:33.368808 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.236841 32225 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.373602 32304 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.358292 32475 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.253875 32546 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:33.644913 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.319346 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.347450 32683 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:33.391803 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.399230 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.407742 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.356308   336 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:33.440927 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.442485 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.658428 24313 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:33.448307 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 95166.5us
I20260809 23:20:33.658597 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.659451 24429 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:33.660638 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:33.451539 32679 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:33.474180 25366 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:33.504470 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 144735us
I20260809 23:20:33.504503 24546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:33.671532 24429 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:33.675616 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.442409 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.349539 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1223) took 2888 ms. Trace:
I20260809 23:20:33.540432 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1242) took 1832 ms. Trace:
W20260809 23:20:33.564670 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 275215us
I20260809 23:20:33.444375 32762 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:33.685149 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:33.711359 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:33.711984 24429 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:33.679028 24543 rpcz_store.cc:276] 0809 23:20:30.460629 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.460787 (+   158us) service_pool.cc:227] Handling call
0809 23:20:30.461133 (+   346us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:30.503613 (+ 42480us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:30.565659 (+ 62046us) log.cc:842] Serialized 14 byte log entry
0809 23:20:32.665620 (+2099961us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b180052f360. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:33.191821 (+526201us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:33.209383 (+ 17562us) log.cc:842] Serialized 52 byte log entry
0809 23:20:33.209605 (+   222us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:33.276673 (+ 67068us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.341834 (+ 65161us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.342014 (+   180us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:33.349263 (+  7249us) raft_consensus.cc:1671] finished
0809 23:20:33.349290 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.349472 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":605981,"prepare.queue_time_us":261427,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":665840,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":32661,"spinlock_wait_cycles":4600704,"thread_start_us":134854,"threads_started":3,"wal-append.queue_time_us":2402131}
I20260809 23:20:33.649770 32699 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.330705 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.427749   365 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.382661 32587 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.654381 24422 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:33.655792 24539 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12.
I20260809 23:20:33.730372 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.662670 32225 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:33.452594 24544 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
W20260809 23:20:33.664193 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 256588us
I20260809 23:20:33.388362 32674 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.452380 32640 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:33.453316 32726 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.644747   367 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:33.679332 24542 rpcz_store.cc:276] 0809 23:20:31.708371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:32.149384 (+441013us) service_pool.cc:227] Handling call
0809 23:20:32.149653 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.149844 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:32.150147 (+   303us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.236655 (+1086508us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.539925 (+303270us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:33.540009 (+    84us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:33.540075 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.540170 (+    95us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:33.540344 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1093526,"prepare.queue_time_us":133983,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":1109953,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":12977,"thread_start_us":193540,"threads_started":2}
W20260809 23:20:33.655299   378 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260809 23:20:33.717590 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.720551 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.714246 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.761649 24543 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:33.762691 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.726432 24546 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:33.763109 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.763989 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.686358 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.630242 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:33.731621 25366 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:33.765035 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.680120 32557 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.669252 32738 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.702209 32678 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:33.764412 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:33.784438 24313 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:33.733163 32763 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.731259 32550 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.766351 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.731112 32683 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:33.714833 32721 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.767876 25375 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:33.715112 32336 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.715674   348 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:33.701505 32711 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:33.715049 32694 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:33.772388 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51935.7us
I20260809 23:20:33.763955 24534 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260809 23:20:33.769029 24539 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:33.775710 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.668746 32278 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:33.776903 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90612.9us
I20260809 23:20:33.736720   337 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.766698 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:33.734140   365 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:33.791632 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.851137 24542 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:33.795331 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:33.851861 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85234.9us
I20260809 23:20:33.746865   346 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:33.747853 32764 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260809 23:20:33.775790   323 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.726s	user 0.013s	sys 0.002s
I20260809 23:20:33.798233   349 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:20:33.855013 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.735865 32699 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.862250 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.648967 32417 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.842415 24422 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:33.843217 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:33.784874 32661 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:20:33.803794 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 361496us
I20260809 23:20:33.867828 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.868515 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:33.863348 31579 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:33.857254 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.801388 24380 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12.
I20260809 23:20:33.881556 24380 leader_election.cc:401] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260809 23:20:33.882022 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80778.7us
I20260809 23:20:33.883270 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:33.859251 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.859728 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:33.888018 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:33.864902 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.889066 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.775306 32726 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:33.851310   378 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260809 23:20:33.788986 32557 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.910486 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:33.911875 24378 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 11 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12.
I20260809 23:20:33.912281 24498 leader_election.cc:398] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13.
I20260809 23:20:33.912875 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.913764 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.770123   336 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.881151 24540 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:33.933825 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:33.895264 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:33.899497 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:33.935076 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:33.935993 25373 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:33.913827 24429 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:33.784119 32469 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260809 23:20:33.935774 25366 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:33.935773 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:33.978768 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:33.980916 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:33.982808 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47140.5us
I20260809 23:20:33.840961   348 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.794836 32678 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.856065   337 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:33.845728 32711 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.789146 32738 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.871477 32278 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:34.007061 24543 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 13
I20260809 23:20:33.882367 32661 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:33.910185 32640 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:33.840656 32336 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.988541   300 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:33.910416   323 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:33.909859   390 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:33.995761 25373 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:33.798961 32721 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.799845 32763 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.941275 32417 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:33.796665 32550 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.754845   353 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.973567   363 raft_consensus.cc:2741] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260809 23:20:33.972957 32600 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.050007 32600 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:34.051432   388 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260809 23:20:33.911470   349 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12.
I20260809 23:20:34.052784 32587 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.055379 32738 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.056510   407 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:34.057803 32557 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.059913   406 raft_consensus.cc:2741] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:34.062000   348 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:33.937130 32683 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.031071 32640 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:34.064298 32640 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:33.864662   346 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.026541   367 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:34.014403 32661 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:34.005191 32475 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:34.031360   389 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260809 23:20:34.086376   367 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.086949 32699 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.032474   390 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:34.088560   390 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:34.052827 32600 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.105894 32721 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.132843 32763 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.133137   346 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.088953 32711 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.142792   407 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:34.147296 32678 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.157267   337 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:34.162282   302 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.008s	sys 0.004s
W20260809 23:20:34.150039   339 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.765s	user 0.013s	sys 0.001s
I20260809 23:20:34.156931   407 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.178306   339 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.181205   339 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:34.177314   302 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.009s	sys 0.005s
W20260809 23:20:34.187168 32712 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.002s
I20260809 23:20:34.255578 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3000) took 1529 ms. Trace:
W20260809 23:20:34.256610 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.002s	sys 0.017s
I20260809 23:20:34.257023 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3004) took 1530 ms. Trace:
I20260809 23:20:33.866078 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.409284 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3069) took 2522 ms. Trace:
I20260809 23:20:34.409868 24423 rpcz_store.cc:276] 0809 23:20:31.887197 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:32.119919 (+232722us) service_pool.cc:227] Handling call
0809 23:20:32.120279 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.423105 (+302826us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.408701 (+1985596us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.409035 (+   334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":672792832}
I20260809 23:20:34.410756 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1070) took 1603 ms. Trace:
I20260809 23:20:33.937186 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.269810 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:33.986514 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:34.272051 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 293026us
I20260809 23:20:34.413585 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:34.413821 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3005) took 1117 ms. Trace:
I20260809 23:20:34.415472 32712 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:34.408535 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.002s	sys 0.018s
I20260809 23:20:34.416003 32712 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:34.416594 32712 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:34.417347 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1348) took 1333 ms. Trace:
W20260809 23:20:34.417425 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 147960us
I20260809 23:20:34.419375 32726 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:34.408592 24320 rpcz_store.cc:276] 0809 23:20:32.726789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.208900 (+482111us) service_pool.cc:227] Handling call
0809 23:20:33.224765 (+ 15865us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:33.225031 (+   266us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:33.225066 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:33.229468 (+  4402us) log.cc:842] Serialized 26 byte log entry
0809 23:20:33.367753 (+138285us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:33.367891 (+   138us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.368020 (+   129us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:34.256496 (+888476us) raft_consensus.cc:1671] finished
0809 23:20:34.256537 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.256824 (+   287us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":105644,"threads_started":1,"wal-append.queue_time_us":411255}
I20260809 23:20:34.416021 24429 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:34.273883 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2897) took 1088 ms. Trace:
I20260809 23:20:34.417755 24317 rpcz_store.cc:276] 0809 23:20:33.083467 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.292641 (+209174us) service_pool.cc:227] Handling call
0809 23:20:33.292940 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:33.293124 (+   184us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:33.748603 (+455479us) log.cc:842] Serialized 14 byte log entry
0809 23:20:34.416609 (+668006us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.416702 (+    93us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:34.416858 (+   156us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.417002 (+   144us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.417277 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55656,"prepare.queue_time_us":78262,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":1047699,"threads_started":2,"wal-append.queue_time_us":340554}
I20260809 23:20:34.423206 24542 rpcz_store.cc:276] 0809 23:20:33.185735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.001943 (+816208us) service_pool.cc:227] Handling call
0809 23:20:34.272741 (+270798us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.273330 (+   589us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.273370 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.273438 (+    68us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:34.273465 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.273541 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.273649 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.425845 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.425952 32644 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:34.428094 32471 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:34.428826 32644 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:34.429709 32644 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:34.430011 32471 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:34.431928 32471 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:33.997892 24539 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:34.436167 32678 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:34.436874 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2892) took 1254 ms. Trace:
I20260809 23:20:34.437120 32678 consensus_queue.cc:1225] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:34.437446 24539 rpcz_store.cc:276] 0809 23:20:33.182922 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.966672 (+783750us) service_pool.cc:227] Handling call
0809 23:20:33.997668 (+ 30996us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.436325 (+438657us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.436695 (+   370us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.015358 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:34.439819 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.440089 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 2997) took 1206 ms. Trace:
I20260809 23:20:34.278666 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:34.027446 31579 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:34.440425 24546 rpcz_store.cc:276] 0809 23:20:33.233175 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.763760 (+530585us) service_pool.cc:227] Handling call
0809 23:20:34.439747 (+675987us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.439903 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":576458112}
I20260809 23:20:34.440961 32231 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:34.441573 31579 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.013033 25366 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260809 23:20:34.312806 32270 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.871s	user 0.010s	sys 0.000s
I20260809 23:20:33.060099 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:34.444478 32270 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.445829 31579 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.446417 25369 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:34.312537 32336 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.447077 32417 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:34.337764 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3007) took 1334 ms. Trace:
I20260809 23:20:34.447912 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1354) took 1231 ms. Trace:
I20260809 23:20:34.447824 24321 rpcz_store.cc:276] 0809 23:20:33.003048 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.034237 (+ 31189us) service_pool.cc:227] Handling call
0809 23:20:34.337253 (+1303016us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.337482 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":129652096}
I20260809 23:20:34.448369 32587 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.344223 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3072) took 2412 ms. Trace:
I20260809 23:20:34.448401 25369 rpcz_store.cc:276] 0809 23:20:33.216803 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.221523 (+  4720us) service_pool.cc:227] Handling call
0809 23:20:33.221823 (+   300us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.446895 (+1225072us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.447562 (+   667us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.447812 (+   250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2815150848}
I20260809 23:20:34.449265 32587 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:34.450492 32679 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.745s	user 0.008s	sys 0.013s
I20260809 23:20:34.345105 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:34.452610 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1346) took 2675 ms. Trace:
I20260809 23:20:34.453091 25363 rpcz_store.cc:276] 0809 23:20:31.777421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.778074 (+   653us) service_pool.cc:227] Handling call
0809 23:20:31.851555 (+ 73481us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.498839 (+647284us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.452067 (+1953228us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54004a5bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.452408 (+   341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1362877824}
W20260809 23:20:34.330235 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 883427us
I20260809 23:20:34.356611 24541 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [10.7-12.8]
W20260809 23:20:34.453445 32657 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.139s	user 0.011s	sys 0.011s
I20260809 23:20:34.455189 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1386) took 2856 ms. Trace:
I20260809 23:20:34.358150 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2896) took 1173 ms. Trace:
W20260809 23:20:34.345677 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1067) took 1539 ms (client timeout 1802 ms). Trace:
I20260809 23:20:34.326263 24164 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:34.368824 31661 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:34.372783 32550 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.375975 32676 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.376688 32475 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.383490 32674 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:34.455650 25371 rpcz_store.cc:276] 0809 23:20:31.598269 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.598545 (+   276us) service_pool.cc:227] Handling call
0809 23:20:31.598876 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.124604 (+525728us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.454686 (+2330082us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.454987 (+   301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1204375936}
I20260809 23:20:34.385676 32212 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:34.465204 32674 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:34.465750 32212 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.467661 31661 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.471625 32471 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.471875 32475 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.401266 32684 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:34.378522 32681 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.474195 24546 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20260809 23:20:34.404820 24316 rpcz_store.cc:276] 0809 23:20:32.725588 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.064717 (+339129us) service_pool.cc:227] Handling call
0809 23:20:33.065016 (+   299us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:33.065369 (+   353us) raft_consensus.cc:1520] Early marking committed up to index 6
0809 23:20:33.079032 (+ 13663us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.263879 (+184847us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:33.275573 (+ 11694us) log.cc:842] Serialized 52 byte log entry
0809 23:20:33.275811 (+   238us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:33.276194 (+   383us) log.cc:842] Serialized 14 byte log entry
0809 23:20:33.276422 (+   228us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:33.276494 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:34.254605 (+978111us) raft_consensus.cc:1671] finished
0809 23:20:34.254690 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.255242 (+   552us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2255925,"prepare.queue_time_us":15949,"raft.queue_time_us":28927,"raft.run_cpu_time_us":765,"raft.run_wall_time_us":2252461,"spinlock_wait_cycles":6185344,"thread_start_us":152671,"threads_started":1,"wal-append.queue_time_us":299262}
I20260809 23:20:34.431488 32644 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.478775 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1251) took 1262 ms. Trace:
I20260809 23:20:34.478826 32032 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:34.473349 32471 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:34.479606 32032 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:34.480121 32475 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.427891 24542 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:34.479343 24546 rpcz_store.cc:276] 0809 23:20:33.216584 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.441753 (+1225169us) service_pool.cc:227] Handling call
0809 23:20:34.473814 (+ 32061us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.478445 (+  4631us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.478686 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:34.434966 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.003s	sys 0.003s
I20260809 23:20:34.422559 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1077) took 1341 ms. Trace:
I20260809 23:20:34.411360 24422 rpcz_store.cc:276] 0809 23:20:32.806686 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.865496 (+1058810us) service_pool.cc:227] Handling call
0809 23:20:34.410487 (+544991us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.440653 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:34.449455 24424 rpcz_store.cc:276] 0809 23:20:31.931973 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.947202 (+ 15229us) service_pool.cc:227] Handling call
0809 23:20:34.343565 (+2396363us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.343871 (+   306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2454779136}
W20260809 23:20:34.496059 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.004s	sys 0.003s
I20260809 23:20:34.496186 24546 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:34.496615 24546 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:34.497841 24546 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.479614 32644 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:34.505170 32278 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:34.505617 32278 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:34.508201 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2577) took 7246 ms. Trace:
I20260809 23:20:34.508587 24431 rpcz_store.cc:276] 0809 23:20:27.262037 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.329920 (+3067883us) service_pool.cc:227] Handling call
0809 23:20:30.421359 (+ 91439us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.576067 (+1154708us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.507670 (+2931603us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540018d154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.508030 (+   360us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2486093056}
I20260809 23:20:34.509948 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.511667 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.447855 32417 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.363085 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1378) took 3557 ms. Trace:
I20260809 23:20:34.512836 25365 rpcz_store.cc:276] 0809 23:20:30.805793 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.824345 (+ 18552us) service_pool.cc:227] Handling call
0809 23:20:30.824779 (+   434us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.990081 (+1165302us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.362350 (+2372269us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.362807 (+   457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2605501184}
I20260809 23:20:34.513688 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:34.360668 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2875) took 3162 ms. Trace:
W20260809 23:20:34.456806 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 178552us
I20260809 23:20:34.485558 24429 rpcz_store.cc:276] 0809 23:20:33.080653 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.936911 (+856258us) service_pool.cc:227] Handling call
0809 23:20:34.422439 (+485528us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.516844 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.517503 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3060) took 3407 ms. Trace:
I20260809 23:20:34.518041 24427 rpcz_store.cc:276] 0809 23:20:31.109573 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.518313 (+408740us) service_pool.cc:227] Handling call
0809 23:20:33.053896 (+1535583us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:33.054611 (+   715us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.517017 (+1462406us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54003b9550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.517292 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3530690816}
I20260809 23:20:34.457031 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.514786 24424 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:34.464315   434 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
W20260809 23:20:34.456526 24425 rpcz_store.cc:269] 0809 23:20:32.805763 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.092942 (+287179us) service_pool.cc:227] Handling call
0809 23:20:34.345286 (+1252344us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.345474 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":298492672}
I20260809 23:20:34.474114 32684 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:34.456386 24544 rpcz_store.cc:276] 0809 23:20:33.185200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.299967 (+114767us) service_pool.cc:227] Handling call
0809 23:20:34.357746 (+1057779us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.357941 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":348151552}
W20260809 23:20:34.466670 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1340) took 1710 ms (client timeout 1538 ms). Trace:
I20260809 23:20:34.515808 24545 rpcz_store.cc:276] 0809 23:20:31.197821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.198038 (+   217us) service_pool.cc:227] Handling call
0809 23:20:31.198348 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.642468 (+1444120us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.359759 (+1717291us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.360358 (+   599us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2865332608}
I20260809 23:20:34.468321 32270 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.475148 32681 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.405846 32304 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.468979 32676 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.476606 32212 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.476733 32663 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:34.477615 32649 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:34.402742 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1330) took 3039 ms (client timeout 1898 ms). Trace:
I20260809 23:20:34.477963 31661 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.414425 24534 rpcz_store.cc:276] 0809 23:20:33.296708 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.913796 (+617088us) service_pool.cc:227] Handling call
0809 23:20:34.413542 (+499746us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.520773 32697 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.520777 31793 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:20:34.467355 32674 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.522681 32417 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.455644 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2982) took 3014 ms. Trace:
I20260809 23:20:34.522966 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:34.524284 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.524423 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3026) took 7355 ms. Trace:
I20260809 23:20:34.510869 24431 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:34.176311   337 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:34.515812 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:34.556556 25370 rpcz_store.cc:269] 0809 23:20:32.755898 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:32.771326 (+ 15428us) service_pool.cc:227] Handling call
0809 23:20:34.465829 (+1694503us) spinlock_profiling.cc:243] Waited 84.9 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.466301 (+   472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195308416}
I20260809 23:20:34.572918 24546 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:34.596078 24423 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:34.456331 32657 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:34.576501 24541 rpcz_store.cc:276] 0809 23:20:31.441169 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.441348 (+   179us) service_pool.cc:227] Handling call
0809 23:20:31.442206 (+   858us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:34.355932 (+2913726us) spinlock_profiling.cc:243] Waited 76.9 ms on lock 0x7b540033e120. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:34.455259 (+ 99327us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.455574 (+   315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176938368}
I20260809 23:20:34.591566 32354 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:34.577396 24430 rpcz_store.cc:276] 0809 23:20:27.169274 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.261555 (+3092281us) service_pool.cc:227] Handling call
0809 23:20:30.261867 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:32.416608 (+2154741us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.523943 (+2107335us) spinlock_profiling.cc:243] Waited 2.13 s on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.524198 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4897343360}
I20260809 23:20:34.581797 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.587738 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:34.590871   420 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.407s	user 0.000s	sys 0.008s
W20260809 23:20:34.591125 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 134832us
W20260809 23:20:34.591629   401 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.593s	user 0.006s	sys 0.012s
W20260809 23:20:34.591811 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.003s	sys 0.029s
I20260809 23:20:34.606369 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:34.593475   400 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.502s	user 0.007s	sys 0.007s
I20260809 23:20:34.594216 25365 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260809 23:20:34.594715 24542 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
W20260809 23:20:34.570917 24323 rpcz_store.cc:269] 0809 23:20:31.362576 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.369189 (+  6613us) service_pool.cc:227] Handling call
0809 23:20:34.402084 (+3032895us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003868d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.402403 (+   319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3805708160}
I20260809 23:20:34.589639 24539 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:34.596184 24429 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:34.613602 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.003s	sys 0.029s
I20260809 23:20:34.603082   434 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:34.601797 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.604087 24545 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:34.604413   403 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.517s	user 0.000s	sys 0.015s
I20260809 23:20:34.604992 32304 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.605882 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2893) took 1422 ms. Trace:
I20260809 23:20:34.596611 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.574888 32712 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:34.592533   428 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.410s	user 0.004s	sys 0.004s
I20260809 23:20:34.606444 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:34.612423 24542 pending_rounds.cc:76] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:34.616765 24540 rpcz_store.cc:276] 0809 23:20:33.183449 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.966844 (+783395us) service_pool.cc:227] Handling call
0809 23:20:33.968768 (+  1924us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.006178 (+ 37410us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.006229 (+    51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.006282 (+    53us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:34.006325 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.605371 (+599046us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:34.605444 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.605672 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85480320}
W20260809 23:20:34.605904 32546 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.943s	user 0.010s	sys 0.014s
W20260809 23:20:34.608752   399 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.572s	user 0.006s	sys 0.032s
I20260809 23:20:34.627166 32304 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.628134   337 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:34.629951 32304 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.629729   337 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.625989 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2993) took 1388 ms. Trace:
I20260809 23:20:34.639118 24545 rpcz_store.cc:276] 0809 23:20:33.231563 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.603223 (+1371660us) service_pool.cc:227] Handling call
0809 23:20:34.603723 (+   500us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.619063 (+ 15340us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.619276 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.625079 24425 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:34.647817 24544 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.14-14.14]
I20260809 23:20:34.631726 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.645824 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:34.650414 32038 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:34.656100   390 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.567s	user 0.012s	sys 0.000s
I20260809 23:20:34.656545 24534 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:34.656679   390 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.658730 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2996) took 1425 ms. Trace:
I20260809 23:20:34.659246 24540 rpcz_store.cc:276] 0809 23:20:33.232864 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.647896 (+1415032us) service_pool.cc:227] Handling call
0809 23:20:34.657994 (+ 10098us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.658415 (+   421us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.658440 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.658479 (+    39us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:34.658508 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.658558 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.658678 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:34.660615 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54441.6us
I20260809 23:20:34.661383 24540 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260809 23:20:34.662072 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3001) took 1427 ms. Trace:
I20260809 23:20:34.662242 32743 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:34.669301   424 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.466s	user 0.007s	sys 0.025s
W20260809 23:20:34.671330 32661 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.561s	user 0.005s	sys 0.016s
I20260809 23:20:34.647500 24542 pending_rounds.cc:97] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260809 23:20:34.674643 32661 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.675282 24542 raft_consensus.cc:2866] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:34.676889 24542 pending_rounds.cc:97] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260809 23:20:34.677248 24542 raft_consensus.cc:2866] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:34.181490   365 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:34.680289 24313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.010s	sys 0.006s
W20260809 23:20:34.650544   427 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.395s	user 0.000s	sys 0.005s
I20260809 23:20:34.657270 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2999) took 1423 ms. Trace:
I20260809 23:20:34.682994 32546 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.624367 32354 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:34.622879 32684 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:34.685036   395 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.612s	user 0.000s	sys 0.018s
I20260809 23:20:34.725229 24545 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260809 23:20:34.725764 32676 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.726027 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2998) took 1492 ms. Trace:
I20260809 23:20:34.591197 32697 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.729899 24430 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [11.9-13.10]
W20260809 23:20:34.676409 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66706.2us
I20260809 23:20:34.730631 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3071) took 2817 ms. Trace:
I20260809 23:20:34.662421 24540 rpcz_store.cc:276] 0809 23:20:33.235077 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.660439 (+1425362us) service_pool.cc:227] Handling call
0809 23:20:34.660882 (+   443us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.661825 (+   943us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.662026 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.731423 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:34.731673 32697 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:34.691640   408 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.527s	user 0.012s	sys 0.022s
I20260809 23:20:34.733078 32697 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:34.733325 24540 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260809 23:20:34.733500 32697 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:34.733853 32697 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:34.733987 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3003) took 1498 ms. Trace:
I20260809 23:20:34.735200 32661 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.735500 32417 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:34.731055 24430 rpcz_store.cc:276] 0809 23:20:31.912923 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.601162 (+2688239us) service_pool.cc:227] Handling call
0809 23:20:34.665222 (+ 64060us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:34.730325 (+ 65103us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.730547 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.718544 24422 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:34.734510 24540 rpcz_store.cc:276] 0809 23:20:33.235930 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.732417 (+1496487us) service_pool.cc:227] Handling call
0809 23:20:34.732940 (+   523us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:34.733715 (+   775us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.733933 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.621071   406 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
W20260809 23:20:34.719854 32380 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.856s	user 0.016s	sys 0.021s
W20260809 23:20:34.739770 32380 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.024s	user 0.016s	sys 0.021s
I20260809 23:20:34.740073   390 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:34.740447 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:34.740984 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1382) took 3153 ms (client timeout 2599 ms). Trace:
I20260809 23:20:34.741439 24422 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 16 because replica has last-logged OpId of term: 14 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 13.
W20260809 23:20:34.741380 25372 rpcz_store.cc:269] 0809 23:20:31.587809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.587959 (+   150us) service_pool.cc:227] Handling call
0809 23:20:34.740535 (+3152576us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:34.740739 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":884610432}
W20260809 23:20:34.742834   432 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.063s	user 0.003s	sys 0.000s
I20260809 23:20:34.721712 24539 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:34.722185 24534 rpcz_store.cc:276] 0809 23:20:33.234324 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.655689 (+1421365us) service_pool.cc:227] Handling call
0809 23:20:34.656227 (+   538us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.656992 (+   765us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.657193 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.744809 31984 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:34.745041 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2994) took 1513 ms. Trace:
I20260809 23:20:34.723057 24431 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:34.745451 24539 rpcz_store.cc:276] 0809 23:20:33.232061 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.624140 (+1392079us) service_pool.cc:227] Handling call
0809 23:20:34.718991 (+ 94851us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.744795 (+ 25804us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.744974 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.724579 32753 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:34.724431 32354 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.685462 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1079) took 1534 ms. Trace:
I20260809 23:20:34.752478 24427 rpcz_store.cc:276] 0809 23:20:33.150714 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.589338 (+1438624us) service_pool.cc:227] Handling call
0809 23:20:34.685393 (+ 96055us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.727943 32546 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.728920 24309 raft_consensus.cc:2354] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13.
I20260809 23:20:34.729300 32038 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:34.729715 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:34.677424   434 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.737321 24545 rpcz_store.cc:276] 0809 23:20:33.233946 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.640129 (+1406183us) service_pool.cc:227] Handling call
0809 23:20:34.663649 (+ 23520us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.725730 (+ 62081us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.725945 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.738103 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 2995) took 1486 ms. Trace:
I20260809 23:20:34.764382 24544 rpcz_store.cc:276] 0809 23:20:33.232488 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.631044 (+1398556us) service_pool.cc:227] Handling call
0809 23:20:34.639067 (+  8023us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.681593 (+ 42526us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.718657 (+ 37064us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.758325 24424 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:34.726086 32278 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.736114 32417 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.644939 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.746980 24431 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260809 23:20:34.723161 24313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.011s	sys 0.007s
I20260809 23:20:34.753521 24423 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:34.741710 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3001) took 2015 ms. Trace:
I20260809 23:20:34.754563 24422 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260809 23:20:34.754720 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.009s	sys 0.034s
W20260809 23:20:34.754779 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.005s	sys 0.017s
I20260809 23:20:34.759008 32753 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.726243 32684 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:34.776877 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3082) took 1276 ms. Trace:
I20260809 23:20:34.777480 24431 rpcz_store.cc:276] 0809 23:20:33.500918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.588692 (+1087774us) service_pool.cc:227] Handling call
0809 23:20:34.776795 (+188103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.778597 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2621) took 2391 ms. Trace:
I20260809 23:20:34.779134 24422 rpcz_store.cc:276] 0809 23:20:32.379417 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.742242 (+2362825us) service_pool.cc:227] Handling call
0809 23:20:34.742627 (+   385us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.770700 (+ 28073us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.771051 (+   351us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:34.776556 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71284.8us
I20260809 23:20:34.787103 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3070) took 2883 ms. Trace:
I20260809 23:20:34.788877 24423 rpcz_store.cc:276] 0809 23:20:31.891025 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.609990 (+2718965us) service_pool.cc:227] Handling call
0809 23:20:34.719011 (+109021us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.773977 (+ 54966us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.774222 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.786028 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2899) took 1545 ms. Trace:
I20260809 23:20:34.792457 24542 rpcz_store.cc:276] 0809 23:20:33.228506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.580873 (+1352367us) service_pool.cc:227] Handling call
0809 23:20:34.593970 (+ 13097us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.770902 (+176932us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.773570 (+  2668us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:34.776582 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.009s	sys 0.035s
I20260809 23:20:34.800202 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:34.787108 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:34.770040 32712 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.810884 32546 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:34.811059 32640 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:34.811568 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 943011us
W20260809 23:20:34.812894 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 953627us
I20260809 23:20:34.813436 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:34.814195 24424 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:20:34.815559 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1256) took 1128 ms. Trace:
I20260809 23:20:34.817970 32661 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:34.818887   406 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:34.771833 24314 rpcz_store.cc:276] 0809 23:20:32.725885 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.180402 (+454517us) service_pool.cc:227] Handling call
0809 23:20:33.180844 (+   442us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:33.201079 (+ 20235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:33.201128 (+    49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:33.220212 (+ 19084us) log.cc:842] Serialized 52 byte log entry
0809 23:20:33.702378 (+482166us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:33.733379 (+ 31001us) log.cc:842] Serialized 14 byte log entry
0809 23:20:34.324152 (+590773us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b180040fd80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:34.610322 (+286170us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.645460 (+ 35138us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:34.741260 (+ 95800us) raft_consensus.cc:1671] finished
0809 23:20:34.741408 (+   148us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.741559 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":441676,"prepare.queue_time_us":30824,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":10240,"raft.queue_time_us":411198,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":478381,"spinlock_wait_cycles":9086208,"thread_start_us":740680,"threads_started":2,"wal-append.queue_time_us":860463}
W20260809 23:20:34.820060 32225 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.992s	user 0.006s	sys 0.014s
I20260809 23:20:34.827235 24539 rpcz_store.cc:276] 0809 23:20:33.686707 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.754306 (+1067599us) service_pool.cc:227] Handling call
0809 23:20:34.814677 (+ 60371us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.815192 (+   515us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.815218 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.815298 (+    80us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:34.815332 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.815392 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.815497 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.803761 32471 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:34.833140 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1066) took 2022 ms. Trace:
I20260809 23:20:34.839216 24424 rpcz_store.cc:276] 0809 23:20:32.805425 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.800538 (+1995113us) service_pool.cc:227] Handling call
0809 23:20:34.813798 (+ 13260us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.826754 (+ 12956us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.827902 (+  1148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13056}
I20260809 23:20:34.838469 32471 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:34.857075 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1254) took 1171 ms. Trace:
I20260809 23:20:34.866065 24540 rpcz_store.cc:276] 0809 23:20:33.686099 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.755074 (+1068975us) service_pool.cc:227] Handling call
0809 23:20:34.781024 (+ 25950us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.848604 (+ 67580us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.848642 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.856748 (+  8106us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:34.856806 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.856883 (+    77us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.857012 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.872229 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:34.874039 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2898) took 1656 ms. Trace:
I20260809 23:20:34.871232 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:34.871984 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3017) took 1139 ms. Trace:
I20260809 23:20:34.849596 32681 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.883757 25371 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:34.874562 24546 rpcz_store.cc:276] 0809 23:20:33.217751 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.495406 (+1277655us) service_pool.cc:227] Handling call
0809 23:20:34.495937 (+   531us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.873565 (+377628us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:34.873611 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.873670 (+    59us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:34.873729 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.873814 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.873970 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5504,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":216}
I20260809 23:20:34.855932 32712 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:34.888818 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1072) took 2081 ms. Trace:
I20260809 23:20:34.889957 24423 rpcz_store.cc:276] 0809 23:20:32.807257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.805651 (+1998394us) service_pool.cc:227] Handling call
0809 23:20:34.861191 (+ 55540us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.878844 (+ 17653us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.878897 (+    53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.888139 (+  9242us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:34.888266 (+   127us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.888405 (+   139us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.888690 (+   285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.869732 32471 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:34.896000 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:34.876569 24313 rpcz_store.cc:276] 0809 23:20:33.732427 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.785210 (+ 52783us) service_pool.cc:227] Handling call
0809 23:20:33.839617 (+ 54407us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.842939 (+1003322us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.848817 (+  5878us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.855394 (+  6577us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:34.855457 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.865449 (+  9992us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.871908 (+  6459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":447247,"raft.queue_time_us":310876,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":451371,"thread_start_us":929717,"threads_started":1}
I20260809 23:20:34.900279 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1255) took 1213 ms. Trace:
I20260809 23:20:34.898154 24544 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:34.859064 32546 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:34.902647 24544 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:34.895494 24496 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:34.911768 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 101619us
I20260809 23:20:34.899631 32681 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.900926 24534 rpcz_store.cc:276] 0809 23:20:33.686390 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.745058 (+1058668us) service_pool.cc:227] Handling call
0809 23:20:34.745448 (+   390us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.745719 (+   271us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:34.752193 (+  6474us) log.cc:842] Serialized 14 byte log entry
0809 23:20:34.896272 (+144079us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.896416 (+   144us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:34.899942 (+  3526us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.900026 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.900211 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4087,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":6718,"thread_start_us":88432,"threads_started":1}
I20260809 23:20:34.915108 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:34.918633 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2624) took 2083 ms. Trace:
I20260809 23:20:34.920413 24423 rpcz_store.cc:276] 0809 23:20:32.835265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.891390 (+2056125us) service_pool.cc:227] Handling call
0809 23:20:34.905345 (+ 13955us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.918268 (+ 12923us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.918309 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:34.918380 (+    71us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:34.918414 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.918463 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.918571 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.918200 24311 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:34.926783 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:34.924913 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111632us
I20260809 23:20:34.936882   416 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:34.937593   416 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.939299 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:34.939771   413 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:34.940596 32761 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.508s	user 0.005s	sys 0.040s
W20260809 23:20:34.941146 32761 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.580s	user 0.005s	sys 0.041s
W20260809 23:20:34.942060 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 225258us
I20260809 23:20:34.942973 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:34.943642 32759 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.508s	user 0.018s	sys 0.037s
W20260809 23:20:34.944021 32759 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.581s	user 0.018s	sys 0.037s
W20260809 23:20:34.945276   310 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.329s	user 0.015s	sys 0.006s
I20260809 23:20:34.945353   416 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:34.945812   365 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:34.924386 32354 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.947052 32354 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.947440 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:34.949491 32354 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:34.951601 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2530) took 3351 ms. Trace:
I20260809 23:20:34.952068 24312 rpcz_store.cc:276] 0809 23:20:31.599620 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.599751 (+   131us) service_pool.cc:227] Handling call
0809 23:20:31.632806 (+ 33055us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:31.645304 (+ 12498us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:31.645352 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:31.651380 (+  6028us) log.cc:842] Serialized 26 byte log entry
0809 23:20:32.066290 (+414910us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:32.066334 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.066411 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:34.800595 (+2734184us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:34.951017 (+150422us) raft_consensus.cc:1671] finished
0809 23:20:34.951047 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:34.951367 (+   320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1359451,"raft.queue_time_us":18994,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":1312274,"spinlock_wait_cycles":30680448,"thread_start_us":414523,"threads_started":1,"wal-append.queue_time_us":760041}
I20260809 23:20:34.940692 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:34.939858 24313 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:34.940641   413 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.958994 24544 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.960197 24544 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:34.960103   406 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:34.960393   413 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:34.959254 32712 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:34.915179 32681 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:34.961184   406 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:34.991706 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:34.999991 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.000815 24423 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 15 index: 15. (index mismatch)
I20260809 23:20:35.001811 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1074) took 2041 ms. Trace:
I20260809 23:20:35.004961 24423 rpcz_store.cc:276] 0809 23:20:32.960449 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.964848 (+2004399us) service_pool.cc:227] Handling call
0809 23:20:34.998401 (+ 33553us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.001565 (+  3164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.001759 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:34.961194   413 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:34.986299 32417 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:34.920264 32032 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.006973 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.015820 24534 raft_consensus.cc:2396] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:35.009146 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:34.961561 32661 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:34.991261   390 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.022856   413 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.023619   390 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.025409   413 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.025977   390 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.026324 24313 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: [11.9-13.10]
I20260809 23:20:34.998595 32761 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.027590 32761 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.028554 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:34.986598 32278 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.029753 32761 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.005090 32753 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.030433 24424 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:35.030874 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.033793 32032 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.018105 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:35.035149 32068 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:35.036054 32068 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.036850 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.038551 32068 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.039312 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.039585 32032 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:35.018585 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.029781 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:34.996053   406 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:35.049604 24424 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:35.057365 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:35.061342 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:35.065091 24424 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:35.039343 32674 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:35.063015 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.074100 24423 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:35.046373 32032 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:35.074266 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.077149 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.079640 24321 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:35.080691   417 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:35.081415   417 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:35.087549 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.071805 32674 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:35.094354 24321 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:35.111207 32674 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:35.112610   366 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.802s	user 0.000s	sys 0.001s
I20260809 23:20:35.115190 24313 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 11.
I20260809 23:20:35.116640   365 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:35.118279   365 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:35.094861   300 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:35.097730 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2620) took 2719 ms. Trace:
I20260809 23:20:35.099895 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3019) took 8418 ms. Trace:
I20260809 23:20:35.079262 32032 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:35.138099 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46857.2us
I20260809 23:20:35.101246   336 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:35.107306 32546 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.107715 24316 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:35.138913 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2890) took 1957 ms. Trace:
I20260809 23:20:35.146999 24543 rpcz_store.cc:276] 0809 23:20:33.181736 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.864655 (+682919us) service_pool.cc:227] Handling call
0809 23:20:35.138660 (+1274005us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000b6154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:35.138853 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271447808}
I20260809 23:20:35.105554 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1259) took 1212 ms. Trace:
I20260809 23:20:35.119810 24428 rpcz_store.cc:276] 0809 23:20:26.681829 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:30.063995 (+3382166us) service_pool.cc:227] Handling call
0809 23:20:30.102440 (+ 38445us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:31.590840 (+1488400us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.099371 (+3508531us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54002f94d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:35.099685 (+   314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3410192128}
I20260809 23:20:35.098155 32697 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.123481 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.124500 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.125377 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:35.127214 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.137926   407 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:35.146323 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.120867 24323 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:35.158602 25365 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:35.151425 24542 rpcz_store.cc:276] 0809 23:20:33.893679 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.800975 (+907296us) service_pool.cc:227] Handling call
0809 23:20:35.104450 (+303475us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:35.104727 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.105176 (+   449us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:35.105204 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.105281 (+    77us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:35.105313 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.105376 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.105498 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":529,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":16088960}
I20260809 23:20:35.154500 24424 raft_consensus.cc:2396] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:35.119721 24425 rpcz_store.cc:276] 0809 23:20:32.378795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.665414 (+2286619us) service_pool.cc:227] Handling call
0809 23:20:34.727403 (+ 61989us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.728191 (+   788us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.728226 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:34.729210 (+   984us) log.cc:842] Serialized 26 byte log entry
0809 23:20:34.820768 (+ 91558us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:34.820809 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:34.820925 (+   116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:35.097451 (+276526us) raft_consensus.cc:1671] finished
0809 23:20:35.097469 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.097639 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":91213,"threads_started":1,"wal-append.queue_time_us":360629}
I20260809 23:20:35.139132 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:35.162499 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.118808   365 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:35.167618 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3012) took 1282 ms. Trace:
I20260809 23:20:35.175773 24544 rpcz_store.cc:276] 0809 23:20:33.885448 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.769935 (+884487us) service_pool.cc:227] Handling call
0809 23:20:34.843330 (+ 73395us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.162182 (+318852us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:35.166272 (+  4090us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.167025 (+   753us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:35.167130 (+   105us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.167239 (+   109us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.167500 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10837}
I20260809 23:20:35.176925 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:35.144955   320 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.503s	user 0.011s	sys 0.003s
I20260809 23:20:35.183750 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.178812 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:35.141954 32640 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:35.142289 32681 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.190173 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:35.192996 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.196034 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.197129 25371 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:35.199426 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.202180 24313 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:35.204586 24316 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:35.205905 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.206238 24314 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:35.206274 24428 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260809 23:20:35.207206 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.210472 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.190583 32640 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:35.212764 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.216930 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.217784 24314 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:35.222644   449 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260809 23:20:35.225018   444 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260809 23:20:35.226666 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2630) took 2150 ms. Trace:
I20260809 23:20:35.227094 24424 rpcz_store.cc:276] 0809 23:20:33.075805 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.173668 (+2097863us) service_pool.cc:227] Handling call
0809 23:20:35.209091 (+ 35423us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.226340 (+ 17249us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.226368 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.226428 (+    60us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:35.226460 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.226515 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.226611 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.229378 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2634) took 2088 ms. Trace:
I20260809 23:20:35.229688 24424 rpcz_store.cc:276] 0809 23:20:33.141056 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.228107 (+2087051us) service_pool.cc:227] Handling call
0809 23:20:35.228483 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.228972 (+   489us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.228997 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.229078 (+    81us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:35.229113 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.229174 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.229311 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.231243 32741 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:35.234259 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2632) took 2117 ms. Trace:
I20260809 23:20:35.234807 24428 rpcz_store.cc:276] 0809 23:20:33.094398 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.205662 (+2111264us) service_pool.cc:227] Handling call
0809 23:20:35.206038 (+   376us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:35.207230 (+  1192us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.211666 (+  4436us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.240168 24323 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:20:35.245270 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2631) took 2167 ms. Trace:
I20260809 23:20:35.249296 24430 rpcz_store.cc:276] 0809 23:20:33.077469 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.172030 (+2094561us) service_pool.cc:227] Handling call
0809 23:20:35.203256 (+ 31226us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.233970 (+ 30714us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.234389 (+   419us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.244962 (+ 10573us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:35.245007 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.245077 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.245205 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:35.252408 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113384us
I20260809 23:20:35.142534   300 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.257014   300 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.261044 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.263731 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.264272 24323 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:35.260579 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1065) took 2455 ms. Trace:
I20260809 23:20:35.273080 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.274487 24316 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:35.276149 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.282488 24323 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:20:35.284243 31984 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.539s	user 0.014s	sys 0.000s
I20260809 23:20:35.268872 24427 rpcz_store.cc:276] 0809 23:20:32.805012 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.773530 (+1968518us) service_pool.cc:227] Handling call
0809 23:20:34.849186 (+ 75656us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.887405 (+ 38219us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.887455 (+    50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:34.919601 (+ 32146us) log.cc:842] Serialized 26 byte log entry
0809 23:20:35.042083 (+122482us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:35.258592 (+216509us) log.cc:842] Serialized 14 byte log entry
0809 23:20:35.259347 (+   755us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.259576 (+   229us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:35.259924 (+   348us) raft_consensus.cc:1671] finished
0809 23:20:35.260149 (+   225us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.260448 (+   299us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":80408,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":16581,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":280098,"threads_started":2,"wal-append.queue_time_us":175217}
I20260809 23:20:35.291455 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:35.267426 24164 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.299078 32634 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.299546 32634 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.301576 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1069) took 2495 ms. Trace:
I20260809 23:20:35.279517 32380 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.313218 32380 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.247406 32471 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.328862 25371 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:35.323030 24423 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:35.302057 24422 rpcz_store.cc:276] 0809 23:20:32.806400 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.804991 (+1998591us) service_pool.cc:227] Handling call
0809 23:20:34.813763 (+  8772us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.814386 (+   623us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:34.886168 (+ 71782us) log.cc:842] Serialized 14 byte log entry
0809 23:20:35.006576 (+120408us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:35.077986 (+ 71410us) log.cc:842] Serialized 26 byte log entry
0809 23:20:35.078141 (+   155us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:35.121680 (+ 43539us) log.cc:842] Serialized 14 byte log entry
0809 23:20:35.121923 (+   243us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.121992 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:35.301318 (+179326us) raft_consensus.cc:1671] finished
0809 23:20:35.301346 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.301508 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43242,"prepare.queue_time_us":352664,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":77,"raft.queue_time_us":3338,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":344,"thread_start_us":157731,"threads_started":2,"wal-append.queue_time_us":276308}
I20260809 23:20:35.328146 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:35.322461 32684 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.522s	user 0.006s	sys 0.006s
I20260809 23:20:35.338689 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.345739 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.348183 32380 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.348944 24545 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260809 23:20:35.351603   442 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260809 23:20:35.357626 32471 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.369283 24422 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:35.370730 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3084) took 1091 ms. Trace:
I20260809 23:20:35.371690 24422 rpcz_store.cc:276] 0809 23:20:34.279141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.332245 (+1053104us) service_pool.cc:227] Handling call
0809 23:20:35.369048 (+ 36803us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.370403 (+  1355us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.370649 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.363392 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3083) took 1520 ms. Trace:
I20260809 23:20:35.384374 24423 rpcz_store.cc:276] 0809 23:20:33.842962 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.362003 (+1519041us) service_pool.cc:227] Handling call
0809 23:20:35.362528 (+   525us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.362797 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.362835 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.362907 (+    72us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:35.363091 (+   184us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.363162 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.363310 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.180932   417 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.146271   406 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.405476 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.399016 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.399999 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:35.407778 24422 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:35.363967 32697 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:35.408910 32202 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.290062   365 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.411945   352 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:35.420092 24423 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:35.421053 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3089) took 1045 ms. Trace:
I20260809 23:20:35.424949 24423 rpcz_store.cc:276] 0809 23:20:34.376014 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.410978 (+1034964us) service_pool.cc:227] Handling call
0809 23:20:35.412818 (+  1840us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.420702 (+  7884us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.420987 (+   285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:35.428208 32471 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:35.431169 24314 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:35.432317 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:35.439154 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 448425us
W20260809 23:20:35.417506 32641 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.005s	sys 0.020s
I20260809 23:20:35.391235   300 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
W20260809 23:20:35.451717 32641 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.005s	sys 0.021s
I20260809 23:20:35.452500 24423 pending_rounds.cc:76] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:20:35.462790 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:35.527686 24423 pending_rounds.cc:97] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260809 23:20:35.529328 24423 raft_consensus.cc:2866] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:35.533989 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1068) took 2723 ms. Trace:
W20260809 23:20:35.542205 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 159240us
I20260809 23:20:35.544190 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.421334 32202 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.557608 24535 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:35.554693 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.545965   365 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:35.568064 32546 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:35.446301   352 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.576463 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:35.579298   457 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260809 23:20:35.579887 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.74527e+06us
I20260809 23:20:35.580132 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.581199 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.582952 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:35.588850 32684 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:35.589123 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3012) took 2293 ms. Trace:
I20260809 23:20:35.589728 24311 rpcz_store.cc:276] 0809 23:20:33.296160 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.651882 (+355722us) service_pool.cc:227] Handling call
0809 23:20:35.588842 (+1936960us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:35.589029 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2482361600}
W20260809 23:20:35.591209   326 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.180s	user 0.003s	sys 0.058s
W20260809 23:20:35.591610   326 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.363s	user 0.003s	sys 0.058s
I20260809 23:20:35.592975 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1348) took 2515 ms. Trace:
I20260809 23:20:35.593364 25375 rpcz_store.cc:276] 0809 23:20:33.077163 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.130108 (+ 52945us) service_pool.cc:227] Handling call
0809 23:20:33.152637 (+ 22529us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:33.795339 (+642702us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.592098 (+1796759us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:35.592805 (+   707us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1414391552}
I20260809 23:20:35.595746 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3063) took 4129 ms. Trace:
I20260809 23:20:35.559012   406 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:35.565759 32038 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
W20260809 23:20:35.596606 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46313.2us
I20260809 23:20:35.606632 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.632582 24316 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:35.611598 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:35.597793 24435 rpcz_store.cc:276] 0809 23:20:31.272314 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.983493 (+711179us) service_pool.cc:227] Handling call
0809 23:20:32.020296 (+ 36803us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:32.020500 (+   204us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:32.020523 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:32.031586 (+ 11063us) log.cc:842] Serialized 26 byte log entry
0809 23:20:32.215928 (+184342us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:32.215994 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:32.216118 (+   124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:35.143903 (+2927785us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:35.339548 (+195645us) raft_consensus.cc:1671] finished
0809 23:20:35.339594 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.401529 (+ 61935us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36011904,"thread_start_us":125126,"threads_started":1,"wal-append.queue_time_us":378299}
I20260809 23:20:35.554759 32380 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260809 23:20:35.563227 32471 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:35.642867 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61828.3us
I20260809 23:20:35.663131 32225 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:35.663614 32225 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:35.664633 32225 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:35.665956 24167 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 371) took 41049 ms (client timeout 15000 ms). Trace:
I20260809 23:20:35.666311 32225 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:35.666472 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:35.611217   433 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:35.686199 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.691421 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.692986 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:35.705480   352 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:35.705840 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49013.9us
W20260809 23:20:35.707871 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 164768us
I20260809 23:20:35.707398 24164 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 12, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.708954 24269 leader_election.cc:398] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:35.709389 24269 leader_election.cc:401] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260809 23:20:35.711974 32202 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:20:35.717324 32640 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.694617 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.674563 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.710038 24164 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 7 to 9, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 9 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.710769 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.674484 24541 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:20:35.728609 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3085) took 1448 ms. Trace:
I20260809 23:20:35.742556 24429 rpcz_store.cc:276] 0809 23:20:34.279955 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.516404 (+236449us) service_pool.cc:227] Handling call
0809 23:20:35.728390 (+1211986us) spinlock_profiling.cc:243] Waited 78.5 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:35.728537 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":180438912}
I20260809 23:20:35.740295 24544 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:35.652087   336 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:35.724529 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:35.751209 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:35.750252 24543 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:35.707420 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:35.761667 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 95341us
W20260809 23:20:35.725819 32674 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.614s	user 0.008s	sys 0.004s
W20260809 23:20:35.767970 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98102.1us
I20260809 23:20:35.768147 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:35.722874 32032 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.594s	user 0.006s	sys 0.003s
I20260809 23:20:35.771066 24164 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 11, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 11 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.777158 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.734084   332 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:35.779630 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:35.788451 25369 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:35.670616 31661 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:35.793232 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 82587.9us
W20260809 23:20:35.789618 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 82339.7us
W20260809 23:20:35.794538 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 183069us
I20260809 23:20:35.803025 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.775774 32681 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:35.659782 32471 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:35.810945 24313 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.000s
I20260809 23:20:35.817196 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:35.809671 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:35.815377 24313 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:35.822410 24164 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 6 to 12, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.788647 32684 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:35.791554   407 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:35.770157   336 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:35.776423 32032 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.774273 32674 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.804596 31984 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:35.845202 31984 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:35.847795 31984 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.851616 32634 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:35.859588 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65480.9us
I20260809 23:20:35.811915 32681 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:35.842180 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.841017 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:35.852430 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 14, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 14 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:35.810833 31661 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:35.887935 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43923.8us
I20260809 23:20:35.888497 24312 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:35.893054 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59332us
I20260809 23:20:35.852967   433 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:35.869670   315 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:35.877517 32681 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:35.904403 24316 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:35.880270   300 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:35.907272 24164 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 12, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:20:35.911410 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.024s	sys 0.057s
W20260809 23:20:35.913271 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.025s	sys 0.057s
W20260809 23:20:35.918620 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52334.3us
I20260809 23:20:35.923377 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3078) took 2529 ms. Trace:
W20260809 23:20:35.899421   465 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.218s	user 0.001s	sys 0.001s
I20260809 23:20:35.927053 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:35.930454 24435 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
W20260809 23:20:35.931437 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70817.2us
I20260809 23:20:35.908751 32640 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:35.925379 32202 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260809 23:20:35.942173 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 11, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 11 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:35.929574 32674 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:35.918473 32694 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:35.926450   300 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:35.936370 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:35.923925 24428 rpcz_store.cc:276] 0809 23:20:33.394277 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.236257 (+1841980us) service_pool.cc:227] Handling call
0809 23:20:35.285037 (+ 48780us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.285388 (+   351us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:35.340159 (+ 54771us) log.cc:842] Serialized 14 byte log entry
0809 23:20:35.923000 (+582841us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:35.923057 (+    57us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:35.923110 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.923202 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.923323 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39055,"prepare.queue_time_us":123814,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"thread_start_us":529699,"threads_started":1}
I20260809 23:20:35.977871 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:35.975178 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.023s	sys 0.015s
W20260809 23:20:35.985623 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.023s	sys 0.015s
W20260809 23:20:35.988269 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75177.9us
W20260809 23:20:35.980254   473 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260809 23:20:35.994117 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:35.987504 24164 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 10, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:36.000248 24435 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260809 23:20:36.000731 24435 pending_rounds.cc:76] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:20:36.000993 24435 pending_rounds.cc:97] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260809 23:20:35.536052 24431 rpcz_store.cc:276] 0809 23:20:32.806092 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.793739 (+1987647us) service_pool.cc:227] Handling call
0809 23:20:34.837530 (+ 43791us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:34.852740 (+ 15210us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:34.852787 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:34.898735 (+ 45948us) log.cc:842] Serialized 26 byte log entry
0809 23:20:35.066673 (+167938us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:35.528606 (+461933us) log.cc:842] Serialized 14 byte log entry
0809 23:20:35.528986 (+   380us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.529117 (+   131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:35.529196 (+    79us) raft_consensus.cc:1671] finished
0809 23:20:35.529208 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:35.529408 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15672,"prepare.queue_time_us":56020,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":8527,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":10565,"thread_start_us":546719,"threads_started":2,"wal-append.queue_time_us":225160}
I20260809 23:20:36.004338 25373 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:20:36.004570 24431 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:20:36.005489 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.006244 25373 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:35.985252   433 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260809 23:20:36.018299   300 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.019032   407 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:36.021697   336 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:35.666410 24167 rpcz_store.cc:269] 0809 23:19:54.687355 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.688914 (+  1559us) service_pool.cc:227] Handling call
0809 23:20:35.665697 (+40976783us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:20:21.501976 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:20:21.502544 (+   568us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:20:21.502604 (+    60us) write_op.cc:435] Acquired schema lock
0809 23:20:21.502616 (+    12us) tablet.cc:660] Decoding operations
0809 23:20:21.504003 (+  1387us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:20:21.504103 (+   100us) write_op.cc:641] Partition lock acquired for write op
0809 23:20:21.504130 (+    27us) tablet.cc:683] Acquiring locks for 14 operations
0809 23:20:21.505266 (+  1136us) tablet.cc:699] Row locks acquired
0809 23:20:21.505308 (+    42us) write_op.cc:260] PREPARE: finished
0809 23:20:21.505472 (+   164us) write_op.cc:270] Start()
0809 23:20:21.505611 (+   139us) write_op.cc:276] Timestamp: P: 1786317621505408 usec, L: 0
0809 23:20:21.505632 (+    21us) op_driver.cc:331] REPLICATION: starting
0809 23:20:21.597485 (+ 91853us) log.cc:842] Serialized 5509 byte log entry
0809 23:20:23.327585 (+1730100us) op_driver.cc:444] REPLICATION: finished
0809 23:20:23.556980 (+229395us) write_op.cc:301] APPLY: starting
0809 23:20:23.557350 (+   370us) tablet.cc:1365] starting BulkCheckPresence
0809 23:20:23.566027 (+  8677us) tablet.cc:1368] finished BulkCheckPresence
0809 23:20:23.753071 (+187044us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:20:24.098512 (+345441us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:20:24.150413 (+ 51901us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0809 23:20:24.157856 (+  7443us) write_op.cc:312] APPLY: finished
0809 23:20:24.171834 (+ 13978us) log.cc:842] Serialized 129 byte log entry
0809 23:20:24.172117 (+   283us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:24.172603 (+   486us) write_op.cc:454] Released schema lock
0809 23:20:24.172725 (+   122us) write_op.cc:341] FINISH: Updating metrics
0809 23:20:24.173258 (+   533us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:24.173285 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":245535,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":174181,"apply.run_cpu_time_us":27328,"apply.run_wall_time_us":616933,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62591,"num_ops":14,"prepare.queue_time_us":147203,"prepare.run_cpu_time_us":50248,"prepare.run_wall_time_us":1346144,"raft.queue_time_us":240185,"raft.run_cpu_time_us":25201,"raft.run_wall_time_us":255534,"replication_time_us":1806738,"thread_start_us":1655732,"threads_started":4,"wal-append.queue_time_us":1053507,"wal-append.run_cpu_time_us":15796,"wal-append.run_wall_time_us":2515401}]]}
I20260809 23:20:36.023273 32032 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:36.023082 32391 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260809 23:20:36.001266 24435 raft_consensus.cc:2866] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:36.026583 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:35.981344 32694 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:36.028097 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:36.028286 24422 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.609s	user 0.005s	sys 0.006s
I20260809 23:20:36.039794 24422 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:36.031579 25369 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:36.048098 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3081) took 2547 ms. Trace:
I20260809 23:20:36.048182 25373 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260809 23:20:36.048714 24427 rpcz_store.cc:276] 0809 23:20:33.500235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.293407 (+1793172us) service_pool.cc:227] Handling call
0809 23:20:35.293842 (+   435us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:35.304100 (+ 10258us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.304142 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:35.313221 (+  9079us) log.cc:842] Serialized 26 byte log entry
0809 23:20:35.760523 (+447302us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:35.760564 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.760632 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:36.047834 (+287202us) raft_consensus.cc:1671] finished
0809 23:20:36.047875 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.048032 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6552,"raft.queue_time_us":11254,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10039,"thread_start_us":446771,"threads_started":1,"wal-append.queue_time_us":631912}
I20260809 23:20:35.995574   434 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:36.055447 24428 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:36.056155 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3097) took 1012 ms. Trace:
I20260809 23:20:36.056511 24428 rpcz_store.cc:276] 0809 23:20:35.044052 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.994737 (+950685us) service_pool.cc:227] Handling call
0809 23:20:36.051347 (+ 56610us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:36.055929 (+  4582us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.056108 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:36.040412   336 raft_consensus.cc:2741] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260809 23:20:36.047070 32587 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:36.077366 24164 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 6 to 8, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 8 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:36.079926 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3096) took 1036 ms. Trace:
I20260809 23:20:36.080413 24435 rpcz_store.cc:276] 0809 23:20:35.043773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.979202 (+935429us) service_pool.cc:227] Handling call
0809 23:20:35.986578 (+  7376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:36.073800 (+ 87222us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.079838 (+  6038us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:36.083177 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 153848us
I20260809 23:20:36.054652 32380 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:36.051966   407 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:36.093844 24164 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 1 to 10. New cstate: current_term: 10 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:20:36.100435 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1345) took 3023 ms (client timeout 2006 ms). Trace:
I20260809 23:20:36.101156 32747 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260809 23:20:36.102070 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1261) took 2111 ms. Trace:
I20260809 23:20:36.102403 24317 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:36.103057 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2537) took 1164 ms. Trace:
I20260809 23:20:36.103297 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2876) took 4742 ms. Trace:
W20260809 23:20:36.104252 32741 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.872s	user 0.014s	sys 0.000s
I20260809 23:20:36.104745 32741 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:36.105362   407 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.105948 32417 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:36.106472 32417 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:36.078040   352 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:36.106907 32694 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.112298 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1090) took 5212 ms. Trace:
I20260809 23:20:36.112727 24427 rpcz_store.cc:276] 0809 23:20:30.900210 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.059315 (+5159105us) service_pool.cc:227] Handling call
0809 23:20:36.112223 (+ 52908us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:36.107867   352 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:36.103368 24317 rpcz_store.cc:276] 0809 23:20:34.938571 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.938745 (+   174us) service_pool.cc:227] Handling call
0809 23:20:34.939282 (+   537us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:36.102247 (+1162965us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5400457a20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:36.102795 (+   548us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.103007 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36806784}
I20260809 23:20:36.107507 32417 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.103223 31661 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:36.116003 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 77739.5us
I20260809 23:20:36.116644 31661 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.102424 24545 rpcz_store.cc:276] 0809 23:20:33.990964 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.809403 (+818439us) service_pool.cc:227] Handling call
0809 23:20:35.139381 (+329978us) spinlock_profiling.cc:243] Waited 9.85 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:35.139609 (+   228us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:35.349467 (+209858us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.101767 (+752300us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:36.101984 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":499135616}
I20260809 23:20:36.103785 24535 rpcz_store.cc:276] 0809 23:20:31.361234 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:31.367429 (+  6195us) service_pool.cc:227] Handling call
0809 23:20:31.367700 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.566153 (+4198453us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.103024 (+536871us) spinlock_profiling.cc:243] Waited 3.94 s on lock 0x7b5400116554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:36.103226 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9050850176}
I20260809 23:20:36.117033 31884 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:36.110580 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:36.125535 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54828.1us
W20260809 23:20:36.100848 24309 rpcz_store.cc:269] 0809 23:20:33.076964 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:33.095230 (+ 18266us) service_pool.cc:227] Handling call
0809 23:20:36.100147 (+3004917us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:36.100342 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3625366912}
I20260809 23:20:36.123186 25363 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:36.129536 32417 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:36.129561 24164 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 13, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:20:36.129994 31884 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:36.112746 32694 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:36.145955 24164 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 10, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:36.152345 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:36.152333 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3079) took 2757 ms. Trace:
I20260809 23:20:36.114542   352 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.141676 31884 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:36.141494 32380 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.009s
I20260809 23:20:36.131584 31661 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.173014 31661 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:36.174697 31661 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:36.175998 31661 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:36.162223 24430 rpcz_store.cc:276] 0809 23:20:33.394655 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.257665 (+1863010us) service_pool.cc:227] Handling call
0809 23:20:35.293080 (+ 35415us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.293436 (+   356us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:35.390579 (+ 97143us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.151947 (+761368us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.152004 (+    57us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:36.152056 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.152124 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.152264 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46695,"prepare.queue_time_us":157518,"prepare.run_cpu_time_us":7177,"prepare.run_wall_time_us":89065,"raft.queue_time_us":210818,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":107,"thread_start_us":742711,"threads_started":2,"wal-append.queue_time_us":573842}
I20260809 23:20:36.179584 32417 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:36.180037 32741 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:36.181330   417 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:20:36.190277 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3077) took 2991 ms. Trace:
I20260809 23:20:36.181456 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:36.191974 24424 rpcz_store.cc:276] 0809 23:20:33.198710 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.230744 (+2032034us) service_pool.cc:227] Handling call
0809 23:20:35.238621 (+  7877us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.260279 (+ 21658us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:35.324436 (+ 64157us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.189362 (+864926us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.189412 (+    50us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:36.189468 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.190093 (+   625us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.190214 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7331,"prepare.queue_time_us":63852,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4905,"raft.queue_time_us":208240,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":801848,"threads_started":2,"wal-append.queue_time_us":660509}
W20260809 23:20:36.183199 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 432129us
I20260809 23:20:36.184151 32741 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:36.190636 32684 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:36.159658 32747 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:36.224678 32741 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:36.231607   417 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:20:36.264324 25373 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:36.227485 32225 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.264819 25369 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:36.287923 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:36.288318 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:36.289387 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.289888 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:36.289963 25368 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:36.292659 25368 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:36.296217   315 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:36.296615   315 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:36.297482   315 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.300246 32380 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.300721 32380 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.311393 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1396) took 1686 ms. Trace:
I20260809 23:20:36.311884 25370 rpcz_store.cc:276] 0809 23:20:34.624713 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.326200 (+701487us) service_pool.cc:227] Handling call
0809 23:20:35.343303 (+ 17103us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:35.419573 (+ 76270us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.419620 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:35.541890 (+122270us) log.cc:842] Serialized 26 byte log entry
0809 23:20:35.994392 (+452502us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:35.994598 (+   206us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:35.994675 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:36.309565 (+314890us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:36.311005 (+  1440us) raft_consensus.cc:1671] finished
0809 23:20:36.311031 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.311294 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10184,"raft.queue_time_us":11600,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":5174,"spinlock_wait_cycles":101664640,"thread_start_us":380830,"threads_started":1,"wal-append.queue_time_us":591267}
W20260809 23:20:36.315390 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 133972us
I20260809 23:20:36.320819 32747 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.321579 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3091) took 1874 ms. Trace:
I20260809 23:20:36.321964 24423 rpcz_store.cc:276] 0809 23:20:34.447347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.431190 (+983843us) service_pool.cc:227] Handling call
0809 23:20:35.452261 (+ 21071us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:35.600475 (+148214us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:35.600524 (+    49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:35.609579 (+  9055us) log.cc:842] Serialized 52 byte log entry
0809 23:20:36.066816 (+457237us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:36.066869 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.066974 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:36.321351 (+254377us) raft_consensus.cc:1671] finished
0809 23:20:36.321374 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.321522 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12465,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":130,"thread_start_us":393641,"threads_started":1,"wal-append.queue_time_us":663496}
I20260809 23:20:36.327134 32225 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.329208 32225 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:36.329712   365 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.726s	user 0.014s	sys 0.000s
I20260809 23:20:36.331132 32634 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.331776 32634 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.334532 32634 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:36.175417 31884 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:36.337846 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45927.8us
I20260809 23:20:36.339382 32471 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:36.298724   315 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:36.344465 24321 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:36.348584 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:36.352196 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:36.302606 32380 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:36.362821 25373 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260809 23:20:36.363653 25371 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.017s	sys 0.042s
W20260809 23:20:36.364153 25371 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.017s	sys 0.042s
I20260809 23:20:36.367599 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:36.346704 32747 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:36.376399 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:36.347246   417 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260809 23:20:36.373172   491 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:36.383236   406 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:36.379084 32681 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.418691   491 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:36.419827   491 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.420712 32681 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:36.411659 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 65308.1us
I20260809 23:20:36.421761   491 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:36.422097 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:36.382460 31884 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:36.417841   332 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.000s
I20260809 23:20:36.424803 32684 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:36.424815 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:36.377071 32471 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:36.428740   417 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:36.432901   417 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:36.401271   494 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:36.430614   332 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:36.439561 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:36.447734 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:36.429666 32471 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:36.463989 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3099) took 1276 ms. Trace:
I20260809 23:20:36.464493 24431 rpcz_store.cc:276] 0809 23:20:35.187817 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.005431 (+817614us) service_pool.cc:227] Handling call
0809 23:20:36.005717 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:36.440341 (+434624us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:36.440400 (+    59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.463674 (+ 23274us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:36.463723 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.463795 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.463929 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":167566,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":424141,"threads_started":1}
I20260809 23:20:36.435318 32634 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:36.441911 32038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.003s	sys 0.008s
I20260809 23:20:36.437089 32681 raft_consensus.cc:2741] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260809 23:20:36.443953   417 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:36.464860 32471 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:36.442174   494 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:36.467167 32640 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:36.487058 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1399) took 1094 ms. Trace:
W20260809 23:20:36.474570 32038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.003s	sys 0.008s
I20260809 23:20:36.497169 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:36.497891   509 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260809 23:20:36.502127 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1262) took 2218 ms. Trace:
W20260809 23:20:36.515136 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 406273us
I20260809 23:20:36.471894   493 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:36.525599 24423 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260809 23:20:36.526197 24423 pending_rounds.cc:76] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 9
I20260809 23:20:36.526479 24423 pending_rounds.cc:97] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260809 23:20:36.526739 24423 raft_consensus.cc:2866] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:36.487535 25372 rpcz_store.cc:276] 0809 23:20:35.393217 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.410098 (+ 16881us) service_pool.cc:227] Handling call
0809 23:20:35.423878 (+ 13780us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.448095 (+ 24217us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:35.990811 (+542716us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.486656 (+495845us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.486707 (+    51us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:36.486760 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.486828 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.487001 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":780429,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":6903,"raft.queue_time_us":87,"thread_start_us":870518,"threads_started":2}
I20260809 23:20:36.488322 32640 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:36.517375 24539 rpcz_store.cc:276] 0809 23:20:34.284024 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:34.840836 (+556812us) service_pool.cc:227] Handling call
0809 23:20:34.856232 (+ 15396us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:34.887942 (+ 31710us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:34.911527 (+ 23585us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.145567 (+1234040us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b1800331c60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:36.501678 (+356111us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.501749 (+    71us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:36.501822 (+    73us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.501918 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.502057 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14150,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":1441,"spinlock_wait_cycles":2993920,"thread_start_us":355833,"threads_started":1,"wal-append.queue_time_us":1582231}
W20260809 23:20:36.530613 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 351965us
I20260809 23:20:36.486819   494 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:36.531914   493 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.536525 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.537449 24424 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:20:36.539724 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:36.540504 24430 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:36.518646   509 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:36.475844 32741 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.478578 32747 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260809 23:20:36.543288 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.550107 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.537873 32640 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:36.560474 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.563915 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.574426 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.575661 24423 raft_consensus.cc:2396] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:36.580415 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.549775   509 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:36.573132 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.560320 32747 consensus_queue.cc:1225] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:36.583330 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.583457 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:36.584021 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.581071 24427 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:36.587196 24323 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 12: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:36.585104 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:36.594882 24314 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:36.594736 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.587953 24313 raft_consensus.cc:2362] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14. Re-sending same reply.
I20260809 23:20:36.589243 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.596422 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.596621 24309 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:36.597574 24320 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:36.599733 24540 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:36.602231 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.602618 24544 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:36.603194 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:36.604234 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:36.605763 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.606567 24542 raft_consensus.cc:2396] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:36.604156 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:36.608464 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:36.552088 32202 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260809 23:20:36.597028 24435 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:36.617718 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2906) took 1451 ms. Trace:
I20260809 23:20:36.618270 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.619611 24542 rpcz_store.cc:276] 0809 23:20:35.166739 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.579302 (+1412563us) service_pool.cc:227] Handling call
0809 23:20:36.617638 (+ 38336us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:36.620641 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:36.621227 24539 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:36.622421 24534 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:36.623817 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 173441us
I20260809 23:20:36.627781 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.628787 24546 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:36.632220 24541 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:36.625173 24545 pending_rounds.cc:76] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 8
I20260809 23:20:36.633423 24545 pending_rounds.cc:97] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260809 23:20:36.633654 24545 raft_consensus.cc:2866] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:36.624863 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:36.635484 24311 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:36.636468 25372 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:36.639139 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:36.629453 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.632584 24541 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:36.633903 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.633572 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1400) took 1201 ms. Trace:
I20260809 23:20:36.651551 25371 rpcz_store.cc:276] 0809 23:20:35.430130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.431049 (+   919us) service_pool.cc:227] Handling call
0809 23:20:35.451401 (+ 20352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:35.561877 (+110476us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:36.375718 (+813841us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.630064 (+254346us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:36.630137 (+    73us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:36.630203 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.631383 (+  1180us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:36.631512 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":759575,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":110,"thread_start_us":970233,"threads_started":2}
W20260809 23:20:36.636611 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 139559us
I20260809 23:20:36.667220 24311 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:36.694314 24427 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:36.624344   434 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:36.702204 24540 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
W20260809 23:20:36.732918 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 211572us
I20260809 23:20:36.632676   332 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.089s
I20260809 23:20:36.709297 25370 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:36.631842   499 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:36.637605 32694 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:36.695531 24317 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:36.635679 32380 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260809 23:20:36.709725 24541 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.719630 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:36.719511 32225 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.633147 32354 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:36.708324 24314 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260809 23:20:36.735001 24541 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:36.721041 31984 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.735039 24544 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:36.735100 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 110357us
I20260809 23:20:36.733654 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:36.719659 32681 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260809 23:20:36.742871 24311 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260809 23:20:36.753216 32380 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.754015 32380 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.762881 24430 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:36.734596   315 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260809 23:20:36.755972   529 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:36.756383   493 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:36.757728 24317 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260809 23:20:36.707315   407 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260809 23:20:36.732481   352 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260809 23:20:36.775594 32380 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:36.815981 24427 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
W20260809 23:20:36.783331 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50460.2us
I20260809 23:20:36.785328 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.732380   434 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:36.782830 25370 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260809 23:20:36.783751 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.778635 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:36.784605 24534 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:20:36.848683 24542 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:20:36.795306 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54187.6us
W20260809 23:20:36.849565 24269 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.538s (SENT)
I20260809 23:20:36.822443 24544 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:36.775914 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.746809 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:36.852011 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.856451 24540 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:36.857239 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:36.739825 32354 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:36.858799 24541 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:36.860543 24546 raft_consensus.cc:2396] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:36.783569   497 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:36.867518 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91196us
I20260809 23:20:36.902088 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:36.903399 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:36.874408 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.883289 32380 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:36.738332 32225 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.863986   434 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:36.869247   492 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:36.785539 32681 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.905007 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.905758 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:36.917387 32380 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:36.792289 32697 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260809 23:20:36.926524 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:36.778074 31984 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:36.926671 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108583us
I20260809 23:20:36.932363 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:36.742532   499 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:36.900971 32354 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:36.936353 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.039119 24546 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:20:36.943043 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:36.944972 31884 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.517s	user 0.013s	sys 0.000s
I20260809 23:20:36.942704 32640 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:36.945914   532 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.946683 32634 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:36.917972 32417 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:36.901178   533 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:36.964453   529 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:36.965870 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.966979   538 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:36.968204 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:36.907028   531 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:36.972079 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.987329 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:36.989600 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 207145us
I20260809 23:20:36.996575 24539 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:36.997090 24540 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:37.020607 24430 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:36.962260 32741 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:37.022737 24541 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:37.025794 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:36.922330   497 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.112568 24543 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:37.099346 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:37.099959 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2560) took 1682 ms (client timeout 1921 ms). Trace:
W20260809 23:20:37.098765 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58489.4us
I20260809 23:20:37.015338   492 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:37.181427 24317 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:37.101359   332 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:37.102211 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.196966 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.197290 25368 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:37.095212 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 238062us
I20260809 23:20:37.110286 24542 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:36.989382 32694 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:37.034054 32687 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260809 23:20:37.208525 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.209316 24314 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:37.035319 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 300822us
I20260809 23:20:37.022282 32681 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.153693 24544 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260809 23:20:37.178884 25371 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:37.181485 24316 rpcz_store.cc:269] 0809 23:20:35.417132 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.417333 (+   201us) service_pool.cc:227] Handling call
0809 23:20:37.038126 (+1620793us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540019d7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.099885 (+ 61759us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":458821888}
I20260809 23:20:37.188165 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:37.131377   532 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.217671 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.192334   533 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.193149   544 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:37.151005 32380 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.097605   407 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.194242   529 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.194804 24534 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:37.106271   502 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:37.229219 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.229334 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:37.204025 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 355431us
I20260809 23:20:37.097470   493 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.231954 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.233067 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.230029 25372 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
W20260809 23:20:37.235541 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2547) took 2108 ms (client timeout 2246 ms). Trace:
I20260809 23:20:37.235736 24309 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:37.236083 24312 rpcz_store.cc:269] 0809 23:20:35.127392 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.189929 (+ 62537us) service_pool.cc:227] Handling call
0809 23:20:37.235266 (+2045337us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b540043bcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.235467 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1545112448}
I20260809 23:20:37.020020   494 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.241603 24424 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:37.215844 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:37.195341   539 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260809 23:20:37.219837 32761 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:37.248633 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:37.192711   508 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260809 23:20:37.217341 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.223590 24311 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260809 23:20:37.223816 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:37.264329 24430 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:37.227610 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.227685 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.265863 24316 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:20:37.202522   492 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:37.178766   531 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:37.116114   509 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.000s
I20260809 23:20:37.229976 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:37.230553 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:37.239938   529 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.244119 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.273937 25368 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:37.240834 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1086) took 1655 ms (client timeout 2066 ms). Trace:
W20260809 23:20:37.274549 24429 rpcz_store.cc:269] 0809 23:20:35.585608 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.766432 (+180824us) service_pool.cc:227] Handling call
0809 23:20:37.240677 (+1474245us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.240779 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123664256}
W20260809 23:20:37.272095 24378 connection.cc:232] RPC call timeout handler was delayed by -1.29955202084966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20260809 23:20:37.217185 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 244836us
I20260809 23:20:37.276597 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:37.179996 32634 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260809 23:20:37.280587 24312 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260809 23:20:37.263585   417 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.775s	user 0.011s	sys 0.000s
I20260809 23:20:37.247735 32380 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:37.265314 24317 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:37.288017 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:37.288899 24430 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:37.180366   538 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.271325 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.293401 24539 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:37.277915 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 78749.4us
I20260809 23:20:37.294479 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:37.266076 24314 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:20:37.282418 25372 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:37.250991   336 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:37.271010 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.297722 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:37.298813 24429 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:37.303586 25373 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:37.304868 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:37.307919 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66918.1us
I20260809 23:20:37.245206 32694 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.232180   407 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.275810   531 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:37.318095 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 129855us
I20260809 23:20:37.318480 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.247138   493 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.320844 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.321663 24539 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:37.321544   493 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.322389 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:37.375835 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67144.1us
I20260809 23:20:37.296797 32687 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260809 23:20:37.378340 24429 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260809 23:20:37.381230 24430 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260809 23:20:37.381629 24430 pending_rounds.cc:76] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 7
I20260809 23:20:37.381850 24430 pending_rounds.cc:97] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 12.8
W20260809 23:20:37.318825   491 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.896s	user 0.013s	sys 0.003s
I20260809 23:20:37.383247 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:37.384020 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61702.4us
I20260809 23:20:37.382124 24430 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:37.276211 32752 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260809 23:20:37.324342   556 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.386166 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.278373 32761 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.278519   509 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.384815 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:37.405768 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.401127 24435 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:37.416373 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 139046us
I20260809 23:20:37.290079   417 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.385564   493 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:37.421954 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:37.422894 24539 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:37.427814 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.314990   336 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.385139 32225 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.424094 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:37.455833 24545 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260809 23:20:37.409947 31984 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.459573 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.476668 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:37.386876   558 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:37.482766 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 170200us
I20260809 23:20:37.414758   491 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:37.480885 24540 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:37.424073   526 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.489914 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:37.429029   493 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:37.389976   502 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260809 23:20:37.496501 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.497668 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.426309   499 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.426131 32741 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:37.503865 24430 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:37.484700 32225 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:37.492471   491 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:37.520380 24540 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:37.502568   493 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:37.524981   417 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.484174   322 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260809 23:20:37.501720   407 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.524714   559 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260809 23:20:37.484117 32634 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.030s
I20260809 23:20:37.559692 24435 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260809 23:20:37.561153   417 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:37.561753   417 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.563946   417 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.561120 24435 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:37.566291 24435 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260809 23:20:37.567890   322 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:37.568471   322 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:37.568925   322 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:37.569412   562 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.569940   562 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.571502   529 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.571914   562 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:37.572718   562 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:37.573200   562 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.575282   562 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.575943   562 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:37.576472   550 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.088s	user 0.003s	sys 0.000s
I20260809 23:20:37.576423   562 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.578950   562 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:37.579690 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:37.521822   491 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:37.514598 32741 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.532228 32761 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.534121 25368 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:37.534550 32471 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.011s	user 0.011s	sys 0.000s
I20260809 23:20:37.522003 32640 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.535564   497 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.490226   558 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:37.617462 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2543) took 2491 ms (client timeout 2084 ms). Trace:
W20260809 23:20:37.617882 24321 rpcz_store.cc:269] 0809 23:20:35.125873 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.189702 (+ 63829us) service_pool.cc:227] Handling call
0809 23:20:37.617292 (+2427590us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.617392 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2564559232}
I20260809 23:20:37.618639   558 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.619028 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.620428 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.621237 24535 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:37.621624 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2908) took 1029 ms. Trace:
I20260809 23:20:37.621906 24535 rpcz_store.cc:276] 0809 23:20:36.592188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.620125 (+1027937us) service_pool.cc:227] Handling call
0809 23:20:37.621568 (+  1443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:37.623795 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3101) took 2085 ms. Trace:
I20260809 23:20:37.624185 24422 rpcz_store.cc:276] 0809 23:20:35.538795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.051686 (+512891us) service_pool.cc:227] Handling call
0809 23:20:36.052058 (+   372us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:36.063343 (+ 11285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:36.063381 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:36.090348 (+ 26967us) log.cc:842] Serialized 26 byte log entry
0809 23:20:36.500803 (+410455us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:37.623238 (+1122435us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54002ab5a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f3fb3 00007fac23860f0f 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c
0809 23:20:37.623293 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.623391 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:37.623434 (+    43us) raft_consensus.cc:1671] finished
0809 23:20:37.623449 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:37.623693 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":110515,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":74,"spinlock_wait_cycles":37613952,"thread_start_us":358315,"threads_started":1,"wal-append.queue_time_us":368006,"wal-append.run_cpu_time_us":2046,"wal-append.run_wall_time_us":1002056}
I20260809 23:20:37.626492   315 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:37.626886   315 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:37.622922 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.627791   315 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.628283 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.629148 24535 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 12: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:37.630229 24535 pending_rounds.cc:76] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:20:37.630482 24535 pending_rounds.cc:97] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260809 23:20:37.630714 24535 raft_consensus.cc:2866] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:20:37.631385   480 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.220s	user 0.001s	sys 0.008s
I20260809 23:20:37.629165   315 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:37.633224 24544 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:37.537086   494 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.537429   532 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.515210 32681 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.539494   538 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.539788   563 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:37.541144   336 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.542294   509 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.542663 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.547299 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.547904 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.409126   556 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.655289 24312 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:37.558214 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:37.523890 32225 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:37.525719   542 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260809 23:20:37.588019 24542 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:37.594574 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.595597 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.597747 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.601835   531 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.602962 32761 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:37.603588 32471 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.604144 32407 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:37.609203 32741 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.609628 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1374) took 1013 ms. Trace:
W20260809 23:20:37.594956   434 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.000s
I20260809 23:20:37.615787 32354 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.536612   525 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:37.638342 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58728.8us
W20260809 23:20:37.746502 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.000s	sys 0.011s
I20260809 23:20:37.746728 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:37.655071 24314 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:37.557914   533 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.581158   326 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:37.642794 32640 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:37.722322 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:20:37.759370 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 165296us
I20260809 23:20:37.644654 32587 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:37.645294   502 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.735867 24309 rpcz_store.cc:276] 0809 23:20:36.595810 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.663867 (+ 68057us) service_pool.cc:227] Handling call
0809 23:20:37.235474 (+571607us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:37.240581 (+  5107us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.609403 (+368822us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540043bcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.609557 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1208282880}
W20260809 23:20:37.739602 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.007s	sys 0.003s
I20260809 23:20:37.739526 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:37.740752   532 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:37.741745 24540 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:37.737258 24323 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:37.768324 24321 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:37.768723 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.007s	sys 0.003s
I20260809 23:20:37.735687   509 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.753661 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.652522   563 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:37.756035 24544 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:37.757149 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:37.661321 32681 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
W20260809 23:20:37.755436 24266 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.632s (SENT) [suppressed 20 similar messages]
I20260809 23:20:37.514374   526 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.768035 24539 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:37.770220 24317 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:20:37.781598 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:37.770828 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.029s	sys 0.025s
I20260809 23:20:37.753840   326 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:37.750591 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.000s	sys 0.011s
W20260809 23:20:37.783999 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.030s	sys 0.025s
I20260809 23:20:37.768067   525 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:37.776338 24545 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:37.777433 24540 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:37.774791 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:37.775269 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.006s	sys 0.005s
W20260809 23:20:37.803059 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56403.2us
W20260809 23:20:37.813642 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.006s	sys 0.005s
I20260809 23:20:37.745924   434 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.800318   533 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:37.769389 32587 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.801749 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:37.749770   491 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.780221   563 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:37.785449   544 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.816998   531 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.817678   525 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.774883   499 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:37.824955 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:37.771175   532 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.828198 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.775096   571 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260809 23:20:37.851397 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96020.6us
W20260809 23:20:37.852543 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1361) took 2652 ms (client timeout 1538 ms). Trace:
W20260809 23:20:37.851178 32380 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.558s	user 0.010s	sys 0.000s
W20260809 23:20:37.839259 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53259.6us
I20260809 23:20:37.781323   509 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.771019   502 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.837198 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:37.862044 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94532.6us
I20260809 23:20:37.823810   533 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:37.863803 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1375) took 1261 ms. Trace:
I20260809 23:20:37.883184 24313 rpcz_store.cc:276] 0809 23:20:36.602407 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.679490 (+ 77083us) service_pool.cc:227] Handling call
0809 23:20:36.679913 (+   423us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:36.680178 (+   265us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:36.680212 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:36.697732 (+ 17520us) log.cc:842] Serialized 26 byte log entry
0809 23:20:36.875329 (+177597us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:36.875390 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.875478 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:37.863529 (+988051us) raft_consensus.cc:1671] finished
0809 23:20:37.863551 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:37.863736 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139360,"threads_started":1,"wal-append.queue_time_us":381914}
I20260809 23:20:37.823832   332 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:37.819543   407 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.868686 25369 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:37.891589 32202 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:37.892124 32202 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:37.892719 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1369) took 1542 ms. Trace:
I20260809 23:20:37.892585   332 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.892976 32202 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.894181 32202 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:37.842378   433 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:20:37.875406   492 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.578s	user 0.011s	sys 0.000s
I20260809 23:20:37.835142 32674 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260809 23:20:37.887554 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:37.887713   499 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:37.891671   407 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.893126 25369 rpcz_store.cc:276] 0809 23:20:36.350296 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.350448 (+   152us) service_pool.cc:227] Handling call
0809 23:20:37.862378 (+1511930us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:37.891649 (+ 29271us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.892528 (+   879us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54007062d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.892660 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3455993216}
W20260809 23:20:37.869805 25363 rpcz_store.cc:269] 0809 23:20:35.200120 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.200256 (+   136us) service_pool.cc:227] Handling call
0809 23:20:37.852335 (+2652079us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:37.852474 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2088441216}
I20260809 23:20:37.894686   332 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:37.896191   524 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.873s	user 0.010s	sys 0.000s
I20260809 23:20:37.800053   322 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.900133 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2910) took 1307 ms. Trace:
I20260809 23:20:37.838210 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:37.936365 24317 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:37.885644   580 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.899823 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:37.866674 32068 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:37.908195 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94031.4us
I20260809 23:20:37.885924 32587 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.919215 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:37.908970 32546 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:37.928268 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:37.945881 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:37.851562 32741 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260809 23:20:37.932361 24539 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260809 23:20:37.899210   533 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:37.919091 32225 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.941470 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:37.834163   544 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:37.948328 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:37.935878 24543 rpcz_store.cc:276] 0809 23:20:36.592824 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.605178 (+ 12354us) service_pool.cc:227] Handling call
0809 23:20:36.627421 (+ 22243us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:37.181188 (+553767us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.899911 (+718723us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:37.900071 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1115049216}
I20260809 23:20:37.920753 32471 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.937093   494 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:37.929697 32354 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:37.930522   499 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:37.931582   556 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:37.966310 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.009s	sys 0.042s
I20260809 23:20:37.974776 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:37.961439   548 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:37.951072 32068 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:37.931354   315 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260809 23:20:37.981984 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:37.954108   433 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
W20260809 23:20:37.985183 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44222.9us
I20260809 23:20:37.985543 24540 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:37.956008 32684 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:37.973277   587 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20260809 23:20:37.983958 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.010s	sys 0.042s
I20260809 23:20:37.988024   507 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:37.988993   577 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:38.037190   568 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20260809 23:20:37.924387   492 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:38.037945   596 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:20:38.038339   336 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:38.040046 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:37.879755   326 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:38.040928 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59029.2us
I20260809 23:20:37.933861   352 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260809 23:20:38.040395 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:37.951238   332 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.035s
I20260809 23:20:37.980233   578 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:38.043478 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 143730us
I20260809 23:20:37.988813   548 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:37.965553 32681 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:37.978853   494 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:38.045746 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 660958us
I20260809 23:20:37.956346 32546 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.048981   507 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.040267 32354 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:38.043083   559 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:38.043130   434 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:37.989292   589 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:38.209631 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.019s	sys 0.014s
W20260809 23:20:38.210011 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.019s	sys 0.015s
I20260809 23:20:38.044523 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.045269   526 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.049139   491 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.193969   494 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.186461 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:38.189945 24430 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.15-15.15]
I20260809 23:20:38.190325 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.224485 24313 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:38.172096 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.225289 24539 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:20:37.954620   580 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.167721 32684 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.204774 24316 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:38.168253   556 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:38.196112 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:38.205101 24540 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:38.174237 32741 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:38.206094 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.175544 32225 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:38.041226   493 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.043283 32674 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.142s
I20260809 23:20:38.208139 24323 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:20:38.183622   336 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:38.213544 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.215688 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.216048 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:38.254660 25370 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:38.218371 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.185205   577 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:38.189026 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 148715us
I20260809 23:20:38.204952   507 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:38.208266   591 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260809 23:20:38.192037 32681 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:38.216427   559 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:38.231487   595 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.230731 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.270385 25373 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:38.240814 32741 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:38.231772 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.276794 32546 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:38.277469 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45790.9us
I20260809 23:20:38.238253 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.215921   326 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.239549 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:38.241380 25363 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:38.242825 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.208009   599 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:38.218657 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.230795   589 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.257277 25366 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:38.234885   509 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.188238   578 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.261193 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:38.332044 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70971.8us
I20260809 23:20:38.231267 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:38.263728 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.334152 24316 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:38.334829 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.232752   532 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.235055 24543 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:38.336673 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.336862 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.338169 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.243068 32471 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260809 23:20:38.270720   584 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
I20260809 23:20:38.245021   322 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.263411   556 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.344429   556 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.254122 32674 consensus_queue.cc:1225] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:38.350948 32546 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:38.351620 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:38.258705   573 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.386s	user 0.000s	sys 0.002s
I20260809 23:20:38.352965   494 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:38.283577 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 239113us
I20260809 23:20:38.355382 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3029) took 1434 ms. Trace:
I20260809 23:20:38.355883 24546 rpcz_store.cc:276] 0809 23:20:36.921550 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.098031 (+176481us) service_pool.cc:227] Handling call
0809 23:20:37.148648 (+ 50617us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:37.149027 (+   379us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:37.149069 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:37.211356 (+ 62287us) log.cc:842] Serialized 26 byte log entry
0809 23:20:37.784568 (+573212us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:37.784624 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.784711 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:38.355010 (+570299us) raft_consensus.cc:1671] finished
0809 23:20:38.355033 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:38.355313 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":88558,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":26336,"thread_start_us":538310,"threads_started":1,"wal-append.queue_time_us":597024}
I20260809 23:20:38.285377 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:34762 (request call id 1091) took 1785 ms. Trace:
I20260809 23:20:38.358474 24431 rpcz_store.cc:276] 0809 23:20:36.500472 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.503553 (+  3081us) service_pool.cc:227] Handling call
0809 23:20:38.285047 (+1781494us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:20:38.285273 (+   226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":119833984}
I20260809 23:20:38.307032 24317 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
W20260809 23:20:38.362112 32640 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.596s	user 0.010s	sys 0.000s
I20260809 23:20:38.362526 32640 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260809 23:20:38.363162   407 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.363891   502 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.363993 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.316471   544 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.365213   544 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:38.365511 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.242620 32684 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.367108 32471 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:38.367595 32471 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:38.368197 32684 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:38.370682 32407 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.623s	user 0.009s	sys 0.001s
I20260809 23:20:38.287782 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.372583 24429 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:38.329864 24313 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:38.263826 32681 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.271648   433 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.352826 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.375357 24543 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:38.375296 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.377503 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.278321 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.375224   433 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:38.379397 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 101148us
I20260809 23:20:38.380105 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.380918   433 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.380956 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.381253 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.381723 24431 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:38.382162 24435 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:38.382678 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:38.368911 32471 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.284861 24320 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:38.383224   363 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.387579   577 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.387725   363 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.389036 25365 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:38.383282 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.390117 24435 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:38.383828 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.383828 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.385028 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:38.224753   548 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.385396 24320 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:38.278230   587 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:38.391520 24424 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260809 23:20:38.382915 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.399945 24543 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:38.402915 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.403231 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.396483 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.406692 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:38.407104 24539 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:38.408136 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.404156 24543 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:38.279798 32391 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:38.287402   497 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:20:38.349103   549 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:20:38.411023 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.412262 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:38.280426   493 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.279203   492 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.346791   556 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:38.410254 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72185.6us
I20260809 23:20:38.396018 24320 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.432344 24320 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:38.348131   599 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:38.433109 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.256431   588 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.178s
I20260809 23:20:38.433724 24543 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:38.423852 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.385568   509 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:38.387161 32684 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:38.349253 32354 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:38.396847   600 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
W20260809 23:20:38.428123 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 233192us
W20260809 23:20:38.428642 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44887us
I20260809 23:20:38.386996   531 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:38.430390 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.447029 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.447293 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.430786 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.449026 24316 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:38.421582 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:38.434343 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.449855 24317 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:38.433917 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.435539 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:38.440851 24546 raft_consensus.cc:2362] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12. Re-sending same reply.
I20260809 23:20:38.450630 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.419391 32546 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:38.414320 32391 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:38.422713   548 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:38.269279 32225 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.414641   497 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:38.417217 32068 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.450716 24323 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:38.441676 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.451879 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.452472 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.427012 32697 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:38.428254 32640 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:38.450165 24314 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:38.462399 24312 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260809 23:20:38.385720   532 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260809 23:20:38.434625 32641 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:38.429531   322 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260809 23:20:38.430461   493 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.454312 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.467406 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.467945 24539 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:38.468009 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:38.442431   492 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:38.481174 24543 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:38.429275   615 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:38.475576 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.464202 24317 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:38.448033   494 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:38.481637 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73150us
I20260809 23:20:38.444712 32684 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.485306 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.512465 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:38.488675 24313 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:38.281921   595 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.491937 25370 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260809 23:20:38.454087 32391 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:38.453577   556 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:38.481354 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.456476 32747 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:38.499732 24314 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:38.500138 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:38.469765   563 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.627s	user 0.010s	sys 0.000s
I20260809 23:20:38.558508 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.504966 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:38.507189 24316 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:20:38.461310   502 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:38.347675   589 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:38.561764 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 109350us
I20260809 23:20:38.563438 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.488045 32068 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:38.560002 24539 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:38.560464 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:38.571497 24546 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:38.496973   615 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:38.479626   532 consensus_queue.cc:1225] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:38.476321   433 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:38.558024 32747 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.448118   559 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.565507   531 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.513815   322 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:38.567266 32202 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:38.577776 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:38.767752 24431 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.767854 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:38.556806 32640 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.553215   494 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.496595 32641 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:38.509285 32471 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:38.488457   492 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:38.584795 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3027) took 1664 ms (client timeout 1809 ms). Trace:
I20260809 23:20:38.593400 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:38.824935 24542 rpcz_store.cc:269] 0809 23:20:36.920250 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.920488 (+   238us) service_pool.cc:227] Handling call
0809 23:20:38.584638 (+1664150us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:38.584739 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176147712}
W20260809 23:20:38.639060 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75732.5us
I20260809 23:20:38.604703 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1279) took 1032 ms. Trace:
I20260809 23:20:38.826121 24545 rpcz_store.cc:276] 0809 23:20:37.572132 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.747541 (+175409us) service_pool.cc:227] Handling call
0809 23:20:37.763978 (+ 16437us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:37.811710 (+ 47732us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:38.604397 (+792687us) spinlock_profiling.cc:243] Waited 8.74 ms on lock 0x7b5400115154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:38.604620 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20112768}
W20260809 23:20:38.595268 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1360) took 3408 ms (client timeout 2675 ms). Trace:
I20260809 23:20:38.655597 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:38.827116 25365 rpcz_store.cc:269] 0809 23:20:35.187213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.192768 (+  5555us) service_pool.cc:227] Handling call
0809 23:20:38.595085 (+3402317us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b540073e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:38.595195 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7348494592}
I20260809 23:20:38.601243 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2648) took 1879 ms. Trace:
W20260809 23:20:38.827785 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 444925us
I20260809 23:20:38.581380 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.827993 24427 rpcz_store.cc:276] 0809 23:20:36.721637 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.721883 (+   246us) service_pool.cc:227] Handling call
0809 23:20:36.815718 (+ 93835us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:36.816882 (+  1164us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:38.601060 (+1784178us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b54003bae50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:38.601187 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177245568}
I20260809 23:20:38.629120 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:38.606148 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.829309 24543 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:38.599730 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2578) took 1008 ms. Trace:
I20260809 23:20:38.830755 24321 rpcz_store.cc:276] 0809 23:20:37.591824 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.618699 (+ 26875us) service_pool.cc:227] Handling call
0809 23:20:38.599527 (+980828us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540019dcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:38.599658 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331583232}
I20260809 23:20:38.828526 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.600826 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2650) took 1808 ms. Trace:
I20260809 23:20:38.832248 24423 rpcz_store.cc:276] 0809 23:20:36.792611 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.792760 (+   149us) service_pool.cc:227] Handling call
0809 23:20:36.793326 (+   566us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:37.107974 (+314648us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:37.180722 (+ 72748us) log.cc:842] Serialized 14 byte log entry
0809 23:20:37.772999 (+592277us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:37.773052 (+    53us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:37.773093 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:38.600595 (+827502us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:38.600646 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:38.600771 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":85426,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":1172,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":4632,"spinlock_wait_cycles":722849152,"thread_start_us":591933,"threads_started":1,"wal-append.queue_time_us":646603}
I20260809 23:20:38.552719 32697 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:38.567457   603 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.231s
I20260809 23:20:38.503198 24323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:38.517531 32684 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:38.577173   602 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260809 23:20:38.583488   533 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.603s	user 0.004s	sys 0.006s
I20260809 23:20:38.496935   493 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.576771   588 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
W20260809 23:20:38.839196   499 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.851s	user 0.008s	sys 0.001s
I20260809 23:20:38.577236   615 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:38.577195   580 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.825291 24546 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:38.471567   562 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260809 23:20:38.582646   334 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.278s
I20260809 23:20:38.829180 24313 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:38.826917 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.597021   560 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:38.828598 24424 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:38.768993   531 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.603595 32354 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.584734   589 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.564798   563 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.840857 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:38.767118   322 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.842258 24323 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.583024   608 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:38.753213   502 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:38.557367 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:20:38.910482 24150 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 494037us
I20260809 23:20:38.851034 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.825430 32471 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:38.812263 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44522.8us
W20260809 23:20:38.852739 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285115us
I20260809 23:20:38.835843   492 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.827587   363 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.856346 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.885703   610 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:38.753222   556 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
I20260809 23:20:38.834122 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:38.925124 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91619us
I20260809 23:20:38.839056   491 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:38.925889 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3107) took 2390 ms. Trace:
I20260809 23:20:38.579404   617 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.282s
I20260809 23:20:38.753286 32747 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:38.868134 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.846481   624 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:38.820626 32640 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.864810 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:38.575063   497 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.824448   494 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.925900 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:38.928434   365 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:38.852682   578 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.893515 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.895977 24323 raft_consensus.cc:603] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:20:38.898597   326 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.584s	user 0.011s	sys 0.000s
I20260809 23:20:38.857465   499 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.582960   599 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.869730   580 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:38.902433 32407 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:38.890388   525 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:38.869343   560 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:38.914778 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.852252   533 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.918200 24427 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:38.856676 24545 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:38.927714   624 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:38.931809   491 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:38.933586 24546 pending_rounds.cc:76] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 9
I20260809 23:20:38.932350   589 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.014s
I20260809 23:20:38.945652 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.945883   580 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.946460   580 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:38.946292   525 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.947082 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:38.947685 24427 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:38.947732   624 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.948596   525 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.949507 24429 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:38.950666   533 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.945509 25368 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:38.915376   532 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.953070   532 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.954187   494 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.955576 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.957212 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:38.958011 24424 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 12.
I20260809 23:20:38.959003   595 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:38.911969   322 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:38.930001 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59675.5us
I20260809 23:20:38.971508 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:38.768811   571 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:38.964221   499 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.980085   556 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:38.980513   556 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:38.930899   497 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:38.981405   556 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.976444 32684 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.983865   617 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:38.983508   502 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:38.984375   617 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.984815   617 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:38.985265 32684 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:38.964624   365 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:38.965672 32640 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.945783 24546 pending_rounds.cc:97] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260809 23:20:38.992162 24546 raft_consensus.cc:2866] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:38.925843   651 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:38.967038 24423 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:38.918440   608 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:39.022158 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 107449us
I20260809 23:20:38.974466   606 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:38.974665 24311 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.556s	user 0.011s	sys 0.000s
I20260809 23:20:39.025597 24311 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:38.935235   639 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260809 23:20:38.976428   578 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:38.979188   571 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:38.959583   532 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:38.989212 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:39.037822 24431 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260809 23:20:38.920712   507 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.654s	user 0.007s	sys 0.004s
W20260809 23:20:38.991142 24320 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.558s	user 0.010s	sys 0.001s
I20260809 23:20:38.951812 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.041527 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:38.983595 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 118884us
W20260809 23:20:38.995173 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3028) took 2074 ms (client timeout 1786 ms). Trace:
W20260809 23:20:39.042461 24534 rpcz_store.cc:269] 0809 23:20:36.920874 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.921101 (+   227us) service_pool.cc:227] Handling call
0809 23:20:38.994922 (+2073821us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:38.995085 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216361728}
I20260809 23:20:38.996192 24427 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:39.044600 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:20:39.004460 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1076) took 6038 ms. Trace:
I20260809 23:20:39.009236   315 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:38.924731   610 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:38.969867 24542 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260809 23:20:38.926702 24428 rpcz_store.cc:276] 0809 23:20:36.535666 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.543598 (+  7932us) service_pool.cc:227] Handling call
0809 23:20:36.558586 (+ 14988us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:36.558793 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:36.573965 (+ 15172us) log.cc:842] Serialized 14 byte log entry
0809 23:20:38.606373 (+2032408us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18003d35a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:38.925405 (+319032us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:38.925490 (+    85us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:38.925567 (+    77us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:38.925657 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:38.925812 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14985,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":36716,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":19094,"spinlock_wait_cycles":33996672,"thread_start_us":312328,"threads_started":2,"wal-append.queue_time_us":2325066}
I20260809 23:20:39.028893 24323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "904621b566f64064bc1a7411ab27c677"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:39.049907 24323 raft_consensus.cc:595] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:38.915962 32471 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.048439 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.051528 25365 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260809 23:20:39.025367   606 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.035280 24317 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:39.015723   365 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.021374   588 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:39.040715 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.059571 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.060243 24313 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:39.040943 24431 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:39.048210 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1367) took 2467 ms. Trace:
I20260809 23:20:39.037829 32225 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.028894   595 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:39.052984 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.033802 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.065119 24427 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:39.053825 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102122us
I20260809 23:20:39.063395 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:39.066402 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.029944   556 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:39.071069 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.051314 25371 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:39.056659 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.032032   332 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:39.060137 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.081378 25366 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:39.052825 32417 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:39.048945   315 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:39.050299 24545 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:38.916455 32641 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.068364 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.050447   507 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.045575 24425 rpcz_store.cc:276] 0809 23:20:32.966058 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.172312 (+2206254us) service_pool.cc:227] Handling call
0809 23:20:35.735570 (+563258us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:36.143775 (+408205us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:36.155964 (+ 12189us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.504110 (+348146us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:36.522118 (+ 18008us) log.cc:842] Serialized 26 byte log entry
0809 23:20:36.522270 (+   152us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:36.523385 (+  1115us) log.cc:842] Serialized 14 byte log entry
0809 23:20:36.664471 (+141086us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:36.664554 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:36.756830 (+ 92276us) raft_consensus.cc:1671] finished
0809 23:20:36.756851 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.004274 (+2247423us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b540017fcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:39.004402 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5377,"prepare.queue_time_us":14706,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":109,"raft.queue_time_us":298661,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":97313,"spinlock_wait_cycles":1188819328,"thread_start_us":869585,"threads_started":3,"wal-append.queue_time_us":428863,"wal-append.run_cpu_time_us":3299,"wal-append.run_wall_time_us":1168017}
W20260809 23:20:39.041213 32681 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.667s	user 0.007s	sys 0.003s
I20260809 23:20:39.096035 24424 pending_rounds.cc:76] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:20:39.096347 24424 pending_rounds.cc:97] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260809 23:20:39.085259 24431 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:39.060088   639 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:39.101724 24424 raft_consensus.cc:2866] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:39.050630   610 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:39.055620   434 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:39.099081 32761 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Stepping down as leader of term 11
I20260809 23:20:39.059875   606 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.096936 24427 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:39.080358   559 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.120882 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.083679 25365 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:39.084590   300 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:39.124598   300 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.125064   300 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.100018 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.127012   315 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.128070   315 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.089527 32417 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.103922   531 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:39.130564 32417 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.132228   507 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:39.133172 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.133425 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:39.135982 32640 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:39.136408 32640 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:39.099620 32681 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.138425 24435 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:39.091284   651 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:39.115710   434 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.141376 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:39.141470   434 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:39.073452 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.143253 25371 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:39.144687 25371 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260809 23:20:39.144980 25371 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:20:39.145183 25371 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260809 23:20:39.145404 25371 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:20:39.099819   509 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.655s	user 0.005s	sys 0.005s
I20260809 23:20:39.146960   563 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:39.147251 32391 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.551s	user 0.007s	sys 0.003s
I20260809 23:20:39.147554   509 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:39.099395   493 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.119830 24431 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260809 23:20:39.136776 32640 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:39.148079   646 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260809 23:20:39.156802 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:39.114091 32761 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:39.178423 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:39.178334 32761 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.179188   562 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:39.179481 32761 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
I20260809 23:20:39.179879   532 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:39.180790   554 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:39.181195   554 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:39.181298 32681 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.182103   554 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.182621 32687 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:39.183189 32687 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:39.184110 32687 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.185030   554 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:39.188695   322 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.189666   562 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.190065 25370 raft_consensus.cc:2354] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:20:39.189257 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.104833 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.147940 32391 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.171447 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.192107 24427 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:20:39.193943 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 89361.3us
I20260809 23:20:39.198424 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.205161 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.201776   365 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.209050 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.210597 24311 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:39.211723   434 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:39.212455   434 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.216694 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.217420 24542 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.220568 24316 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260809 23:20:39.222067 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.223366 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.224768 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.225410 24314 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:39.228348 24542 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:39.230250 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:39.231514   545 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:39.232105 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.233621 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.234160   365 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.234463 24312 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.232169   545 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:39.236661   545 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.222762 24316 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:39.242911 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.244413 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.245913 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.246766   434 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.248015   493 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:39.251806   493 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.253575 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.254449   493 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.256315 24312 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:39.258095   509 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:39.264138 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73128.8us
I20260809 23:20:39.259567 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.260129 24316 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260809 23:20:39.195865 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 117805us
I20260809 23:20:39.264672 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.268111 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.270481 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.273097 24543 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:39.267428 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:39.203251   651 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader election won for term 12
I20260809 23:20:39.190203   562 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:39.195461   548 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.734s	user 0.005s	sys 0.009s
I20260809 23:20:39.276839 24314 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260809 23:20:39.279829 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 366174us
I20260809 23:20:39.282493 25372 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:39.284255 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.285394 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.285944 24542 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
W20260809 23:20:39.267259   660 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260809 23:20:39.286767 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.291443 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.196156 32681 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:39.297451 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.298239 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.298552 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.108197   608 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.301142 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.301748 24534 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:39.302600 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.302865   559 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:39.303165 24534 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.305620 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:39.307647 24542 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:39.305383 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.308638 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.345134 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.348244 24542 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:39.297016   497 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.349427 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.189265   300 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.350276 25372 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:39.351979 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.359004 24543 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:39.361159 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:39.361934 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.13193e+06us
I20260809 23:20:39.362627 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:39.363544 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.364923 24543 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:39.365628 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.276240   532 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260809 23:20:39.303848 32417 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:39.279687   577 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.066s
I20260809 23:20:39.294137   651 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:39.371024 24540 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
I20260809 23:20:39.348097   493 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.371718 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:39.413939 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.373190 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.415475 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:39.416765 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 128582us
I20260809 23:20:39.367302 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.364157 24431 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:39.330471   595 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:39.381506 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.391896 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:39.424151 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 116630us
I20260809 23:20:39.313303   548 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260809 23:20:39.424199 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:39.373580   518 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260809 23:20:39.392362 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.373798   624 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:39.370564   509 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:39.148581   588 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.394269   651 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.370956 32681 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260809 23:20:39.396205   300 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.378381   580 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.397619 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.452705 24314 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.399745 32747 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:39.401319 24266 connection.cc:232] RPC call timeout handler was delayed by -1.1155431289225817s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23670000000000002s for a response
I20260809 23:20:39.404174   617 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.408886 32391 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.296947 31984 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260809 23:20:39.427086 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.370275   559 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:39.388666 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.430339 24431 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:39.431270 25372 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:39.393023 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.374984   365 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:39.378852 32641 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:39.379516   571 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.446713 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1278) took 1874 ms. Trace:
I20260809 23:20:39.446973 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.447158 24312 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: []
I20260809 23:20:39.594470 24544 rpcz_store.cc:276] 0809 23:20:37.571849 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.634064 (+ 62215us) service_pool.cc:227] Handling call
0809 23:20:37.634356 (+   292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:37.781088 (+146732us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.446466 (+1665378us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:39.446650 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279239552}
I20260809 23:20:39.371604   563 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.070s
I20260809 23:20:39.425318   494 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:39.460428 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.424767   652 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.442963   624 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.460036   580 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.430434 32684 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:39.378337   326 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260809 23:20:39.425165   595 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.464695 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.436527   499 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:39.436012   548 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260809 23:20:39.471055 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44047.7us
I20260809 23:20:39.501996 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:39.454031   530 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.209s	user 0.002s	sys 0.016s
I20260809 23:20:39.501921 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:39.609187   532 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260809 23:20:39.621371   548 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:39.609799   624 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:39.626338 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.627977 32417 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:39.628382 32417 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:39.629274 32417 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.630384 32417 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.584451 32640 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.632179 24545 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:39.633049 32640 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.633719   494 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:39.634675   494 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:39.636457   610 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.000s
I20260809 23:20:39.636884   610 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:39.611332   674 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260809 23:20:39.612746   595 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:39.611585 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.1586e+06us
I20260809 23:20:39.613627 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.586534 24543 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:39.619030 25373 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 16. Ops: [14.11-14.11]
I20260809 23:20:39.591907 24425 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:39.619160 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.593384 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:39.618980 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 117137us
I20260809 23:20:39.643586 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.663789 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:39.614866 24431 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:39.461823   592 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.422s	user 0.002s	sys 0.002s
I20260809 23:20:39.601352 24313 raft_consensus.cc:595] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:39.605247   671 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260809 23:20:39.605892   551 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.222s	user 0.006s	sys 0.000s
I20260809 23:20:39.461966 24321 raft_consensus.cc:2354] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17.
I20260809 23:20:39.671288 24534 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:39.714516 25372 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 11. Current term is 12. Ops: [11.11-11.11]
W20260809 23:20:39.715960 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112608us
I20260809 23:20:39.714354 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.718245 24540 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:39.719771 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.502455 32681 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.699277 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.683001 24314 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:39.662004 25371 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:39.489909 32354 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:39.729681 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.709229 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:39.617790   530 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.373s	user 0.003s	sys 0.016s
I20260809 23:20:39.642057   570 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:39.741906 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.730083 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.743856 24534 raft_consensus.cc:2362] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14. Re-sending same reply.
W20260809 23:20:39.664405   606 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.512s	user 0.008s	sys 0.000s
I20260809 23:20:39.658358   336 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.749301 25373 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:39.731256 24431 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:39.750190 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:39.750767 24431 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:39.619324   499 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:20:39.644876   594 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.601s	user 0.002s	sys 0.000s
I20260809 23:20:39.615365   577 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.132s
I20260809 23:20:39.641126   332 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.645915 32747 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:39.741071 25372 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:39.730748 24544 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:39.745105 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.773801 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:39.746750 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.747545 24425 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:39.755627 25371 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260809 23:20:39.600143   322 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:39.615181   571 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.133s
I20260809 23:20:39.762303 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.762794 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.763038 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.782600 24534 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 16
W20260809 23:20:39.764581 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 298631us
I20260809 23:20:39.742457   570 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:39.824684 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:39.827076 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 391818us
I20260809 23:20:39.769764 24431 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:39.719239 31884 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.037s
I20260809 23:20:39.766515 24540 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:39.835353 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.785521 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:39.786465 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.787279 32320 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.079s
I20260809 23:20:39.788259 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1264) took 2561 ms. Trace:
I20260809 23:20:39.790246   555 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:39.791783   519 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:39.793875 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2933) took 1388 ms. Trace:
I20260809 23:20:39.862134 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:39.864866 24546 rpcz_store.cc:276] 0809 23:20:38.405576 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.866995 (+461419us) service_pool.cc:227] Handling call
0809 23:20:38.889634 (+ 22639us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:38.992902 (+103268us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:38.992945 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:38.993553 (+   608us) log.cc:842] Serialized 52 byte log entry
0809 23:20:39.084432 (+ 90879us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:39.084942 (+   510us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.793479 (+708537us) spinlock_profiling.cc:243] Waited 43.7 ms on lock 0x7b54000b5c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:39.793536 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:39.793585 (+    49us) raft_consensus.cc:1671] finished
0809 23:20:39.793597 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.793815 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1334,"raft.queue_time_us":127,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4060,"spinlock_wait_cycles":100527872,"thread_start_us":45972,"threads_started":1,"wal-append.queue_time_us":76877}
I20260809 23:20:39.796770 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:39.760221   610 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:39.802356 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.803190 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.804003 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.804549   599 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:39.804677 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.871215 24313 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:39.745884 32354 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.806448 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2661) took 1426 ms. Trace:
W20260809 23:20:39.765609   672 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260809 23:20:39.807054 32038 raft_consensus.cc:979] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:20:39.807490   365 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:39.807706   497 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:39.891717   433 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.892071 24546 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260809 23:20:39.892894   492 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.924s	user 0.011s	sys 0.000s
I20260809 23:20:39.893376   492 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.893567 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:39.808470 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.894990 24425 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:39.808321 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1276) took 2237 ms. Trace:
I20260809 23:20:39.809959 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3108) took 1336 ms. Trace:
I20260809 23:20:39.811302 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.899289 25365 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:20:39.899026 24435 rpcz_store.cc:276] 0809 23:20:38.473330 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.473481 (+   151us) service_pool.cc:227] Handling call
0809 23:20:38.502702 (+ 29221us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.138786 (+636084us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.809748 (+670962us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:39.809904 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1461541248}
W20260809 23:20:39.900589 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1377) took 1687 ms (client timeout 2018 ms). Trace:
I20260809 23:20:39.812252   569 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:39.812139   315 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:39.813584 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.816015 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:39.817057 32407 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:39.773659   332 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:39.757895   499 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:39.788194 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.824455 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.831264 24544 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:39.831907 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:39.833812 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:39.835706 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1375) took 3017 ms. Trace:
I20260809 23:20:39.825132   580 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260809 23:20:39.923741 24544 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:20:39.837483 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:39.821926 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:39.840524 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1094) took 2907 ms. Trace:
I20260809 23:20:39.840422 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:39.922981 24431 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:39.839828 25373 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:39.837471 24423 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 12.
I20260809 23:20:39.751803   606 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.873445 24422 rpcz_store.cc:276] 0809 23:20:38.379869 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.380056 (+   187us) service_pool.cc:227] Handling call
0809 23:20:38.380473 (+   417us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:38.414367 (+ 33894us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:38.414397 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:38.414437 (+    40us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:38.414468 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.806226 (+1391758us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b54003aaf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:39.806271 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.806388 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":77500288}
W20260809 23:20:39.875303 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89850.5us
I20260809 23:20:39.862174 24541 rpcz_store.cc:276] 0809 23:20:37.226794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.227025 (+   231us) service_pool.cc:227] Handling call
0809 23:20:37.249666 (+ 22641us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:37.279275 (+ 29609us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:37.279303 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:37.292423 (+ 13120us) log.cc:842] Serialized 52 byte log entry
0809 23:20:37.637129 (+344706us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b1800502a80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:20:37.871259 (+234130us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:37.871313 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.871480 (+   167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:38.289621 (+418141us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b1800502a80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938799 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:39.644205 (+1354584us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:39.788037 (+143832us) raft_consensus.cc:1671] finished
0809 23:20:39.788058 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.788201 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29193,"raft.queue_time_us":47025,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":3830,"spinlock_wait_cycles":23189376,"thread_start_us":233887,"threads_started":1,"wal-append.queue_time_us":630218}
W20260809 23:20:39.867337 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70674.4us
I20260809 23:20:39.877120 24534 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:39.806456   617 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:39.841020 32471 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.773s	user 0.009s	sys 0.002s
I20260809 23:20:39.870324 32752 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.883742 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:39.884074   684 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:39.946339 24313 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
W20260809 23:20:39.946810 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 124960us
I20260809 23:20:39.947908 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.884457   599 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:39.949182 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 111779us
I20260809 23:20:39.951601 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.885051 24543 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:39.952638 24422 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:39.872035   519 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:39.746799 32741 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.839561 31884 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:20:39.887354 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.887857 32354 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:39.888794 32038 raft_consensus.cc:1067] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:20:39.889616 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.889058   365 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:39.062021 24320 rpcz_store.cc:276] 0809 23:20:36.580682 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.626147 (+ 45465us) service_pool.cc:227] Handling call
0809 23:20:36.668654 (+ 42507us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:38.284768 (+1616114us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b5400458420. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:39.047890 (+763122us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:39.047920 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:39.047960 (+    40us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:39.048000 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.048057 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.048160 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5852,"raft.queue_time_us":11701,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5902,"spinlock_wait_cycles":1119374720}
I20260809 23:20:39.892179   433 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:39.896284 24546 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:39.897218 24428 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:39.809140 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2659) took 1430 ms. Trace:
I20260809 23:20:39.835343   530 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:39.894923 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:39.901060 25368 rpcz_store.cc:269] 0809 23:20:38.213534 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.218420 (+  4886us) service_pool.cc:227] Handling call
0809 23:20:39.895645 (+1677225us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:39.900540 (+  4895us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1476464896}
I20260809 23:20:39.979939 24429 rpcz_store.cc:276] 0809 23:20:38.379143 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.383608 (+  4465us) service_pool.cc:227] Handling call
0809 23:20:39.808942 (+1425334us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b5400061654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:39.809081 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1283113856}
I20260809 23:20:39.907411 24424 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:39.867509 32225 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:39.949120 32761 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.752s	user 0.003s	sys 0.008s
I20260809 23:20:39.897927 24535 rpcz_store.cc:276] 0809 23:20:37.571144 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.629717 (+ 58573us) service_pool.cc:227] Handling call
0809 23:20:37.630052 (+   335us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:37.631265 (+  1213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:37.631288 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:37.639899 (+  8611us) log.cc:842] Serialized 26 byte log entry
0809 23:20:37.830522 (+190623us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:37.830558 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.830623 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:39.461176 (+1630553us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:39.808017 (+346841us) raft_consensus.cc:1671] finished
0809 23:20:39.808041 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.808262 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3955584,"thread_start_us":109986,"threads_started":1,"wal-append.queue_time_us":301673}
I20260809 23:20:39.913103 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:39.924907   597 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:39.918159 25323 leader_election.cc:398] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17.
I20260809 23:20:39.917982 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.878449   322 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:39.925125 24430 rpcz_store.cc:276] 0809 23:20:36.933080 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:37.514166 (+581086us) service_pool.cc:227] Handling call
0809 23:20:37.585656 (+ 71490us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:38.224185 (+638529us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:39.835793 (+1611608us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:39.840448 (+  4655us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1389143936}
I20260809 23:20:39.925431 24540 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:39.983554 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:39.927208 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:39.984690 24423 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:39.929924 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:39.931383 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.932543 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:39.986703 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54248.5us
I20260809 23:20:39.934206 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:39.936123 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:39.988039 24425 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:39.988286 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54193us
W20260809 23:20:39.912397   639 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.809s	user 0.010s	sys 0.001s
I20260809 23:20:39.941836 25365 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:39.923096 25375 rpcz_store.cc:276] 0809 23:20:36.818463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.818647 (+   184us) service_pool.cc:227] Handling call
0809 23:20:36.868373 (+ 49726us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:36.895570 (+ 27197us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:36.895636 (+    66us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:36.935670 (+ 40034us) log.cc:842] Serialized 26 byte log entry
0809 23:20:37.207162 (+271492us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:37.207216 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:37.207304 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:39.444268 (+2236964us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:39.823088 (+378820us) raft_consensus.cc:1671] finished
0809 23:20:39.823108 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:39.835639 (+ 12531us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2465,"raft.queue_time_us":2625,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":29089024,"thread_start_us":220947,"threads_started":1,"wal-append.queue_time_us":287867}
I20260809 23:20:39.930305 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:39.926064   499 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:39.973603 24323 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:39.816331   595 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.002904 24422 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260809 23:20:39.867758   578 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:39.982450 25323 leader_election.cc:401] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:39.924479   610 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260809 23:20:39.884479   682 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
W20260809 23:20:39.993717 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 294510us
W20260809 23:20:39.993594 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63750.8us
I20260809 23:20:39.995033 24312 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260809 23:20:40.035871 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:40.002746 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.976233 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:40.002678 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.984973 24431 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
W20260809 23:20:40.038362 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 126774us
I20260809 23:20:39.920442   569 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:39.951318   684 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:40.040268 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:40.037252 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:40.041157 24435 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:40.026505 24313 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:40.042303 24546 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260809 23:20:40.042403 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66318.1us
I20260809 23:20:40.043386 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:40.043882 25366 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:40.020545 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:40.027812 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:39.975013   497 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.021803 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:40.046005 24316 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:39.968395   545 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:40.036136 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:40.010027 25372 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:40.051311 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:40.036892 25371 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:40.052019 24431 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:40.051543 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:20:40.023682   615 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.158s	user 0.007s	sys 0.012s
I20260809 23:20:39.997293   555 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:39.901997   570 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.003697   507 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:39.992993 32032 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:40.063923 24313 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:40.040650 24535 pending_rounds.cc:76] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 10
I20260809 23:20:40.066202 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:40.052646 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:40.109772 24424 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:39.999228 32747 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:40.000147   597 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:40.069410 24423 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:20:39.946468 32471 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.005762   639 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:40.072454 24314 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:40.072187 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51756.8us
I20260809 23:20:40.128501 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:40.063041 24431 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:40.130975 24431 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260809 23:20:40.009018 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:40.022320   322 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:40.004520 32225 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.079627 24316 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260809 23:20:40.079465 24540 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:40.080479 25368 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:40.067114 32038 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:40.051275 32684 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:40.103826   545 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:40.105055 24541 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "904621b566f64064bc1a7411ab27c677"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:20:40.110857 24535 pending_rounds.cc:97] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260809 23:20:40.158423 24535 raft_consensus.cc:2866] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:40.050421   518 raft_consensus.cc:2741] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260809 23:20:40.051647   578 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:40.064193   300 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17.
W20260809 23:20:40.133663 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 124793us
I20260809 23:20:40.064277   497 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.173025 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:40.079278 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:40.034533   588 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260809 23:20:40.152562 24427 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:40.077591   615 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.161329 24539 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
W20260809 23:20:40.162801 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 170502us
I20260809 23:20:40.167176 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:40.167313   578 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.050611   684 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:40.122623   569 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:40.075263   531 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.080s
I20260809 23:20:40.123076 32747 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:40.142616 32687 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:40.135417   507 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:40.079111   494 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:40.156782 32684 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:40.166381   554 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.975s	user 0.000s	sys 0.013s
I20260809 23:20:40.181653 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:40.186213   492 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:40.183559   606 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:40.160585   570 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:40.180001 24424 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
W20260809 23:20:40.170126   692 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260809 23:20:40.157048   545 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:40.160719 31884 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:40.186861 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:40.123457   624 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.135047 32354 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260809 23:20:40.196168   494 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:40.183179 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:40.237010 25368 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:40.187384 32380 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.200850   569 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:40.193194 32747 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:40.220067 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:40.220633 24424 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260809 23:20:40.221493 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 166149us
I20260809 23:20:40.200951 32684 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:40.235742   545 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:40.245954 32380 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.248066 32380 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:40.248966 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:40.249859 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:40.251622   606 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:20:40.252240 32354 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:40.252661 32354 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:40.253002 32354 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:40.255336 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:40.255774   569 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:40.256338 25368 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:40.203426 24542 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:40.256841   569 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.258880 24424 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:40.206014   599 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:40.199420   699 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:40.260288 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2941) took 1066 ms. Trace:
I20260809 23:20:40.259160 24316 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:40.259702 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1394) took 1216 ms. Trace:
I20260809 23:20:40.260980 24435 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:40.263628 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:40.268784 24540 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:40.267946 24311 rpcz_store.cc:276] 0809 23:20:39.043481 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.043752 (+   271us) service_pool.cc:227] Handling call
0809 23:20:39.044048 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.211129 (+167081us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.259479 (+1048350us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:40.259647 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":382498176}
I20260809 23:20:40.261693 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:40.269618 24545 rpcz_store.cc:276] 0809 23:20:39.193492 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.193665 (+   173us) service_pool.cc:227] Handling call
0809 23:20:39.270064 (+ 76399us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.632635 (+362571us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.260063 (+627428us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400270bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:40.260227 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832085376}
I20260809 23:20:40.207406 24425 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:40.287115 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:40.289670 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:40.294108 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 227984us
I20260809 23:20:40.243716   492 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.245537   494 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:40.253005   606 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:40.297837 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:40.187465   597 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.306560   599 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.232313   693 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20260809 23:20:40.221053 31984 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260809 23:20:40.293675   499 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.295950   530 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:40.293932   699 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:40.296715   688 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:40.299808 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 255119us
I20260809 23:20:40.293179   559 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:40.248589   433 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.322333   606 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:40.357770   595 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:40.362089   595 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:40.362603   595 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:40.239648   322 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.434314 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:40.314308   492 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.372146 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:40.432448 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1419) took 1395 ms. Trace:
I20260809 23:20:40.336090   639 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.343387 32471 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:40.346917   597 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:40.497732 25369 rpcz_store.cc:276] 0809 23:20:39.037421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.038736 (+  1315us) service_pool.cc:227] Handling call
0809 23:20:39.050866 (+ 12130us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.054507 (+  3641us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:39.064732 (+ 10225us) log.cc:842] Serialized 14 byte log entry
0809 23:20:40.186431 (+1121699us) spinlock_profiling.cc:243] Waited 5.74 ms on lock 0x7b18003db4c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:20:40.431986 (+245555us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:40.432055 (+    69us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:40.432121 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.432205 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:40.432374 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13299,"prepare.queue_time_us":10140,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":1701,"spinlock_wait_cycles":13196544,"thread_start_us":240229,"threads_started":1}
W20260809 23:20:40.504015 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 131998us
W20260809 23:20:40.505456 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 377055us
I20260809 23:20:40.293965   507 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.365489 32225 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:40.495736 32417 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.865s	user 0.011s	sys 0.004s
I20260809 23:20:40.361145   615 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:40.564926 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:40.578018 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:40.330899   599 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.327904   705 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.185s
I20260809 23:20:40.352412 32380 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.213s
I20260809 23:20:40.497032 32761 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.507828   497 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:40.657513 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 362672us
W20260809 23:20:40.661399 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 96610.2us
I20260809 23:20:40.663609 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:40.562371   639 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.567591   577 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.360750   713 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.284s
I20260809 23:20:40.365226   706 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.304s
I20260809 23:20:40.508219   580 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.151s
I20260809 23:20:40.648465   578 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:40.657612 32225 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:40.731914 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 472134us
I20260809 23:20:40.657991   556 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:40.648655   705 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:40.494729   595 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.669548   651 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:40.649753   499 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:40.667798   718 raft_consensus.cc:2741] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:40.566632   719 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.221s
I20260809 23:20:40.690064 32752 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:40.739722   556 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:40.748688   599 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:40.779748   705 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:40.668485 32761 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:40.688091 32038 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.104s
I20260809 23:20:40.737713   615 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:40.745487 32225 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:40.836706   727 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260809 23:20:40.836529   722 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260809 23:20:40.678117   606 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:40.861158 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 293927us
I20260809 23:20:40.983224 24543 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:40.983628   546 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:40.691453   577 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:40.881419 32587 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.011s	sys 0.033s
I20260809 23:20:40.877017 32752 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:41.044503 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.017s	sys 0.074s
W20260809 23:20:41.078312 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.017s	sys 0.074s
W20260809 23:20:41.078707 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 779427us
I20260809 23:20:40.855769   721 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:41.026666 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:41.080418 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53864.1us
I20260809 23:20:40.959760 32038 consensus_queue.cc:1225] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260809 23:20:41.084583 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:41.086220 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:41.086706 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.014s	sys 0.034s
W20260809 23:20:41.087801 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.014s	sys 0.034s
W20260809 23:20:41.021517 32032 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.018s	sys 0.050s
W20260809 23:20:41.059196   608 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.022s	sys 0.057s
I20260809 23:20:40.644405   699 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.078312 32587 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.012s	sys 0.033s
I20260809 23:20:40.811995   651 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.113871 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1305826168972999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260809 23:20:41.116091 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0918826169800013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20260809 23:20:41.116537 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0000826169271022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
I20260809 23:20:41.117580 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:41.120803 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.022s	sys 0.030s
W20260809 23:20:41.121153 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.023s	sys 0.030s
I20260809 23:20:41.078353   433 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:41.095479 32032 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.018s	sys 0.051s
W20260809 23:20:41.124039 24270 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.030s	sys 0.038s
W20260809 23:20:41.092685 32684 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.846s	user 0.011s	sys 0.000s
I20260809 23:20:41.086135   705 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.090826   557 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:41.126633 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:41.090148   718 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:41.089952 32380 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:41.094033 32761 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:41.136387 24270 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.030s	sys 0.038s
I20260809 23:20:41.184723 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1385) took 1433 ms. Trace:
W20260809 23:20:41.127931   545 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.881s	user 0.004s	sys 0.007s
I20260809 23:20:41.090721 32752 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.127355   562 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.830s	user 0.015s	sys 0.000s
I20260809 23:20:41.114815   747 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.115535   492 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:41.179733 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53205.7us
I20260809 23:20:41.077721   546 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.007805   615 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.106257   719 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:41.136112   706 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260809 23:20:41.160679 32684 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.096694   721 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.188838 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 660829us
I20260809 23:20:41.281416 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1392) took 2772 ms. Trace:
W20260809 23:20:41.270071 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.024s	sys 0.016s
I20260809 23:20:41.295758 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1383) took 1558 ms. Trace:
I20260809 23:20:41.229815 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3138) took 1216 ms. Trace:
I20260809 23:20:41.189213 25365 rpcz_store.cc:276] 0809 23:20:39.751448 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.063419 (+311971us) service_pool.cc:227] Handling call
0809 23:20:40.108697 (+ 45278us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.135192 (+ 26495us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.135272 (+    80us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.154585 (+ 19313us) log.cc:842] Serialized 26 byte log entry
0809 23:20:40.293083 (+138498us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:40.293126 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.293194 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.184457 (+891263us) raft_consensus.cc:1671] finished
0809 23:20:41.184488 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.184648 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15000,"raft.queue_time_us":76277,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":3320,"thread_start_us":125174,"threads_started":1,"wal-append.queue_time_us":405811}
I20260809 23:20:41.331267 24317 rpcz_store.cc:276] 0809 23:20:38.508635 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.509262 (+   627us) service_pool.cc:227] Handling call
0809 23:20:38.509551 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.063010 (+553459us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.281161 (+2218151us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540043bcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:41.281353 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1208473344}
I20260809 23:20:41.332348 25372 rpcz_store.cc:276] 0809 23:20:39.737347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.003747 (+266400us) service_pool.cc:227] Handling call
0809 23:20:40.004173 (+   426us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.067298 (+ 63125us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.295441 (+1228143us) spinlock_profiling.cc:243] Waited 78 ms on lock 0x7b5400718520. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:41.295686 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179499264}
I20260809 23:20:41.215533   556 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:41.213294   729 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260809 23:20:41.333984   723 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.462s	user 0.000s	sys 0.001s
I20260809 23:20:41.335366 32752 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.138355   750 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:41.191160   705 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.336503   747 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.329602   597 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.753s	user 0.006s	sys 0.007s
W20260809 23:20:41.224515 32640 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.014s	sys 0.019s
I20260809 23:20:41.337695   518 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:41.272789 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.008s	sys 0.031s
W20260809 23:20:41.381611 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.014s	sys 0.033s
I20260809 23:20:41.311317   760 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:41.248231 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.011s	sys 0.027s
I20260809 23:20:41.319459 32225 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:41.339102 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.024s	sys 0.026s
W20260809 23:20:41.384557 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.019s	sys 0.031s
W20260809 23:20:41.268982 25373 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.000s
I20260809 23:20:41.360100 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:41.361104 24431 rpcz_store.cc:276] 0809 23:20:40.012938 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.134564 (+121626us) service_pool.cc:227] Handling call
0809 23:20:40.134903 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.135256 (+   353us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.135291 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.153251 (+ 17960us) log.cc:842] Serialized 26 byte log entry
0809 23:20:40.641201 (+487950us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:40.641243 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.641320 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.229505 (+588185us) raft_consensus.cc:1671] finished
0809 23:20:41.229535 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.229743 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":835,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":22133,"thread_start_us":457088,"threads_started":1,"wal-append.queue_time_us":915395}
W20260809 23:20:41.383147 32640 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.023s	sys 0.026s
W20260809 23:20:41.290139   735 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260809 23:20:41.368779 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:41.296347 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.019s	sys 0.028s
I20260809 23:20:41.381170   750 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:41.382251 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 718682us
I20260809 23:20:41.338599 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:41.311910   741 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.360579   714 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:41.293462   509 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:41.405810 25372 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:41.180284   608 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.406376 25373 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:41.182469 32380 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:41.416651 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:41.419202 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.019s	sys 0.032s
I20260809 23:20:41.419279 24317 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:41.466501 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 127986us
W20260809 23:20:41.379009 25369 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.001s	sys 0.013s
I20260809 23:20:41.193862   639 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.363245 32032 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260809 23:20:41.452149 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:41.371223 32391 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260809 23:20:41.332723   545 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.400017   651 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:41.403297   332 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.467s	user 0.013s	sys 0.003s
W20260809 23:20:41.470088 25369 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.001s	sys 0.013s
I20260809 23:20:41.393705   562 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.473304 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:41.467880 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:41.420068   686 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:41.437052   577 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.452711 32761 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:41.493424 32747 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.246s	user 0.011s	sys 0.003s
I20260809 23:20:41.464274   526 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.467274   509 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:41.472517 32380 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:41.394344   518 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:41.501775 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56248.8us
I20260809 23:20:41.503185 24317 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:41.472894   750 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:41.505450 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53751us
I20260809 23:20:41.493892   615 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:41.495595 32354 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.242s	user 0.013s	sys 0.000s
I20260809 23:20:41.464087   493 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.510672 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.492027   706 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:41.513392 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:41.571420 24534 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:41.401713   760 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.465041 32752 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260809 23:20:41.512475 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3062) took 1493 ms. Trace:
I20260809 23:20:41.526728 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1409) took 1219 ms. Trace:
I20260809 23:20:41.621546 24317 rpcz_store.cc:276] 0809 23:20:40.307470 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.368514 (+1061044us) service_pool.cc:227] Handling call
0809 23:20:41.526648 (+158134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:41.526062 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:41.529037 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:41.623682 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94737.2us
I20260809 23:20:41.624459 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:41.581818   775 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
I20260809 23:20:41.473641   608 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:41.503575   684 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.309s	user 0.008s	sys 0.007s
W20260809 23:20:41.579340   773 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20260809 23:20:41.580246   770 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:20:41.681224 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3066) took 1436 ms. Trace:
I20260809 23:20:41.681494   615 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:41.681655 24546 rpcz_store.cc:276] 0809 23:20:40.245247 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.245406 (+   159us) service_pool.cc:227] Handling call
0809 23:20:40.259835 (+ 14429us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.645374 (+385539us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.645413 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.656636 (+ 11223us) log.cc:842] Serialized 26 byte log entry
0809 23:20:41.108291 (+451655us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:41.108331 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.108414 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.680990 (+572576us) raft_consensus.cc:1671] finished
0809 23:20:41.681014 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.681155 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":168880,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":12402,"thread_start_us":813239,"threads_started":2,"wal-append.queue_time_us":789766}
I20260809 23:20:41.683300   684 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.684113 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.685302 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.685480   760 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.686092 24546 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:41.686223   684 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.686748 24423 raft_consensus.cc:595] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:41.687151 24423 raft_consensus.cc:603] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:20:41.581035   755 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260809 23:20:41.543109   560 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.688300   560 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:41.690302   719 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.690871   719 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.582816   771 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260809 23:20:41.528201   526 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.586849 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.524186 32761 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.631151 32640 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.521456   322 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260809 23:20:41.690917   494 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.369s	user 0.009s	sys 0.003s
I20260809 23:20:41.591735   557 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:20:41.594393 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3136) took 1582 ms (client timeout 1815 ms). Trace:
I20260809 23:20:41.621382 24535 rpcz_store.cc:276] 0809 23:20:40.019339 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.020187 (+   848us) service_pool.cc:227] Handling call
0809 23:20:40.040476 (+ 20289us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.168441 (+127965us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.168478 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.168966 (+   488us) log.cc:842] Serialized 26 byte log entry
0809 23:20:40.571200 (+402234us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:40.571247 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:40.571342 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.512270 (+940928us) raft_consensus.cc:1671] finished
0809 23:20:41.512294 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.512419 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":401842,"threads_started":1,"wal-append.queue_time_us":644320}
W20260809 23:20:41.505822 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1413) took 2472 ms (client timeout 1535 ms). Trace:
W20260809 23:20:41.628818 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102667us
I20260809 23:20:41.630009   711 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:41.786379 25370 rpcz_store.cc:269] 0809 23:20:39.033331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.033470 (+   139us) service_pool.cc:227] Handling call
0809 23:20:41.505550 (+2472080us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.505732 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":286036224}
I20260809 23:20:41.541208 32391 consensus_queue.cc:1225] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:20:41.644419 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:41.667081   774 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260809 23:20:41.472302   714 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.669157   606 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260809 23:20:41.799721 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 117017us
I20260809 23:20:41.669607   763 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.670266   790 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:41.671854   721 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.673455   545 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.467466   741 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.603333   639 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.675892 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1398) took 1741 ms. Trace:
I20260809 23:20:41.807017 25369 rpcz_store.cc:276] 0809 23:20:39.934816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.576430 (+641614us) service_pool.cc:227] Handling call
0809 23:20:40.576841 (+   411us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.577122 (+   281us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:41.487622 (+910500us) log.cc:842] Serialized 14 byte log entry
0809 23:20:41.629493 (+141871us) log.cc:842] Serialized 14 byte log entry
0809 23:20:41.675527 (+ 46034us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:41.675587 (+    60us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:41.675639 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.675706 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.675832 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":120382,"prepare.queue_time_us":505422,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":22744,"raft.queue_time_us":58701,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":102084,"spinlock_wait_cycles":22933248,"thread_start_us":913879,"threads_started":2,"wal-append.queue_time_us":219628}
I20260809 23:20:41.686856 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.695305 24534 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:41.616992   706 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:41.697238 25372 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:41.812629 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:41.812428   706 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.813552   706 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:41.500064   332 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.815435   332 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.521729 32747 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.817548 32747 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:41.818320 24309 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:41.819042   559 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.819536   559 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.820298 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2581) took 2023 ms. Trace:
I20260809 23:20:41.820657 24309 rpcz_store.cc:276] 0809 23:20:39.796430 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.802902 (+  6472us) service_pool.cc:227] Handling call
0809 23:20:41.820116 (+2017214us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.820246 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4478602624}
I20260809 23:20:41.821749 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:41.822335 24309 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:41.705225   526 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.823009   526 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.823259 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3130) took 2733 ms. Trace:
I20260809 23:20:41.823450   526 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.823626 24427 rpcz_store.cc:276] 0809 23:20:39.090299 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.196982 (+106683us) service_pool.cc:227] Handling call
0809 23:20:39.280621 (+ 83639us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.176309 (+895688us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.823026 (+1646717us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:41.823202 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1991999872}
I20260809 23:20:41.825466   526 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.825956   526 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.826339   526 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.682154   615 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:41.653012   597 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:41.676527 32684 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.504817   686 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.758912   684 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:41.702162   570 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.470s	user 0.011s	sys 0.003s
I20260809 23:20:41.600760 32354 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:41.630801 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 104818us
W20260809 23:20:41.784942 24435 rpcz_store.cc:269] 0809 23:20:40.012347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.050588 (+ 38241us) service_pool.cc:227] Handling call
0809 23:20:41.594152 (+1543564us) spinlock_profiling.cc:243] Waited 86.9 ms on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.594327 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199820672}
I20260809 23:20:41.784181 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.597546   493 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.792681 24317 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:41.796797 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:41.706140   322 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:41.898964   322 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:41.858474   509 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.901209   509 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.903195 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:41.903812 24435 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:41.904734 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:41.905414 24435 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:41.906100 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:41.906766 24435 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:41.753865 32752 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.795888 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:41.753621   767 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:41.801482 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.928750 32684 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.936340 24535 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:41.936131   767 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.936410 32684 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.937505 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:41.938056 32752 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.822409 31984 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:41.699605   760 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.827134 24542 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:41.590389   518 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.828753   719 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.830067 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3067) took 1535 ms. Trace:
I20260809 23:20:41.831712 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1293) took 1764 ms. Trace:
I20260809 23:20:41.831956   741 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:41.833432   519 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.834012   639 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.837734   608 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.839687 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2594) took 2018 ms. Trace:
I20260809 23:20:41.793773   711 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.785766   494 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.813753 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:41.805454   790 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:42.018762 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:41.802021   546 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.775184 32587 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.848706 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:41.811024   763 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.850766   714 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.945787 24545 rpcz_store.cc:276] 0809 23:20:40.294640 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.295027 (+   387us) service_pool.cc:227] Handling call
0809 23:20:40.317979 (+ 22952us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:40.726664 (+408685us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.726705 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:40.744068 (+ 17363us) log.cc:842] Serialized 52 byte log entry
0809 23:20:41.077177 (+333109us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:41.077228 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.077331 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.829235 (+751904us) raft_consensus.cc:1671] finished
0809 23:20:41.829257 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.829656 (+   399us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6395,"raft.queue_time_us":252746,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":44274,"thread_start_us":696612,"threads_started":2,"wal-append.queue_time_us":560587}
W20260809 23:20:41.855772   792 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260809 23:20:41.859807   617 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.862599   336 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 11 FOLLOWER]: Advancing to term 12
W20260809 23:20:41.863545 32687 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.666s	user 0.013s	sys 0.000s
I20260809 23:20:41.866169 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:41.872614 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:41.806203   557 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:41.881347 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:41.892818   526 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.864939   597 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:41.893441 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.028067 32684 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:41.896670 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.028105   790 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:41.754076 32640 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:41.907383 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.062247 24424 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:41.911136 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1104) took 2236 ms. Trace:
I20260809 23:20:42.062870 24422 rpcz_store.cc:276] 0809 23:20:39.674709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.953230 (+278521us) service_pool.cc:227] Handling call
0809 23:20:39.953501 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.003451 (+ 49950us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.910888 (+1907437us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:41.911080 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":112672000}
W20260809 23:20:41.911898 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1402) took 2459 ms (client timeout 2241 ms). Trace:
W20260809 23:20:42.064292 24323 rpcz_store.cc:269] 0809 23:20:39.452828 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.452966 (+   138us) service_pool.cc:227] Handling call
0809 23:20:41.911703 (+2458737us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.911822 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":847803264}
I20260809 23:20:41.914336 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:41.914768   684 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.917359 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1406) took 2037 ms. Trace:
I20260809 23:20:41.922206   726 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:41.871950   560 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:41.922966 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2926) took 3519 ms (client timeout 2128 ms). Trace:
I20260809 23:20:41.924474 32354 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.926295   315 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:41.932828 24546 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:41.849970   747 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.938647 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:41.947050 24543 rpcz_store.cc:276] 0809 23:20:40.067073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.067258 (+   185us) service_pool.cc:227] Handling call
0809 23:20:40.982961 (+915703us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:41.059499 (+ 76538us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.831529 (+772030us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54000131d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.831655 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2022829824}
I20260809 23:20:42.038067 32587 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:41.948117 24312 rpcz_store.cc:276] 0809 23:20:39.821331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.065705 (+244374us) service_pool.cc:227] Handling call
0809 23:20:40.066006 (+   301us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.176778 (+110772us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.176811 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.177187 (+   376us) log.cc:842] Serialized 26 byte log entry
0809 23:20:40.336747 (+159560us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:40.336814 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.839279 (+1502465us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:41.839329 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:41.839394 (+    65us) raft_consensus.cc:1671] finished
0809 23:20:41.839406 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.839626 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2851,"raft.queue_time_us":3824,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":8343,"spinlock_wait_cycles":247324928,"thread_start_us":135193,"threads_started":1,"wal-append.queue_time_us":467028}
I20260809 23:20:41.950894 24270 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:42.168709 24270 leader_election.cc:401] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:41.934346 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:41.946976 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 151132us
W20260809 23:20:42.016994 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1416) took 2982 ms (client timeout 1832 ms). Trace:
I20260809 23:20:41.847616   693 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:42.025890 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:41.938571   767 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.030179 25369 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
W20260809 23:20:41.853935 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1431) took 2584 ms (client timeout 3194 ms). Trace:
I20260809 23:20:41.945693 31984 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.015630   554 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.045872 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2952) took 2148 ms. Trace:
I20260809 23:20:42.025954   741 raft_consensus.cc:2741] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:42.050319 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.021641   559 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:41.895955   493 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.034052   519 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:42.057789 24317 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.183925 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:42.184693 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1400) took 2234 ms. Trace:
I20260809 23:20:41.898358 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:42.058349 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 185831us
I20260809 23:20:42.185132 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.037690   565 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:41.785919   720 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.402s	user 0.001s	sys 0.000s
I20260809 23:20:42.190383 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:42.167205 32761 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.099184 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:42.092232 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2587) took 2294 ms. Trace:
I20260809 23:20:42.106750 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.092412 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.018854   760 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.117241 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:42.057351   597 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.115307 24534 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:42.118525 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.200234 24323 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:20:42.177294   578 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 15
I20260809 23:20:42.143323 24424 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:42.165719 24539 rpcz_store.cc:269] 0809 23:20:38.403498 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:38.570692 (+167194us) service_pool.cc:227] Handling call
0809 23:20:41.922659 (+3351967us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.922828 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3067156864}
I20260809 23:20:42.141245 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:42.166180 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.205992 24425 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:42.206461 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107377us
I20260809 23:20:42.061260   526 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:42.170032 25363 rpcz_store.cc:269] 0809 23:20:39.034359 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.034487 (+   128us) service_pool.cc:227] Handling call
0809 23:20:42.016765 (+2982278us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:42.016907 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":865868544}
W20260809 23:20:42.172492 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 221784us
I20260809 23:20:42.174623 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:42.171564 25366 rpcz_store.cc:269] 0809 23:20:39.269912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.613361 (+343449us) service_pool.cc:227] Handling call
0809 23:20:41.853728 (+2240367us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:41.853871 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":880381824}
I20260809 23:20:42.092136   587 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.048924   617 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:42.339627   720 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260809 23:20:42.174579 24544 rpcz_store.cc:276] 0809 23:20:39.897654 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.944624 (+ 46970us) service_pool.cc:227] Handling call
0809 23:20:39.974015 (+ 29391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.974241 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:41.817328 (+1843087us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b18002194c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:20:41.817466 (+   138us) log.cc:842] Serialized 14 byte log entry
0809 23:20:42.025958 (+208492us) log.cc:842] Serialized 14 byte log entry
0809 23:20:42.045440 (+ 19482us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:42.045507 (+    67us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:42.045577 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:42.045663 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:42.045803 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10239,"prepare.queue_time_us":216570,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":3381,"raft.queue_time_us":19390,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10465,"spinlock_wait_cycles":48313472,"thread_start_us":406668,"threads_started":2}
W20260809 23:20:42.139008   688 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.637s	user 0.027s	sys 0.062s
W20260809 23:20:42.383610   688 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.888s	user 0.029s	sys 0.066s
I20260809 23:20:42.383533 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.384024   688 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.384824   760 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.376601   714 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.384966   688 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.185994 24543 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:42.357409   711 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.387653 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.283077   526 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.389009 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:42.390280 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.185114 25365 rpcz_store.cc:276] 0809 23:20:39.950014 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.368429 (+1418415us) service_pool.cc:227] Handling call
0809 23:20:41.400173 (+ 31744us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:41.400570 (+   397us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:41.400599 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:41.406473 (+  5874us) log.cc:842] Serialized 26 byte log entry
0809 23:20:41.629274 (+222801us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:41.629322 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.629408 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:42.184483 (+555075us) raft_consensus.cc:1671] finished
0809 23:20:42.184507 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:42.184636 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":19595,"thread_start_us":170575,"threads_started":1,"wal-append.queue_time_us":262408}
I20260809 23:20:42.391458   526 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.391639   705 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.377151   570 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.392753   570 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.394642   570 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.116578   599 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:42.399186 32225 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.191489 25370 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:42.399736 32225 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.401878 32225 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.402521   809 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.190069 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.114575   509 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.403072   809 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.403478   509 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.405213   809 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.405977   509 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.194698 24317 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:42.406731 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2598) took 1273 ms. Trace:
I20260809 23:20:42.407117 24317 rpcz_store.cc:276] 0809 23:20:41.133430 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.896332 (+762902us) service_pool.cc:227] Handling call
0809 23:20:42.406664 (+510332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.407737 32640 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.408107 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.121029   802 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:42.411559 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:42.170912   700 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.196673 24545 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:42.044557   615 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:42.182842 32380 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.000s
I20260809 23:20:42.428030 32380 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:42.429656   784 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.762s	user 0.008s	sys 0.000s
I20260809 23:20:42.430536 32380 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.200727 24422 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:42.055088 32687 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:42.200659 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3060) took 1863 ms. Trace:
W20260809 23:20:42.201432 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84308.3us
I20260809 23:20:42.433764 24323 rpcz_store.cc:276] 0809 23:20:40.336902 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.071114 (+1734212us) service_pool.cc:227] Handling call
0809 23:20:42.200604 (+129490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.435014 24541 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:20:42.435446 24541 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:42.437950   810 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.246s	user 0.000s	sys 0.000s
I20260809 23:20:42.437736 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.198175 24321 rpcz_store.cc:276] 0809 23:20:39.798071 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.982470 (+184399us) service_pool.cc:227] Handling call
0809 23:20:40.021574 (+ 39104us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.107404 (+ 85830us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:40.107450 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:40.107506 (+    56us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:40.107551 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:42.090871 (+1983320us) spinlock_profiling.cc:243] Waited 81.8 ms on lock 0x7b54003868d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:42.092048 (+  1177us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:42.092177 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3557,"raft.queue_time_us":49733,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5508,"spinlock_wait_cycles":188244864}
I20260809 23:20:42.170910   693 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.202302 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:42.439878 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 237688us
I20260809 23:20:42.440239   595 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:42.440093   693 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.440577 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:42.441231   693 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:42.055576   557 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.442714   502 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:42.443305   758 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.443653   557 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.443819   758 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.444284   682 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:42.444787   682 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.444927   802 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:42.445768   682 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.446009   802 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.163188   726 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.447225 32687 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:42.280629   801 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.282600 32761 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:42.181655   554 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.307655 24425 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260809 23:20:42.449615 24425 pending_rounds.cc:76] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
W20260809 23:20:42.389024   706 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.000s
I20260809 23:20:42.182668   747 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260809 23:20:42.204391   546 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.307173 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.190234 32641 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.088s
I20260809 23:20:42.307200 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:42.453406 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.455233 24543 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:42.458215 24541 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.461192 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:42.283514 24540 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:42.279784 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:42.205653   719 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:42.463683   719 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.464092   719 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:42.467386 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.162798   322 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.294284 24427 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:20:42.187912   767 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:42.373572 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.000s	sys 0.006s
I20260809 23:20:42.374270 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1407) took 2454 ms. Trace:
I20260809 23:20:42.471235 24427 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 13.11
W20260809 23:20:42.471414 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.000s	sys 0.006s
I20260809 23:20:42.303504 24534 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:42.471601 24427 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:42.471510 24316 rpcz_store.cc:276] 0809 23:20:39.919511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.176567 (+257056us) service_pool.cc:227] Handling call
0809 23:20:40.194214 (+ 17647us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.271874 (+ 77660us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:42.373992 (+2102118us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:42.374199 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":148955008}
I20260809 23:20:42.283902 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.181766 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.475116 24435 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:42.195922   565 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.426796 32752 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.187181 24535 raft_consensus.cc:2354] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
W20260809 23:20:42.395418   750 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.800s	user 0.011s	sys 0.000s
W20260809 23:20:42.395434   800 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260809 23:20:42.377944   639 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.385800   714 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.398833   599 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.067818 24313 rpcz_store.cc:276] 0809 23:20:39.880456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.084158 (+203702us) service_pool.cc:227] Handling call
0809 23:20:40.105752 (+ 21594us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.202001 (+ 96249us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.202044 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:40.202097 (+    53us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:40.202141 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:41.917145 (+1715004us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:41.917189 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:41.917305 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220802304}
I20260809 23:20:42.494246 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.391685 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.311513   780 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:42.167078   315 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:42.416005 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:42.381179   587 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.420933 25375 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:42.308380   686 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.425458 25370 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:42.172895   560 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.449928 24425 pending_rounds.cc:97] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260809 23:20:42.507192 24425 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:42.279781 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:42.450639 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.477164   509 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.451884 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.448417   700 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.450166 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:42.462821 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 183124us
I20260809 23:20:42.514827 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:42.279939 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71617.5us
I20260809 23:20:42.433236   617 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.440629   595 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:42.359606   789 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.493s	user 0.001s	sys 0.001s
I20260809 23:20:42.451104 32640 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.478196 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.531839 24435 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:42.493873   750 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:42.478813 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.479719 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:42.533977 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.479758 24312 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:42.451432 32761 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:42.307442 31984 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.491020 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.536755 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.537045 24316 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:42.534770 25366 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:42.538352 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:42.492194 24540 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:42.537518 25365 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:42.419427 24317 raft_consensus.cc:2354] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:42.496923 24544 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:42.539959 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.470163 32417 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:42.540889 24540 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:42.477640   530 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.513132 24323 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260809 23:20:42.508363   684 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.547416 25368 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:20:42.522899 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:42.523957 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 72194.1us
I20260809 23:20:42.549314 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.491154 32587 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.529274 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.490571   763 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.470572   790 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.470275   546 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.448691   801 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:42.497856 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81966.9us
W20260809 23:20:42.535885   805 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260809 23:20:42.542769 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.543627 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:42.521492   526 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:42.546476 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:42.548123 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1393) took 2753 ms (client timeout 2048 ms). Trace:
I20260809 23:20:42.476972   767 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.522454   315 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.531073   497 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:42.532322   617 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:42.553488 24378 leader_election.cc:398] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:42.614507 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.612588 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.563299 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.597174 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:42.568867 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.598631 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:42.560802 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.550770 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.529757   639 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:42.592114 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.563531 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.560779   814 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:42.539005   532 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:42.698477   532 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:42.572590   336 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.520s	user 0.007s	sys 0.003s
I20260809 23:20:42.699576   532 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.700069   801 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.700676   532 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:42.700953 24431 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260809 23:20:42.701339 24431 pending_rounds.cc:76] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:20:42.701558 24431 pending_rounds.cc:97] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260809 23:20:42.701785 24431 raft_consensus.cc:2866] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:42.544270 32417 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:42.670320 25372 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:42.695480 24539 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260809 23:20:42.648008 24317 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:20:42.697369 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100288us
I20260809 23:20:42.694562 24378 leader_election.cc:401] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260809 23:20:42.698015 24435 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:42.579581   599 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.738075 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.741863 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.697707 25365 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:42.702848 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.738562 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:42.662117 25373 rpcz_store.cc:269] 0809 23:20:39.795128 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.930012 (+134884us) service_pool.cc:227] Handling call
0809 23:20:42.546813 (+2616801us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:42.548061 (+  1248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1497870720}
I20260809 23:20:42.744241 24543 raft_consensus.cc:2396] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:42.739480 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.697299 24425 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:42.523800   509 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.574405   714 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.746753 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.746867 24431 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:42.672695 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 129169us
I20260809 23:20:42.748334 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.748564 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.744711 24424 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:42.716300 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 165613us
I20260809 23:20:42.756937 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:42.696257 32380 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:42.537017 32761 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.577404   565 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.763731 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.760443 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.568717   587 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.760787 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:42.761253 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.578388   554 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.761519 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.580560   322 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.703215   814 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:42.696996   315 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:42.742348 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 195690us
I20260809 23:20:42.751055   639 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:42.769356 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:42.819075 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62337.9us
I20260809 23:20:42.822561 24540 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:42.773319 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.765540 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.806173 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.824877 24425 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:42.825809 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.671582   526 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:42.826702   526 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.827263 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.828018 24425 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:42.804944 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.808266 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.829720 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.830291 24424 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:42.830487 24425 raft_consensus.cc:2396] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260809 23:20:42.738116   801 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:42.703104 32354 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.831655 24424 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:42.833209 24424 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:42.778757 24316 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:42.706081   502 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:42.575776   700 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.703001   531 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.553517   530 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.837044 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.542547 32640 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:42.806077 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47161.3us
I20260809 23:20:42.810313 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.560938 32752 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:42.695446   560 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:42.746762 32417 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:42.842070 24435 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:42.842366 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.812768 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3115) took 4337 ms. Trace:
I20260809 23:20:42.699607   546 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.843070 24428 rpcz_store.cc:276] 0809 23:20:38.475664 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.049698 (+574034us) service_pool.cc:227] Handling call
0809 23:20:39.049955 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:39.984332 (+934377us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:42.812478 (+2828146us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:42.812694 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1948080384}
I20260809 23:20:42.844689 24428 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:42.763792 32391 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:42.849037   790 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.849246 32391 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:42.849728 32391 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.851202 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.851902 24540 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:42.852794 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.853590 24540 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:42.854344 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.855804 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3134) took 2844 ms. Trace:
I20260809 23:20:42.856194 24430 rpcz_store.cc:276] 0809 23:20:40.011659 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.017778 (+  6119us) service_pool.cc:227] Handling call
0809 23:20:40.051039 (+ 33261us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:40.198454 (+147415us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:40.198500 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:40.199042 (+   542us) log.cc:842] Serialized 26 byte log entry
0809 23:20:40.505462 (+306420us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:40.505510 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:42.855347 (+2349837us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54003b9f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:42.855416 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:42.855484 (+    68us) raft_consensus.cc:1671] finished
0809 23:20:42.855502 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:42.855745 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3355,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3811,"spinlock_wait_cycles":325715712,"thread_start_us":246935,"threads_started":1,"wal-append.queue_time_us":528124,"wal-append.run_cpu_time_us":8249,"wal-append.run_wall_time_us":1266952}
I20260809 23:20:42.859848   525 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:42.860232   525 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.861215   525 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.862542   525 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:42.859007 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.869122 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.869963 24321 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:42.804881 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1401) took 1050 ms. Trace:
I20260809 23:20:42.589084   434 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.215s
I20260809 23:20:42.580544   595 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 15.15, Last appended by leader: 15, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.871986   595 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:42.819046 24266 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.690s (SENT) [suppressed 29 similar messages]
I20260809 23:20:42.875732   565 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:42.808583 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.876794 25365 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:42.877070   803 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.572s	user 0.003s	sys 0.000s
I20260809 23:20:42.878312   571 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:20:42.880478   571 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:42.880808   571 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:42.877280 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1402) took 1122 ms. Trace:
I20260809 23:20:42.881385 25363 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:42.881590 25365 rpcz_store.cc:276] 0809 23:20:41.754378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.759989 (+1005611us) service_pool.cc:227] Handling call
0809 23:20:42.877226 (+117237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.882618 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.883073 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.883741   624 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.883700 25363 raft_consensus.cc:2362] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15. Re-sending same reply.
I20260809 23:20:42.703382 32587 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:42.831310 32354 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.885305   767 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.844486   763 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.886811 32354 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.889024 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40954 (request call id 5) took 2909 ms. Trace:
I20260809 23:20:42.889492 24423 rpcz_store.cc:276] 0809 23:20:40.010224 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.013417 (+  3193us) service_pool.cc:227] Handling call
0809 23:20:42.888766 (+2875349us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:42.888903 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3586947456}
I20260809 23:20:42.837779 24313 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:42.891855 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.892711   509 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:42.893100 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.894024 24313 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:42.838299 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.895785 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1408) took 1139 ms. Trace:
I20260809 23:20:42.897596 25363 rpcz_store.cc:276] 0809 23:20:41.756265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.882822 (+1126557us) service_pool.cc:227] Handling call
0809 23:20:42.895712 (+ 12890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.836658   587 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:42.848857 24428 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260809 23:20:42.813616 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.565486   597 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.844853 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.807858   714 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.905750 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.906565 24316 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:42.906744 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:42.890726 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.910277 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.883514 25373 rpcz_store.cc:276] 0809 23:20:41.754038 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.760553 (+1006515us) service_pool.cc:227] Handling call
0809 23:20:42.804816 (+ 44263us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.838378 24424 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: [16.12-17.13]
I20260809 23:20:42.911896 24317 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:42.907855 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:42.823068 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.913770 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.813120 32380 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:42.921022 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.910640 24435 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:42.817960   814 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:42.904578 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.924719 24428 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:42.851914   782 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.172s	user 0.002s	sys 0.011s
I20260809 23:20:42.923141 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.846906   700 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.935591   700 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:42.914389   767 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:42.938234   767 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.924253   595 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:42.936100 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.924270 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:42.923782 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 475018us
I20260809 23:20:42.915356 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.934837 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.930903 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:42.840188   531 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.939720 24430 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:42.912583   809 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:42.927623   578 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.652s	user 0.008s	sys 0.002s
W20260809 23:20:42.918660 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95715.8us
I20260809 23:20:42.962962 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:42.940256 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.966070 24435 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:42.943490 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.952455 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:42.968986 24424 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:42.944790 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.925249 32587 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:42.927289 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:42.979378 24428 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:42.975616 24317 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:42.980950 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.983285 24435 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:42.942109 24423 pending_rounds.cc:76] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:20:42.887240   763 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.004097 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:42.988482 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:42.991406 24423 pending_rounds.cc:97] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260809 23:20:43.006237 24423 raft_consensus.cc:2866] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:42.949714 24540 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:42.947129 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:43.007965 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60923.2us
I20260809 23:20:43.009011 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:42.948675 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:42.946529 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.943315 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.946640 24313 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:42.887274 32417 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.013370 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.011487 25363 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:43.015194 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66632.2us
I20260809 23:20:43.025005 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1399) took 1336 ms. Trace:
I20260809 23:20:43.025416 25363 rpcz_store.cc:276] 0809 23:20:41.688582 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.943022 (+1254440us) service_pool.cc:227] Handling call
0809 23:20:43.024928 (+ 81906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:42.939342   595 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:42.939363 32354 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.041177 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.017340 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:42.921985   714 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.938905   700 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.036085 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.079607 24316 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:20:43.079872 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43865.6us
I20260809 23:20:43.082420 32417 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:42.915983   587 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.023836   686 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.083873   686 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:43.024330   624 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:43.084120 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.085318 24539 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:43.044961 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:43.086390 24323 raft_consensus.cc:2354] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:43.086311 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:42.984467   434 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:43.104980   434 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:42.926451 32391 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.011436 25373 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:43.115286 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.053325 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.058293   615 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:43.065225   562 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.071043 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.978438   494 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.071944   530 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:43.127511 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2586) took 3329 ms (client timeout 1897 ms). Trace:
W20260809 23:20:43.128111 24314 rpcz_store.cc:269] 0809 23:20:39.797794 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.889365 (+ 91571us) service_pool.cc:227] Handling call
0809 23:20:43.127265 (+3237900us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.127425 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225560960}
I20260809 23:20:43.129814 24311 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:43.013228   502 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.096s
I20260809 23:20:43.087455 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:42.939553   571 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.089322 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.106035   577 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:43.091006 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:42.923692   526 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:42.989166   578 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:43.143322 24431 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.143456   615 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:43.098384 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.099712 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:43.144460 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53569.9us
I20260809 23:20:43.144824 24316 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260809 23:20:43.145238 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3070) took 1051 ms. Trace:
I20260809 23:20:43.145633 24316 rpcz_store.cc:276] 0809 23:20:42.093868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.082372 (+988504us) service_pool.cc:227] Handling call
0809 23:20:43.145183 (+ 62811us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:43.100601 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1394) took 1700 ms (client timeout 2117 ms). Trace:
W20260809 23:20:43.146164 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47901.1us
W20260809 23:20:43.146440 25370 rpcz_store.cc:269] 0809 23:20:41.400202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.898120 (+497918us) service_pool.cc:227] Handling call
0809 23:20:43.094792 (+1196672us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.100526 (+  5734us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13423488}
I20260809 23:20:43.147589 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:42.940655   767 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:43.103458 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62385.8us
I20260809 23:20:43.151319 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.152482 24378 leader_election.cc:398] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:43.152828 24378 leader_election.cc:401] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260809 23:20:43.153843 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.053030 24428 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260809 23:20:43.023447   750 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:43.106848 24539 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260809 23:20:43.102490   682 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:43.040493   595 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.116950 25373 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260809 23:20:43.174235 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.120123 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.036924 32380 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.099462   747 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:42.999781 32752 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.119s
I20260809 23:20:42.977138   809 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.040742 32354 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:43.124637 25363 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:43.185053 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1405) took 1021 ms. Trace:
I20260809 23:20:43.185356 25363 rpcz_store.cc:276] 0809 23:20:42.164218 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.044115 (+879897us) service_pool.cc:227] Handling call
0809 23:20:43.184994 (+140879us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.087167   617 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.137115 24323 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:43.018327 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3067) took 1494 ms. Trace:
I20260809 23:20:43.189823 24317 rpcz_store.cc:276] 0809 23:20:41.523729 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.003768 (+1480039us) service_pool.cc:227] Handling call
0809 23:20:43.018266 (+ 14498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.108942   760 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:43.194595 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 375549us
I20260809 23:20:43.084483   587 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.111701   434 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.141006 25365 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:43.122705   719 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.658s	user 0.007s	sys 0.006s
W20260809 23:20:43.124018   688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.024s	sys 0.053s
I20260809 23:20:43.145153 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:43.100837 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 112418us
W20260809 23:20:43.213156 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68089.2us
I20260809 23:20:43.214761 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.159487 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:43.161180 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:43.210011 25366 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:43.171598 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.171273 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:43.176113 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56099.7us
I20260809 23:20:43.220892 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.123253 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2601) took 1615 ms. Trace:
I20260809 23:20:43.182523   747 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:43.177103 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:43.221683 24323 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:43.168905   492 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.177963 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.231163   700 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:43.231236 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53363.3us
I20260809 23:20:43.232679 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:43.212505   688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.025s	sys 0.053s
I20260809 23:20:43.183182 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.246013 24539 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:43.169317   509 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:43.173282   532 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.252554 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.254132 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3063) took 1851 ms. Trace:
I20260809 23:20:43.176460   571 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:43.175230   597 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:43.139828 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50608.5us
I20260809 23:20:43.176311   684 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:43.179955   578 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:43.232929   502 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.210974 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.234613   492 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:43.221704 24313 rpcz_store.cc:276] 0809 23:20:41.508186 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.012371 (+1504185us) service_pool.cc:227] Handling call
0809 23:20:43.123176 (+110805us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.223393 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.223604 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:43.218214 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.158725   577 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:43.285111 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61572.7us
I20260809 23:20:43.286038 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.192025 32354 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.140208   494 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.212277   315 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.236140 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:43.195037   760 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.263674   578 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:43.237046 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.271965   492 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.173949   682 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.193608 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.276618   684 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:43.255473 24269 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.537s (SENT)
W20260809 23:20:43.349933 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94535.4us
I20260809 23:20:43.257303 24323 rpcz_store.cc:276] 0809 23:20:41.402366 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.227520 (+1825154us) service_pool.cc:227] Handling call
0809 23:20:43.254056 (+ 26536us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.346792 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3076) took 1149 ms. Trace:
I20260809 23:20:43.351564 24317 rpcz_store.cc:276] 0809 23:20:42.197513 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.190679 (+993166us) service_pool.cc:227] Handling call
0809 23:20:43.346719 (+156040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.225689   525 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.229861   691 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:43.229422   747 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:43.358022   691 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.358534   691 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.360690   691 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.229532 32640 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.218892   719 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.265069 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1402) took 1170 ms. Trace:
I20260809 23:20:43.177295 25373 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:43.341940   578 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.251592   532 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.251714   587 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.286123 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.150087   530 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:20:43.395195 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 109245us
I20260809 23:20:43.396265 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.398370 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:43.291922 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68635.7us
I20260809 23:20:43.296221 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:43.338233 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102206us
I20260809 23:20:43.321780   315 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.344812 24539 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:43.352782 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.294184   711 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:43.356675 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.408015 24323 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:43.357290   525 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:43.362327 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1128) took 1337 ms. Trace:
I20260809 23:20:43.409508 24422 rpcz_store.cc:276] 0809 23:20:42.026410 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.431406 (+404996us) service_pool.cc:227] Handling call
0809 23:20:42.431814 (+   408us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:42.432234 (+   420us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:42.807222 (+374988us) log.cc:842] Serialized 14 byte log entry
0809 23:20:43.138113 (+330891us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:43.142807 (+  4694us) log.cc:842] Serialized 52 byte log entry
0809 23:20:43.156928 (+ 14121us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:43.157158 (+   230us) log.cc:842] Serialized 14 byte log entry
0809 23:20:43.188392 (+ 31234us) log.cc:842] Serialized 14 byte log entry
0809 23:20:43.192647 (+  4255us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.192716 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:43.362092 (+169376us) raft_consensus.cc:1671] finished
0809 23:20:43.362117 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.362266 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32268,"prepare.queue_time_us":274178,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":74,"raft.queue_time_us":39242,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10259,"spinlock_wait_cycles":28718848,"thread_start_us":696291,"threads_started":2,"wal-append.queue_time_us":526516}
I20260809 23:20:43.412302 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.307365   565 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:43.423403   565 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.423898   565 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:43.335516   750 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.427286   750 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.297693   577 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.309556   526 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.313709   688 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:43.432262   688 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:43.432780   688 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:43.369978   829 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:20:43.265673 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.435389 24316 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:43.372182 25363 rpcz_store.cc:276] 0809 23:20:42.094709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.186036 (+1091327us) service_pool.cc:227] Handling call
0809 23:20:43.264994 (+ 78958us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.436332   750 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.436874 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.437100   750 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:43.381569   563 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.438048 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.438339   597 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.438910 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1398) took 1750 ms. Trace:
I20260809 23:20:43.439358   525 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.439487 25363 rpcz_store.cc:276] 0809 23:20:41.688233 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.437782 (+1749549us) service_pool.cc:227] Handling call
0809 23:20:43.438834 (+  1052us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.440712 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.441998 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.442634 25363 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.344445   509 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:43.295312   763 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.326500   714 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:43.376601 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 139662us
I20260809 23:20:43.254463 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.381131   532 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.348385   682 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:43.371111   824 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260809 23:20:43.274266   502 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.383767 24540 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:43.467514 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.307530   494 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.403134 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.405431 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:43.471717 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68675.9us
I20260809 23:20:43.472425 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.373976 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:43.473614 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 99752.5us
W20260809 23:20:43.453940   693 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.012s	user 0.007s	sys 0.005s
I20260809 23:20:43.444617   509 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:43.448139 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.378153 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.436327 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.311136 32391 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.455145 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.455757 25363 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:43.457203 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.483296 24313 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260809 23:20:43.381381   587 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:43.462067 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:43.403018 32640 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:43.468294 24422 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:43.240175 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.495838 24428 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.468284 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.344583   814 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.375866   760 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.406371 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:43.501472 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 95210.5us
I20260809 23:20:43.474382 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.438961   597 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:43.461223   827 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260809 23:20:43.507231 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.507578 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:43.507890 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.508067 24428 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.483631   763 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:43.485160 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.511117 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.513917 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1418) took 1077 ms. Trace:
I20260809 23:20:43.489085 24317 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:43.480408 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1443) took 1947 ms. Trace:
I20260809 23:20:43.493625 24543 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:43.468381 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.514979 25373 rpcz_store.cc:276] 0809 23:20:41.533147 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.377773 (+1844626us) service_pool.cc:227] Handling call
0809 23:20:43.480344 (+102571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.494987   587 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.505995 24309 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:43.484432 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.483346   711 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.511577 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.520802 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.521206 24314 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:43.514374 24422 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.496922 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.515007 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.522464 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.522851 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.498394 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.517746 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.514394 25363 rpcz_store.cc:276] 0809 23:20:42.436605 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.482896 (+1046291us) service_pool.cc:227] Handling call
0809 23:20:43.513838 (+ 30942us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.520200 24425 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.517211 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.523186 24316 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 11
I20260809 23:20:43.524273 24539 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:43.525161 25373 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:43.544353 25365 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:43.544986 24316 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260809 23:20:43.508846   597 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.546209 24323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:43.547827 24316 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:43.552116 24323 raft_consensus.cc:595] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.545634 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1408) took 1013 ms. Trace:
I20260809 23:20:43.552485 24323 raft_consensus.cc:603] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:43.552625 25365 rpcz_store.cc:276] 0809 23:20:42.532187 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.499267 (+967080us) service_pool.cc:227] Handling call
0809 23:20:43.545578 (+ 46311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.522912 24422 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:43.554080 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:43.559221 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.554051 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.549278 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.560787 24314 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:43.562191 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1410) took 2907 ms. Trace:
I20260809 23:20:43.506894 32391 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:43.562682 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.563048 24314 rpcz_store.cc:276] 0809 23:20:40.654880 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.525355 (+2870475us) service_pool.cc:227] Handling call
0809 23:20:43.560527 (+ 35172us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.561873 (+  1346us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.562124 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.551088   588 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:43.568157 32741 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.574991   509 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.588582 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1395) took 2161 ms. Trace:
I20260809 23:20:43.588511 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.549054   526 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:43.577231   315 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.578482   615 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260809 23:20:43.579617   826 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
W20260809 23:20:43.580576   831 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260809 23:20:43.581350 24317 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:43.641065 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3061) took 3291 ms. Trace:
I20260809 23:20:43.641449 24317 rpcz_store.cc:276] 0809 23:20:40.350044 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.545610 (+3195566us) service_pool.cc:227] Handling call
0809 23:20:43.560801 (+ 15191us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.640712 (+ 79911us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.640997 (+   285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.523847   530 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:43.547566   825 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260809 23:20:43.585610 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.587212 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.649010 25363 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.543783 32752 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:43.523612   711 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.589205 25373 rpcz_store.cc:276] 0809 23:20:41.402866 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.520579 (+2117713us) service_pool.cc:227] Handling call
0809 23:20:43.560081 (+ 39502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.543982   563 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:43.509864   809 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.659459   809 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
W20260809 23:20:43.659917   833 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260809 23:20:43.659982   809 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:43.661089   809 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.593739 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:43.662629 24314 raft_consensus.cc:595] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.663000 24314 raft_consensus.cc:603] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:43.595726   700 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.600625 24540 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:43.663947   532 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.614408   326 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.665337   326 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:43.623615   832 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260809 23:20:43.666607   532 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.604553   615 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.607304   315 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.667339 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:20:43.667342   615 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:43.608039   588 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:43.644470 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.666679 25371 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:20:43.634189   565 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.592178   526 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.654788 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.477345   684 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.608436   719 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.591513 32391 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:43.678648 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1407) took 1150 ms. Trace:
I20260809 23:20:43.671663 24428 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:43.583324   747 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.684037 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.736443 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.737381 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:43.685367 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 287180us
I20260809 23:20:43.692582 25363 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:43.739058 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.739317 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.637588 32354 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.713794   532 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.715185 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1396) took 3780 ms. Trace:
I20260809 23:20:43.748826 25375 rpcz_store.cc:276] 0809 23:20:39.934356 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.072958 (+138602us) service_pool.cc:227] Handling call
0809 23:20:40.073571 (+   613us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:42.504021 (+2430450us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.714661 (+1210640us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b540064c8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:43.715069 (+   408us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5397484416}
I20260809 23:20:43.750113 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.750700 25375 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
W20260809 23:20:43.716499 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1412) took 1913 ms (client timeout 1537 ms). Trace:
I20260809 23:20:43.717926 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1397) took 3783 ms. Trace:
W20260809 23:20:43.751888 25369 rpcz_store.cc:269] 0809 23:20:41.803354 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.848433 (+ 45079us) service_pool.cc:227] Handling call
0809 23:20:43.716248 (+1867815us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.716408 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10043648}
I20260809 23:20:43.751960 25368 rpcz_store.cc:276] 0809 23:20:39.934587 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.256936 (+322349us) service_pool.cc:227] Handling call
0809 23:20:42.547064 (+2290128us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:42.547783 (+   719us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.717676 (+1169893us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b540001ac50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.717856 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5265926400}
I20260809 23:20:43.719511 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:43.720583 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2597) took 2587 ms (client timeout 2234 ms). Trace:
W20260809 23:20:43.753448 24312 rpcz_store.cc:269] 0809 23:20:41.133027 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.174958 (+1041931us) service_pool.cc:227] Handling call
0809 23:20:43.720378 (+1545420us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.720503 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107381632}
W20260809 23:20:43.722026 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1403) took 1626 ms (client timeout 1706 ms). Trace:
I20260809 23:20:43.754698 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:43.754803 25372 rpcz_store.cc:269] 0809 23:20:42.096129 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.283619 (+187490us) service_pool.cc:227] Handling call
0809 23:20:43.721828 (+1438209us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.721960 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449099648}
I20260809 23:20:43.755800 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34762 (request call id 1094) took 4364 ms. Trace:
I20260809 23:20:43.672067   711 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:43.722918 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3075) took 1831 ms (client timeout 2022 ms). Trace:
W20260809 23:20:43.756801 24535 rpcz_store.cc:269] 0809 23:20:41.891899 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.938409 (+ 46510us) service_pool.cc:227] Handling call
0809 23:20:43.722693 (+1784284us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.722842 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43527424}
I20260809 23:20:43.724018 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3090) took 1327 ms. Trace:
I20260809 23:20:43.758129 24543 rpcz_store.cc:276] 0809 23:20:42.396570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.842088 (+445518us) service_pool.cc:227] Handling call
0809 23:20:43.723825 (+881737us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b5400147154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.723953 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1496044416}
I20260809 23:20:43.734108   531 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.735790 25373 rpcz_store.cc:276] 0809 23:20:42.528118 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.654473 (+1126355us) service_pool.cc:227] Handling call
0809 23:20:43.678566 (+ 24093us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.740355 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.745483 24539 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:43.753057 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.763140 24312 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:43.756157 24429 rpcz_store.cc:276] 0809 23:20:39.374669 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.980931 (+606262us) service_pool.cc:227] Handling call
0809 23:20:43.738894 (+3757963us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.739490 (+   596us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3926888704}
I20260809 23:20:43.766506 24430 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.15-15.15]
I20260809 23:20:43.767369 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1131) took 1098 ms. Trace:
I20260809 23:20:43.767745 24430 rpcz_store.cc:276] 0809 23:20:42.668511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.765743 (+1097232us) service_pool.cc:227] Handling call
0809 23:20:43.766174 (+   431us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.767047 (+   873us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.767293 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:43.772531   434 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.001s
I20260809 23:20:43.772939   434 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:43.773464 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2958) took 2301 ms (client timeout 2680 ms). Trace:
W20260809 23:20:43.773850 24544 rpcz_store.cc:269] 0809 23:20:41.481968 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.381152 (+899184us) service_pool.cc:227] Handling call
0809 23:20:43.773302 (+1392150us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.773408 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258994432}
I20260809 23:20:43.683245   526 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.764940 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.774909 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:43.775475   595 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.003s
I20260809 23:20:43.788599   595 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.776513 25370 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:43.791292 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.792176   814 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.792425 24539 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260809 23:20:43.793018 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3091) took 1127 ms. Trace:
I20260809 23:20:43.777997 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3058) took 3706 ms. Trace:
I20260809 23:20:43.793792 24311 rpcz_store.cc:276] 0809 23:20:40.071532 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.294534 (+223002us) service_pool.cc:227] Handling call
0809 23:20:43.129376 (+2834842us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.130285 (+   909us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.777796 (+647511us) spinlock_profiling.cc:243] Waited 2.83 s on lock 0x7b5400253fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.777938 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6518699904}
I20260809 23:20:43.795003   502 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.795337 24311 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260809 23:20:43.795668 24311 pending_rounds.cc:76] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 7
I20260809 23:20:43.795866 24311 pending_rounds.cc:97] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260809 23:20:43.796101 24311 raft_consensus.cc:2866] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:43.738459   808 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:43.798488 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3062) took 2397 ms. Trace:
I20260809 23:20:43.798882 24311 rpcz_store.cc:276] 0809 23:20:41.400981 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.794687 (+2393706us) service_pool.cc:227] Handling call
0809 23:20:43.795006 (+   319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.797481 (+  2475us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.797657 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.780205   525 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.747474   707 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:43.781490 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1404) took 1670 ms (client timeout 2146 ms). Trace:
I20260809 23:20:43.808374   494 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.786574 32391 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:43.808643 25366 rpcz_store.cc:269] 0809 23:20:42.111409 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.551680 (+440271us) service_pool.cc:227] Handling call
0809 23:20:43.781330 (+1229650us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.781433 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326450560}
I20260809 23:20:43.809100 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.809779 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.810343 25366 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:43.782963 25372 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:20:43.811216 24314 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:43.812043 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1436) took 3746 ms. Trace:
I20260809 23:20:43.812534 32354 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.812666 25372 rpcz_store.cc:276] 0809 23:20:40.065711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.782344 (+3716633us) service_pool.cc:227] Handling call
0809 23:20:43.782669 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.811792 (+ 29123us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.811991 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.813438 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1413) took 2507 ms. Trace:
I20260809 23:20:43.814275 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.814407   492 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.800827   565 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.783154   587 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.793385 24539 rpcz_store.cc:276] 0809 23:20:42.665664 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.760647 (+1094983us) service_pool.cc:227] Handling call
0809 23:20:43.792952 (+ 32305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.816491   492 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.783218 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.816903 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.817485 24535 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:43.817673   763 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:43.818285 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.783505 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.783628 24380 leader_election.cc:398] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15.
I20260809 23:20:43.816927   587 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.764511   532 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:43.819788 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59816.4us
I20260809 23:20:43.820536 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:43.821890   532 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:43.823568   836 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
I20260809 23:20:43.785861 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1414) took 2422 ms. Trace:
I20260809 23:20:43.824240 24312 rpcz_store.cc:276] 0809 23:20:41.363200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.754214 (+2391014us) service_pool.cc:227] Handling call
0809 23:20:43.754547 (+   333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.785610 (+ 31063us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.785797 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.813956 24314 rpcz_store.cc:276] 0809 23:20:41.306378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.663724 (+2357346us) service_pool.cc:227] Handling call
0809 23:20:43.664041 (+   317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.812424 (+148383us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.813157 (+   733us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:43.813360 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":337677056}
W20260809 23:20:43.781911 32761 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.973s	user 0.004s	sys 0.008s
I20260809 23:20:43.786420 24429 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 12.
I20260809 23:20:43.834076 25373 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260809 23:20:43.834473 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.819231 24543 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:43.818917 24535 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:43.835812 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.836410 24535 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:43.837432 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.838195 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:43.838737 24429 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.787400 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2962) took 2277 ms. Trace:
I20260809 23:20:43.839533 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.839673 24544 rpcz_store.cc:276] 0809 23:20:41.512429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.774653 (+2262224us) service_pool.cc:227] Handling call
0809 23:20:43.787336 (+ 12683us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.792490 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1406) took 1265 ms. Trace:
I20260809 23:20:43.768354 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.783926 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.779906 24541 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.826639 24314 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260809 23:20:43.798580 24428 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:43.810285 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.805472   595 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.822837 25375 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:43.834633 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1393) took 3762 ms. Trace:
I20260809 23:20:43.844995 25373 rpcz_store.cc:276] 0809 23:20:40.071809 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.828403 (+3756594us) service_pool.cc:227] Handling call
0809 23:20:43.828681 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.834409 (+  5728us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.834584 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.846143 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.848929 32752 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:43.849438 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:38102 (request call id 2954) took 3806 ms. Trace:
I20260809 23:20:43.838007 24535 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.757557   719 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:43.852381 24541 rpcz_store.cc:276] 0809 23:20:40.036287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.036444 (+   157us) service_pool.cc:227] Handling call
0809 23:20:43.778779 (+3742335us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b5400116554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23857da7 00007fac287a3ee0 00007fac21e95784 00007fac21e95712 00007fac21e95641 00007fac21e955bc 00007fac21e94872 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.842396 (+ 63617us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5235879168}
I20260809 23:20:43.808152   525 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:43.840433 25370 rpcz_store.cc:276] 0809 23:20:42.527371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.147290 (+619919us) service_pool.cc:227] Handling call
0809 23:20:43.792306 (+645016us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.792434 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1444757248}
W20260809 23:20:43.853870 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85591.5us
I20260809 23:20:43.854414 25370 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:43.829981 24311 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:20:43.827314 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2689) took 1263 ms. Trace:
I20260809 23:20:43.855118 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1418) took 1791 ms. Trace:
I20260809 23:20:43.856101 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.854977 24424 rpcz_store.cc:276] 0809 23:20:42.563621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.969788 (+406167us) service_pool.cc:227] Handling call
0809 23:20:42.970073 (+   285us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:42.970513 (+   440us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:42.970553 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:42.990236 (+ 19683us) log.cc:842] Serialized 26 byte log entry
0809 23:20:43.471297 (+481061us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:43.471334 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.471393 (+    59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:43.827118 (+355725us) raft_consensus.cc:1671] finished
0809 23:20:43.827138 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.827261 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2812,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":16718,"thread_start_us":480672,"threads_started":1,"wal-append.queue_time_us":562530}
I20260809 23:20:43.857466 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.858412 24424 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.817677 24539 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.850663 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1392) took 3779 ms. Trace:
I20260809 23:20:43.860033 25375 rpcz_store.cc:276] 0809 23:20:40.071399 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.822383 (+3750984us) service_pool.cc:227] Handling call
0809 23:20:43.822666 (+   283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.850369 (+ 27703us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.850596 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.861526 25375 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:43.862160 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1441) took 2355 ms. Trace:
I20260809 23:20:43.851642 25373 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:43.828619 25372 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:43.864939 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1442) took 2357 ms. Trace:
I20260809 23:20:43.865329 25372 rpcz_store.cc:276] 0809 23:20:41.507238 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.864012 (+2356774us) service_pool.cc:227] Handling call
0809 23:20:43.864325 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.864536 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.864570 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:43.864625 (+    55us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:43.864680 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.864761 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.864881 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.842873 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3143) took 1276 ms. Trace:
I20260809 23:20:43.867098 24428 rpcz_store.cc:276] 0809 23:20:42.566871 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.797846 (+1230975us) service_pool.cc:227] Handling call
0809 23:20:43.798322 (+   476us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.842569 (+ 44247us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.842799 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.856029 24311 rpcz_store.cc:276] 0809 23:20:42.063992 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.829428 (+1765436us) service_pool.cc:227] Handling call
0809 23:20:43.829688 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.854834 (+ 25146us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.855065 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.866878 25372 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:43.862504 25375 rpcz_store.cc:276] 0809 23:20:41.506372 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.860981 (+2354609us) service_pool.cc:227] Handling call
0809 23:20:43.861313 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.861927 (+   614us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.862107 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.871057 25375 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:43.871821 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1404) took 2116 ms. Trace:
I20260809 23:20:43.872251 25375 rpcz_store.cc:276] 0809 23:20:41.755026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.870491 (+2115465us) service_pool.cc:227] Handling call
0809 23:20:43.870822 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.871545 (+   723us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.871766 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.843641 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1415) took 1995 ms. Trace:
I20260809 23:20:43.875761 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.874296 24314 rpcz_store.cc:276] 0809 23:20:41.848729 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.825911 (+1977182us) service_pool.cc:227] Handling call
0809 23:20:43.826259 (+   348us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.843364 (+ 17105us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.843580 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.878571 24314 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260809 23:20:43.880793   839 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260809 23:20:43.882042   502 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:43.883361 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1403) took 2128 ms. Trace:
I20260809 23:20:43.884011 25372 rpcz_store.cc:276] 0809 23:20:41.754645 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.866344 (+2111699us) service_pool.cc:227] Handling call
0809 23:20:43.866656 (+   312us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.883060 (+ 16404us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.883298 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.884378 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.886279 25372 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:20:43.887079 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1415) took 1832 ms. Trace:
I20260809 23:20:43.887516 25372 rpcz_store.cc:276] 0809 23:20:42.054696 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.885583 (+1830887us) service_pool.cc:227] Handling call
0809 23:20:43.885909 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.886751 (+   842us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.887024 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.889215 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:43.879442 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1419) took 1705 ms. Trace:
I20260809 23:20:43.883172   719 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:43.891077 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:43.891371   719 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.892338 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.893129 24544 raft_consensus.cc:2396] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260809 23:20:43.893869 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:43.895370 25366 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:43.895989 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1438) took 2722 ms. Trace:
I20260809 23:20:43.896275 25366 rpcz_store.cc:276] 0809 23:20:41.174012 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.844318 (+2670306us) service_pool.cc:227] Handling call
0809 23:20:43.895190 (+ 50872us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.895765 (+   575us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.895940 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.897370 25366 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:43.897930 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1416) took 1789 ms. Trace:
I20260809 23:20:43.898339 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1439) took 2684 ms. Trace:
I20260809 23:20:43.898831 25370 rpcz_store.cc:276] 0809 23:20:41.174333 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.853952 (+2679619us) service_pool.cc:227] Handling call
0809 23:20:43.854246 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.858430 (+  4184us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.858641 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.900030 25370 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:43.900571 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1447) took 1705 ms. Trace:
I20260809 23:20:43.900862 25370 rpcz_store.cc:276] 0809 23:20:42.195267 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.899571 (+1704304us) service_pool.cc:227] Handling call
0809 23:20:43.899859 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.900337 (+   478us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.900523 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.890645 24314 rpcz_store.cc:276] 0809 23:20:42.173630 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.877862 (+1704232us) service_pool.cc:227] Handling call
0809 23:20:43.878183 (+   321us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.879169 (+   986us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.879383 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.904974 24314 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:43.905627 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3074) took 1724 ms. Trace:
I20260809 23:20:43.906064 24314 rpcz_store.cc:276] 0809 23:20:42.180966 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.904235 (+1723269us) service_pool.cc:227] Handling call
0809 23:20:43.904760 (+   525us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.905378 (+   618us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.905571 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.908325 25375 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:43.909134 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1406) took 2153 ms. Trace:
I20260809 23:20:43.909485 25375 rpcz_store.cc:276] 0809 23:20:41.755541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.873891 (+2118350us) service_pool.cc:227] Handling call
0809 23:20:43.879975 (+  6084us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.908823 (+ 28848us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.909075 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:43.911164 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53024.2us
I20260809 23:20:43.911906 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:43.913033 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.914083   562 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:43.915175   843 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:43.915602   843 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:43.827654   841 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:43.915840 32640 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:43.916523 32640 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.844338   808 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:43.916975 32640 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:43.917493   808 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:43.919538   562 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:43.920071   562 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:43.920581   747 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:43.921308   434 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:43.844442   587 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:43.921033   562 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.921974   587 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.922350   587 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:43.922699   562 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:43.928071 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2588) took 4129 ms. Trace:
I20260809 23:20:43.928462 24320 rpcz_store.cc:276] 0809 23:20:39.798350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:39.985702 (+187352us) service_pool.cc:227] Handling call
0809 23:20:39.985978 (+   276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:40.003551 (+ 17573us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:40.211943 (+208392us) log.cc:842] Serialized 14 byte log entry
0809 23:20:40.658015 (+446072us) log.cc:842] Serialized 14 byte log entry
0809 23:20:41.856116 (+1198101us) spinlock_profiling.cc:243] Waited 9.18 ms on lock 0x7b1800171a00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:42.066113 (+209997us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:42.066191 (+    78us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:42.066263 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.927849 (+1861586us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b540043c1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:43.927901 (+    52us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.928018 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8401,"prepare.queue_time_us":239666,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":92,"raft.queue_time_us":267419,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":172,"spinlock_wait_cycles":61073536,"thread_start_us":818625,"threads_started":3,"wal-append.queue_time_us":883604,"wal-append.run_cpu_time_us":9114,"wal-append.run_wall_time_us":1349576}
I20260809 23:20:43.932437 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1427) took 1026 ms. Trace:
I20260809 23:20:43.932778 24320 rpcz_store.cc:276] 0809 23:20:42.905827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.930279 (+1024452us) service_pool.cc:227] Handling call
0809 23:20:43.930580 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.932047 (+  1467us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.932082 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:43.932123 (+    41us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:43.932154 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.932214 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.932318 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":917,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87}
I20260809 23:20:43.917061   740 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.018s
I20260809 23:20:43.935647   740 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:43.936093   740 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:43.936954   740 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.938184   740 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:43.919603   808 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:43.927429 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2963) took 2404 ms (client timeout 2340 ms). Trace:
W20260809 23:20:43.939921 24534 rpcz_store.cc:269] 0809 23:20:41.522856 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.865888 (+343032us) service_pool.cc:227] Handling call
0809 23:20:43.927250 (+2061362us) spinlock_profiling.cc:243] Waited 67.3 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:43.927361 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154699008}
I20260809 23:20:43.940471 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:20:43.941342 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:43.855626 32752 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:43.942631 32752 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:43.942718 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2955) took 3691 ms. Trace:
I20260809 23:20:43.943893 24542 rpcz_store.cc:276] 0809 23:20:40.251368 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.258605 (+  7237us) service_pool.cc:227] Handling call
0809 23:20:40.258885 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:41.945403 (+1686518us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.942494 (+1997091us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:43.942660 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3606199040}
I20260809 23:20:43.950033 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3073) took 1777 ms. Trace:
I20260809 23:20:43.950531 24311 rpcz_store.cc:276] 0809 23:20:42.172735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.949077 (+1776342us) service_pool.cc:227] Handling call
0809 23:20:43.949449 (+   372us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.949660 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.949695 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:43.949747 (+    52us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:43.949796 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.949856 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.949968 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.953253 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:43.898227 25366 rpcz_store.cc:276] 0809 23:20:42.108498 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.896935 (+1788437us) service_pool.cc:227] Handling call
0809 23:20:43.897207 (+   272us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.897721 (+   514us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.897886 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.954963 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.955606 24428 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:43.955843 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:43.956851 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 368088us
I20260809 23:20:43.958981 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:43.944761 24429 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:43.960988 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:43.961589 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:43.914721 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:43.964112 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49468.1us
I20260809 23:20:43.965879 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3083) took 1775 ms. Trace:
I20260809 23:20:43.961089 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2685) took 1456 ms. Trace:
I20260809 23:20:43.966419 24544 rpcz_store.cc:276] 0809 23:20:42.190671 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.894738 (+1704067us) service_pool.cc:227] Handling call
0809 23:20:43.965806 (+ 71068us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.973367 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:43.974227 24424 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:43.975672 24424 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:43.966729 24423 rpcz_store.cc:276] 0809 23:20:42.504818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.941426 (+436608us) service_pool.cc:227] Handling call
0809 23:20:42.941861 (+   435us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:43.103313 (+161452us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.103345 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:43.103861 (+   516us) log.cc:842] Serialized 52 byte log entry
0809 23:20:43.499617 (+395756us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:43.954599 (+454982us) log.cc:842] Serialized 14 byte log entry
0809 23:20:43.960677 (+  6078us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.960776 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:43.960836 (+    60us) raft_consensus.cc:1671] finished
0809 23:20:43.960853 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:43.961025 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48929,"prepare.queue_time_us":57316,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":49816,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7479,"thread_start_us":804652,"threads_started":2,"wal-append.queue_time_us":570502}
I20260809 23:20:43.988811 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3145) took 1318 ms. Trace:
I20260809 23:20:44.007439 24424 rpcz_store.cc:276] 0809 23:20:42.669990 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.975074 (+1305084us) service_pool.cc:227] Handling call
0809 23:20:43.975445 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.988491 (+ 13046us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.988737 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:43.994201 24428 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:43.915990   843 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:44.029249 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70376.2us
I20260809 23:20:44.029893 25366 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:43.883750   707 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:44.022583 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:43.848999 32761 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.020887 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.030263 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:43.883109   682 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.034852 24429 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:43.910328   530 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:44.039340 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3148) took 1219 ms. Trace:
I20260809 23:20:44.039496 24428 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.039695 24429 rpcz_store.cc:276] 0809 23:20:42.819676 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.034124 (+1214448us) service_pool.cc:227] Handling call
0809 23:20:44.034535 (+   411us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.038828 (+  4293us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.039288 (+   460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.040024 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:44.042496 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.043120 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.044373 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:44.044998 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.045567 24544 raft_consensus.cc:2362] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17. Re-sending same reply.
I20260809 23:20:44.046463 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.048564 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:44.023420   688 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.000s
I20260809 23:20:43.971825   525 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:44.054649 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3151) took 1200 ms. Trace:
I20260809 23:20:44.057379 25366 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.057356 24423 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.058961 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3147) took 1292 ms. Trace:
I20260809 23:20:44.021210   300 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:44.059496 24423 rpcz_store.cc:276] 0809 23:20:42.766505 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.012801 (+1246296us) service_pool.cc:227] Handling call
0809 23:20:44.057143 (+ 44342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.058459 (+  1316us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.058894 (+   435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.061185 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.063282 24543 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:20:44.063145 24539 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:44.063916 24435 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.035705   707 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:44.065246   707 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:44.065150 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:44.066344 25372 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:44.068830   300 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:43.986470   315 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.077s
I20260809 23:20:44.070444   315 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:44.071003   315 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:44.071909   315 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.070021   300 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:43.925418   711 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.080121 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:44.080435   711 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.045817 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.056567 24428 rpcz_store.cc:276] 0809 23:20:42.854541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.039060 (+1184519us) service_pool.cc:227] Handling call
0809 23:20:44.039321 (+   261us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.051512 (+ 12191us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.054582 (+  3070us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.083828 24313 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:44.038357   808 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:44.052829   578 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.667s	user 0.005s	sys 0.007s
I20260809 23:20:44.053746   493 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:44.102605 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:44.104907 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1133) took 1057 ms. Trace:
W20260809 23:20:44.105322 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 2968) took 2484 ms (client timeout 2027 ms). Trace:
W20260809 23:20:44.107951 24546 rpcz_store.cc:269] 0809 23:20:41.621314 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.686628 (+ 65314us) service_pool.cc:227] Handling call
0809 23:20:44.105152 (+2418524us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.105254 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":283339520}
I20260809 23:20:44.058010 32354 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:44.056344   509 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:44.056493   763 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election won for term 13
I20260809 23:20:44.051951   814 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:44.153347   763 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:44.106799 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.064s	sys 0.114s
W20260809 23:20:44.154295 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.065s	sys 0.114s
W20260809 23:20:44.154886 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 682489us
I20260809 23:20:44.154559   763 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.155732 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:44.081095   711 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:44.086584   525 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:44.110888 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:44.107399 24425 rpcz_store.cc:276] 0809 23:20:43.047115 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.170773 (+123658us) service_pool.cc:227] Handling call
0809 23:20:44.104635 (+933862us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.104829 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":688120192}
I20260809 23:20:44.167817 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3075) took 1980 ms. Trace:
I20260809 23:20:44.168303 24313 rpcz_store.cc:276] 0809 23:20:42.187232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.519536 (+1332304us) service_pool.cc:227] Handling call
0809 23:20:44.167592 (+648056us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.167752 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1200182656}
I20260809 23:20:44.113597 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:44.169801 24317 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:44.170486 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57016.7us
I20260809 23:20:44.029525 32640 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.035885 32471 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260809 23:20:44.083460   615 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.104785   494 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260809 23:20:44.125371   509 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:44.128888 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.129523 32354 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:44.019966   597 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.131188 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:44.190222 25366 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:44.107650   578 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.105567   531 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:44.061815   492 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.078s
I20260809 23:20:44.101428   530 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:44.199250   554 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:44.139737 24540 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: []
W20260809 23:20:44.141357   760 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.000s
I20260809 23:20:44.142299 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1395) took 4208 ms. Trace:
I20260809 23:20:44.234135 25371 rpcz_store.cc:276] 0809 23:20:39.934075 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:40.073888 (+139813us) service_pool.cc:227] Handling call
0809 23:20:40.103459 (+ 29571us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.671786 (+3568327us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.142057 (+470271us) spinlock_profiling.cc:243] Waited 3.56 s on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.142233 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8193867648}
I20260809 23:20:44.144872   843 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.148689   758 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.237491   758 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:44.239049 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2965) took 2710 ms. Trace:
I20260809 23:20:44.239547 24540 rpcz_store.cc:276] 0809 23:20:41.528848 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.139239 (+2610391us) service_pool.cc:227] Handling call
0809 23:20:44.139561 (+   322us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.238698 (+ 99137us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.238978 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.245518 25365 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:44.181797   499 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:44.043460   841 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.081s
I20260809 23:20:44.192101   509 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:44.184463 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:44.062568   688 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.128127   326 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.183482 24535 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:44.299542 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.300395 24540 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:44.191250 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.300987 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2969) took 2622 ms. Trace:
I20260809 23:20:44.160981   711 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:44.189733   700 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.010s
I20260809 23:20:44.139577   493 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:44.186836   624 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260809 23:20:44.199678 32471 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.249454 32354 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.250833   726 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.327036 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:44.182489   525 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:44.177965 24422 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:44.327680 24313 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:44.274413 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.010s	sys 0.024s
I20260809 23:20:44.275955   497 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:44.277516 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2599) took 3143 ms (client timeout 1846 ms). Trace:
I20260809 23:20:44.278378 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3142) took 1711 ms. Trace:
I20260809 23:20:44.329059 24435 rpcz_store.cc:276] 0809 23:20:42.566545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.046644 (+480099us) service_pool.cc:227] Handling call
0809 23:20:43.054313 (+  7669us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.064356 (+1010043us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.278139 (+213783us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.278312 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2321414528}
W20260809 23:20:44.328964 24309 rpcz_store.cc:269] 0809 23:20:41.133788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.055020 (+921232us) service_pool.cc:227] Handling call
0809 23:20:44.277329 (+2222309us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b54003c23d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.277450 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2174419200}
I20260809 23:20:44.285699 31984 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:44.290011 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1117) took 2405 ms. Trace:
I20260809 23:20:44.340220 24427 rpcz_store.cc:276] 0809 23:20:41.884312 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:41.884649 (+   337us) service_pool.cc:227] Handling call
0809 23:20:41.897504 (+ 12855us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:42.478956 (+581452us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:42.478986 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:43.091426 (+612440us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b18004cf120. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:20:43.091944 (+   518us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:43.093334 (+  1390us) log.cc:842] Serialized 52 byte log entry
0809 23:20:43.453585 (+360251us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:43.502777 (+ 49192us) log.cc:842] Serialized 14 byte log entry
0809 23:20:43.503074 (+   297us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.289635 (+786561us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54003903d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.289695 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:44.289751 (+    56us) raft_consensus.cc:1671] finished
0809 23:20:44.289762 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.289941 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3107,"prepare.queue_time_us":49569,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":4519,"raft.queue_time_us":1686,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":19374,"spinlock_wait_cycles":919486336,"thread_start_us":359664,"threads_started":1,"wal-append.queue_time_us":517393}
I20260809 23:20:44.291327 24534 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.292351 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1419) took 1855 ms. Trace:
I20260809 23:20:44.343084 25365 rpcz_store.cc:276] 0809 23:20:42.436931 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.553733 (+1116802us) service_pool.cc:227] Handling call
0809 23:20:44.292122 (+738389us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.292280 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1575720832}
I20260809 23:20:44.301294 24540 rpcz_store.cc:276] 0809 23:20:41.678536 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.240425 (+2561889us) service_pool.cc:227] Handling call
0809 23:20:44.300210 (+ 59785us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.300748 (+   538us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.300932 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.304492 24535 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260809 23:20:44.344928 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2970) took 2651 ms. Trace:
I20260809 23:20:44.345363 24535 rpcz_store.cc:276] 0809 23:20:41.693242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.300421 (+2607179us) service_pool.cc:227] Handling call
0809 23:20:44.300720 (+   299us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.344608 (+ 43888us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.344848 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.200037   531 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:44.254220 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.016s	sys 0.033s
W20260809 23:20:44.254067   706 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.023s	sys 0.012s
W20260809 23:20:44.267169   809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.018s	sys 0.020s
W20260809 23:20:44.308604 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 124252us
I20260809 23:20:44.205108 25363 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:44.185222   588 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
W20260809 23:20:44.346654   684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.010s	sys 0.015s
I20260809 23:20:44.366873 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:44.367085   588 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:44.367575   588 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:44.327135 32471 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.368427   588 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.369721   588 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:44.370612 32471 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.253744   554 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:44.152181   707 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.196702   747 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.117s
I20260809 23:20:44.280889 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2600) took 3146 ms. Trace:
I20260809 23:20:44.393378 24317 rpcz_store.cc:276] 0809 23:20:41.134099 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.642398 (+2508299us) service_pool.cc:227] Handling call
0809 23:20:44.169564 (+527166us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.170316 (+   752us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.280652 (+110336us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54003a48d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.280818 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1141957504}
I20260809 23:20:44.330837 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:44.394783 24309 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:44.328280 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.010s	sys 0.024s
W20260809 23:20:44.338289 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.008s	sys 0.012s
W20260809 23:20:44.396165 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.008s	sys 0.013s
I20260809 23:20:44.329820 32761 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:44.338562 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.009s	sys 0.013s
I20260809 23:20:44.339025 24435 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260809 23:20:44.360874   809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.018s	sys 0.020s
W20260809 23:20:44.396994 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.009s	sys 0.013s
I20260809 23:20:44.397336 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1135) took 1349 ms. Trace:
I20260809 23:20:44.304067   691 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.347460 24535 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:44.400763 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2972) took 2640 ms. Trace:
I20260809 23:20:44.401199 24535 rpcz_store.cc:276] 0809 23:20:41.760629 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.346396 (+2585767us) service_pool.cc:227] Handling call
0809 23:20:44.346741 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.400451 (+ 53710us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.400700 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.402650 24535 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:44.345228 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1297) took 2943 ms. Trace:
I20260809 23:20:44.403543 24534 rpcz_store.cc:276] 0809 23:20:41.402020 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.940708 (+2538688us) service_pool.cc:227] Handling call
0809 23:20:44.290825 (+350117us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:44.291104 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.344925 (+ 53821us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.345163 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39885696}
W20260809 23:20:44.357982 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.020s	sys 0.037s
I20260809 23:20:44.252756   760 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.186915   814 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.088s
I20260809 23:20:44.384019   326 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.376858   531 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.377554   525 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.345688   497 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:44.385452 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.388036 24313 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:44.329291   700 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:44.357683   706 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.023s	sys 0.012s
I20260809 23:20:44.397785 24540 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 16. Ops: []
I20260809 23:20:44.397686 24435 rpcz_store.cc:276] 0809 23:20:43.047738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.330310 (+1282572us) service_pool.cc:227] Handling call
0809 23:20:44.338573 (+  8263us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.397049 (+ 58476us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.397278 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:44.405046 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 920669us
I20260809 23:20:44.422611 24429 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 15. Ops: []
W20260809 23:20:44.367101   684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.010s	sys 0.015s
I20260809 23:20:44.407019 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3073) took 2524 ms. Trace:
I20260809 23:20:44.444265 24535 rpcz_store.cc:276] 0809 23:20:41.882833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.402090 (+2519257us) service_pool.cc:227] Handling call
0809 23:20:44.402442 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.406733 (+  4291us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.406953 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.445487 24535 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:44.446005 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2975) took 2553 ms. Trace:
I20260809 23:20:44.446101 25373 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 17. Ops: []
W20260809 23:20:44.408629 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3153) took 1553 ms (client timeout 1936 ms). Trace:
W20260809 23:20:44.446543 24431 rpcz_store.cc:269] 0809 23:20:42.855211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.913394 (+ 58183us) service_pool.cc:227] Handling call
0809 23:20:44.408377 (+1494983us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.408561 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":519972608}
I20260809 23:20:44.446326 24535 rpcz_store.cc:276] 0809 23:20:41.892736 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.445042 (+2552306us) service_pool.cc:227] Handling call
0809 23:20:44.445327 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.445817 (+   490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.445955 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.447901 24431 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:44.448480 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1140) took 1301 ms. Trace:
I20260809 23:20:44.383698   682 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:44.409927 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:44.410986 25375 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:44.413412 24544 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:44.416258 24534 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:44.421175 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.425660 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.428495 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1139) took 1379 ms. Trace:
I20260809 23:20:44.461215 24422 rpcz_store.cc:276] 0809 23:20:43.048895 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.427336 (+1378441us) service_pool.cc:227] Handling call
0809 23:20:44.427700 (+   364us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.428090 (+   390us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.428129 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.428183 (+    54us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:44.428226 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.428295 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.428428 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12065}
I20260809 23:20:44.440603 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2971) took 2747 ms. Trace:
I20260809 23:20:44.462229 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.462332 24540 rpcz_store.cc:276] 0809 23:20:41.693837 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.365486 (+2671649us) service_pool.cc:227] Handling call
0809 23:20:44.397618 (+ 32132us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.440337 (+ 42719us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.440544 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.409313   844 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.442945 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1410) took 1873 ms. Trace:
I20260809 23:20:44.468130 25363 rpcz_store.cc:276] 0809 23:20:42.569115 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.739086 (+1169971us) service_pool.cc:227] Handling call
0809 23:20:44.442684 (+703598us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.442854 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069339520}
I20260809 23:20:44.449113 24535 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:44.424924   700 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:44.455140 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:44.469754 25366 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:44.456017   848 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260809 23:20:44.412487   849 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:44.458168 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3152) took 1603 ms. Trace:
I20260809 23:20:44.443719 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:44.448768 24431 rpcz_store.cc:276] 0809 23:20:43.147109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.447387 (+1300278us) service_pool.cc:227] Handling call
0809 23:20:44.447687 (+   300us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.448266 (+   579us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.448436 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.460829 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3077) took 2568 ms. Trace:
W20260809 23:20:44.470610 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49542.5us
I20260809 23:20:44.469910 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3079) took 2576 ms. Trace:
I20260809 23:20:44.473749 24423 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:20:44.476466 24429 rpcz_store.cc:276] 0809 23:20:42.854875 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.040718 (+1185843us) service_pool.cc:227] Handling call
0809 23:20:44.041070 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.425347 (+384277us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.457936 (+ 32589us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.458090 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":876921088}
I20260809 23:20:44.487082   554 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.15, Last appended by leader: 15, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.488891   565 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:44.487865   700 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.489298   565 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:44.490625 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.484089 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:44.492797 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1420) took 2055 ms. Trace:
I20260809 23:20:44.560807 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.560818 25375 rpcz_store.cc:276] 0809 23:20:42.437279 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.911097 (+1473818us) service_pool.cc:227] Handling call
0809 23:20:43.911615 (+   518us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.467887 (+556272us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.492539 (+ 24652us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.492737 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1147878144}
I20260809 23:20:44.561986 24309 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:44.482784   844 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.483795   597 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.483106 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:44.475399   530 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:44.478292 24534 rpcz_store.cc:276] 0809 23:20:41.892613 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.404745 (+2512132us) service_pool.cc:227] Handling call
0809 23:20:44.415986 (+ 11241us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.460604 (+ 44618us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.460780 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.416739   326 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.553908 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:44.490149   565 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.480803 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.601836 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.484526 32640 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:44.589864   845 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.199s	user 0.003s	sys 0.000s
I20260809 23:20:44.554548 24422 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:20:44.610109 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1141) took 1418 ms. Trace:
I20260809 23:20:44.610469 24422 rpcz_store.cc:276] 0809 23:20:43.191727 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.462169 (+1270442us) service_pool.cc:227] Handling call
0809 23:20:44.554052 (+ 91883us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.609839 (+ 55787us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.610056 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.481462 24535 rpcz_store.cc:276] 0809 23:20:41.893329 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.448267 (+2554938us) service_pool.cc:227] Handling call
0809 23:20:44.448511 (+   244us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.469581 (+ 21070us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.469845 (+   264us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:44.471395   562 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.004s
I20260809 23:20:44.558055 24540 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
W20260809 23:20:44.619510 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.016s	sys 0.007s
I20260809 23:20:44.410387   691 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:44.522780   740 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.584s	user 0.004s	sys 0.008s
I20260809 23:20:44.587677 24544 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.583563 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1449) took 2115 ms. Trace:
I20260809 23:20:44.583812 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3156) took 1713 ms. Trace:
I20260809 23:20:44.630400 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3085) took 2423 ms. Trace:
I20260809 23:20:44.630169 25373 rpcz_store.cc:276] 0809 23:20:42.468188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.954313 (+1486125us) service_pool.cc:227] Handling call
0809 23:20:44.445933 (+491620us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.463632 (+ 17699us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.583353 (+119721us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b5400403b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:44.583505 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1129933184}
I20260809 23:20:44.631068 24544 rpcz_store.cc:276] 0809 23:20:42.206541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.587091 (+2380550us) service_pool.cc:227] Handling call
0809 23:20:44.587491 (+   400us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.630099 (+ 42608us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.630337 (+   238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.630604 24423 rpcz_store.cc:276] 0809 23:20:42.870080 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.064434 (+1194354us) service_pool.cc:227] Handling call
0809 23:20:44.121525 (+ 57091us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.486284 (+364759us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.583568 (+ 97284us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:44.583753 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":809272192}
W20260809 23:20:44.589761 24543 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.023s	sys 0.026s
I20260809 23:20:44.393276   711 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:44.581925 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.026s	sys 0.021s
I20260809 23:20:44.475713   615 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:44.596611 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.008s	sys 0.032s
W20260809 23:20:44.602771 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 122062us
I20260809 23:20:44.612423   688 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:44.519928   578 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:44.585877   526 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.802s	user 0.008s	sys 0.002s
I20260809 23:20:44.469304   699 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.123s
I20260809 23:20:44.605732 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.430253   531 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260809 23:20:44.564404   719 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.013s	sys 0.013s
I20260809 23:20:44.567636   700 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:44.412092   493 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.490172 32417 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.113s
I20260809 23:20:44.458608   502 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:44.620929 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.016s	sys 0.008s
W20260809 23:20:44.594805   300 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.017s	sys 0.032s
I20260809 23:20:44.623035 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3082) took 2433 ms. Trace:
I20260809 23:20:44.737675 24540 rpcz_store.cc:276] 0809 23:20:42.189179 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.463181 (+2274002us) service_pool.cc:227] Handling call
0809 23:20:44.557822 (+ 94641us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.619584 (+ 61762us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.622960 (+  3376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.619122 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.739079 24535 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:44.619928 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.006s	sys 0.018s
I20260809 23:20:44.566072 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:44.606369   843 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.634514 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:44.634812 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1147) took 1051 ms. Trace:
W20260809 23:20:44.636116 24543 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.023s	sys 0.026s
I20260809 23:20:44.752344 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2964) took 3225 ms. Trace:
W20260809 23:20:44.735765 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 360837us
I20260809 23:20:44.752775 24543 rpcz_store.cc:276] 0809 23:20:41.526518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.071758 (+2545240us) service_pool.cc:227] Handling call
0809 23:20:44.072124 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.752022 (+679898us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.752054 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.752094 (+    40us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:44.752124 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.752187 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.752294 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6748,"raft.queue_time_us":89333,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":14818,"thread_start_us":679478,"threads_started":1}
I20260809 23:20:44.632927 24544 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:44.640820 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.014s	sys 0.040s
W20260809 23:20:44.754602 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.014s	sys 0.040s
I20260809 23:20:44.754774 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3088) took 2358 ms. Trace:
W20260809 23:20:44.620445   587 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.695s	user 0.012s	sys 0.000s
W20260809 23:20:44.612121 32380 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.003s	sys 0.036s
W20260809 23:20:44.646171 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.026s	sys 0.021s
I20260809 23:20:44.665702   760 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.476871 24435 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:44.772735 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2614) took 2318 ms. Trace:
I20260809 23:20:44.608086   588 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:44.654624 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.008s	sys 0.032s
I20260809 23:20:44.656148 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3162) took 1761 ms. Trace:
I20260809 23:20:44.635689   565 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:44.663314 24423 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260809 23:20:44.750908   300 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.018s	sys 0.032s
I20260809 23:20:44.665092 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1411) took 3974 ms. Trace:
I20260809 23:20:44.583016   846 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260809 23:20:44.606750   758 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
W20260809 23:20:44.654866   315 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.004s	sys 0.027s
I20260809 23:20:44.619652   434 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.066s
I20260809 23:20:44.618506   814 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
W20260809 23:20:44.748454 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.006s	sys 0.018s
W20260809 23:20:44.817673 32752 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.000s
I20260809 23:20:44.751834 24422 rpcz_store.cc:276] 0809 23:20:43.583271 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.614754 (+1031483us) service_pool.cc:227] Handling call
0809 23:20:44.634366 (+ 19612us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.634528 (+   162us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.634554 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.634594 (+    40us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:44.634624 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.634672 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.634762 (+    90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.760324 24317 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:44.760573 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:44.762916 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.019s	sys 0.019s
W20260809 23:20:44.842159 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.020s	sys 0.019s
W20260809 23:20:44.711251   719 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.014s	sys 0.014s
W20260809 23:20:44.768046 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 745504us
I20260809 23:20:44.654065 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.766477 24535 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:44.856155 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.857601 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:44.763855   531 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260809 23:20:44.663614 24309 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:44.781301 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:44.710733   707 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.851576 32752 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.780326 24540 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:44.767644   624 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
W20260809 23:20:44.893913 25365 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.021s	sys 0.033s
I20260809 23:20:44.791043 24316 rpcz_store.cc:276] 0809 23:20:40.691232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.522564 (+2831332us) service_pool.cc:227] Handling call
0809 23:20:43.523009 (+   445us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.560414 (+ 37405us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.560447 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:43.560864 (+   417us) log.cc:842] Serialized 26 byte log entry
0809 23:20:43.976331 (+415467us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:43.976373 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:43.976443 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:44.664822 (+688379us) raft_consensus.cc:1671] finished
0809 23:20:44.664863 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.665023 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":408327,"threads_started":1,"wal-append.queue_time_us":743060}
W20260809 23:20:44.894282 25365 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.021s	sys 0.033s
I20260809 23:20:44.792037 24543 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:20:44.895618 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1308) took 2426 ms. Trace:
I20260809 23:20:44.896015 24543 rpcz_store.cc:276] 0809 23:20:42.468813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.754312 (+2285499us) service_pool.cc:227] Handling call
0809 23:20:44.791676 (+ 37364us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.895335 (+103659us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.895554 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.734869   726 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:44.772389 32380 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.003s	sys 0.037s
I20260809 23:20:44.796653   748 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:20:44.692564   526 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:44.771150   686 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.772017   740 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.748493   578 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:44.755108 24544 rpcz_store.cc:276] 0809 23:20:42.396016 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.632316 (+2236300us) service_pool.cc:227] Handling call
0809 23:20:44.632695 (+   379us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.754531 (+121836us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.754724 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.917745 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.918444 24544 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:44.864454 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:44.824260   300 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.867300 25370 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.919590 24535 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:44.921247 24535 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:44.921957 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1316) took 1419 ms. Trace:
I20260809 23:20:44.774493 24425 rpcz_store.cc:276] 0809 23:20:42.894512 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.164983 (+1270471us) service_pool.cc:227] Handling call
0809 23:20:44.165288 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.165495 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:44.185745 (+ 20250us) log.cc:842] Serialized 14 byte log entry
0809 23:20:44.655449 (+469704us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.655568 (+   119us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:44.655684 (+   116us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.655814 (+   130us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.656032 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16038,"prepare.queue_time_us":21105,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":20355,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":29866,"thread_start_us":453096,"threads_started":1}
I20260809 23:20:44.891404   843 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:44.901409 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3089) took 2505 ms. Trace:
I20260809 23:20:44.931468 24540 rpcz_store.cc:276] 0809 23:20:42.396299 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.738708 (+2342409us) service_pool.cc:227] Handling call
0809 23:20:44.780090 (+ 41382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.894090 (+114000us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.901347 (+  7257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.767424   587 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:44.902544 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45062.6us
I20260809 23:20:44.756121   502 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:44.906088 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:44.909793 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2959) took 3421 ms. Trace:
I20260809 23:20:44.933370 24542 rpcz_store.cc:276] 0809 23:20:41.489057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.944874 (+2455817us) service_pool.cc:227] Handling call
0809 23:20:44.293920 (+349046us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:44.294119 (+   199us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.294565 (+   446us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.294603 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:44.295181 (+   578us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.691303 (+396122us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:44.691344 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.691412 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:44.894081 (+202669us) raft_consensus.cc:1671] finished
0809 23:20:44.894103 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.909710 (+ 15607us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4954,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":8276,"spinlock_wait_cycles":24620288,"thread_start_us":395686,"threads_started":1,"wal-append.queue_time_us":554337}
I20260809 23:20:44.935916 24542 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260809 23:20:44.936347 24542 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 10
I20260809 23:20:44.936558 24542 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260809 23:20:44.936782 24542 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:44.914762 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:44.922292 24535 rpcz_store.cc:276] 0809 23:20:43.502735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.920498 (+1417763us) service_pool.cc:227] Handling call
0809 23:20:44.920799 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.921665 (+   866us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.921896 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.936566 24544 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:44.939780 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2981) took 1883 ms. Trace:
I20260809 23:20:44.940086 24535 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:20:44.940057 24544 rpcz_store.cc:276] 0809 23:20:43.056670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.936031 (+1879361us) service_pool.cc:227] Handling call
0809 23:20:44.936400 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.939578 (+  3178us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.939732 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:44.919541 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.943404 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.943977 24535 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:44.949224 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 295227us
I20260809 23:20:44.950526 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.943172 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:44.953065 24544 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:44.955051 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 874980us
I20260809 23:20:44.957135 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:20:44.949249 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 188761us
I20260809 23:20:44.903729   844 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.979282 24424 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 13. Current term is 15. Ops: []
I20260809 23:20:44.980438 24541 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:44.920557   588 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:44.995857 24540 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260809 23:20:45.014163 24544 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:20:45.206539 24534 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.013s	sys 0.013s
W20260809 23:20:45.207216 24534 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.014s	sys 0.013s
I20260809 23:20:45.207645 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1413) took 3357 ms. Trace:
I20260809 23:20:45.016000 24535 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260809 23:20:45.209170 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2967) took 3622 ms. Trace:
I20260809 23:20:45.208191 25368 rpcz_store.cc:276] 0809 23:20:41.849893 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.880494 (+2030601us) service_pool.cc:227] Handling call
0809 23:20:43.880819 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.881208 (+   389us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.881237 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:44.164287 (+283050us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b180013b100. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:20:44.165258 (+   971us) spinlock_profiling.cc:243] Waited 2.99 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:44.198810 (+ 33552us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.863794 (+664984us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:44.863859 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.863947 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.206896 (+342949us) raft_consensus.cc:1671] finished
0809 23:20:45.206967 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.207324 (+   357us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6730,"raft.queue_time_us":3164,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":81205760,"thread_start_us":468261,"threads_started":1,"wal-append.queue_time_us":813524}
I20260809 23:20:45.212076 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1437) took 4038 ms. Trace:
W20260809 23:20:45.051059 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 303293us
I20260809 23:20:45.212510 25369 rpcz_store.cc:276] 0809 23:20:41.173610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.817750 (+2644140us) service_pool.cc:227] Handling call
0809 23:20:43.818046 (+   296us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.818257 (+   211us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.818283 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:43.818724 (+   441us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.773548 (+954824us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:44.773590 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.773666 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.211525 (+437859us) raft_consensus.cc:1671] finished
0809 23:20:45.211574 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.211884 (+   310us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":954444,"threads_started":1,"wal-append.queue_time_us":1268185}
W20260809 23:20:45.185336 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.003s	sys 0.007s
W20260809 23:20:45.185963 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.007s	sys 0.007s
W20260809 23:20:45.216454 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.003s	sys 0.008s
W20260809 23:20:45.216899 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.008s	sys 0.008s
I20260809 23:20:44.904808 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2622) took 1688 ms. Trace:
I20260809 23:20:45.219291 24317 rpcz_store.cc:276] 0809 23:20:43.216687 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.850774 (+1634087us) service_pool.cc:227] Handling call
0809 23:20:44.904207 (+ 53433us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.904425 (+   218us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.904463 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.904512 (+    49us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:44.904558 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.904619 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.904740 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:45.184144 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 721858us
I20260809 23:20:45.221838 24317 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:45.223791 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.06796e+06us
I20260809 23:20:45.225337 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.229460 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1296) took 3828 ms. Trace:
I20260809 23:20:45.230062 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:45.231472 31984 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.871s	user 0.008s	sys 0.022s
I20260809 23:20:45.232822 24545 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:45.236321 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1143) took 2001 ms. Trace:
I20260809 23:20:45.198664 24316 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:45.200120 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3161) took 2321 ms. Trace:
W20260809 23:20:45.131987   875 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260809 23:20:45.188815 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 238168us
I20260809 23:20:45.239392 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3146) took 2536 ms. Trace:
I20260809 23:20:45.197050 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1432) took 1746 ms. Trace:
W20260809 23:20:45.240703 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3084) took 3037 ms (client timeout 2937 ms). Trace:
I20260809 23:20:45.240082 32354 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:45.090905 32761 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:44.951387   300 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.330226   809 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.971140   434 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:45.331418   434 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:45.331540   901 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260809 23:20:45.083195   509 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.007s
I20260809 23:20:45.242231   530 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:44.964216   531 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.069s
W20260809 23:20:45.335544 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 110569us
I20260809 23:20:45.335422   809 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.336828   531 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:45.337380   531 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.090358   587 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.030539   588 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.335551   530 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.341646   588 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.342229 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:44.920028   686 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.336091   846 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260809 23:20:45.335557   844 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.009s
I20260809 23:20:45.062743   691 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.344944   315 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:45.345721   686 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:44.920603   688 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260809 23:20:44.906899   726 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.347626   905 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.333779   509 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.350036   726 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.350371 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2618) took 2134 ms. Trace:
I20260809 23:20:44.920490   740 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:45.098881   864 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:20:45.351200 24317 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:45.351755 24317 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:45.098003   876 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260809 23:20:45.244096   884 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:45.350766 24311 rpcz_store.cc:276] 0809 23:20:43.215458 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.952055 (+736597us) service_pool.cc:227] Handling call
0809 23:20:43.952428 (+   373us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.956615 (+  4187us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:44.137200 (+180585us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b18004f5a60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:20:44.628610 (+491410us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.193613 (+565003us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:45.194921 (+  1308us) log.cc:842] Serialized 26 byte log entry
0809 23:20:45.240107 (+ 45186us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:45.240537 (+   430us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.240790 (+   253us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.240932 (+   142us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.349877 (+108945us) raft_consensus.cc:1671] finished
0809 23:20:45.349908 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.350126 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3427,"prepare.queue_time_us":802064,"prepare.run_cpu_time_us":167,"prepare.run_wall_time_us":8255,"raft.queue_time_us":51718,"raft.run_cpu_time_us":494,"raft.run_wall_time_us":26615,"spinlock_wait_cycles":77832448,"thread_start_us":963902,"threads_started":2,"wal-append.queue_time_us":689761}
I20260809 23:20:45.243824   758 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:45.353778 24317 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.110603 32752 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.990207   763 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.062s
I20260809 23:20:45.143551   760 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:45.355639 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2960) took 3857 ms. Trace:
I20260809 23:20:45.356065   867 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.356360   760 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.037328   699 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:45.358323   699 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:44.979888   326 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:44.906101   863 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.116s
W20260809 23:20:45.037781   861 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.190s	user 0.002s	sys 0.000s
I20260809 23:20:44.963989   877 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:45.104660   858 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.353s	user 0.000s	sys 0.002s
I20260809 23:20:44.994297   682 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.058s
I20260809 23:20:45.130678   918 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260809 23:20:45.107558   860 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260809 23:20:45.229920 24541 rpcz_store.cc:276] 0809 23:20:41.400888 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.898067 (+2497179us) service_pool.cc:227] Handling call
0809 23:20:44.296169 (+398102us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:44.296347 (+   178us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.980933 (+684586us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.228811 (+247878us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:45.229264 (+   453us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1704947200}
W20260809 23:20:45.120922   865 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
W20260809 23:20:45.140774   894 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:20:45.135859   711 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:45.137423   717 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:45.202493 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1138) took 2153 ms. Trace:
I20260809 23:20:45.191377 24309 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260809 23:20:45.209736 24546 rpcz_store.cc:276] 0809 23:20:41.586150 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.188744 (+2602594us) service_pool.cc:227] Handling call
0809 23:20:44.202081 (+ 13337us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.202322 (+   241us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:44.714728 (+512406us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.208102 (+493374us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:45.208315 (+   213us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:45.208435 (+   120us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.208582 (+   147us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.208835 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":762326,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"thread_start_us":899314,"threads_started":2}
I20260809 23:20:45.251883 24316 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.252296 24428 rpcz_store.cc:276] 0809 23:20:42.878967 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.087266 (+1208299us) service_pool.cc:227] Handling call
0809 23:20:44.087660 (+   394us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.109455 (+ 21795us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.109504 (+    49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:44.110059 (+   555us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.415102 (+305043us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b180033b7a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:20:44.792819 (+377717us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:44.792870 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.793554 (+   684us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.199265 (+405711us) raft_consensus.cc:1671] finished
0809 23:20:45.199320 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.199874 (+   554us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":20086,"spinlock_wait_cycles":43104000,"thread_start_us":377516,"threads_started":1,"wal-append.queue_time_us":967899}
I20260809 23:20:45.300122 24313 rpcz_store.cc:276] 0809 23:20:43.450239 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.866822 (+1416583us) service_pool.cc:227] Handling call
0809 23:20:45.195624 (+328802us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:45.196179 (+   555us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:45.196232 (+    53us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:45.196372 (+   140us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:45.196527 (+   155us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.196647 (+   120us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.196820 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:45.299639 24424 rpcz_store.cc:276] 0809 23:20:42.703155 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.008459 (+1305304us) service_pool.cc:227] Handling call
0809 23:20:44.008892 (+   433us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.979707 (+970815us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.238869 (+259162us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:45.239194 (+   325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2231268608}
W20260809 23:20:45.300489 24545 rpcz_store.cc:269] 0809 23:20:42.203674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.556473 (+352799us) service_pool.cc:227] Handling call
0809 23:20:45.240403 (+2683930us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:45.240582 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5832753408}
W20260809 23:20:45.318658 32417 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.005s	sys 0.000s
W20260809 23:20:45.319096 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 325661us
I20260809 23:20:45.320156 32354 consensus_queue.cc:1225] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:20:45.321311 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.323083   546 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.323454 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:45.323722   814 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:45.121706   808 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.015s	user 0.013s	sys 0.012s
I20260809 23:20:45.413165   808 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:45.414899 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:45.415906 25369 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:45.343827 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:45.356717   763 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:44.964486 32391 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.002s	sys 0.013s
W20260809 23:20:45.052285   854 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.407s	user 0.000s	sys 0.001s
I20260809 23:20:45.359615 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1412) took 4262 ms. Trace:
W20260809 23:20:45.463965 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1420) took 3249 ms (client timeout 2156 ms). Trace:
W20260809 23:20:45.464439 24321 rpcz_store.cc:269] 0809 23:20:42.204125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:42.870819 (+666694us) service_pool.cc:227] Handling call
0809 23:20:45.452945 (+2582126us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b5400385ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:45.453143 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5226648832}
I20260809 23:20:45.464229 24323 rpcz_store.cc:276] 0809 23:20:41.097141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.583048 (+2485907us) service_pool.cc:227] Handling call
0809 23:20:43.604652 (+ 21604us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.605131 (+   479us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:44.398258 (+793127us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.217930 (+819672us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:45.218768 (+   838us) log.cc:842] Serialized 26 byte log entry
0809 23:20:45.223018 (+  4250us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:45.223051 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.223144 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.359274 (+136130us) raft_consensus.cc:1671] finished
0809 23:20:45.359297 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.359552 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18017,"prepare.queue_time_us":1000201,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":84375,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":240,"thread_start_us":1499192,"threads_started":2,"wal-append.queue_time_us":695566}
I20260809 23:20:45.359110   707 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:45.467665 24545 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
W20260809 23:20:45.362988 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 277260us
I20260809 23:20:45.471352   706 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260809 23:20:45.472442 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:45.473407 24535 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:45.358768   760 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.401736 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2624) took 2170 ms. Trace:
I20260809 23:20:45.403080   562 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:45.428431   531 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.377189   740 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:45.476706 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3087) took 3081 ms. Trace:
I20260809 23:20:45.476322   546 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.477123 24534 rpcz_store.cc:276] 0809 23:20:42.395170 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.600998 (+2205828us) service_pool.cc:227] Handling call
0809 23:20:44.621241 (+ 20243us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.621510 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:45.208014 (+586504us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.398699 (+190685us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:45.432233 (+ 33534us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:45.459475 (+ 27242us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.476437 (+ 16962us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.476653 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":417689,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":71071,"thread_start_us":656780,"threads_started":2}
I20260809 23:20:45.477442   531 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:45.412062   684 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:45.480528 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99934.7us
I20260809 23:20:45.480286   684 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.389207   809 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.485001   597 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:45.253073 24431 rpcz_store.cc:276] 0809 23:20:43.235311 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.477599 (+1242288us) service_pool.cc:227] Handling call
0809 23:20:44.477995 (+   396us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.605631 (+127636us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.605675 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:44.606189 (+   514us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.828004 (+221815us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:44.828071 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.235242 (+407171us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:45.235524 (+   282us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.235647 (+   123us) raft_consensus.cc:1671] finished
0809 23:20:45.235671 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.236133 (+   462us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121,"raft.queue_time_us":944,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7828,"spinlock_wait_cycles":292180224,"thread_start_us":199146,"threads_started":1,"wal-append.queue_time_us":483219}
I20260809 23:20:45.419778 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.379690 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.355594   758 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:45.124050   526 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:45.398118 32417 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.007s	sys 0.000s
I20260809 23:20:45.488242   814 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:45.443049 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 108089us
I20260809 23:20:45.447945 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:45.490481 24541 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:45.475224 24309 rpcz_store.cc:276] 0809 23:20:43.222899 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.865953 (+1643054us) service_pool.cc:227] Handling call
0809 23:20:45.190289 (+324336us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:45.392871 (+202582us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.393111 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:45.491425   867 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.449374 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.422742   763 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.375581 24427 rpcz_store.cc:276] 0809 23:20:43.048594 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.342767 (+1294173us) service_pool.cc:227] Handling call
0809 23:20:44.381634 (+ 38867us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:44.382079 (+   445us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.382120 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:44.382839 (+   719us) log.cc:842] Serialized 52 byte log entry
0809 23:20:44.779188 (+396349us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:45.200681 (+421493us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.201565 (+   884us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.201774 (+   209us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.201946 (+   172us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.202003 (+    57us) raft_consensus.cc:1671] finished
0809 23:20:45.202015 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.202227 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":738419,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":94,"raft.queue_time_us":90501,"raft.run_cpu_time_us":598,"raft.run_wall_time_us":4151,"thread_start_us":740819,"threads_started":2,"wal-append.queue_time_us":693119}
I20260809 23:20:45.451957 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.360599   844 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:45.342289   530 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.465955 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3144) took 2796 ms. Trace:
I20260809 23:20:45.466250   509 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.475498 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:45.475320   760 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:45.405315 24540 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:45.356318 24539 rpcz_store.cc:276] 0809 23:20:41.498252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.063898 (+2565646us) service_pool.cc:227] Handling call
0809 23:20:44.064237 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.064584 (+   347us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:44.064624 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:44.065095 (+   471us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.754996 (+689901us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:44.755036 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.755101 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.355051 (+599950us) raft_consensus.cc:1671] finished
0809 23:20:45.355095 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.355423 (+   328us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":91923,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":689580,"threads_started":1,"wal-append.queue_time_us":868991}
I20260809 23:20:45.485368   918 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:45.487581   758 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.538372 24317 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:45.451478 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:45.499466 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:45.494129 25371 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:45.452965   300 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.475870   562 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:45.498980   326 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260809 23:20:45.532967   814 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:45.554162 32391 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.099s	user 0.002s	sys 0.014s
I20260809 23:20:45.359148   699 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:45.544636 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 140102us
I20260809 23:20:45.497808   684 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.548135   562 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.548806   758 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:44.773119 24314 rpcz_store.cc:276] 0809 23:20:42.454374 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.907113 (+1452739us) service_pool.cc:227] Handling call
0809 23:20:43.907422 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:43.907654 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:44.394392 (+486738us) log.cc:842] Serialized 14 byte log entry
0809 23:20:44.757426 (+363034us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:44.757492 (+    66us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:44.757561 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.757639 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:44.772673 (+ 15034us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":997,"prepare.queue_time_us":784330,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":849024,"threads_started":2}
I20260809 23:20:45.550038   706 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:45.551195   492 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
W20260809 23:20:45.552268   849 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.988s	user 0.013s	sys 0.012s
W20260809 23:20:45.552541 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 213959us
I20260809 23:20:45.552711   874 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.553602   711 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.519296   844 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.488029   597 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.056s
I20260809 23:20:45.523499 24430 rpcz_store.cc:276] 0809 23:20:42.669629 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.768590 (+1098961us) service_pool.cc:227] Handling call
0809 23:20:43.768973 (+   383us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.804437 (+ 35464us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.804475 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:43.804901 (+   426us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.397502 (+592601us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:44.397555 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.397645 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.465530 (+1067885us) raft_consensus.cc:1671] finished
0809 23:20:45.465575 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.465784 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18103,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":591273,"threads_started":1,"wal-append.queue_time_us":793301}
I20260809 23:20:45.562878 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2617) took 2340 ms. Trace:
I20260809 23:20:45.494040   867 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.563898 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1277) took 1128 ms. Trace:
W20260809 23:20:45.568132 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75880.8us
W20260809 23:20:45.569108   700 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.831s	user 0.014s	sys 0.018s
I20260809 23:20:45.574223   884 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:45.634562 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:45.624825 32417 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260809 23:20:45.621598   849 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:45.637892   562 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.575130 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3066) took 4120 ms. Trace:
I20260809 23:20:45.638633 24312 rpcz_store.cc:276] 0809 23:20:41.455762 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.825011 (+2369249us) service_pool.cc:227] Handling call
0809 23:20:43.825373 (+   362us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:43.825590 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.825616 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:43.826060 (+   444us) log.cc:842] Serialized 26 byte log entry
0809 23:20:44.396618 (+570558us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:44.396673 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.396762 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.518466 (+1121704us) raft_consensus.cc:1671] finished
0809 23:20:45.525678 (+  7212us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.574947 (+ 49269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79471,"thread_start_us":570156,"threads_started":1,"wal-append.queue_time_us":785816}
I20260809 23:20:45.639715 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:45.640599 24431 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:20:45.575749 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 232213us
I20260809 23:20:45.585951   492 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:45.641686 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:45.560619 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70685.9us
I20260809 23:20:45.642454   492 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.599231 24320 rpcz_store.cc:276] 0809 23:20:43.215107 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.933855 (+718748us) service_pool.cc:227] Handling call
0809 23:20:43.941353 (+  7498us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:43.942519 (+  1166us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:43.942550 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:43.943148 (+   598us) log.cc:842] Serialized 52 byte log entry
0809 23:20:44.420126 (+476978us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:44.844257 (+424131us) log.cc:842] Serialized 14 byte log entry
0809 23:20:44.871205 (+ 26948us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:44.871320 (+   115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.244407 (+373087us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:20:45.514651 (+270244us) raft_consensus.cc:1671] finished
0809 23:20:45.533988 (+ 19337us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.555121 (+ 21133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45027,"prepare.queue_time_us":351628,"prepare.run_cpu_time_us":8349,"prepare.run_wall_time_us":23997,"raft.queue_time_us":71479,"raft.run_cpu_time_us":5192,"raft.run_wall_time_us":330248,"spinlock_wait_cycles":23160064,"thread_start_us":852564,"threads_started":2,"wal-append.queue_time_us":750100}
I20260809 23:20:45.615831   929 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:45.644827 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:45.644593   929 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.565997 24316 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.577550   526 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:45.646991 24316 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:45.647675   925 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:45.647738   526 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:45.648828   945 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:45.574906   699 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:45.586382   706 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.651230   901 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:45.652738   901 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:45.652583   706 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.654239   901 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:45.604355   905 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:45.655406 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1419) took 1291 ms. Trace:
I20260809 23:20:45.655593   905 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:45.656104   748 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:45.605266 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:45.656838   748 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:45.655447   686 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:45.659723   686 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:45.660483   682 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.660984   686 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.661304   682 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.608002   707 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:45.615525   624 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.553905   763 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.669307   763 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:45.618161 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.618940   874 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.595353   300 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:45.621181 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:45.610670   502 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.638s	user 0.015s	sys 0.009s
I20260809 23:20:45.635509 24427 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:45.640735 24424 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:20:45.627010 24313 rpcz_store.cc:276] 0809 23:20:44.457470 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.440964 (+983494us) service_pool.cc:227] Handling call
0809 23:20:45.563786 (+122822us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:45.656165   905 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.658150   748 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.609370   844 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.563827   926 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:45.605911 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 263899us
I20260809 23:20:45.638568 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:45.617676   497 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.174s	user 0.011s	sys 0.007s
I20260809 23:20:45.667335   867 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.666893   624 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.635675   884 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:45.667084   707 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.668319   758 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:45.665267 24535 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:45.706171 32354 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:45.707072   874 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.706400   926 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:45.708426   917 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260809 23:20:45.695073   502 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.710024   926 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.712920   502 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.715075   700 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:45.715615 25373 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:45.715749   700 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.683490   300 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.718086   700 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.719403   300 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.720687   929 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.722374   711 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:45.723212 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1153) took 1124 ms. Trace:
I20260809 23:20:45.723565 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:45.725085   867 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.620442   684 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.076s
I20260809 23:20:45.723589 24423 rpcz_store.cc:276] 0809 23:20:44.598639 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.827658 (+229019us) service_pool.cc:227] Handling call
0809 23:20:44.948058 (+120400us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.948339 (+   281us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:45.350125 (+401786us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.626215 (+276090us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:45.664061 (+ 37846us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:45.679835 (+ 15774us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.684585 (+  4750us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.723036 (+ 38451us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64338,"prepare.queue_time_us":411153,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":83955,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":147,"thread_start_us":533213,"threads_started":2,"wal-append.queue_time_us":272908}
I20260809 23:20:45.703925   884 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.731915 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:45.671584 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45716.1us
I20260809 23:20:45.683427 24541 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:45.681078 24435 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:45.646780 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:45.662823   686 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:45.638818   853 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.871s	user 0.002s	sys 0.002s
I20260809 23:20:45.670950   706 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:45.687388 24430 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:45.664171   901 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:45.690267 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1440) took 1230 ms. Trace:
I20260809 23:20:45.655849 25365 rpcz_store.cc:276] 0809 23:20:44.363775 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.378344 (+ 14569us) service_pool.cc:227] Handling call
0809 23:20:44.378681 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:44.379120 (+   439us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:44.894665 (+515545us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.594969 (+700304us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:45.619613 (+ 24644us) log.cc:842] Serialized 26 byte log entry
0809 23:20:45.619803 (+   190us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:45.620120 (+   317us) log.cc:842] Serialized 14 byte log entry
0809 23:20:45.620325 (+   205us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.620437 (+   112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.655009 (+ 34572us) raft_consensus.cc:1671] finished
0809 23:20:45.655031 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.655185 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81623,"prepare.queue_time_us":697549,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":123554,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":20000,"thread_start_us":933352,"threads_started":2,"wal-append.queue_time_us":667790}
I20260809 23:20:45.744169 24317 rpcz_store.cc:276] 0809 23:20:44.459969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.302586 (+842617us) service_pool.cc:227] Handling call
0809 23:20:45.350916 (+ 48330us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:45.689598 (+338682us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:45.689642 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:45.689687 (+    45us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:45.689730 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.689790 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.689925 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3065,"raft.queue_time_us":28926,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152}
W20260809 23:20:45.687178 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.815s (SENT)
I20260809 23:20:45.742790 25371 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:20:45.673056 24316 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:20:45.747195 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1433) took 2294 ms. Trace:
I20260809 23:20:45.747787 24316 rpcz_store.cc:276] 0809 23:20:43.453287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.894831 (+1441544us) service_pool.cc:227] Handling call
0809 23:20:45.198254 (+303423us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:45.565415 (+367161us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b54001b3b20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f6951 00007fac23857321 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612
0809 23:20:45.746811 (+181396us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.747124 (+   313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":73351552}
W20260809 23:20:45.747462 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84068.5us
I20260809 23:20:45.731854   884 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:45.706256 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88293.9us
I20260809 23:20:45.750784 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:45.752554 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:45.756395 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.760035   684 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.760895   684 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.764163   684 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:45.744498 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 123680us
I20260809 23:20:45.754678   884 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:45.746281 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.797708   929 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:45.740381   497 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.823889   905 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.827138   867 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:45.847134 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 120696us
I20260809 23:20:45.835243 24429 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:45.835457 32640 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:45.835954   565 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.044s	user 0.013s	sys 0.010s
I20260809 23:20:45.836268   497 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.837141   615 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:45.837209   624 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:45.831408 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 184910us
I20260809 23:20:45.791178   846 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:45.831382   844 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:45.850656 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 104600us
W20260809 23:20:45.852640 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 120800us
I20260809 23:20:45.865787 24427 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:20:45.869210 32741 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.872355   964 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:45.872987   964 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:45.872575 32741 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.873543   964 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:45.875234   494 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:20:45.875864 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119833us
I20260809 23:20:45.878115   969 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:45.878904 32640 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.879727 32640 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:45.880867 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 205121us
I20260809 23:20:45.882644   682 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.882647   502 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:45.884598   526 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:45.884856   502 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.885742 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3095) took 1405 ms. Trace:
I20260809 23:20:45.886147 24311 rpcz_store.cc:276] 0809 23:20:44.479974 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.354709 (+874735us) service_pool.cc:227] Handling call
0809 23:20:45.418787 (+ 64078us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:45.500468 (+ 81681us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:45.500513 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:45.541700 (+ 41187us) log.cc:842] Serialized 26 byte log entry
0809 23:20:45.773879 (+232179us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:45.774728 (+   849us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.774836 (+   108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:45.885511 (+110675us) raft_consensus.cc:1671] finished
0809 23:20:45.885534 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:45.885680 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48474,"raft.queue_time_us":42813,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":37776,"thread_start_us":148272,"threads_started":1,"wal-append.queue_time_us":305591}
I20260809 23:20:45.888593 32741 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:45.890307   502 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.891106 32640 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:45.891278   901 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.894448   326 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.894899   562 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:45.895195   326 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.895699   562 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.860078   929 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:45.890565   905 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.901430   326 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:45.902738   929 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:45.904770 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 266733us
I20260809 23:20:45.905551 24546 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:45.864244   615 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:45.871289   809 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:45.926281 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:45.925895 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.846s (SENT)
I20260809 23:20:45.907797   758 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:45.935721 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.936492 25373 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:45.941951   502 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:45.942509   502 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:45.942909   502 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:45.943822 24541 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:20:45.927964   809 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:45.950134 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.925406   969 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
I20260809 23:20:45.952783 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:45.954015 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.960062 25366 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:45.949570   809 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:45.966313 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1145) took 2503 ms. Trace:
I20260809 23:20:45.966692 24429 rpcz_store.cc:276] 0809 23:20:43.463266 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.485936 (+1022670us) service_pool.cc:227] Handling call
0809 23:20:44.486318 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:45.866614 (+1380296us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.965997 (+ 99383us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:45.966246 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3101055744}
I20260809 23:20:45.967869 24541 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260809 23:20:45.967464   615 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:45.968765   615 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.969815   971 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:45.970410   971 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:45.972898   971 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:45.975275 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.976063 24543 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:45.967794   846 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:45.983986 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:45.987246 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.988588 24545 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260809 23:20:45.989291 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:45.990060 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:45.991209 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.992249 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:45.993029 25375 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260809 23:20:45.993505   492 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:45.993818 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:45.959620   901 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:45.990010 24545 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:45.944103 32354 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260809 23:20:45.991066 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.033716 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:45.990450 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.035337 25372 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 14.
I20260809 23:20:45.981607   846 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:46.037264 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.035699 24544 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.038599 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.036432 24429 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 16. (index mismatch)
W20260809 23:20:46.040690 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55529.1us
I20260809 23:20:46.042376 24430 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:46.040382 24543 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:45.977317   315 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:46.041842   969 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:46.044715 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52670.1us
I20260809 23:20:46.049437 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:46.051903 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69939us
I20260809 23:20:46.054153 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:46.062810 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:46.067631 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 147338us
I20260809 23:20:46.073400 25369 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:46.098078 25373 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:46.158267 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:46.161960 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:46.161728 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.164580 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.045838   315 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:46.168766 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.167716 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.151921   494 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:46.189126   300 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:46.189692   300 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:46.190910 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.159905   562 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.193755   300 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:46.197224 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.218797 24423 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:46.221666 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.226348 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.228171 24545 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.165951 32741 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260809 23:20:46.223754 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.180084   509 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.233175 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.233644 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.210678 24544 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:46.188283   494 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:46.241937 24431 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:46.247457 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:46.250597 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 330403us
I20260809 23:20:46.246914 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.253374   984 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.093s
I20260809 23:20:46.254213   984 consensus_queue.cc:1225] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:46.255081   434 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.255403   509 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.256489   434 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:46.257602   509 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.259934 25375 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 15. Preceding OpId from leader: term: 16 index: 16. (index mismatch)
I20260809 23:20:46.261341 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.264434 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.223202   969 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.202268   706 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.044s
I20260809 23:20:46.259086 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.308746 24423 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.303665 24543 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:46.242120   494 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:46.237696 32640 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:46.310405 25372 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:20:46.312045 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 117157us
I20260809 23:20:46.313859 24541 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:46.307746 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68622us
I20260809 23:20:46.312120 24422 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:46.317303 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 153019us
I20260809 23:20:46.313843 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.309237 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.259402   686 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.217881   964 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:46.316177   976 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260809 23:20:46.313385 32741 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.341305 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:46.343793 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.321799   684 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:46.355301 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.354568 25375 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260809 23:20:46.349383 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.351063   970 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:46.354563   492 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:46.357713   688 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.399825 24425 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260809 23:20:46.368803 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.234s (SENT)
I20260809 23:20:46.334856   686 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.408912   686 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:46.348848 32741 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.412251 32741 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.417886 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.365988   509 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.423266 24317 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260809 23:20:46.378324   684 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:46.350188   969 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.389149   970 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:46.442113 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:46.450497   884 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.626s	user 0.015s	sys 0.000s
I20260809 23:20:46.451941   964 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.456215 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.456794 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.457541 24323 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 12.
I20260809 23:20:46.437415 31984 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.375638   809 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.377596 32640 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:46.474682 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 149062us
I20260809 23:20:46.473455 25366 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:46.481339 24313 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:46.438400   492 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:46.483681 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:46.439909   526 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.508426   526 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.509500 24542 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:46.509905 24542 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:46.458868 31984 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.515870  1001 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:46.516382  1001 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:46.516017   684 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.516718  1001 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:46.355727   315 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:46.489835 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74368us
I20260809 23:20:46.520246 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.521908 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.523116   682 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:46.522981   565 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:20:46.523635   682 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:46.524889 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:46.497252   981 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260809 23:20:46.525925   846 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.498220 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.530483 24430 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:46.530357 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.498523   809 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.532323 24428 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:46.533937   526 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.489688   999 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
W20260809 23:20:46.501780   905 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.583s	user 0.010s	sys 0.003s
I20260809 23:20:46.501986 32640 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:46.536626 24309 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:46.537470   846 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.541330 31984 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.541507   546 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:46.545493 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1448) took 4348 ms. Trace:
I20260809 23:20:46.549726 25370 rpcz_store.cc:276] 0809 23:20:42.197202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:43.901545 (+1704343us) service_pool.cc:227] Handling call
0809 23:20:43.901830 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:44.930011 (+1028181us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.238197 (+308186us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b540065d220. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:46.543252 (+1305055us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b540065d220. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5cbd 00007fac23863b97 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:46.545009 (+  1757us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:46.545277 (+   268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2645660672}
I20260809 23:20:46.507296   492 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.555811   969 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.482633   530 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.556813   530 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.486696   706 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.505136 24316 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
W20260809 23:20:46.496955 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 151740us
I20260809 23:20:46.559398   706 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.561126 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:46.561738   706 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:46.563266 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67112.8us
W20260809 23:20:46.499289   983 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260809 23:20:46.535229 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.490430   884 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:46.566087 24317 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:46.566280   884 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.568207 31984 raft_consensus.cc:2741] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:46.568425   884 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:46.546478   844 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.676s	user 0.009s	sys 0.002s
I20260809 23:20:46.547230   562 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:46.570191   562 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:46.524614   682 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.571264   562 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.572926 25363 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260809 23:20:46.510856 24542 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.444082   970 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:46.549068 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:46.507624 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:46.565403 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:46.553659   686 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:20:46.576145 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.575930 24321 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260809 23:20:46.600137 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:46.602165 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53213us
I20260809 23:20:46.574859 24542 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:46.607213 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 227034us
I20260809 23:20:46.422519   688 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:46.630905 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 135060us
I20260809 23:20:46.631810 25371 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:46.559422 24316 pending_rounds.cc:76] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 7
I20260809 23:20:46.437933   509 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.633976 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.635023 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:46.624396 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.018s	sys 0.021s
W20260809 23:20:46.636930 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.018s	sys 0.021s
I20260809 23:20:46.637671 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:46.593977 24317 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:46.633132 24316 pending_rounds.cc:97] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 12.8
W20260809 23:20:46.633569 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68252.9us
I20260809 23:20:46.644484 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:46.617055   615 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.002s
I20260809 23:20:46.647585 24544 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.650441 24316 raft_consensus.cc:2866] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:46.654429 25375 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260809 23:20:46.655184 25375 pending_rounds.cc:76] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 11
I20260809 23:20:46.655465 25375 pending_rounds.cc:97] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260809 23:20:46.558882   530 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.655618 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.625104   984 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.632613   884 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:46.655786 25375 raft_consensus.cc:2866] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:46.673063 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:46.672034 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.650471   526 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:46.681316 24544 raft_consensus.cc:2354] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
W20260809 23:20:46.685076   993 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.202s	user 0.000s	sys 0.000s
I20260809 23:20:46.685449 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:46.685902 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:46.687309 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.687631   509 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.625327   300 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:46.686187   943 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.813s	user 0.018s	sys 0.026s
I20260809 23:20:46.680716 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.693076 24321 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:46.692929 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.631325   546 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.040s
I20260809 23:20:46.684010   526 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:46.653882   763 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260809 23:20:46.705075   526 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:46.705497   763 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:46.706028   763 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:46.706159 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1460) took 1295 ms. Trace:
I20260809 23:20:46.707136   763 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.708745   763 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:46.715027 32761 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.115s
I20260809 23:20:46.716665   300 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.717532 25369 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:46.676151   530 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.722947  1001 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.723392 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.724682 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.725423 24313 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:46.725135 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.726140 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.726438 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.727767 24323 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:46.691543   531 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:46.730208   531 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:46.731464   531 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.674919   849 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260809 23:20:46.735605 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:46.718802   530 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.736407 24428 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:20:46.664422   884 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.706588 25368 rpcz_store.cc:276] 0809 23:20:45.410569 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.417240 (+  6671us) service_pool.cc:227] Handling call
0809 23:20:45.417808 (+   568us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:45.418625 (+   817us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:45.418661 (+    36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:45.419527 (+   866us) log.cc:842] Serialized 52 byte log entry
0809 23:20:45.530523 (+110996us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:45.530565 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:45.530635 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:46.522473 (+991838us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:46.705848 (+183375us) raft_consensus.cc:1671] finished
0809 23:20:46.705868 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:46.706097 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24290,"raft.queue_time_us":35266,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":103557,"spinlock_wait_cycles":40292096,"thread_start_us":110510,"threads_started":1,"wal-append.queue_time_us":324448}
I20260809 23:20:46.745628 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:46.747534 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.749683 24430 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:46.749857 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.749857 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.746582 24535 pending_rounds.cc:76] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 9
I20260809 23:20:46.751827 24535 pending_rounds.cc:97] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260809 23:20:46.752182 24535 raft_consensus.cc:2866] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:46.726742 24313 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:46.690194 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.690086 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:46.755000 25370 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:46.685493   502 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.742s	user 0.010s	sys 0.001s
I20260809 23:20:46.692005 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.757223 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.754506 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.753813 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:46.758627 24321 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260809 23:20:46.752494 25371 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:46.760171 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.743978   884 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:46.762878 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:46.767704 32752 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:46.768605 32752 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:46.769045 32752 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:46.770550 32761 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:46.743769   686 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:46.744664   497 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260809 23:20:46.771210 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.771476 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:46.771592 32380 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260809 23:20:46.760389   502 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.772924 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.797118 24321 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:46.773042   748 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:46.691645   984 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.775442 24425 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:46.800246 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.776479 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.725113  1001 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.786700 24316 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260809 23:20:46.788717 25371 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:46.760174 32417 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.801090 24424 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:46.803010   688 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.803085   530 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.803742 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.805696   502 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:46.807156 25373 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 17. Ops: []
I20260809 23:20:46.747038   494 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.813167 32640 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:46.815532 24313 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:46.816545   901 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.774s	user 0.010s	sys 0.001s
I20260809 23:20:46.784499   686 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:46.822760 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:46.872305 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49648.6us
I20260809 23:20:46.872611 32380 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:20:46.760236   849 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260809 23:20:46.875957   849 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:46.827122   688 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:46.877756 24423 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260809 23:20:46.877914   688 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:46.827625 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.881544   970 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.882972 24320 raft_consensus.cc:2354] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18.
I20260809 23:20:46.834005 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.883965   686 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:46.883975 24430 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.888375 24312 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:46.834713 32417 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.889729 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.836422 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:46.837424 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:46.839701   526 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:46.928689 24312 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15. Re-sending same reply.
I20260809 23:20:46.944869 25370 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:46.840951 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.946749 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.948119 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.836553 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.854733 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.861171 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.953403 24309 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:46.954066 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:46.954756 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.955381 25371 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:46.956261 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.956820 25371 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260809 23:20:46.957842 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.887089   884 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:46.855975   991 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.330s	user 0.000s	sys 0.004s
I20260809 23:20:46.892066   688 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.835533 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:46.958503 25371 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:20:46.961273 24428 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:46.898319 24266 leader_election.cc:398] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:20:46.962610 25371 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:46.911231 24427 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:46.857765   530 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:46.964304 25371 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:20:46.861903 32640 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:46.967195 32640 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:46.912838 24317 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:46.971817 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.973194 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:46.974030 24317 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:46.896167   686 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:46.976784   686 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.919683 24321 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:46.978278 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.978974 24321 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:46.982106 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66685.7us
I20260809 23:20:46.983429 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:46.944165 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:46.960741   884 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:46.893836   970 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:46.962713 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64590.1us
I20260809 23:20:46.998137 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:46.993621 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:46.993063 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:46.994680 24435 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260809 23:20:46.974870 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.006618 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:47.007378 24422 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 18
I20260809 23:20:47.010001 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.000842 24435 pending_rounds.cc:76] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:20:47.011571 24435 pending_rounds.cc:97] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260809 23:20:47.012295 24435 raft_consensus.cc:2866] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:46.958562 24309 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:47.016693 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.025213 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:47.026250 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.033s	sys 0.012s
I20260809 23:20:47.026026 24312 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:20:47.033991 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.034s	sys 0.012s
I20260809 23:20:47.038791 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:47.076314 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:47.078243 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 594592us
I20260809 23:20:47.072181 24313 raft_consensus.cc:2362] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16. Re-sending same reply.
I20260809 23:20:47.005970   849 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:47.002519   901 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:46.995903   497 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:47.079231 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.089298 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.083043 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:47.100381 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 106868us
I20260809 23:20:47.102810 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.103075 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.105100 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:47.102110 24321 raft_consensus.cc:2362] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14. Re-sending same reply.
I20260809 23:20:47.117707 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.073616   492 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:47.125379 24313 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:47.088130   999 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260809 23:20:47.086671   809 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:47.188932 24316 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:47.189989 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.190685 24313 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:47.099009 32741 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.193410 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.193699 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:47.124325 31984 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.013s	sys 0.018s
I20260809 23:20:47.082465   970 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260809 23:20:47.185613 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 188051us
W20260809 23:20:47.186250 24545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.003s	sys 0.030s
W20260809 23:20:47.202293 24545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.004s	sys 0.031s
I20260809 23:20:47.127087   884 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:47.197054 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:47.180555 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47562.9us
I20260809 23:20:47.197981 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.199057 24541 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260809 23:20:47.183888   706 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.203240 24309 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:47.195604 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.125700   492 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:47.104291   849 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:47.206622 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.204025 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2997) took 1139 ms. Trace:
I20260809 23:20:47.209499 24312 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260809 23:20:47.213944 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:47.199826 31984 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.013s	sys 0.018s
I20260809 23:20:47.210076 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.211455 32417 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.201121 32741 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.186400 32380 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.026s
I20260809 23:20:47.184156   844 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260809 23:20:47.191571   969 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260809 23:20:47.179729   497 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.106063   901 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.213233 24545 rpcz_store.cc:276] 0809 23:20:46.064587 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.241731 (+177144us) service_pool.cc:227] Handling call
0809 23:20:46.315787 (+ 74056us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:46.324955 (+  9168us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:46.395025 (+ 70070us) log.cc:842] Serialized 14 byte log entry
0809 23:20:47.202650 (+807625us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:47.203754 (+  1104us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:47.203803 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.203865 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.203973 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":153223,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":175386,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":827355,"threads_started":3,"wal-append.queue_time_us":477554}
I20260809 23:20:47.218091   492 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:47.232844 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.233280 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:47.242398 24540 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:47.221442   970 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:47.244412 24309 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:47.220636   494 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:47.250051 24313 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:47.229640 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.016s	sys 0.016s
W20260809 23:20:47.253629 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.016s	sys 0.016s
W20260809 23:20:47.254052 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 568728us
W20260809 23:20:47.235189   562 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.005s	sys 0.016s
W20260809 23:20:47.243376 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.009s	sys 0.014s
W20260809 23:20:47.258275 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.009s	sys 0.015s
W20260809 23:20:47.258723 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 623764us
W20260809 23:20:47.255805 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.007s	sys 0.027s
W20260809 23:20:47.259670 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.007s	sys 0.027s
I20260809 23:20:47.264825 24317 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:47.212440   706 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.243839   763 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:47.269454 24309 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260809 23:20:47.269668 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67195.2us
I20260809 23:20:47.271344 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1163) took 1104 ms. Trace:
W20260809 23:20:47.255975   546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.021s	sys 0.020s
I20260809 23:20:47.248382   877 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.255076 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.008s	sys 0.016s
I20260809 23:20:47.290588 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:47.290836 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.008s	sys 0.016s
W20260809 23:20:47.257898   562 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.006s	sys 0.016s
W20260809 23:20:47.267071   682 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.012s	sys 0.013s
I20260809 23:20:47.279661 24429 rpcz_store.cc:276] 0809 23:20:46.167391 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.167609 (+   218us) service_pool.cc:227] Handling call
0809 23:20:46.202127 (+ 34518us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:46.258375 (+ 56248us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:46.258770 (+   395us) log.cc:842] Serialized 14 byte log entry
0809 23:20:47.081597 (+822827us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:47.081723 (+   126us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:47.081810 (+    87us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.271126 (+189316us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:47.271174 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.271288 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36936,"prepare.queue_time_us":681,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":52626,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":10513,"spinlock_wait_cycles":128669440,"thread_start_us":775260,"threads_started":1}
I20260809 23:20:47.286538   849 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.289529   901 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:47.293116 25363 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:47.296276 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1469) took 1092 ms. Trace:
I20260809 23:20:47.265672   494 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:47.296679 25369 rpcz_store.cc:276] 0809 23:20:46.200309 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.200474 (+   165us) service_pool.cc:227] Handling call
0809 23:20:47.292024 (+1091550us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:47.292172 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1127514496}
I20260809 23:20:47.297837 24316 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260809 23:20:47.298755 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.266121   526 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:47.335887 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.009s	sys 0.029s
I20260809 23:20:47.335970 24435 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 18 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260809 23:20:47.336344 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.009s	sys 0.029s
I20260809 23:20:47.336542 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:47.282150   546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.022s	sys 0.020s
I20260809 23:20:47.307736 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.305478 24423 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:47.293272 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.335098 24422 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:47.297259 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.343007 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:47.350764 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:47.296026   682 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.013s	sys 0.013s
I20260809 23:20:47.351223 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.357272 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:47.318553 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:47.306360  1021 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260809 23:20:47.303965 32752 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.534s	user 0.010s	sys 0.002s
I20260809 23:20:47.361850 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:47.301090 32417 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
W20260809 23:20:47.366605 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48175.7us
I20260809 23:20:47.367594 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:47.368646   717 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:47.368738 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:47.370270 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:47.371317  1015 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:47.372030 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:47.372217  1015 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:47.373404   502 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:47.374073   502 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:47.358373   531 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.016s	sys 0.020s
I20260809 23:20:47.374475   502 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:47.374665   531 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.016s	sys 0.020s
I20260809 23:20:47.308456   844 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:47.374888   502 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:47.374300 32417 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:47.302480 32741 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.377113 24430 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
W20260809 23:20:47.380419 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 83221.7us
I20260809 23:20:47.384316 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.386150 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.339495   562 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.389082  1001 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.002s	sys 0.017s
I20260809 23:20:47.313746 32640 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.377110   969 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.400400 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.011s	sys 0.011s
W20260809 23:20:47.393780  1001 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.003s	sys 0.018s
I20260809 23:20:47.296983   877 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:47.398957 32761 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.010s	sys 0.011s
I20260809 23:20:47.391762   844 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.393751   970 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:47.394064 32741 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
W20260809 23:20:47.414335   525 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260809 23:20:47.398751   562 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.416204   315 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:47.421079 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:47.459851   970 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:47.464089   970 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:47.464939 25373 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260809 23:20:47.419591 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.013s	sys 0.012s
W20260809 23:20:47.410256   684 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.023s	sys 0.042s
W20260809 23:20:47.424839 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.012s	sys 0.011s
W20260809 23:20:47.426066 24535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.013s	sys 0.028s
I20260809 23:20:47.377525 32752 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.409325   984 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:47.423547 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1322) took 2641 ms. Trace:
I20260809 23:20:47.414842 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3103) took 1049 ms. Trace:
W20260809 23:20:47.470041 24535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.013s	sys 0.028s
W20260809 23:20:47.425967 32761 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.010s	sys 0.011s
I20260809 23:20:47.473487 24320 rpcz_store.cc:276] 0809 23:20:46.365519 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.380867 (+ 15348us) service_pool.cc:227] Handling call
0809 23:20:47.414585 (+1033718us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:47.414759 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1067129216}
I20260809 23:20:47.476928 24320 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260809 23:20:47.479367  1002 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.966s	user 0.003s	sys 0.002s
I20260809 23:20:47.458918 32741 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:47.479880 32640 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:47.480927 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1333) took 1452 ms. Trace:
I20260809 23:20:47.480635 32741 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.481256 24546 rpcz_store.cc:276] 0809 23:20:46.028109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.035997 (+  7888us) service_pool.cc:227] Handling call
0809 23:20:46.147186 (+111189us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.242198 (+ 95012us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.242228 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.314893 (+ 72665us) log.cc:842] Serialized 26 byte log entry
0809 23:20:46.437351 (+122458us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:46.437403 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:46.437473 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:47.460114 (+1022641us) spinlock_profiling.cc:243] Waited 85.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:47.480616 (+ 20502us) raft_consensus.cc:1671] finished
0809 23:20:47.480641 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.480857 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31001,"raft.queue_time_us":49532,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":3504,"spinlock_wait_cycles":196892544,"thread_start_us":100052,"threads_started":1,"wal-append.queue_time_us":191958}
I20260809 23:20:47.483043   300 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:47.483596   300 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.484640   809 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:47.485816   300 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:47.487038   998 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.989s	user 0.003s	sys 0.001s
I20260809 23:20:47.483614 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:47.488157 24422 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:47.488489 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1332) took 1492 ms. Trace:
I20260809 23:20:47.459465   969 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.488838 24534 rpcz_store.cc:276] 0809 23:20:45.995705 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.995935 (+   230us) service_pool.cc:227] Handling call
0809 23:20:46.038860 (+ 42925us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.070090 (+ 31230us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.070135 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.167385 (+ 97250us) log.cc:842] Serialized 26 byte log entry
0809 23:20:46.373656 (+206271us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:46.373713 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:46.373807 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:47.399752 (+1025945us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:47.488142 (+ 88390us) raft_consensus.cc:1671] finished
0809 23:20:47.488168 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.488431 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17726,"raft.queue_time_us":6465,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":16479,"spinlock_wait_cycles":232794496,"thread_start_us":142775,"threads_started":1,"wal-append.queue_time_us":235376}
I20260809 23:20:47.490684   969 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.492139   877 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.462153   844 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.420728   546 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.461478 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.004s	sys 0.036s
W20260809 23:20:47.494095 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.004s	sys 0.037s
I20260809 23:20:47.494125   546 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.495195   844 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.496639   546 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:47.498023 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1166) took 1277 ms. Trace:
I20260809 23:20:47.498411 24427 rpcz_store.cc:276] 0809 23:20:46.220684 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.223431 (+  2747us) service_pool.cc:227] Handling call
0809 23:20:47.497790 (+1274359us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b540017f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:47.497941 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1520504704}
I20260809 23:20:47.499373 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1434) took 1054 ms. Trace:
I20260809 23:20:47.499807 25375 rpcz_store.cc:276] 0809 23:20:46.445017 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.445178 (+   161us) service_pool.cc:227] Handling call
0809 23:20:46.451100 (+  5922us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:46.671923 (+220823us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.499131 (+827208us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540071dc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:47.499311 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463934848}
W20260809 23:20:47.501197 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.024s	sys 0.041s
W20260809 23:20:47.501528 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.024s	sys 0.041s
W20260809 23:20:47.502822   996 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.005s	user 0.006s	sys 0.000s
W20260809 23:20:47.461931  1018 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260809 23:20:47.503301  1024 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:47.503733 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:47.504034  1024 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:47.504403  1024 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:47.504438 32640 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260809 23:20:47.488782  1014 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260809 23:20:47.504878  1024 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:47.468912 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.014s	sys 0.012s
I20260809 23:20:47.499926 24431 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:47.427773   565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.024s	sys 0.010s
I20260809 23:20:47.506269 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:47.515061 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89023.3us
I20260809 23:20:47.460825   497 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:47.518990 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2999) took 1273 ms. Trace:
I20260809 23:20:47.527364 24270 leader_election.cc:398] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:47.528017 24270 leader_election.cc:401] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
W20260809 23:20:47.500715   748 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.682s	user 0.004s	sys 0.006s
W20260809 23:20:47.469545   684 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.024s	sys 0.042s
I20260809 23:20:47.514607 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.474114   717 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260809 23:20:47.460471 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 152841us
I20260809 23:20:47.575275 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.526311 24543 rpcz_store.cc:276] 0809 23:20:46.246071 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.524171 (+278100us) service_pool.cc:227] Handling call
0809 23:20:46.809059 (+284888us) spinlock_profiling.cc:243] Waited 51.5 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:20:46.809273 (+   214us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:46.809541 (+   268us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:46.809868 (+   327us) log.cc:842] Serialized 14 byte log entry
0809 23:20:47.511540 (+701672us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:47.518708 (+  7168us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.518750 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.518812 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.518918 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12270,"prepare.queue_time_us":844,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"spinlock_wait_cycles":118407040,"thread_start_us":684267,"threads_started":1}
W20260809 23:20:47.557075 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.014s	sys 0.014s
W20260809 23:20:47.576416 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.014s	sys 0.015s
I20260809 23:20:47.560931 24422 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:47.533098 24546 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:47.557629 25372 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:47.530906 24545 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:47.588325 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:47.527442   707 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.006s	sys 0.025s
I20260809 23:20:47.505615   901 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260809 23:20:47.508361 32761 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:47.575637 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.527109 24542 rpcz_store.cc:276] 0809 23:20:44.770257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:44.937574 (+167317us) service_pool.cc:227] Handling call
0809 23:20:44.937832 (+   258us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:46.509191 (+1571359us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54002e2820. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:47.336797 (+827606us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.336828 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:47.336871 (+    43us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.336911 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.364988 (+ 28077us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.411144 (+ 46156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":489284,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":13168,"spinlock_wait_cycles":1591520128,"thread_start_us":577758,"threads_started":1}
W20260809 23:20:47.527573   509 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.000s	sys 0.055s
I20260809 23:20:47.526717 32752 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.528707   877 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.518788   565 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.951s	user 0.028s	sys 0.010s
I20260809 23:20:47.607017 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:47.526145   706 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.556527   844 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 15
I20260809 23:20:47.604557 24540 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:47.604755 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:47.586678   726 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.608399 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.560577   884 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:47.616470 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.617211 31984 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:47.617750 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.631560 24427 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260809 23:20:47.593293   688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.018s	sys 0.013s
I20260809 23:20:47.540970   546 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:47.605365   707 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.008s	sys 0.026s
W20260809 23:20:47.613090   509 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.000s	sys 0.055s
I20260809 23:20:47.516206   300 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.101s
I20260809 23:20:47.627103 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3003) took 1075 ms. Trace:
I20260809 23:20:47.612964   969 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:47.670552 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:47.633167 24543 raft_consensus.cc:2396] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:47.629957 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:47.301402 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:20:47.669405 24546 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:47.671912 24540 rpcz_store.cc:276] 0809 23:20:46.548293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.566214 (+ 17921us) service_pool.cc:227] Handling call
0809 23:20:47.233086 (+666872us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:20:47.623932 (+390846us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21548416}
I20260809 23:20:47.676013 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:47.622502 24498 connection.cc:232] RPC call timeout handler was delayed by -1.1513472239021212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
I20260809 23:20:47.625594   682 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
W20260809 23:20:47.681645 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106099us
I20260809 23:20:47.682497 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:47.683631 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:47.634572   688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.018s	sys 0.013s
I20260809 23:20:47.686604 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.685467 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:47.703785 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.704756 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:47.704850 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.705916 24542 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:20:47.681473   809 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:47.630653   492 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.710366   809 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:47.711166 32761 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.712318   492 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:47.710747   809 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:47.715690   984 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:47.716403   984 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:47.707581  1033 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.719790  1033 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.720805 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:20:47.721987  1033 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:47.704525 32380 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:47.723315 32380 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:47.723632 32380 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:47.724908   562 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:47.728315   315 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:47.728621 24314 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260809 23:20:47.710722   990 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.178s	user 0.002s	sys 0.005s
I20260809 23:20:47.622789   877 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.735860 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:47.736021 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:47.733937 24546 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:20:47.728323  1030 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:20:47.742808 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.743216 25373 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:47.734262 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.007s	sys 0.057s
W20260809 23:20:47.745927 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69708.8us
I20260809 23:20:47.685075   300 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:20:47.729121   706 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.751799 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.752162 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:47.751387 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:47.756085 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:47.752903 24430 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:47.753588 24431 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:47.745921 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.007s	sys 0.057s
I20260809 23:20:47.632889   726 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:47.769722 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.006s	sys 0.031s
W20260809 23:20:47.772346 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.006s	sys 0.031s
I20260809 23:20:47.776585 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.777328 24316 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:47.746358   748 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:47.780978   748 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:47.783386  1010 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.285s	user 0.001s	sys 0.005s
I20260809 23:20:47.780452   877 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:47.728883   315 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:47.792555 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1446) took 1071 ms. Trace:
I20260809 23:20:47.793049 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1433) took 1080 ms. Trace:
I20260809 23:20:47.786058 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:47.787385 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1431) took 1075 ms. Trace:
I20260809 23:20:47.789783 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.793103 24323 rpcz_store.cc:276] 0809 23:20:46.721306 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.728806 (+  7500us) service_pool.cc:227] Handling call
0809 23:20:46.729150 (+   344us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.729519 (+   369us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.729551 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.729979 (+   428us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.260204 (+530225us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:47.260257 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.260347 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:47.792338 (+531991us) raft_consensus.cc:1671] finished
0809 23:20:47.792361 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.792497 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":529771,"threads_started":1,"wal-append.queue_time_us":892931}
I20260809 23:20:47.794975 25365 rpcz_store.cc:276] 0809 23:20:46.712113 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.767603 (+ 55490us) service_pool.cc:227] Handling call
0809 23:20:46.768249 (+   646us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.817298 (+ 49049us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.817346 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.817932 (+   586us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.525054 (+707122us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.525113 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.525198 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:47.787173 (+261975us) raft_consensus.cc:1671] finished
0809 23:20:47.787195 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.787314 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18585,"raft.queue_time_us":51121,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":13815,"thread_start_us":690042,"threads_started":1,"wal-append.queue_time_us":963269}
I20260809 23:20:47.800594 24427 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260809 23:20:47.802737 24543 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:47.795255 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 2996) took 1890 ms. Trace:
I20260809 23:20:47.791177   315 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:47.803751 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:47.759933 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.006s	sys 0.030s
I20260809 23:20:47.803376 24539 rpcz_store.cc:276] 0809 23:20:45.904498 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:45.913953 (+  9455us) service_pool.cc:227] Handling call
0809 23:20:45.916911 (+  2958us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:45.917181 (+   270us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:45.917224 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:45.917788 (+   564us) log.cc:842] Serialized 26 byte log entry
0809 23:20:46.223626 (+305838us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:46.223669 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:46.223771 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:47.594418 (+1370647us) spinlock_profiling.cc:243] Waited 92.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:47.784283 (+189865us) raft_consensus.cc:1671] finished
0809 23:20:47.784308 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:47.795184 (+ 10876us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212143360,"thread_start_us":305419,"threads_started":1,"wal-append.queue_time_us":614253}
W20260809 23:20:47.820549 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.006s	sys 0.030s
I20260809 23:20:47.793808 25372 rpcz_store.cc:276] 0809 23:20:46.712696 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.726189 (+ 13493us) service_pool.cc:227] Handling call
0809 23:20:47.792722 (+1066533us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54006004d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:47.792864 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1909798144}
W20260809 23:20:47.821156 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 716090us
I20260809 23:20:47.831548 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:47.831877   326 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.009s	sys 0.023s
W20260809 23:20:47.834424   326 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.010s	sys 0.023s
I20260809 23:20:47.835954   326 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:47.836428   326 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:47.836521 24542 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260809 23:20:47.837524   326 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:47.772264   707 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.046s
I20260809 23:20:47.768548   706 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:47.842818 24316 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:47.846403 25373 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:47.879954 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.842192 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:47.889353 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49145.9us
W20260809 23:20:47.895057 25370 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.010s	sys 0.010s
W20260809 23:20:47.920262 25370 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.012s	sys 0.012s
I20260809 23:20:47.931496 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:47.933175 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:47.939247 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.006s	sys 0.023s
W20260809 23:20:47.939592 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.006s	sys 0.023s
W20260809 23:20:47.891671   960 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.017s	sys 0.013s
W20260809 23:20:47.942052 25369 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.016s	sys 0.016s
W20260809 23:20:47.942390 25369 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.017s	sys 0.017s
I20260809 23:20:47.942979 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:47.943810 24542 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:47.931006  1010 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260809 23:20:47.948462 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:47.879113   546 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260809 23:20:47.951927 24309 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.000s	sys 0.018s
W20260809 23:20:47.952432 24309 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.000s	sys 0.018s
I20260809 23:20:47.950366 24323 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:47.954476 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:47.956969 24320 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:47.891072   884 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:47.954102  1015 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.006s	sys 0.023s
I20260809 23:20:47.960795 24313 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:47.935107   748 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:47.840344   849 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.010s	sys 0.005s
I20260809 23:20:47.971784 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:47.946149   960 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.017s	sys 0.014s
W20260809 23:20:47.948221  1029 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260809 23:20:47.945465 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.005s	sys 0.024s
W20260809 23:20:47.976963 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.005s	sys 0.024s
W20260809 23:20:47.977908 24321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.000s	sys 0.026s
W20260809 23:20:47.980257 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.016s	sys 0.027s
W20260809 23:20:47.966841  1015 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.006s	sys 0.023s
W20260809 23:20:47.982299 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 609416us
I20260809 23:20:47.983207 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:47.982501 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.016s	sys 0.027s
W20260809 23:20:47.982349 24321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.000s	sys 0.027s
I20260809 23:20:47.985401 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:47.990262   726 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:47.991892 24165 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 10, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:20:48.005664 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.018s	sys 0.012s
I20260809 23:20:47.934151   525 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260809 23:20:47.891222  1026 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.095s
I20260809 23:20:47.957666   546 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:48.006094 24428 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.014s	sys 0.024s
W20260809 23:20:48.022382 24428 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.014s	sys 0.024s
W20260809 23:20:48.009174 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.018s	sys 0.012s
I20260809 23:20:47.958901   492 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:47.974818   849 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.014s	sys 0.005s
W20260809 23:20:47.995724   530 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.032s	user 0.013s	sys 0.001s
W20260809 23:20:48.040783 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.008s	sys 0.019s
W20260809 23:20:48.041131 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.008s	sys 0.019s
I20260809 23:20:48.039358 24378 leader_election.cc:398] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18.
I20260809 23:20:48.042153 24378 leader_election.cc:401] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:20:48.039717 24542 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:47.980105  1001 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.050014  1001 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.051846 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 495681us
I20260809 23:20:48.054438 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:48.051640 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3096) took 2097 ms. Trace:
I20260809 23:20:47.999869   624 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:47.994212 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:48.032445   546 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:48.085057 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:48.073098 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 287230us
I20260809 23:20:48.118261 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:47.996091   884 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:48.092905 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.864s (SENT) [suppressed 26 similar messages]
I20260809 23:20:48.052481   686 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
W20260809 23:20:48.128080 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 141236us
I20260809 23:20:48.038062   969 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:48.088088   530 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:48.091477   726 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:20:48.086493   999 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:48.142976 24430 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
W20260809 23:20:48.153869 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 114689us
I20260809 23:20:48.158074 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:48.165797 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:48.032918   684 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.098s
I20260809 23:20:48.070179 24311 rpcz_store.cc:276] 0809 23:20:46.030752 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.035908 (+  5156us) service_pool.cc:227] Handling call
0809 23:20:46.076315 (+ 40407us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:46.249997 (+173682us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.250305 (+   308us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:46.334054 (+ 83749us) log.cc:842] Serialized 52 byte log entry
0809 23:20:46.438443 (+104389us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:46.481001 (+ 42558us) log.cc:842] Serialized 14 byte log entry
0809 23:20:46.496213 (+ 15212us) log.cc:842] Serialized 14 byte log entry
0809 23:20:46.600962 (+104749us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:46.601037 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:46.895729 (+294692us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b1800227f20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511
0809 23:20:47.707865 (+812136us) spinlock_profiling.cc:243] Waited 75.7 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:48.051388 (+343523us) raft_consensus.cc:1671] finished
0809 23:20:48.051412 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.051575 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131395,"prepare.queue_time_us":33838,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":62,"raft.queue_time_us":156004,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":67085,"spinlock_wait_cycles":198830336,"thread_start_us":63792,"threads_started":1,"wal-append.queue_time_us":149165}
I20260809 23:20:48.019573   809 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.175119 24425 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:48.052448   492 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.169557 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:20:48.137696   844 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.512s	user 0.015s	sys 0.002s
W20260809 23:20:48.192847 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 489145us
I20260809 23:20:48.138852  1049 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.134846   686 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:48.201629 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43667.7us
I20260809 23:20:48.202693 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:48.087044 32380 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.153156  1065 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.166301   849 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18.
I20260809 23:20:48.165486   748 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:48.153229   497 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:48.223469   497 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.225593   497 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.094182   624 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.228403   624 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.228948   624 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:48.229403   624 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:48.094033   509 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260809 23:20:48.166841   999 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:48.166036  1063 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:48.235579  1054 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
I20260809 23:20:48.235288  1063 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.237334  1047 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260809 23:20:48.237594  1063 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.238559 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1429) took 1527 ms. Trace:
I20260809 23:20:48.239481   686 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.240566   686 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:48.240381 24165 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 12 to 11, leader changed from <none> to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 11 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:48.241358   995 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:48.241724   995 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:48.190889  1048 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.367s	user 0.000s	sys 0.001s
W20260809 23:20:48.244007  1042 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.443s	user 0.002s	sys 0.002s
I20260809 23:20:48.238993 25368 rpcz_store.cc:276] 0809 23:20:46.711293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.748212 (+ 36919us) service_pool.cc:227] Handling call
0809 23:20:46.766126 (+ 17914us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.777398 (+ 11272us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.777435 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.777948 (+   513us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.761940 (+983992us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:48.165840 (+403900us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.166041 (+   201us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.166165 (+   124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:48.238304 (+ 72139us) raft_consensus.cc:1671] finished
0809 23:20:48.238331 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.238493 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3143,"prepare.queue_time_us":444742,"raft.queue_time_us":11910,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":13860,"thread_start_us":1346660,"threads_started":2,"wal-append.queue_time_us":1248403}
I20260809 23:20:48.212560 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:48.171799  1016 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:48.220031  1057 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260809 23:20:48.242633   995 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.190896   525 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.226629   497 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:48.253331 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87633.3us
I20260809 23:20:48.242204 24165 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 9, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 9 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:48.204452 32761 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.204924   844 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15.
I20260809 23:20:48.274518 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:48.213743  1025 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
W20260809 23:20:48.268034 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87132.6us
I20260809 23:20:48.208734   530 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.286798 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:48.200340   726 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:48.279373 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:48.283067 24545 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.001s
I20260809 23:20:48.214164   300 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.291596 24545 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:48.285775 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.014s	sys 0.017s
W20260809 23:20:48.301707 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.014s	sys 0.017s
W20260809 23:20:48.302058 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 588835us
I20260809 23:20:48.230387   624 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.191277   969 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.264647   315 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:20:48.290940 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.002s	sys 0.027s
W20260809 23:20:48.303244 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.002s	sys 0.027s
I20260809 23:20:48.290762 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:48.303866 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 600104us
W20260809 23:20:48.270587   965 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.684s	user 0.005s	sys 0.023s
W20260809 23:20:48.265007   971 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.700s	user 0.005s	sys 0.020s
W20260809 23:20:48.287590   502 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.912s	user 0.010s	sys 0.003s
I20260809 23:20:48.288750   995 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:48.271948   748 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:48.199054  1034 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.105s
I20260809 23:20:48.335944 24540 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260809 23:20:48.344723 24165 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:48.356367 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1441) took 1480 ms. Trace:
I20260809 23:20:48.286558 32761 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.328685   971 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.764s	user 0.005s	sys 0.020s
I20260809 23:20:48.362033 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:48.324671   965 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.754s	user 0.005s	sys 0.023s
I20260809 23:20:48.284144   525 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:48.367146 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0592289478518069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2983s for a response
I20260809 23:20:48.361714 25371 rpcz_store.cc:276] 0809 23:20:46.875637 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.965091 (+ 89454us) service_pool.cc:227] Handling call
0809 23:20:46.965428 (+   337us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:46.967161 (+  1733us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.967202 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:46.967717 (+   515us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.577047 (+609330us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.577091 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.577152 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:48.356113 (+778961us) raft_consensus.cc:1671] finished
0809 23:20:48.356140 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.356296 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1005,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":608732,"threads_started":1,"wal-append.queue_time_us":855335}
I20260809 23:20:48.385722 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3198) took 1085 ms. Trace:
I20260809 23:20:48.246762  1065 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.326195  1033 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:48.394922 25368 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:48.354900   726 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:48.412362 24428 rpcz_store.cc:276] 0809 23:20:47.300595 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.362352 (+ 61757us) service_pool.cc:227] Handling call
0809 23:20:47.390138 (+ 27786us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:47.390339 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:48.034989 (+644650us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.370641 (+335652us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.385325 (+ 14684us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:48.385380 (+    55us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:48.385442 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.385515 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.385653 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13750,"prepare.queue_time_us":903020,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":19222,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":17701,"spinlock_wait_cycles":63492224,"thread_start_us":947162,"threads_started":2}
I20260809 23:20:48.326323 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1438) took 1644 ms. Trace:
W20260809 23:20:48.388322 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 117148us
I20260809 23:20:48.273964  1078 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.131s
I20260809 23:20:48.208539  1049 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.304787   844 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.087s
I20260809 23:20:48.300702   530 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:48.324174   300 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.423679 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 220775us
I20260809 23:20:48.431038 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:48.357290   502 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.467868 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:48.479113 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:48.430348 25366 rpcz_store.cc:276] 0809 23:20:46.681966 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.682109 (+   143us) service_pool.cc:227] Handling call
0809 23:20:46.682466 (+   357us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:46.719245 (+ 36779us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.719292 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:46.720032 (+   740us) log.cc:842] Serialized 52 byte log entry
0809 23:20:47.291411 (+571379us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:47.833234 (+541823us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.326003 (+492769us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.326079 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:48.326120 (+    41us) raft_consensus.cc:1671] finished
0809 23:20:48.326135 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.326269 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":222,"prepare.queue_time_us":846120,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":6527,"raft.queue_time_us":222588,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":12286,"thread_start_us":1546124,"threads_started":3,"wal-append.queue_time_us":980613}
I20260809 23:20:48.482793 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:48.456017   998 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:20:48.493201 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:48.495108 24431 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:48.450943   717 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:48.501842 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:48.506696 24429 raft_consensus.cc:2396] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:48.493115 24165 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 11, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:48.502709 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:48.508421 24320 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.551s	user 0.013s	sys 0.000s
W20260809 23:20:48.510245 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79313.6us
I20260809 23:20:48.456421  1001 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.467736   726 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:48.520630 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:48.482553   844 consensus_queue.cc:1225] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:20:48.489151   684 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:48.530439 25366 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:20:48.532568 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:48.508795 24320 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:20:48.533593 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 408834us
I20260809 23:20:48.489076  1033 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.396663  1098 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.118s
I20260809 23:20:48.506187  1063 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:48.528932 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:48.537654 24428 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260809 23:20:48.495289   884 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:48.510339   717 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:48.519399 32380 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:48.529186   970 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.065s	user 0.015s	sys 0.000s
I20260809 23:20:48.558331 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:48.566540 25366 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:48.528499   492 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:48.537032   684 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:48.457515  1101 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.121s
I20260809 23:20:48.574064 24165 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:20:48.555693  1073 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260809 23:20:48.623265 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.619393 24317 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:48.630957 24317 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:48.631080 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:48.548404  1063 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:48.556140  1034 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.490121   748 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.584411  1074 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.321s	user 0.000s	sys 0.001s
I20260809 23:20:48.563616   717 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:48.634073 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 240) took 36960 ms (client timeout 15000 ms). Trace:
W20260809 23:20:48.646327 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82369us
W20260809 23:20:48.650406 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 161961us
W20260809 23:20:48.659291  1111 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
W20260809 23:20:48.660260  1109 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260809 23:20:48.663770 24320 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260809 23:20:48.589156   684 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:48.692191 24428 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:48.645749 24317 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.699499 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 335054us
I20260809 23:20:48.641590  1063 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:48.713241 24165 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:20:48.759413  1085 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.350s	user 0.001s	sys 0.002s
I20260809 23:20:48.771377 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:48.640681   809 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.772113 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:48.771378 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.584677   970 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.711820   726 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:48.800738 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 302432us
W20260809 23:20:48.703074 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 168532us
I20260809 23:20:48.646229  1001 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.773129 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:48.790997 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:20:48.800343 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 181147us
I20260809 23:20:48.806572 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.803006 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:48.823850 25366 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:48.711380 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:48.796619 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:48.721217   969 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:48.790695 24323 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18. Re-sending same reply.
I20260809 23:20:48.815644 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:48.844409 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52200.5us
I20260809 23:20:48.845294 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:48.846621 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:48.842849 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3114) took 2078 ms. Trace:
W20260809 23:20:48.837314  1117 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260809 23:20:48.839421 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.840421 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:48.692734 24164 rpcz_store.cc:269] 0809 23:20:11.385269 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:11.387142 (+  1873us) service_pool.cc:227] Handling call
0809 23:20:48.284912 (+36897770us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:20:36.795161 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:20:36.795957 (+   796us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:20:36.796014 (+    57us) write_op.cc:435] Acquired schema lock
0809 23:20:36.796038 (+    24us) tablet.cc:660] Decoding operations
0809 23:20:36.951732 (+155694us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:20:36.951859 (+   127us) write_op.cc:641] Partition lock acquired for write op
0809 23:20:36.951917 (+    58us) tablet.cc:683] Acquiring locks for 25 operations
0809 23:20:36.954420 (+  2503us) tablet.cc:699] Row locks acquired
0809 23:20:36.954456 (+    36us) write_op.cc:260] PREPARE: finished
0809 23:20:36.954656 (+   200us) write_op.cc:270] Start()
0809 23:20:36.954833 (+   177us) write_op.cc:276] Timestamp: P: 1786317636954581 usec, L: 0
0809 23:20:36.954876 (+    43us) op_driver.cc:331] REPLICATION: starting
0809 23:20:36.961140 (+  6264us) log.cc:842] Serialized 9740 byte log entry
0809 23:20:38.207161 (+1246021us) op_driver.cc:444] REPLICATION: finished
0809 23:20:38.343646 (+136485us) write_op.cc:301] APPLY: starting
0809 23:20:38.343754 (+   108us) tablet.cc:1365] starting BulkCheckPresence
0809 23:20:38.981866 (+638112us) tablet.cc:1368] finished BulkCheckPresence
0809 23:20:38.981988 (+   122us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:20:41.699058 (+2717070us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:20:41.830116 (+131058us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0809 23:20:41.830221 (+   105us) write_op.cc:312] APPLY: finished
0809 23:20:41.831059 (+   838us) log.cc:842] Serialized 220 byte log entry
0809 23:20:42.066822 (+235763us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:42.106740 (+ 39918us) write_op.cc:454] Released schema lock
0809 23:20:42.106830 (+    90us) write_op.cc:341] FINISH: Updating metrics
0809 23:20:42.160449 (+ 53619us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:42.160494 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5736389,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":118043,"apply.run_cpu_time_us":108406,"apply.run_wall_time_us":3817647,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108527,"num_ops":25,"prepare.queue_time_us":530047,"prepare.run_cpu_time_us":28596,"prepare.run_wall_time_us":453846,"raft.queue_time_us":597900,"raft.run_cpu_time_us":10760,"raft.run_wall_time_us":163593,"replication_time_us":1238039,"thread_start_us":1210687,"threads_started":5,"wal-append.queue_time_us":802245,"wal-append.run_cpu_time_us":40973,"wal-append.run_wall_time_us":3123863}]]}
I20260809 23:20:48.841352  1160 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:48.865368 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:48.839447 25371 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:48.865638 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:48.866153 25371 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:48.792713  1024 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.235s	user 0.014s	sys 0.000s
W20260809 23:20:48.854602 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 143274us
I20260809 23:20:48.866446 24428 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:48.867096 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:48.825762 24165 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 11 to 12, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 12 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:48.848162 24321 rpcz_store.cc:276] 0809 23:20:46.764475 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.223957 (+459482us) service_pool.cc:227] Handling call
0809 23:20:47.235865 (+ 11908us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.985259 (+749394us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.985295 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:47.985744 (+   449us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.304715 (+318971us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:48.304757 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.304841 (+    84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:48.842610 (+537769us) raft_consensus.cc:1671] finished
0809 23:20:48.842633 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.842783 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8890,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":7299,"thread_start_us":1067494,"threads_started":2,"wal-append.queue_time_us":497375}
I20260809 23:20:48.664199  1034 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.896752 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.900638 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:48.902869 25373 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:48.897879 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1441) took 1609 ms. Trace:
W20260809 23:20:48.828177   686 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.001s
I20260809 23:20:48.905050 25370 rpcz_store.cc:276] 0809 23:20:47.288375 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.297336 (+  8961us) service_pool.cc:227] Handling call
0809 23:20:47.309787 (+ 12451us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.310116 (+   329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.310146 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:47.310531 (+   385us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.920783 (+610252us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.920825 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.920919 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:48.897609 (+976690us) raft_consensus.cc:1671] finished
0809 23:20:48.897638 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:48.897807 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":989,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":609847,"threads_started":1,"wal-append.queue_time_us":836061}
I20260809 23:20:48.790241  1126 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:48.910759 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.946401 24320 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:48.916812 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.822769   525 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:48.826040 32761 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:48.956399 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.827881  1015 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:48.983610 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:48.984650 24321 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:48.965842 24165 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 11, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 11 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:48.842135   971 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:48.867681   844 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:48.977447 24313 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:48.966643  1092 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
I20260809 23:20:49.009776 24320 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:48.975371 25371 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.866271   969 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:48.945678   300 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:48.993307 31984 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.330s	user 0.009s	sys 0.005s
I20260809 23:20:49.016837   844 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:48.969177  1065 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:49.009363  1093 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.576s	user 0.001s	sys 0.001s
I20260809 23:20:48.864802  1001 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.075673 25371 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:49.049790 24165 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 9, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 9 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:20:49.024914   999 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.793s	user 0.018s	sys 0.000s
I20260809 23:20:49.067469 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3197) took 1775 ms. Trace:
I20260809 23:20:48.983880   686 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:49.083487 24435 rpcz_store.cc:276] 0809 23:20:47.291815 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.364125 (+ 72310us) service_pool.cc:227] Handling call
0809 23:20:47.390512 (+ 26387us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:47.408772 (+ 18260us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.408818 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:47.419867 (+ 11049us) log.cc:842] Serialized 52 byte log entry
0809 23:20:48.023235 (+603368us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:48.023281 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.023367 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.067236 (+1043869us) raft_consensus.cc:1671] finished
0809 23:20:49.067262 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.067404 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17834,"raft.queue_time_us":22079,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3756,"thread_start_us":550182,"threads_started":1,"wal-append.queue_time_us":748024}
W20260809 23:20:49.035844 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 170562us
W20260809 23:20:49.082095 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 570354us
W20260809 23:20:49.098610 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 254392us
W20260809 23:20:49.076642   315 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.069s	sys 0.095s
I20260809 23:20:49.122210 24545 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260809 23:20:48.904769  1024 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.135737 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3124) took 1295 ms. Trace:
I20260809 23:20:49.066733  1049 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:49.059405  1033 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.223645 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:49.129217 24165 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 9, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 9 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:49.211669 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:49.211018 25366 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:49.017257  1123 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.257205 25371 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260809 23:20:49.267764 25371 pending_rounds.cc:76] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:20:49.268118 25371 pending_rounds.cc:97] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260809 23:20:49.268522 25371 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:49.067898 31984 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:49.265764 24435 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:49.125084   315 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.070s	sys 0.095s
I20260809 23:20:49.237962 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:49.271572 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.049989  1026 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.096s
I20260809 23:20:49.277052 24165 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 6 to 12, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 12 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:49.011683   971 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.290817 24546 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260809 23:20:49.292523 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 172219us
I20260809 23:20:49.227685 24543 rpcz_store.cc:276] 0809 23:20:47.840461 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.840641 (+   180us) service_pool.cc:227] Handling call
0809 23:20:47.930845 (+ 90204us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.942922 (+ 12077us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.942984 (+    62us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:48.225493 (+282509us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:20:48.226053 (+   560us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.397763 (+171710us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:48.397838 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.400491 (+  2653us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.115037 (+714546us) raft_consensus.cc:1671] finished
0809 23:20:49.115065 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.135543 (+ 20478us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":785,"raft.queue_time_us":933,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6001,"spinlock_wait_cycles":30477184,"thread_start_us":149240,"threads_started":1,"wal-append.queue_time_us":326264}
I20260809 23:20:49.066030   969 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:48.970363  1126 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:49.324810 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 457409us
I20260809 23:20:49.330756 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:49.317744 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 118683us
I20260809 23:20:49.141501  1101 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:49.291446  1092 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260809 23:20:49.340463 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.223242  1065 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:49.347589 24165 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:49.349531 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:49.366546 24545 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:49.359735 24534 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:49.276954   315 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:49.290817  1123 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:49.399581 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 280702us
I20260809 23:20:49.340888  1033 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.172s
I20260809 23:20:49.395169   970 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.401315 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.431057  1127 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:49.416522   809 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:49.325513  1026 consensus_queue.cc:1225] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:49.627168 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:49.627247 24543 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:49.533180 24545 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:49.630810 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.450078  1024 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:49.414026  1094 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.583s	user 0.001s	sys 0.000s
W20260809 23:20:49.366470  1082 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.490s	user 0.000s	sys 0.008s
I20260809 23:20:49.423259 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1443) took 2105 ms. Trace:
I20260809 23:20:49.347218   844 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.424052  1036 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:49.436647  1112 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:49.434182  1154 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:49.400676 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:49.431545  1071 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.541s	user 0.000s	sys 0.004s
I20260809 23:20:49.453091  1126 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:49.123639   999 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:49.439384  1084 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.453s	user 0.001s	sys 0.000s
I20260809 23:20:49.428911  1165 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.435163  1121 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.679615   624 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:49.710330  1121 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.710796   624 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.318109   530 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:49.434965 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:49.706833  1126 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:49.713481 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 306858us
W20260809 23:20:49.713846  1106 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.311s	user 0.000s	sys 0.000s
W20260809 23:20:49.714115   717 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.021s	user 0.011s	sys 0.001s
I20260809 23:20:49.715173 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:49.716542 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:49.418027  1153 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:49.717262 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1477) took 1986 ms. Trace:
I20260809 23:20:49.717741 25363 rpcz_store.cc:276] 0809 23:20:47.731347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.748210 (+ 16863us) service_pool.cc:227] Handling call
0809 23:20:47.748513 (+   303us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.754663 (+  6150us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.754699 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:47.783187 (+ 28488us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.184022 (+400835us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:48.184074 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.184157 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.717043 (+1532886us) raft_consensus.cc:1671] finished
0809 23:20:49.717066 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.717202 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":86869,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":400383,"threads_started":1,"wal-append.queue_time_us":1918971}
I20260809 23:20:49.414872  1137 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:49.714534   717 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.415964 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.416967  1141 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:20:49.581990 24165 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:20:49.400624   546 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.263s	user 0.006s	sys 0.007s
I20260809 23:20:49.586750 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3000) took 3272 ms. Trace:
W20260809 23:20:49.405139  1110 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.634s	user 0.001s	sys 0.001s
I20260809 23:20:49.424459  1081 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:49.370369  1101 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.414875  1034 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.734225 24535 rpcz_store.cc:276] 0809 23:20:46.314314 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.701031 (+386717us) service_pool.cc:227] Handling call
0809 23:20:46.746393 (+ 45362us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:47.508520 (+762127us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.508555 (+    35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:47.510072 (+  1517us) log.cc:842] Serialized 52 byte log entry
0809 23:20:48.042093 (+532021us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:48.042140 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.042226 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.586456 (+1544230us) raft_consensus.cc:1671] finished
0809 23:20:49.586483 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.586677 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":490505,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":12415,"thread_start_us":1139487,"threads_started":2,"wal-append.queue_time_us":747028}
I20260809 23:20:49.739878 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.594391  1169 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:49.732044  1121 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.744266 24535 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:49.657444 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.745786 24430 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 13: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:49.746052 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.747721 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.748311 24535 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:49.749509 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:49.749882 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.006s	sys 0.019s
W20260809 23:20:49.750278 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.960s	user 0.006s	sys 0.019s
W20260809 23:20:49.438210  1116 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.388s	user 0.000s	sys 0.001s
I20260809 23:20:49.667825 32741 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
W20260809 23:20:49.679113 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3106) took 3223 ms (client timeout 1515 ms). Trace:
I20260809 23:20:49.401626  1065 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.691917 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:49.756778 24314 rpcz_store.cc:269] 0809 23:20:46.455659 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.455810 (+   151us) service_pool.cc:227] Handling call
0809 23:20:49.668596 (+3212786us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:49.679020 (+ 10424us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2923933952}
I20260809 23:20:49.708135  1001 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.709412 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3001) took 3313 ms. Trace:
I20260809 23:20:49.709723   844 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:49.709299  1165 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:49.717483  1136 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.783s	user 0.002s	sys 0.000s
I20260809 23:20:49.423588  1173 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:49.136525   686 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.719033 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3116) took 2831 ms. Trace:
I20260809 23:20:49.720975 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:49.759205 24544 rpcz_store.cc:276] 0809 23:20:46.396320 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.736866 (+340546us) service_pool.cc:227] Handling call
0809 23:20:46.750360 (+ 13494us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.496611 (+746251us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b18004ac320. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac23858f6b 00007fac2386378b
0809 23:20:47.496779 (+   168us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:47.497129 (+   350us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.041592 (+544463us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:48.072001 (+ 30409us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.072262 (+   261us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:48.072342 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.073312 (+   970us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.709047 (+1635735us) raft_consensus.cc:1671] finished
0809 23:20:49.709179 (+   132us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.709335 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2499,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":470332,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":43360,"spinlock_wait_cycles":7850880,"thread_start_us":1031180,"threads_started":2,"wal-append.queue_time_us":1282200}
I20260809 23:20:49.701382 32761 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260809 23:20:49.656119   970 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:49.651952 25369 rpcz_store.cc:276] 0809 23:20:47.317505 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.364193 (+ 46688us) service_pool.cc:227] Handling call
0809 23:20:47.371541 (+  7348us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.947877 (+576336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.947921 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:47.954187 (+  6266us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.340834 (+386647us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:48.340890 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.340992 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.423002 (+1082010us) raft_consensus.cc:1671] finished
0809 23:20:49.423022 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.423196 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2114,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":172,"thread_start_us":931058,"threads_started":2,"wal-append.queue_time_us":1450389}
I20260809 23:20:49.729458 24428 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:49.736912  1114 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.736538   546 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.738051   315 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.697470  1113 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:49.742673 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.002s	sys 0.019s
I20260809 23:20:49.676749  1099 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.740540   565 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:49.745080 24545 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:49.742272 24165 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 10, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 10 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:20:49.690436 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:49.725886   326 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.888s	user 0.014s	sys 0.000s
I20260809 23:20:49.683717  1036 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:49.758859 24543 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260809 23:20:49.717236  1105 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.760550 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.831421 24435 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:49.832020 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2724) took 1054 ms. Trace:
I20260809 23:20:49.832412 24435 rpcz_store.cc:276] 0809 23:20:48.777152 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.729522 (+952370us) service_pool.cc:227] Handling call
0809 23:20:49.831961 (+102439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:49.833984 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.834445 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.764505 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:49.835261 24428 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:49.736460  1033 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:49.722589   995 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.360s	user 0.009s	sys 0.001s
I20260809 23:20:49.847667 24542 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:49.683993  1024 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:49.801229 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80329.8us
I20260809 23:20:49.810766 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.760704  1081 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:49.813512 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1340) took 3264 ms. Trace:
I20260809 23:20:49.815009 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3194) took 2533 ms. Trace:
I20260809 23:20:49.774919  1173 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:49.774292 24309 rpcz_store.cc:276] 0809 23:20:46.887821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.296685 (+408864us) service_pool.cc:227] Handling call
0809 23:20:47.297004 (+   319us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.311454 (+ 14450us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:47.952932 (+641478us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.325464 (+372532us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:48.326207 (+   743us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.326390 (+   183us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:48.326427 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.326513 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.718679 (+1392166us) raft_consensus.cc:1671] finished
0809 23:20:49.718705 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.718831 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22574,"prepare.queue_time_us":908795,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":226140,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":7645,"thread_start_us":984710,"threads_started":2,"wal-append.queue_time_us":1185391}
I20260809 23:20:49.771075   999 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:49.823617 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.003s	sys 0.019s
I20260809 23:20:49.825517 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:49.825678 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 135349us
I20260809 23:20:49.826793 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.796983   624 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:49.811589   684 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.023s	user 0.009s	sys 0.004s
I20260809 23:20:49.834723 24435 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:49.837168 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.891101   684 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.891855   624 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:49.892249 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.724685  1137 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:49.820569  1116 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260809 23:20:49.811936  1034 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:49.850198 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:49.844780 24165 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 11, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 11 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:20:49.839834   565 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.855883 25368 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:49.744608  1169 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.853940 24544 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:49.859329 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:49.860317 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95950.8us
I20260809 23:20:49.687650  1127 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.768385  1141 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260809 23:20:49.857486 24423 rpcz_store.cc:276] 0809 23:20:47.273172 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.361984 (+ 88812us) service_pool.cc:227] Handling call
0809 23:20:47.375516 (+ 13532us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:47.375921 (+   405us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.375954 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:47.376588 (+   634us) log.cc:842] Serialized 52 byte log entry
0809 23:20:47.940082 (+563494us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:47.940137 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.940223 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.806638 (+1866415us) raft_consensus.cc:1671] finished
0809 23:20:49.806668 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.806838 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15104,"raft.queue_time_us":4486,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":8655,"thread_start_us":563073,"threads_started":1,"wal-append.queue_time_us":889555}
I20260809 23:20:49.830669  1001 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:49.852721  1175 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:49.856172 24541 rpcz_store.cc:276] 0809 23:20:46.548647 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.216261 (+667614us) service_pool.cc:227] Handling call
0809 23:20:47.216696 (+   435us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.242282 (+ 25586us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:47.242315 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:47.256953 (+ 14638us) log.cc:842] Serialized 26 byte log entry
0809 23:20:47.985097 (+728144us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:47.985137 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.985200 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.813238 (+1828038us) raft_consensus.cc:1671] finished
0809 23:20:49.813267 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:49.813433 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17508,"raft.queue_time_us":145561,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":10276,"thread_start_us":686626,"threads_started":1,"wal-append.queue_time_us":1151761}
I20260809 23:20:49.891156 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1182) took 1133 ms. Trace:
I20260809 23:20:49.812736   970 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.855391  1144 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.896939 24535 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:49.899798 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1173) took 1254 ms. Trace:
I20260809 23:20:49.910420 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:49.968410 24428 rpcz_store.cc:276] 0809 23:20:48.645030 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.849802 (+1204772us) service_pool.cc:227] Handling call
0809 23:20:49.899737 (+ 49935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:49.824177   969 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.879927  1173 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:49.969780 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.970799 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1178) took 1277 ms. Trace:
I20260809 23:20:49.971238 24428 rpcz_store.cc:276] 0809 23:20:48.693291 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.969386 (+1276095us) service_pool.cc:227] Handling call
0809 23:20:49.970729 (+  1343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:49.913800 24165 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:20:49.894058   809 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:49.912384 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:49.943377 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.980109 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:49.835945   326 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.980846 24423 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:49.912016   565 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:49.945063 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 85828.4us
I20260809 23:20:49.821832  1114 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.997429  1114 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:49.998172  1096 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260809 23:20:49.954603  1172 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:49.999567  1172 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:49.955087 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64875.2us
I20260809 23:20:49.955289 24435 rpcz_store.cc:276] 0809 23:20:48.758273 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.833381 (+1075108us) service_pool.cc:227] Handling call
0809 23:20:49.891090 (+ 57709us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:49.909719  1177 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.859383  1024 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:49.841312  1113 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:49.946103  1026 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:49.944399  1186 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:49.982285 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:49.980430 24543 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:49.912149  1123 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:49.841202  1099 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.002972 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:49.843950  1105 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:50.055675  1190 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260809 23:20:50.056427   686 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.056957  1105 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.057664  1172 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:50.005960 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.057999 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:49.950711  1112 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:49.975536  1141 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:49.960318  1175 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:50.030704 24545 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:49.981283   969 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.031200 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.027987 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1450) took 3305 ms. Trace:
I20260809 23:20:50.030682 25372 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:50.027104 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 114813us
I20260809 23:20:50.053932  1127 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:50.054338  1185 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260809 23:20:50.060575  1063 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.255s	user 0.015s	sys 0.000s
I20260809 23:20:50.048483 24428 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:50.089444 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3215) took 1264 ms. Trace:
I20260809 23:20:49.965826   717 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:50.066291 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.048084 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:50.092639 24435 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:49.964824 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
W20260809 23:20:50.075078 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 164715us
I20260809 23:20:50.026824   809 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:49.906001  1036 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.054710  1099 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:50.086306 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55181.2us
W20260809 23:20:50.096534 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 90699.6us
I20260809 23:20:50.038901  1101 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.083109 24312 rpcz_store.cc:276] 0809 23:20:46.722297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.214027 (+491730us) service_pool.cc:227] Handling call
0809 23:20:47.229141 (+ 15114us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:47.252963 (+ 23822us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:47.568280 (+315317us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b1800375fa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:20:48.035656 (+467376us) log.cc:842] Serialized 14 byte log entry
0809 23:20:48.649257 (+613601us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:48.693112 (+ 43855us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.693550 (+   438us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:48.693625 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:48.693951 (+   326us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:49.991464 (+1297513us) raft_consensus.cc:1671] finished
0809 23:20:50.027757 (+ 36293us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:50.027920 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":866119,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":315964,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":29759,"spinlock_wait_cycles":17406848,"thread_start_us":1076454,"threads_started":3,"wal-append.queue_time_us":829033}
I20260809 23:20:50.093647 25369 raft_consensus.cc:595] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:50.057250   546 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:49.983175  1144 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.070399  1123 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:50.089811 24428 rpcz_store.cc:276] 0809 23:20:48.825034 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.978509 (+1153475us) service_pool.cc:227] Handling call
0809 23:20:50.089378 (+110869us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:50.066447   684 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.108982 24312 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:50.109294   970 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.109596 24321 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:50.109956   970 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:50.110355   970 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:50.110667 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.111400 24313 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:50.111470  1137 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.112497 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.069203  1176 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.102415 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.097504 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.104585 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.106518 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.107820  1167 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:50.108183   565 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:50.095976  1172 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:20:50.117990 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.086009  1165 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.093739  1114 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:50.123250 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.131407 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.119464 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:20:50.131892 24313 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:50.094655  1127 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:50.122074 25366 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:50.114902 24541 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:20:50.075083  1175 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:50.130308 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:50.133713 24544 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
W20260809 23:20:50.130388 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148194us
I20260809 23:20:50.125460 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.110747   970 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:50.116565  1036 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:50.088107  1141 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.135886 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:50.159250  1169 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.090042  1186 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.141652  1207 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:50.001968  1081 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.143101  1113 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.143394 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.133718  1127 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.145020  1179 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:50.144929 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.146219 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.143438 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3133) took 2121 ms. Trace:
I20260809 23:20:50.175273 24542 rpcz_store.cc:276] 0809 23:20:48.021792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.084709 (+ 62917us) service_pool.cc:227] Handling call
0809 23:20:50.143252 (+2058543us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:50.143380 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3954711296}
I20260809 23:20:50.177060 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.134536 24544 pending_rounds.cc:76] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
I20260809 23:20:50.179163 24544 pending_rounds.cc:97] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 13.13
I20260809 23:20:50.179499 24544 raft_consensus.cc:2866] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:50.180965 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:50.181615 24544 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:50.181526  1169 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:50.147594   326 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.184407 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.130793 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.133018 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.149596 24423 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:50.178675 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.116585  1137 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.127589  1176 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.158336  1199 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:50.160466 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:50.141458   706 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.196646 24428 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:50.030813  1177 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.089782   969 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.128638  1024 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.123204  1123 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:50.209564  1123 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.210286   969 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.198655  1137 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.213297  1137 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:50.213968 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81033.3us
I20260809 23:20:50.164685 24435 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:50.096203  1063 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.215197   530 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:50.215574   530 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.215915   530 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:50.216492  1063 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.133523  1114 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.217535  1114 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:50.166738 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:50.219219 24309 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:50.174710 25366 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:50.186678 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2726) took 1408 ms. Trace:
I20260809 23:20:50.221123 24423 rpcz_store.cc:276] 0809 23:20:48.777771 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.950045 (+1172274us) service_pool.cc:227] Handling call
0809 23:20:50.186595 (+236550us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:50.222113 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.155480 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.185135 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
W20260809 23:20:50.171645   434 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260809 23:20:50.192981  1172 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:50.204521  1203 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260809 23:20:50.185647  1179 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:50.131003  1167 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.168568  1207 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:50.222922 25321 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.577s (SENT)
I20260809 23:20:50.267921 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.224387 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1174) took 1579 ms. Trace:
I20260809 23:20:50.268662 24435 rpcz_store.cc:276] 0809 23:20:48.645297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.016294 (+1370997us) service_pool.cc:227] Handling call
0809 23:20:50.224308 (+208014us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:50.224436 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:50.222856 24423 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:50.225829 25370 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:50.229260 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:50.269873 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46993.7us
W20260809 23:20:50.270022 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45683.6us
I20260809 23:20:50.185510  1127 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:50.205576 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102220us
I20260809 23:20:50.262707 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.271371 24541 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:50.271476 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.271652 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.272218 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.272771 24541 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:50.273388  1205 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:50.273561   717 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:50.270890 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.274098  1205 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:50.274622 24542 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:20:50.274657  1205 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:50.273391 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.276075 24541 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:50.277177 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.278313 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.195938  1199 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.171453  1065 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.296165 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:50.296814 24313 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:50.296917  1144 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.283756  1101 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:50.299397  1101 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.299543  1179 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.300972  1179 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:50.220885  1079 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.286784 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.302057  1079 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.304100  1173 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.284405  1169 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:50.307659  1026 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:50.307888 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.308595 24321 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:50.310124 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.311486 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:50.213317  1186 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.231854   999 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:50.189949  1220 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.119s
I20260809 23:20:50.286546 32741 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:50.338675 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:50.339943 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.311691 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.302778 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.288492 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.317867 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.342118 25371 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:50.241834   684 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:50.347702 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.347383 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.348776 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.349018 25365 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:50.291326  1176 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.343405 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.344521 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:50.329244 24428 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:50.345028 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.348124 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:50.352027 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.353585 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1187) took 1038 ms. Trace:
I20260809 23:20:50.353978 24435 rpcz_store.cc:276] 0809 23:20:49.315561 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.288179 (+972618us) service_pool.cc:227] Handling call
0809 23:20:50.353519 (+ 65340us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:50.284036  1207 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:50.355399 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83987.3us
I20260809 23:20:50.357366 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.311393  1173 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.346472 25363 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260809 23:20:50.353749 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.313382  1199 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:50.401880 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48199.2us
I20260809 23:20:50.405205 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.356649 25366 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:50.403290 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44803.5us
I20260809 23:20:50.409996 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.405359 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:50.347378   434 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
W20260809 23:20:50.374248   844 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.599s	user 0.008s	sys 0.004s
I20260809 23:20:50.413666 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:50.409180 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.415652 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.372917   315 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:50.353015  1065 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260809 23:20:50.492267 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 335975us
I20260809 23:20:50.382081  1024 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:50.494992  1024 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:50.495990  1024 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.193400  1112 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.436422 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:50.381110  1186 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:50.343842  1169 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260809 23:20:50.406069  1001 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:50.500821  1001 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.449482   844 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:50.501608  1001 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:50.269042   809 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.451552  1027 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:50.503337  1063 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:50.503435 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 95656.4us
I20260809 23:20:50.503990  1063 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.505995  1063 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.506176 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.347224 32741 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.372995   748 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:50.372923   684 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:50.500867 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 75697us
I20260809 23:20:50.403083  1026 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:50.342787   999 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.510684 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:50.431941 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.516383 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.501648  1220 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:50.536927 24544 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:50.490231  1065 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.407841  1127 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.509902  1186 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.543118 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:50.543852 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:50.553805 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.557389 24435 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:50.554505 24544 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:50.561031 24313 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:50.569725 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:50.566299 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:50.572289 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:50.547773  1105 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:50.583885  1213 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
W20260809 23:20:50.584566  1228 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260809 23:20:50.585096  1176 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260809 23:20:50.586790  1227 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260809 23:20:50.591111 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (TS 808fed3ed3f04882ad757a3db9abfdd9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42134
I20260809 23:20:50.554922   434 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:50.592468   434 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:50.593330   434 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.594506   434 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:50.543396  1220 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:50.555497  1226 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
W20260809 23:20:50.675386 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 132346us
I20260809 23:20:50.601058 24309 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:50.608608 24544 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:50.696059 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:50.696810 24544 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:50.574594   706 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:50.545033  1225 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:50.615929   999 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:50.616372   684 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.542825  1065 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:50.559556  1186 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:50.535104   686 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.082s
I20260809 23:20:50.571758   844 raft_consensus.cc:2741] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260809 23:20:50.620463 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2658) took 1844 ms (client timeout 2054 ms). Trace:
I20260809 23:20:50.542984  1027 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.079s
I20260809 23:20:50.622272  1223 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.623342   502 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:50.625365  1015 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:50.627719  1120 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.104s	user 0.004s	sys 0.000s
I20260809 23:20:50.347553  1016 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.224s
I20260809 23:20:50.632872 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:50.734133 24323 rpcz_store.cc:269] 0809 23:20:48.775811 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.866802 (+ 90991us) service_pool.cc:227] Handling call
0809 23:20:50.620189 (+1753387us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:50.620370 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3264095104}
I20260809 23:20:50.656620  1220 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:50.523566  1026 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.687928   965 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:50.740216 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 168028us
I20260809 23:20:50.598821  1114 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:50.743541  1081 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.004s	sys 0.010s
W20260809 23:20:50.739006  1175 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.006s
I20260809 23:20:50.735381   326 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:50.781055 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:50.740806   684 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:50.743053 24544 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:50.749369  1015 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:50.758471 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 252356us
I20260809 23:20:50.756539  1112 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:50.786293 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:50.741667   717 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:50.779127  1178 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.043s	sys 0.043s
I20260809 23:20:50.783499   965 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.791306  1024 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:20:50.792896 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:50.805949 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:50.758100 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 125322us
I20260809 23:20:50.749931   706 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:50.807971 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.809062 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:50.827471 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 422329us
I20260809 23:20:50.781003   849 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:50.818552  1178 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.044s	sys 0.044s
W20260809 23:20:50.788024  1081 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.006s	sys 0.013s
I20260809 23:20:50.787084  1167 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.827215 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:50.783152   999 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:50.805105  1205 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.000s
I20260809 23:20:50.905331 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:50.811913  1112 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:50.824360 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:50.919313 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.909257 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:50.834978  1179 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.001s
I20260809 23:20:50.807044   326 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:50.984938 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79817.7us
I20260809 23:20:50.979481 25366 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260809 23:20:50.987694 24435 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:50.988462 24544 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:20:51.003919 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 176761us
I20260809 23:20:50.829540  1016 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:51.011310 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 187039us
I20260809 23:20:51.012549 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:50.979485   965 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:50.901942  1165 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:20:50.907109  1212 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
W20260809 23:20:51.107906 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 301998us
I20260809 23:20:51.056377 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:51.175930 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:50.787683  1175 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:50.994855  1112 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:50.901176   706 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.026021   849 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:50.924345   686 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260809 23:20:51.194896 24435 raft_consensus.cc:2396] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260809 23:20:50.751710  1223 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:50.907107   624 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.202s
W20260809 23:20:51.181433  1063 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.018s	sys 0.043s
I20260809 23:20:51.029488   717 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:51.035989  1081 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:50.805626   684 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.240131 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:51.258919 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3222) took 1190 ms. Trace:
I20260809 23:20:51.259490 24435 rpcz_store.cc:276] 0809 23:20:50.068553 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.000829 (+932276us) service_pool.cc:227] Handling call
0809 23:20:51.258832 (+258003us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.192682 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:20:51.265545 24543 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:50.982678   999 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.269734 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:50.993243  1205 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:51.407691 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 213624us
I20260809 23:20:51.397753 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:51.232786  1212 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:51.440611 25321 leader_election.cc:334] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.692s (SENT)
I20260809 23:20:51.419457 24544 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:51.397405 24427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.031s	sys 0.052s
W20260809 23:20:51.442443 24427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.033s	sys 0.056s
W20260809 23:20:51.423437 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.003s	sys 0.013s
I20260809 23:20:51.406869 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3213) took 3196 ms. Trace:
I20260809 23:20:51.216126  1169 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:20:51.266892 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74814.4us
W20260809 23:20:51.244028  1199 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.807s	user 0.010s	sys 0.000s
W20260809 23:20:51.460709 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.002s	sys 0.019s
W20260809 23:20:51.461056 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.002s	sys 0.019s
I20260809 23:20:51.437256 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:20:51.462191 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2733) took 1318 ms. Trace:
I20260809 23:20:51.462630 24435 rpcz_store.cc:276] 0809 23:20:50.143725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.261038 (+1117313us) service_pool.cc:227] Handling call
0809 23:20:51.462108 (+201070us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:51.409315 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.020s	sys 0.021s
W20260809 23:20:51.464262 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.025s	sys 0.022s
I20260809 23:20:51.460371 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3209) took 3340 ms. Trace:
I20260809 23:20:51.466678 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:51.443387 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.005s	sys 0.014s
I20260809 23:20:51.274506  1167 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:51.443454 24431 rpcz_store.cc:276] 0809 23:20:48.210876 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.211151 (+   275us) service_pool.cc:227] Handling call
0809 23:20:48.262052 (+ 50901us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:48.505540 (+243488us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.406505 (+2900965us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54003bae54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.406783 (+   278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":535261568}
I20260809 23:20:51.435261 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:51.405761  1036 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.233s	user 0.011s	sys 0.000s
W20260809 23:20:51.317618  1233 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.744s	user 0.001s	sys 0.000s
I20260809 23:20:51.472576 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:51.472929 25371 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260809 23:20:51.267300  1063 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.018s	sys 0.044s
I20260809 23:20:51.496982 24542 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:20:51.497246 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:51.416718   565 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.001s	sys 0.014s
W20260809 23:20:51.454834  1236 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.723s	user 0.001s	sys 0.001s
I20260809 23:20:51.443097   877 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:51.465979 24427 rpcz_store.cc:276] 0809 23:20:48.120128 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.123657 (+  3529us) service_pool.cc:227] Handling call
0809 23:20:48.152741 (+ 29084us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:48.389516 (+236775us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:48.389553 (+    37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:48.390205 (+   652us) log.cc:842] Serialized 52 byte log entry
0809 23:20:48.532767 (+142562us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:50.635052 (+2102285us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b18002729e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:20:50.635196 (+   144us) log.cc:842] Serialized 14 byte log entry
0809 23:20:51.442910 (+807714us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.443027 (+   117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:51.443079 (+    52us) raft_consensus.cc:1671] finished
0809 23:20:51.460134 (+ 17055us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:51.460309 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29525,"prepare.queue_time_us":885456,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":170144,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":5876,"spinlock_wait_cycles":8045312,"thread_start_us":1136653,"threads_started":4,"wal-append.queue_time_us":215655}
I20260809 23:20:51.498170 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3224) took 1381 ms. Trace:
I20260809 23:20:51.499500 25366 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:51.511178 24435 rpcz_store.cc:276] 0809 23:20:50.116348 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.463519 (+1347171us) service_pool.cc:227] Handling call
0809 23:20:51.498095 (+ 34576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.504242 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:51.513041 25363 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:51.509963 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:51.514458 24431 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:51.518977 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:51.466809  1199 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.519384 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2737) took 1010 ms. Trace:
I20260809 23:20:51.518776 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:51.640810  1050 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 3.663s	user 0.008s	sys 0.008s
I20260809 23:20:51.440367   326 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:51.641146  1270 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:51.641862  1233 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.126s	user 0.002s	sys 0.000s
W20260809 23:20:51.642307  1245 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.614s	user 0.003s	sys 0.000s
I20260809 23:20:51.642490   706 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:51.644064 24435 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:51.685330 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2709) took 4853 ms. Trace:
I20260809 23:20:51.717710 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1201) took 1411 ms. Trace:
I20260809 23:20:51.644505 24431 rpcz_store.cc:276] 0809 23:20:50.508765 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.478350 (+969585us) service_pool.cc:227] Handling call
0809 23:20:51.519304 (+ 40954us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:51.506194   530 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.290s	user 0.013s	sys 0.000s
I20260809 23:20:51.731328   530 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.509637  1114 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:51.650302 24544 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:51.650352 25363 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:51.683352  1098 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:51.707232  1249 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:51.710557  1177 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:51.713294 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2727) took 2927 ms (client timeout 3345 ms). Trace:
I20260809 23:20:51.718255 24435 rpcz_store.cc:276] 0809 23:20:50.306720 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.512619 (+1205899us) service_pool.cc:227] Handling call
0809 23:20:51.717631 (+205012us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.648095  1165 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:51.885618 24423 rpcz_store.cc:269] 0809 23:20:48.786258 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.431249 (+1644991us) service_pool.cc:227] Handling call
0809 23:20:51.713106 (+1281857us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:51.713234 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116255104}
I20260809 23:20:51.887457 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:51.645716   624 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:51.716099 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1351) took 2595 ms. Trace:
I20260809 23:20:51.727468 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:51.729403 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1359) took 1667 ms. Trace:
I20260809 23:20:51.896219 24543 rpcz_store.cc:276] 0809 23:20:50.061833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.184891 (+123058us) service_pool.cc:227] Handling call
0809 23:20:50.207290 (+ 22399us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:51.273022 (+1065732us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.729163 (+456141us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400147654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.729335 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2383466240}
I20260809 23:20:51.730870  1255 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:20:51.732512  1166 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.329s	user 0.004s	sys 0.008s
I20260809 23:20:51.733557  1199 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:51.735316  1119 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:20:51.735833  1250 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.298s	user 0.001s	sys 0.002s
I20260809 23:20:51.737978 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3144) took 2013 ms. Trace:
I20260809 23:20:51.907042 24545 rpcz_store.cc:276] 0809 23:20:49.724496 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.891901 (+167405us) service_pool.cc:227] Handling call
0809 23:20:51.737807 (+1845906us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001fd954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:51.737915 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316865792}
I20260809 23:20:51.908960 24545 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260809 23:20:51.914096 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3148) took 1694 ms. Trace:
I20260809 23:20:51.496419   849 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.220s
I20260809 23:20:51.739902   684 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:51.740193   884 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:51.742096  1079 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:51.742892  1223 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:51.503463   565 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.004s	sys 0.014s
I20260809 23:20:51.747368  1266 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:51.750188 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1443) took 3970 ms. Trace:
W20260809 23:20:51.751030 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2711) took 4918 ms (client timeout 2506 ms). Trace:
I20260809 23:20:51.751120 24429 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:51.519599   877 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:51.941242 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 429400us
W20260809 23:20:51.752156  1065 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.012s	user 0.005s	sys 0.005s
I20260809 23:20:51.752082 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3145) took 1533 ms. Trace:
W20260809 23:20:51.520599  1001 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.018s	user 0.012s	sys 0.000s
W20260809 23:20:51.536873   970 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.373s	user 0.010s	sys 0.003s
I20260809 23:20:51.980475  1001 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:51.937019   434 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.339s	user 0.011s	sys 0.005s
I20260809 23:20:51.793804 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:51.978964 24425 rpcz_store.cc:269] 0809 23:20:46.832688 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.837108 (+  4420us) service_pool.cc:227] Handling call
0809 23:20:51.750741 (+4913633us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:51.750943 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2847080704}
I20260809 23:20:51.518656  1247 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:51.982465  1247 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.518883  1252 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:51.984673 24544 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:20:51.885905  1098 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:51.717705 24424 rpcz_store.cc:276] 0809 23:20:46.831950 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:46.834322 (+  2372us) service_pool.cc:227] Handling call
0809 23:20:46.834739 (+   417us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:46.853997 (+ 19258us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:46.854044 (+    47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:46.854948 (+   904us) log.cc:842] Serialized 52 byte log entry
0809 23:20:47.500960 (+646012us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:47.501005 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:47.501077 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:51.433900 (+3932823us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:51.684961 (+251061us) raft_consensus.cc:1671] finished
0809 23:20:51.684990 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:51.685245 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11492,"raft.queue_time_us":20435,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":411,"spinlock_wait_cycles":43711744,"thread_start_us":614442,"threads_started":1,"wal-append.queue_time_us":1035810}
I20260809 23:20:51.646548  1175 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:51.756551  1015 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.937s	user 0.007s	sys 0.003s
I20260809 23:20:51.519168  1214 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:51.679824  1270 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:20:51.880959  1186 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.141s	user 0.010s	sys 0.001s
W20260809 23:20:51.997944 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.009s	sys 0.006s
I20260809 23:20:51.506435   748 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:51.893203 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 415046us
I20260809 23:20:51.895735 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2742) took 1162 ms. Trace:
I20260809 23:20:52.001374 24431 rpcz_store.cc:276] 0809 23:20:50.733062 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.719007 (+985945us) service_pool.cc:227] Handling call
0809 23:20:51.895663 (+176656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.895169 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2741) took 1192 ms. Trace:
I20260809 23:20:52.001957 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:52.002305 24423 rpcz_store.cc:276] 0809 23:20:50.703127 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.887120 (+1183993us) service_pool.cc:227] Handling call
0809 23:20:51.895107 (+  7987us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.511140  1167 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.893973 24540 rpcz_store.cc:276] 0809 23:20:49.120523 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.120679 (+   156us) service_pool.cc:227] Handling call
0809 23:20:49.252252 (+131573us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:49.811945 (+559693us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:49.811996 (+    51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:49.822281 (+ 10285us) log.cc:842] Serialized 26 byte log entry
0809 23:20:50.007124 (+184843us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:50.007175 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.715696 (+1708521us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.715751 (+    55us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:51.715803 (+    52us) raft_consensus.cc:1671] finished
0809 23:20:51.715815 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:51.716037 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1286482816,"thread_start_us":145451,"threads_started":1,"wal-append.queue_time_us":266485,"wal-append.run_cpu_time_us":8719,"wal-append.run_wall_time_us":1099298}
I20260809 23:20:51.900833 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:51.915333 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33980 (request call id 7) took 4302 ms. Trace:
W20260809 23:20:52.002389 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.010s	sys 0.005s
W20260809 23:20:52.024927 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.011s	sys 0.006s
W20260809 23:20:52.025681 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 590469us
I20260809 23:20:51.917117 24545 rpcz_store.cc:276] 0809 23:20:50.219807 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.908216 (+1688409us) service_pool.cc:227] Handling call
0809 23:20:51.908663 (+   447us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:51.913742 (+  5079us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.914024 (+   282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.026768 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:52.028261 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:52.028569 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3029) took 1793 ms. Trace:
W20260809 23:20:51.918229 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.060s	sys 0.151s
W20260809 23:20:51.738197  1235 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.215s	user 0.003s	sys 0.008s
I20260809 23:20:51.905978   971 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:51.932072  1260 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:51.937667 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 471101us
I20260809 23:20:51.919956  1119 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:51.979570 24542 rpcz_store.cc:276] 0809 23:20:50.219014 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.280541 (+ 61527us) service_pool.cc:227] Handling call
0809 23:20:50.280842 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:51.497426 (+1216584us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.751843 (+254417us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.752020 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2796039936}
I20260809 23:20:51.980546   970 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:51.563119  1237 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.776s	user 0.001s	sys 0.000s
I20260809 23:20:51.490971  1026 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:51.981941 24427 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:51.935388  1178 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.015s	sys 0.044s
I20260809 23:20:51.903170   624 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:51.654227  1207 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.223s	user 0.004s	sys 0.008s
I20260809 23:20:52.008965   748 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:51.996421  1015 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.001389  1186 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:51.990278 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.001s	sys 0.013s
I20260809 23:20:51.993767 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3212) took 3847 ms. Trace:
W20260809 23:20:51.998502 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.009s	sys 0.006s
I20260809 23:20:51.941344   999 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.029659 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:52.165871 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1448) took 3666 ms (client timeout 2191 ms). Trace:
W20260809 23:20:52.235219 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.003s	sys 0.023s
W20260809 23:20:52.235296 25368 rpcz_store.cc:269] 0809 23:20:48.499449 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.501493 (+  2044us) service_pool.cc:227] Handling call
0809 23:20:52.165549 (+3664056us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:52.165722 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3101776640}
W20260809 23:20:52.047042 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.060s	sys 0.200s
W20260809 23:20:52.237315 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.829s	user 0.060s	sys 0.204s
W20260809 23:20:52.181926 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1342) took 4655 ms (client timeout 2494 ms). Trace:
I20260809 23:20:52.186866 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1168) took 3543 ms. Trace:
I20260809 23:20:52.242440 24429 rpcz_store.cc:276] 0809 23:20:48.643378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.643530 (+   152us) service_pool.cc:227] Handling call
0809 23:20:51.750913 (+3107383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:51.987063 (+236150us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.186594 (+199531us) spinlock_profiling.cc:243] Waited 3.06 s on lock 0x7b54003331d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:52.186788 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7026694912}
I20260809 23:20:52.047840 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:52.001919  1001 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.233428 24422 rpcz_store.cc:276] 0809 23:20:48.146685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.146841 (+   156us) service_pool.cc:227] Handling call
0809 23:20:48.181916 (+ 35075us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:48.494990 (+313074us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:48.495026 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:48.512648 (+ 17622us) log.cc:842] Serialized 26 byte log entry
0809 23:20:48.728709 (+216061us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:48.728793 (+    84us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.984836 (+3256043us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.984908 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:51.984979 (+    71us) raft_consensus.cc:1671] finished
0809 23:20:51.984996 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:51.993715 (+  8719us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11802,"raft.queue_time_us":22736,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":9782,"spinlock_wait_cycles":691754368,"thread_start_us":173248,"threads_started":1,"wal-append.queue_time_us":901700,"wal-append.run_cpu_time_us":1664,"wal-append.run_wall_time_us":1003198}
I20260809 23:20:52.000222 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.029104 24545 rpcz_store.cc:276] 0809 23:20:50.235130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.027601 (+1792471us) service_pool.cc:227] Handling call
0809 23:20:52.027968 (+   367us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.028183 (+   215us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.028218 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.028274 (+    56us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:52.028316 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.028390 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.028506 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.761561  1212 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:52.240453 24534 rpcz_store.cc:269] 0809 23:20:47.526395 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.575649 (+ 49254us) service_pool.cc:227] Handling call
0809 23:20:52.181650 (+4606001us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b540029cf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:52.181813 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3891263744}
I20260809 23:20:51.876777  1259 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:52.003705 24435 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:52.003609 25375 rpcz_store.cc:276] 0809 23:20:47.604308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.604520 (+   212us) service_pool.cc:227] Handling call
0809 23:20:51.906049 (+4301529us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:51.906521 (+   472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1840544640}
W20260809 23:20:52.311337 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2736) took 1803 ms (client timeout 2076 ms). Trace:
W20260809 23:20:52.010443 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.014s	sys 0.002s
W20260809 23:20:52.311754 24435 rpcz_store.cc:269] 0809 23:20:50.508435 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.886285 (+1377850us) service_pool.cc:227] Handling call
0809 23:20:52.311271 (+424986us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:52.312259 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.305s	user 0.014s	sys 0.002s
I20260809 23:20:52.017150 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.313310 24431 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:51.999264  1252 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:52.005795  1261 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.530s	user 0.000s	sys 0.001s
W20260809 23:20:52.326719 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1197) took 2071 ms (client timeout 2610 ms). Trace:
W20260809 23:20:52.327220 24424 rpcz_store.cc:269] 0809 23:20:50.255636 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.991019 (+1735383us) service_pool.cc:227] Handling call
0809 23:20:52.326650 (+335631us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.008490 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.020613 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3128) took 4073 ms. Trace:
W20260809 23:20:52.016181 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.000s	sys 0.015s
W20260809 23:20:52.329161 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.000s	sys 0.016s
W20260809 23:20:52.329948 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 857423us
W20260809 23:20:52.330168 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2731) took 2343 ms (client timeout 3068 ms). Trace:
W20260809 23:20:52.330654 24423 rpcz_store.cc:269] 0809 23:20:49.987048 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.008206 (+2021158us) service_pool.cc:227] Handling call
0809 23:20:52.330098 (+321892us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:52.071013  1104 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 3.252s	user 0.004s	sys 0.006s
I20260809 23:20:51.932411  1165 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.006757  1114 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.021481 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3150) took 1790 ms. Trace:
I20260809 23:20:52.345496 24540 rpcz_store.cc:276] 0809 23:20:50.231554 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.008105 (+1776551us) service_pool.cc:227] Handling call
0809 23:20:52.020972 (+ 12867us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.021179 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.021201 (+    22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.021240 (+    39us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:52.021272 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.021322 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.021423 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:51.993283  1205 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:51.935629   884 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:52.003913 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:51.999214   717 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.004s	sys 0.011s
W20260809 23:20:52.029891 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.060s	sys 0.151s
I20260809 23:20:51.988881  1065 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:52.018646  1081 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.004s	sys 0.011s
I20260809 23:20:52.319232   999 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:51.985579  1266 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:51.997634  1214 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:52.019465   565 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.046759 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1463) took 1943 ms. Trace:
W20260809 23:20:52.358158   717 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.004s	sys 0.011s
I20260809 23:20:52.187657 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3021) took 2555 ms. Trace:
W20260809 23:20:52.163009 24430 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2718) took 3906 ms (client timeout 1965 ms). Trace:
W20260809 23:20:52.414629 24430 rpcz_store.cc:269] 0809 23:20:48.256119 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.256356 (+   237us) service_pool.cc:227] Handling call
0809 23:20:52.162674 (+3906318us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:52.162867 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3381752704}
I20260809 23:20:52.307540 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3156) took 1859 ms. Trace:
I20260809 23:20:52.416601 24534 rpcz_store.cc:276] 0809 23:20:50.448084 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.306481 (+1858397us) service_pool.cc:227] Handling call
0809 23:20:52.306863 (+   382us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.307131 (+   268us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.307169 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.307225 (+    56us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:52.307268 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.307332 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.307474 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:52.308635 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 260894us
I20260809 23:20:52.299947 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:52.312695 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.30615e+06us
I20260809 23:20:51.985442 25372 rpcz_store.cc:276] 0809 23:20:47.779863 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.831350 (+ 51487us) service_pool.cc:227] Handling call
0809 23:20:47.891656 (+ 60306us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:50.081828 (+2190172us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.749944 (+1668116us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:51.750120 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4913190272}
W20260809 23:20:52.127928  1134 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 3.151s	user 0.000s	sys 0.014s
W20260809 23:20:52.147331  1250 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260809 23:20:51.997965  1270 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:52.328820 24546 rpcz_store.cc:276] 0809 23:20:47.947634 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.154716 (+207082us) service_pool.cc:227] Handling call
0809 23:20:48.155160 (+   444us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:49.293713 (+1138553us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.020255 (+2726542us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.020532 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2579065600}
W20260809 23:20:52.268301  1178 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.285s	user 0.036s	sys 0.059s
I20260809 23:20:52.447736  1121 raft_consensus.cc:2741] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260809 23:20:52.337303 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.448879 24422 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260809 23:20:52.449340 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2740) took 1788 ms. Trace:
I20260809 23:20:52.449692 24422 rpcz_store.cc:276] 0809 23:20:50.661263 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.304560 (+1643297us) service_pool.cc:227] Handling call
0809 23:20:52.449270 (+144710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.132014  1244 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:52.451332  1244 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:52.452637  1192 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.619s	user 0.003s	sys 0.000s
I20260809 23:20:52.452435  1244 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:52.454917  1230 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.958s	user 0.008s	sys 0.015s
I20260809 23:20:52.454208  1244 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:52.456784 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.457811 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:52.334806  1252 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.450695 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.459853 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:52.153738  1017 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:20:52.034025   530 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.342531   624 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.343343  1140 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:52.344166 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.464341 24435 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:52.129462  1303 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:52.035148   971 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.468819 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.469349   971 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.471660   971 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.473096 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.131337  1098 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:52.494295 24378 connection.cc:232] RPC call timeout handler was delayed by -3.0509063717909157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
I20260809 23:20:52.348690 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.148288  1127 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.068537  1260 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.163543  1143 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:52.074716  1159 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:52.044045  1302 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:52.032446  1220 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.245s	user 0.012s	sys 0.000s
I20260809 23:20:52.322302  1026 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:52.365015 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.529434 24424 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:20:52.150352  1075 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.206s
I20260809 23:20:51.934013  1253 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.370296 24314 rpcz_store.cc:276] 0809 23:20:50.103123 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.103246 (+   123us) service_pool.cc:227] Handling call
0809 23:20:50.110723 (+  7477us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:50.112991 (+  2268us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:50.113020 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:50.113452 (+   432us) log.cc:842] Serialized 26 byte log entry
0809 23:20:50.248520 (+135068us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:50.248571 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:50.248663 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:51.697517 (+1448854us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:51.737772 (+ 40255us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b240019ba58. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:20:52.046462 (+308690us) raft_consensus.cc:1671] finished
0809 23:20:52.046493 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.046680 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47176832,"thread_start_us":134485,"threads_started":1,"wal-append.queue_time_us":258944}
I20260809 23:20:52.181174 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 19) took 2232 ms. Trace:
I20260809 23:20:52.562464 25369 rpcz_store.cc:276] 0809 23:20:49.948422 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.948569 (+   147us) service_pool.cc:227] Handling call
0809 23:20:52.180654 (+2232085us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5400404820. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f5dc0 00007fac2385a9c2 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.181057 (+   403us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33619840}
I20260809 23:20:52.375185  1290 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:52.375866  1291 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:52.376048  1295 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:52.379258  1065 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.380424  1266 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.627614   434 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.357908 32417 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:52.391486  1254 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.181s	user 0.005s	sys 0.000s
I20260809 23:20:52.396541  1119 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:52.335701  1259 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:52.362048  1081 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.005s	sys 0.011s
I20260809 23:20:52.402772   877 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.182355 24539 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
W20260809 23:20:52.049474  1251 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260809 23:20:51.999805 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.440702 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3035) took 1886 ms. Trace:
I20260809 23:20:52.663580 24534 rpcz_store.cc:276] 0809 23:20:50.554545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.417756 (+1863211us) service_pool.cc:227] Handling call
0809 23:20:52.440160 (+ 22404us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.440384 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.440407 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.440443 (+    36us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:52.440472 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.440520 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.440641 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.418533 24429 raft_consensus.cc:2362] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17. Re-sending same reply.
I20260809 23:20:52.366115   999 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:52.460023   726 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:52.173991   969 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.303704  1207 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.461320 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.460067 24422 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:52.462495 25373 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:52.412528 24535 rpcz_store.cc:276] 0809 23:20:49.632386 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:49.982415 (+350029us) service_pool.cc:227] Handling call
0809 23:20:50.004592 (+ 22177us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:50.058754 (+ 54162us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:50.059051 (+   297us) log.cc:842] Serialized 14 byte log entry
0809 23:20:50.234825 (+175774us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:50.234986 (+   161us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:50.235061 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.187370 (+1952309us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.187443 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.187589 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":728,"prepare.queue_time_us":6818,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":44,"raft.queue_time_us":158,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":20794,"spinlock_wait_cycles":124000512,"thread_start_us":175171,"threads_started":1,"wal-append.queue_time_us":269362,"wal-append.run_cpu_time_us":2304,"wal-append.run_wall_time_us":1394411}
I20260809 23:20:52.465075 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1362) took 2279 ms. Trace:
I20260809 23:20:52.679746 24541 rpcz_store.cc:276] 0809 23:20:50.185951 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.279434 (+ 93483us) service_pool.cc:227] Handling call
0809 23:20:50.279766 (+   332us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:50.284756 (+  4990us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:50.284786 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:50.289909 (+  5123us) log.cc:842] Serialized 26 byte log entry
0809 23:20:50.451676 (+161767us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:50.451732 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:50.451825 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:52.464874 (+2013049us) raft_consensus.cc:1671] finished
0809 23:20:52.464896 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.465017 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4502,"raft.queue_time_us":28231,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":161245,"threads_started":1,"wal-append.queue_time_us":350186}
I20260809 23:20:52.464535 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.460660  1017 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:52.473712  1214 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.474630  1247 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:52.689052 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:52.689320 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1191) took 2449 ms (client timeout 2494 ms). Trace:
W20260809 23:20:52.689700 24425 rpcz_store.cc:269] 0809 23:20:50.233117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.990767 (+1757650us) service_pool.cc:227] Handling call
0809 23:20:52.682723 (+691956us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.478117 24431 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:52.690559 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3227) took 1274 ms. Trace:
W20260809 23:20:52.690822 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1196) took 2448 ms (client timeout 2766 ms). Trace:
I20260809 23:20:52.651870  1186 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.492197  1296 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:52.492760 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.403517   717 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:52.067924  1289 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.495872 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1199) took 2211 ms. Trace:
I20260809 23:20:52.703109 24423 rpcz_store.cc:276] 0809 23:20:50.284152 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.348420 (+2064268us) service_pool.cc:227] Handling call
0809 23:20:52.495795 (+147375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.704237 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.704968 24423 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:52.705969 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:52.355432  1114 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.164139   682 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:52.501417 24378 connection.cc:232] RPC call timeout handler was delayed by -2.3171256210189313s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260809 23:20:52.497597 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:52.707185 24378 connection.cc:232] RPC call timeout handler was delayed by -1.9517256210092455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2191s for a response
W20260809 23:20:52.707515 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2992256209254265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2916s for a response
I20260809 23:20:52.700299 25365 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:52.707988 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:52.645501 32417 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.405452   565 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.561139 25375 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:52.499696  1141 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:52.505231   809 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.011s	sys 0.001s
I20260809 23:20:52.576749 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:52.502087  1049 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.501943  1001 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:52.552603  1199 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.009s	sys 0.028s
I20260809 23:20:52.581136 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.725579 24311 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:52.653610  1252 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:52.581964 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2734) took 2437 ms (client timeout 3104 ms). Trace:
I20260809 23:20:52.709333   682 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:52.729207 24424 rpcz_store.cc:269] 0809 23:20:50.144243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.351493 (+2207250us) service_pool.cc:227] Handling call
0809 23:20:52.581891 (+230398us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.505015   530 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.713886   877 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:52.629594 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.388060   884 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.507463  1143 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:52.523620  1159 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:52.729722  1199 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.009s	sys 0.028s
I20260809 23:20:52.638326   434 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.475692  1140 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:52.632839   509 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s	user 0.022s	sys 0.080s
W20260809 23:20:52.768383   509 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.031s	user 0.022s	sys 0.081s
I20260809 23:20:52.770387 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1465) took 2666 ms. Trace:
W20260809 23:20:52.665343 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1193) took 2431 ms (client timeout 2836 ms). Trace:
W20260809 23:20:52.752942  1239 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.019s	sys 0.021s
W20260809 23:20:52.771087 24429 rpcz_store.cc:269] 0809 23:20:50.233619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.243812 (+2010193us) service_pool.cc:227] Handling call
0809 23:20:52.665270 (+421458us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:52.771492  1239 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.019s	sys 0.022s
I20260809 23:20:52.772539 24429 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 13. Current term is 14. Ops: []
I20260809 23:20:52.773216 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1171) took 4128 ms. Trace:
I20260809 23:20:52.773535 24429 rpcz_store.cc:276] 0809 23:20:48.644446 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.771960 (+4127514us) service_pool.cc:227] Handling call
0809 23:20:52.772320 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.772950 (+   630us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.773140 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.666415 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.775779 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.650918  1015 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.684152 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3037) took 2092 ms. Trace:
I20260809 23:20:52.776889 24534 rpcz_store.cc:276] 0809 23:20:50.592092 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.664893 (+2072801us) service_pool.cc:227] Handling call
0809 23:20:52.683515 (+ 18622us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.683753 (+   238us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.683790 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.683845 (+    55us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:52.683897 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.683964 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.684085 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.732308   682 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:52.778079 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:52.552278  1220 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.690984 24430 rpcz_store.cc:276] 0809 23:20:51.416279 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.416685 (+1000406us) service_pool.cc:227] Handling call
0809 23:20:52.690498 (+273813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.132647  1167 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:52.691218 24431 rpcz_store.cc:269] 0809 23:20:50.242584 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.002146 (+1759562us) service_pool.cc:227] Handling call
0809 23:20:52.690758 (+688612us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.494529  1303 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.654856  1081 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:52.671262   999 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:52.664639   970 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:52.759428  1305 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260809 23:20:52.708623  1296 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.708802 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.709322 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:52.503098 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:52.582878  1290 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.712165 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:52.714881 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 685250us
W20260809 23:20:52.803587 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94343.9us
I20260809 23:20:52.560524  1302 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:52.718146 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1202) took 1252 ms. Trace:
I20260809 23:20:52.804574 24435 rpcz_store.cc:276] 0809 23:20:51.466259 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.327203 (+860944us) service_pool.cc:227] Handling call
0809 23:20:52.718045 (+390842us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.581918 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.505082  1113 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.683416   969 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:52.627552  1291 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.736058  1252 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:52.737866 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:52.717250   809 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.011s	sys 0.002s
W20260809 23:20:52.740715 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.002s	sys 0.018s
I20260809 23:20:52.579110  1253 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:52.720031   497 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.002s	sys 0.025s
W20260809 23:20:52.831197   497 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.002s	sys 0.026s
W20260809 23:20:52.742297 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 738433us
I20260809 23:20:52.832283 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:52.636787 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.005s	sys 0.009s
I20260809 23:20:52.833285  1220 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:52.833797  1317 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260809 23:20:52.752370 24316 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:52.820317  1113 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.840689  1113 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.732524  1001 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.844146 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:52.844935 24316 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 18
I20260809 23:20:52.665637  1259 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } attrs { replace: true } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.753271 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.733019  1287 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.301s
I20260809 23:20:52.754506 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:52.756189 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 190004us
W20260809 23:20:52.865417 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.494s (SENT)
I20260809 23:20:52.865940 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:52.743314   546 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.008s	sys 0.011s
W20260809 23:20:52.763682  1308 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260809 23:20:52.763473  1143 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.765980  1101 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.912228 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3153) took 2657 ms. Trace:
I20260809 23:20:52.767016   434 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:52.743101  1247 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:52.767167   562 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:52.937798   562 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.938571  1143 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:52.782582 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:52.783123 24429 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 16. Ops: []
I20260809 23:20:52.941031 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1172) took 4296 ms. Trace:
I20260809 23:20:52.898797  1259 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.941421 24429 rpcz_store.cc:276] 0809 23:20:48.644742 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.774434 (+4129692us) service_pool.cc:227] Handling call
0809 23:20:52.774747 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.940678 (+165931us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.940961 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.783776 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:52.900175  1255 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Raft consensus shutting down.
I20260809 23:20:52.943220 24321 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:52.785426 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1176) took 4092 ms. Trace:
I20260809 23:20:52.947108 24431 rpcz_store.cc:276] 0809 23:20:48.692703 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.784604 (+4091901us) service_pool.cc:227] Handling call
0809 23:20:52.784886 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.785093 (+   207us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:52.785120 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.785169 (+    49us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:52.785205 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.785265 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.785370 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.948395  1291 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.948621 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:52.770792 24320 rpcz_store.cc:276] 0809 23:20:50.104373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.104502 (+   129us) service_pool.cc:227] Handling call
0809 23:20:50.109044 (+  4542us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:50.242282 (+133238us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:50.242328 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:50.242861 (+   533us) log.cc:842] Serialized 26 byte log entry
0809 23:20:50.743858 (+500997us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:20:51.217962 (+474104us) log.cc:842] Serialized 14 byte log entry
0809 23:20:52.235972 (+1018010us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.769948 (+533976us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54003854d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.770002 (+    54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:52.770055 (+    53us) raft_consensus.cc:1671] finished
0809 23:20:52.770164 (+   109us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.770317 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110679,"prepare.queue_time_us":913588,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":277918,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":25634,"spinlock_wait_cycles":305308928,"thread_start_us":1889787,"threads_started":3,"wal-append.queue_time_us":939188}
I20260809 23:20:52.741829   748 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.697768 24535 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260809 23:20:52.753520  1159 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:52.798138 24430 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:52.700996 24428 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17.
I20260809 23:20:52.751911  1165 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.724925  1049 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.752735   877 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:52.922492   300 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:52.719678  1297 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.324s
W20260809 23:20:52.814476 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.002s	sys 0.019s
I20260809 23:20:52.947139  1207 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.784814  1295 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:52.824748  1115 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:52.827407  1296 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.827694 32741 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:20:52.833535 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.006s	sys 0.009s
I20260809 23:20:53.008934 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3147) took 2789 ms. Trace:
W20260809 23:20:53.009145 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203821us
W20260809 23:20:52.733511  1112 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.494s	user 0.013s	sys 0.003s
I20260809 23:20:52.853917 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:53.013578 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 159756us
I20260809 23:20:52.714555  1289 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.753768 24423 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:52.840860 32417 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:52.869439 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 65402us
I20260809 23:20:52.794133   717 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.870419  1253 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:52.859760 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3155) took 2551 ms. Trace:
I20260809 23:20:52.799371   509 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
I20260809 23:20:52.872241  1302 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.873236   565 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:52.840619  1248 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Stepping down as leader of term 12
I20260809 23:20:52.875542   965 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:52.880286 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:52.743197   530 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.894663  1220 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:52.896131  1198 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:52.890424 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 135998us
W20260809 23:20:52.903877 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.836s (SENT)
W20260809 23:20:53.078151  1036 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.004s	sys 0.028s
W20260809 23:20:52.898730  1300 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.521s	user 0.004s	sys 0.004s
I20260809 23:20:52.918078 24544 rpcz_store.cc:276] 0809 23:20:50.255032 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.183733 (+1928701us) service_pool.cc:227] Handling call
0809 23:20:52.184204 (+   471us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.184486 (+   282us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:52.653530 (+469044us) log.cc:842] Serialized 14 byte log entry
0809 23:20:52.911765 (+258235us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.911839 (+    74us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:52.911913 (+    74us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.912001 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.912153 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30488,"prepare.queue_time_us":518274,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":4221,"raft.queue_time_us":31523,"thread_start_us":695586,"threads_started":2}
I20260809 23:20:53.051535  1290 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:52.891549  1312 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.171s	user 0.001s	sys 0.001s
W20260809 23:20:52.904420   546 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.009s	sys 0.011s
I20260809 23:20:52.743695  1017 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.952456 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:52.939368 24534 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:52.954380 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3121) took 5088 ms. Trace:
I20260809 23:20:53.108419 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.111116 24539 rpcz_store.cc:276] 0809 23:20:47.833923 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:48.262697 (+428774us) service_pool.cc:227] Handling call
0809 23:20:52.182101 (+3919404us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.667711 (+485610us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.922406 (+254695us) spinlock_profiling.cc:243] Waited 3.89 s on lock 0x7b540029cf50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:52.922591 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8957627264}
I20260809 23:20:52.957551 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1478) took 2383 ms. Trace:
I20260809 23:20:53.112095 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:53.095691  1036 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.004s	sys 0.028s
I20260809 23:20:53.023152  1112 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.956579 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3038) took 2169 ms. Trace:
I20260809 23:20:52.957260 24316 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:52.978837 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1175) took 4286 ms. Trace:
I20260809 23:20:53.119518 24535 rpcz_store.cc:276] 0809 23:20:50.787269 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.679246 (+1891977us) service_pool.cc:227] Handling call
0809 23:20:52.697301 (+ 18055us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.956257 (+258956us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.956503 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.119700 24430 rpcz_store.cc:276] 0809 23:20:48.692392 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.779482 (+4087090us) service_pool.cc:227] Handling call
0809 23:20:52.797965 (+ 18483us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.978446 (+180481us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.978751 (+   305us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:52.806278 24312 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
W20260809 23:20:52.998004 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 993245us
I20260809 23:20:53.079701   686 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260809 23:20:53.000604 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1479) took 1530 ms. Trace:
I20260809 23:20:53.006261 24424 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:52.984087 32640 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:53.009392 24543 rpcz_store.cc:276] 0809 23:20:50.219566 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.901268 (+1681702us) service_pool.cc:227] Handling call
0809 23:20:51.913284 (+ 12016us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:51.928656 (+ 15372us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:51.928704 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:51.929251 (+   547us) log.cc:842] Serialized 26 byte log entry
0809 23:20:53.008493 (+1079242us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:53.008549 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.008643 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:53.008701 (+    58us) raft_consensus.cc:1671] finished
0809 23:20:53.008718 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.008865 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":69940,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3557,"thread_start_us":1021651,"threads_started":1,"wal-append.queue_time_us":883939}
I20260809 23:20:53.007459  1295 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:52.985281  1140 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.957957  1247 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:53.029138 24545 rpcz_store.cc:276] 0809 23:20:50.308648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.310740 (+2002092us) service_pool.cc:227] Handling call
0809 23:20:52.311106 (+   366us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:52.319212 (+  8106us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.319243 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:52.341104 (+ 21861us) log.cc:842] Serialized 26 byte log entry
0809 23:20:52.664731 (+323627us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:52.664765 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.832483 (+167718us) spinlock_profiling.cc:243] Waited 7.84 ms on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.832547 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:52.842086 (+  9539us) raft_consensus.cc:1671] finished
0809 23:20:52.842118 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.859672 (+ 17554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18029824,"thread_start_us":323121,"threads_started":1,"wal-append.queue_time_us":494637}
I20260809 23:20:53.074288 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:53.074402 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1170) took 4430 ms. Trace:
W20260809 23:20:53.078548 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53079.8us
I20260809 23:20:53.023860  1297 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:53.119529  1303 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:52.984637  1165 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:53.091145 24311 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:53.028112  1176 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:53.175978 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:20:53.184219 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:53.094671 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 190820us
I20260809 23:20:53.197002 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.141912 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:53.198065 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.766s (SENT) [suppressed 15 similar messages]
I20260809 23:20:52.813094  1265 raft_consensus.cc:2741] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260809 23:20:52.855641  1260 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:53.097396 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:20:53.210945 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 116695us
I20260809 23:20:53.079926  1248 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:53.112437 24313 rpcz_store.cc:276] 0809 23:20:50.573836 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.573997 (+   161us) service_pool.cc:227] Handling call
0809 23:20:50.574343 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:50.728158 (+153815us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:51.225772 (+497614us) log.cc:842] Serialized 14 byte log entry
0809 23:20:52.233665 (+1007893us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.233730 (+    65us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:52.233795 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.898071 (+664276us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:52.957324 (+ 59253us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:52.957381 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:52.957495 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1203,"prepare.queue_time_us":835391,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":142069,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":161,"spinlock_wait_cycles":411036416,"thread_start_us":1488848,"threads_started":2,"wal-append.queue_time_us":1151656}
I20260809 23:20:52.942551 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.212982 24429 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:53.211315   984 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
W20260809 23:20:53.213465  1344 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260809 23:20:53.214008 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:53.214679 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3034) took 2725 ms. Trace:
I20260809 23:20:53.120172 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.215142 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:53.215613   530 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.215941 24316 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:20:53.216138  1306 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.820s	user 0.003s	sys 0.000s
I20260809 23:20:53.216367  1049 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.216776  1112 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:53.217133  1181 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:20:53.216912  1297 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:53.218070  1342 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260809 23:20:52.994802 24321 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:20:53.093838  1198 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:52.820042 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2743) took 1349 ms. Trace:
I20260809 23:20:53.219545  1297 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:53.219761 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.219702 24422 rpcz_store.cc:276] 0809 23:20:51.470982 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.692835 (+1221853us) service_pool.cc:227] Handling call
0809 23:20:52.819965 (+127130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.220129 24423 rpcz_store.cc:276] 0809 23:20:48.644162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.749457 (+4105295us) service_pool.cc:227] Handling call
0809 23:20:52.749881 (+   424us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.051790 (+301909us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.074343 (+ 22553us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.220942 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.221167 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.221565 24422 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:53.221868 24423 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:53.221167 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.222469 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:53.224828 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 150598us
I20260809 23:20:53.016515 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:53.131887 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2738) took 2622 ms (client timeout 1907 ms). Trace:
W20260809 23:20:53.233008 24427 rpcz_store.cc:269] 0809 23:20:50.509093 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.518684 (+1009591us) service_pool.cc:227] Handling call
0809 23:20:53.131725 (+1613041us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:53.131820 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784397056}
I20260809 23:20:53.234508 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.235838 24427 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:53.133203 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.002916  1081 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:53.130184 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:53.026465 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1363) took 2750 ms. Trace:
I20260809 23:20:53.236764 24542 rpcz_store.cc:276] 0809 23:20:50.275674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.308169 (+2032495us) service_pool.cc:227] Handling call
0809 23:20:52.331832 (+ 23663us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.332113 (+   281us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:52.738141 (+406028us) log.cc:842] Serialized 14 byte log entry
0809 23:20:53.026038 (+287897us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:53.026109 (+    71us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:53.026183 (+    74us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.026267 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.026396 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2352,"prepare.queue_time_us":431274,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":1007,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":22704,"thread_start_us":653740,"threads_started":2,"wal-append.queue_time_us":448382}
I20260809 23:20:53.049165   877 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:53.129475   562 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.142163 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.013454  1114 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.141912 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:53.198900 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1190) took 2966 ms (client timeout 1692 ms). Trace:
I20260809 23:20:53.197656 24534 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:53.250916 24428 rpcz_store.cc:269] 0809 23:20:50.232844 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.402503 (+169659us) service_pool.cc:227] Handling call
0809 23:20:53.198695 (+2796192us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:53.198821 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5258973056}
I20260809 23:20:53.199580 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.200872 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:53.252803 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3220) took 3625 ms. Trace:
I20260809 23:20:53.202301 24543 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:20:53.126725 24323 rpcz_store.cc:276] 0809 23:20:51.469826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.470030 (+   204us) service_pool.cc:227] Handling call
0809 23:20:51.470555 (+   525us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:51.470847 (+   292us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:51.470872 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:51.471325 (+   453us) log.cc:842] Serialized 26 byte log entry
0809 23:20:52.012688 (+541363us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:52.012722 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.012798 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:53.000381 (+987583us) raft_consensus.cc:1671] finished
0809 23:20:53.000404 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.000543 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":371313,"threads_started":1,"wal-append.queue_time_us":963913}
I20260809 23:20:53.223865 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:53.216926 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:53.098790  1220 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:53.214613 24429 raft_consensus.cc:2362] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15. Re-sending same reply.
I20260809 23:20:53.215092 24540 rpcz_store.cc:276] 0809 23:20:50.489601 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.346581 (+1856980us) service_pool.cc:227] Handling call
0809 23:20:52.362143 (+ 15562us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:52.726889 (+364746us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:20:52.726950 (+    61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:52.727505 (+   555us) log.cc:842] Serialized 26 byte log entry
0809 23:20:52.979957 (+252452us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:52.979996 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:52.980084 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:53.214482 (+234398us) raft_consensus.cc:1671] finished
0809 23:20:53.214506 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.214624 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":97341,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":616351,"threads_started":2,"wal-append.queue_time_us":373396}
I20260809 23:20:53.248734 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3036) took 2657 ms. Trace:
I20260809 23:20:53.224336  1247 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.222718 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:53.239657 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.266021 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.198734  1017 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:53.267787 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.245110 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:53.268815 24429 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:53.245793 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 229367us
I20260809 23:20:53.267119 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:53.270210 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:53.230741  1098 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:53.233057 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1480) took 1762 ms. Trace:
I20260809 23:20:53.136185 32640 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:53.246776 24430 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:53.281773 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.195889  1176 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:53.225476  1289 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.201696 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:53.253242 24427 rpcz_store.cc:276] 0809 23:20:49.627534 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.252000 (+3624466us) service_pool.cc:227] Handling call
0809 23:20:53.252324 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.252534 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:53.252557 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:53.252593 (+    36us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:53.252630 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.252677 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.252757 (+    80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.259632 24546 rpcz_store.cc:276] 0809 23:20:50.591322 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.444241 (+1852919us) service_pool.cc:227] Handling call
0809 23:20:52.444546 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.901252 (+456706us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.901301 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:52.901359 (+    58us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:52.901405 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.248442 (+347037us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:53.248509 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.248668 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1049737088}
I20260809 23:20:53.123121   969 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:53.262606 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.266381 24311 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:53.272199 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:53.211777  1326 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.292s	user 0.000s	sys 0.001s
I20260809 23:20:53.272701 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.273273 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:53.274139 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.276237 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:53.278157 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 77395us
I20260809 23:20:53.184851  1001 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:53.279016 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2729) took 3367 ms. Trace:
I20260809 23:20:53.277410 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.232105  1352 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:20:53.278096 24309 rpcz_store.cc:276] 0809 23:20:51.477252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:51.488130 (+ 10878us) service_pool.cc:227] Handling call
0809 23:20:51.488545 (+   415us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:51.488778 (+   233us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:51.488812 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:51.489362 (+   550us) log.cc:842] Serialized 26 byte log entry
0809 23:20:51.993184 (+503822us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:51.993233 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:51.993325 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:53.136940 (+1143615us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:53.217271 (+ 80331us) raft_consensus.cc:1671] finished
0809 23:20:53.217293 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.232969 (+ 15676us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":110379392,"thread_start_us":350209,"threads_started":1,"wal-append.queue_time_us":844662}
I20260809 23:20:53.283028 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.282606 24535 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:53.290495 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:53.291138 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:53.292868 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1188) took 3928 ms. Trace:
W20260809 23:20:53.268499  1350 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260809 23:20:53.251577 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:53.261370  1211 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:53.293157 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:53.265889   562 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:53.261260  1290 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260809 23:20:53.302277 24427 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:53.339987 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:54.681725 24534 raft_consensus.cc:2354] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:20:53.340340 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:54.682233 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3051) took 1903 ms (client timeout 2000 ms). Trace:
I20260809 23:20:54.682780 25375 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:54.682714 24534 rpcz_store.cc:269] 0809 23:20:52.779197 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.251347 (+472150us) service_pool.cc:227] Handling call
0809 23:20:54.682160 (+1430813us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.683648 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1479) took 1392 ms (client timeout 1778 ms). Trace:
W20260809 23:20:54.684180 25375 rpcz_store.cc:269] 0809 23:20:53.291244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.295739 (+  4495us) service_pool.cc:227] Handling call
0809 23:20:54.683569 (+1387830us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.684670 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3164) took 1903 ms (client timeout 2077 ms). Trace:
I20260809 23:20:53.393056 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:54.685205 24545 rpcz_store.cc:269] 0809 23:20:52.781466 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.275845 (+494379us) service_pool.cc:227] Handling call
0809 23:20:54.684596 (+1408751us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.372313 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.032s	user 0.003s	sys 0.020s
W20260809 23:20:54.686208 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.005s	sys 0.022s
I20260809 23:20:54.686582 24540 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:53.504362 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 158342us
W20260809 23:20:54.687352 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.48561e+06us
I20260809 23:20:54.684808 24539 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:53.438105 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:54.689128 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:54.689322 25373 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:20:54.680251 24431 rpcz_store.cc:276] 0809 23:20:49.364470 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.949950 (+3585480us) service_pool.cc:227] Handling call
0809 23:20:52.950297 (+   347us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.950534 (+   237us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:53.182046 (+231512us) log.cc:842] Serialized 14 byte log entry
0809 23:20:53.292269 (+110223us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:53.292350 (+    81us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:53.292425 (+    75us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.292506 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:53.292660 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8401,"prepare.queue_time_us":473699,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":16711,"raft.queue_time_us":8770,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":38576,"thread_start_us":299394,"threads_started":2}
I20260809 23:20:53.579386 25365 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 16. Ops: []
W20260809 23:20:54.690670 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1367) took 1890 ms (client timeout 2005 ms). Trace:
W20260809 23:20:54.691087 24535 rpcz_store.cc:269] 0809 23:20:52.801055 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.267025 (+465970us) service_pool.cc:227] Handling call
0809 23:20:54.690539 (+1423514us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.691576 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1375) took 1814 ms (client timeout 1899 ms). Trace:
I20260809 23:20:54.691886 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:54.692102 24429 rpcz_store.cc:276] 0809 23:20:49.911319 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.265632 (+3354313us) service_pool.cc:227] Handling call
0809 23:20:53.268342 (+  2710us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.278746 (+ 10404us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:53.278954 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.451953 24422 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.161s	user 0.002s	sys 0.009s
W20260809 23:20:54.693437 24422 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.403s	user 0.003s	sys 0.010s
W20260809 23:20:54.693714 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3132) took 2060 ms (client timeout 2009 ms). Trace:
W20260809 23:20:54.694238 24311 rpcz_store.cc:269] 0809 23:20:52.633774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.210253 (+576479us) service_pool.cc:227] Handling call
0809 23:20:54.693628 (+1483375us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.695604 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1206) took 2221 ms (client timeout 2066 ms). Trace:
W20260809 23:20:54.696105 24424 rpcz_store.cc:269] 0809 23:20:52.474559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.737483 (+262924us) service_pool.cc:227] Handling call
0809 23:20:54.695399 (+1957916us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:54.695526 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448854784}
I20260809 23:20:54.688149 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:53.277599  1098 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:54.680912 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.32591e+06us
W20260809 23:20:54.704317 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1489) took 1518 ms (client timeout 1720 ms). Trace:
W20260809 23:20:54.704758 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3149) took 1904 ms (client timeout 2077 ms). Trace:
W20260809 23:20:54.705313 24323 rpcz_store.cc:269] 0809 23:20:53.185870 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.273832 (+ 87962us) service_pool.cc:227] Handling call
0809 23:20:54.704256 (+1430424us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.705542 24321 rpcz_store.cc:269] 0809 23:20:52.801113 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.254145 (+453032us) service_pool.cc:227] Handling call
0809 23:20:54.704701 (+1450556us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.277410  1302 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:54.706312 24270 connection.cc:232] RPC call timeout handler was delayed by -1.0322725549340248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2256s for a response
I20260809 23:20:54.707011 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1482) took 1308 ms. Trace:
I20260809 23:20:54.707469 25365 rpcz_store.cc:276] 0809 23:20:53.399177 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.422410 (+ 23233us) service_pool.cc:227] Handling call
0809 23:20:53.561482 (+139072us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.706739 (+1145257us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.706948 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.708081 24316 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 13: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:53.410764 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:54.687121 24544 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
W20260809 23:20:54.709225 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3049) took 1954 ms (client timeout 2092 ms). Trace:
I20260809 23:20:54.709391 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1481) took 1410 ms. Trace:
W20260809 23:20:54.709623 24544 rpcz_store.cc:269] 0809 23:20:52.756354 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.269154 (+512800us) service_pool.cc:227] Handling call
0809 23:20:54.709164 (+1440010us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.691922 24542 rpcz_store.cc:269] 0809 23:20:52.877066 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.238806 (+361740us) service_pool.cc:227] Handling call
0809 23:20:54.691520 (+1452714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.709803 25372 rpcz_store.cc:276] 0809 23:20:53.298992 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.299108 (+   116us) service_pool.cc:227] Handling call
0809 23:20:54.709321 (+1410213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.690263 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1484) took 1252 ms. Trace:
I20260809 23:20:54.721319 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1194) took 4487 ms. Trace:
W20260809 23:20:54.723632 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3162) took 1942 ms (client timeout 2035 ms). Trace:
I20260809 23:20:54.724159 24429 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260809 23:20:54.724210 24540 rpcz_store.cc:269] 0809 23:20:52.780957 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.255846 (+474889us) service_pool.cc:227] Handling call
0809 23:20:54.723554 (+1467708us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.724819 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.716089 25373 rpcz_store.cc:276] 0809 23:20:53.437565 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.437761 (+   196us) service_pool.cc:227] Handling call
0809 23:20:54.690195 (+1252434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.725713 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.727300 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.728252 24534 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:54.728677 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1382) took 1608 ms (client timeout 1877 ms). Trace:
W20260809 23:20:54.729229 24534 rpcz_store.cc:269] 0809 23:20:53.120549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.727007 (+1606458us) service_pool.cc:227] Handling call
0809 23:20:54.728620 (+  1613us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.731405 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1486) took 1595 ms (client timeout 1825 ms). Trace:
W20260809 23:20:54.731875 24312 rpcz_store.cc:269] 0809 23:20:53.136483 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.282759 (+146276us) service_pool.cc:227] Handling call
0809 23:20:54.731322 (+1448563us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.721964 24427 rpcz_store.cc:276] 0809 23:20:50.233884 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.300286 (+3066402us) service_pool.cc:227] Handling call
0809 23:20:53.300666 (+   380us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.721011 (+1420345us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.721235 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.722700 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3167) took 1912 ms (client timeout 2027 ms). Trace:
I20260809 23:20:54.740360 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2739) took 4172 ms. Trace:
W20260809 23:20:54.747460 24543 rpcz_store.cc:269] 0809 23:20:52.810070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.267934 (+457864us) service_pool.cc:227] Handling call
0809 23:20:54.722636 (+1454702us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.725311 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:54.747866 24429 rpcz_store.cc:276] 0809 23:20:50.568053 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.693031 (+4124978us) service_pool.cc:227] Handling call
0809 23:20:54.712609 (+ 19578us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.724746 (+ 12137us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.740286 (+ 15540us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.031296  1345 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.765s	user 0.000s	sys 0.002s
I20260809 23:20:53.298388  1176 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:54.387558   999 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.575s	user 0.006s	sys 0.010s
I20260809 23:20:54.726765 24545 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:53.649365   965 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.002s
I20260809 23:20:54.778472 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:54.746624 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.779446 24323 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:54.779496 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:54.780345 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1490) took 1589 ms (client timeout 1899 ms). Trace:
I20260809 23:20:53.302379  1303 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:54.780980 24323 rpcz_store.cc:269] 0809 23:20:53.190512 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.734772 (+1544260us) service_pool.cc:227] Handling call
0809 23:20:54.780273 (+ 45501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:53.339903  1049 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:54.730707 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:54.258780  1159 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.275s	user 0.010s	sys 0.002s
W20260809 23:20:54.258409  1355 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.826s	user 0.000s	sys 0.001s
W20260809 23:20:54.450081   682 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.650s	user 0.010s	sys 0.005s
W20260809 23:20:53.972283  1143 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.033s	user 0.009s	sys 0.003s
W20260809 23:20:53.646360  1348 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.382s	user 0.000s	sys 0.000s
W20260809 23:20:54.754242 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3056) took 1478 ms (client timeout 1726 ms). Trace:
I20260809 23:20:54.757344 25375 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:54.754151 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:54.772007 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:54.802973 24535 rpcz_store.cc:269] 0809 23:20:53.276213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.724562 (+1448349us) service_pool.cc:227] Handling call
0809 23:20:54.754154 (+ 29592us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.770289 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:20:54.723341  1270 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 2.275s	user 0.022s	sys 0.002s
W20260809 23:20:54.774953 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.277s	user 0.001s	sys 0.013s
W20260809 23:20:54.804335 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s	user 0.002s	sys 0.013s
I20260809 23:20:54.804888 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1483) took 1386 ms. Trace:
I20260809 23:20:54.804659 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1485) took 1252 ms. Trace:
I20260809 23:20:54.722548  1302 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:54.776484 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.782270 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.734441 24431 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260809 23:20:53.393311  1279 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.016s	user 0.019s	sys 0.037s
W20260809 23:20:54.812422 24540 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3171) took 1690 ms (client timeout 2145 ms). Trace:
I20260809 23:20:54.778483 24429 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:54.760450 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:54.785281 24545 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3055) took 1509 ms (client timeout 1723 ms). Trace:
W20260809 23:20:54.813777 24545 rpcz_store.cc:269] 0809 23:20:53.275945 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.725415 (+1449470us) service_pool.cc:227] Handling call
0809 23:20:54.785206 (+ 59791us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.814877 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.815778 24545 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:54.816365 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1380) took 1788 ms. Trace:
I20260809 23:20:53.719149  1295 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:54.807247   969 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:54.818223 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.784992   965 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:54.723575  1098 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:54.793738 24534 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:54.826063 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3173) took 1598 ms. Trace:
I20260809 23:20:54.818068   969 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.801878  1159 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.829280   969 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:54.826545 24534 rpcz_store.cc:276] 0809 23:20:53.227384 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.730388 (+1503004us) service_pool.cc:227] Handling call
0809 23:20:54.825985 (+ 95597us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.832465 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:54.833325 24534 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:54.802778  1303 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:54.745663  1352 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:54.799786 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:53.596387  1248 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.811874  1302 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:54.773303 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.805846 25375 rpcz_store.cc:276] 0809 23:20:53.552398 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.725078 (+1172680us) service_pool.cc:227] Handling call
0809 23:20:54.804594 (+ 79516us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.780452 24321 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:54.811542  1049 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:54.806692 24424 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:20:54.805418 25366 rpcz_store.cc:276] 0809 23:20:53.418297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.418553 (+   256us) service_pool.cc:227] Handling call
0809 23:20:54.804814 (+1386261us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.807446 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3054) took 1531 ms (client timeout 1730 ms). Trace:
I20260809 23:20:54.847266 24311 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
W20260809 23:20:54.849912 24544 rpcz_store.cc:269] 0809 23:20:53.275665 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.725985 (+1450320us) service_pool.cc:227] Handling call
0809 23:20:54.807385 (+ 81400us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.850392 24311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3152) took 1910 ms (client timeout 2027 ms). Trace:
I20260809 23:20:54.807404 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1195) took 4565 ms. Trace:
I20260809 23:20:54.851080 24431 rpcz_store.cc:276] 0809 23:20:50.242301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.721816 (+4479515us) service_pool.cc:227] Handling call
0809 23:20:54.722172 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.807115 (+ 84943us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.807335 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.851413 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2744) took 3355 ms. Trace:
W20260809 23:20:54.850809 24311 rpcz_store.cc:269] 0809 23:20:52.939939 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.732624 (+1792685us) service_pool.cc:227] Handling call
0809 23:20:54.850324 (+117700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.852265 24424 rpcz_store.cc:276] 0809 23:20:51.496298 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.697291 (+3200993us) service_pool.cc:227] Handling call
0809 23:20:54.771597 (+ 74306us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.851067 (+ 79470us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.851343 (+   276us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.754106 32380 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:20:54.723464  1165 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:54.813022 24540 rpcz_store.cc:269] 0809 23:20:53.122337 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.725329 (+1602992us) service_pool.cc:227] Handling call
0809 23:20:54.812265 (+ 86936us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.820708 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1203) took 2356 ms. Trace:
I20260809 23:20:54.859494 24429 rpcz_store.cc:276] 0809 23:20:52.465098 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.749048 (+2283950us) service_pool.cc:227] Handling call
0809 23:20:54.755337 (+  6289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.820313 (+ 64976us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.820619 (+   306us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.860095 24431 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:20:54.703541  1036 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:54.821107 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1463) took 2367 ms. Trace:
I20260809 23:20:54.816955 24545 rpcz_store.cc:276] 0809 23:20:53.027942 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.814576 (+1786634us) service_pool.cc:227] Handling call
0809 23:20:54.816301 (+  1725us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.861703 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:54.755116  1267 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:54.722592   562 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.836947 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1465) took 2364 ms. Trace:
I20260809 23:20:54.802320  1289 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:54.819653  1279 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.435s	user 0.055s	sys 0.115s
I20260809 23:20:53.612561  1198 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.839779 24309 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:20:54.897521 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1491) took 1703 ms (client timeout 1975 ms). Trace:
I20260809 23:20:54.868309  1255 pending_rounds.cc:62] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Trying to abort 2 pending ops.
I20260809 23:20:54.800689 25368 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:54.795029 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.784766   999 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.803731 24542 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:54.809078 24323 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:54.910230 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1488) took 1767 ms (client timeout 2129 ms). Trace:
I20260809 23:20:54.846560 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:54.857296 24534 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:20:54.910701 24323 rpcz_store.cc:269] 0809 23:20:53.142845 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.781969 (+1639124us) service_pool.cc:227] Handling call
0809 23:20:54.910157 (+128188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.860929 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1205) took 2386 ms. Trace:
I20260809 23:20:54.911449 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3169) took 2030 ms. Trace:
I20260809 23:20:54.910859 24546 raft_consensus.cc:2354] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18.
I20260809 23:20:54.861208 25363 rpcz_store.cc:276] 0809 23:20:52.454041 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.463575 (+  9534us) service_pool.cc:227] Handling call
0809 23:20:52.499238 (+ 35663us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:52.812770 (+313532us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:53.071290 (+258520us) log.cc:842] Serialized 14 byte log entry
0809 23:20:53.295346 (+224056us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:53.295413 (+    67us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:53.295481 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.820782 (+1525301us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:54.820862 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:54.821033 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59796,"prepare.queue_time_us":289808,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":51388,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":71653,"spinlock_wait_cycles":735518848,"thread_start_us":404631,"threads_started":2,"wal-append.queue_time_us":1552533}
I20260809 23:20:54.911754 24431 rpcz_store.cc:276] 0809 23:20:52.474258 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.859240 (+2384982us) service_pool.cc:227] Handling call
0809 23:20:54.859672 (+   432us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.860626 (+   954us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.860856 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.881304   877 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:54.756222  1275 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:54.888296  1303 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:54.872051 24435 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:54.880093 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3231) took 2183 ms. Trace:
I20260809 23:20:54.808138   682 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.885165 24321 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
W20260809 23:20:54.917133 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1493) took 1695 ms (client timeout 1927 ms). Trace:
W20260809 23:20:54.917487 24321 rpcz_store.cc:269] 0809 23:20:53.221910 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.733726 (+1511816us) service_pool.cc:227] Handling call
0809 23:20:54.917073 (+183347us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.852035  1371 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.400s	user 0.000s	sys 0.001s
I20260809 23:20:54.890120 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:54.918689 24540 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:54.919183 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3163) took 2138 ms. Trace:
I20260809 23:20:54.891211 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:54.861258  1354 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
W20260809 23:20:54.855433   717 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.631s	user 0.024s	sys 0.021s
I20260809 23:20:54.873422 25369 rpcz_store.cc:276] 0809 23:20:52.472424 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.636967 (+164543us) service_pool.cc:227] Handling call
0809 23:20:52.652760 (+ 15793us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:52.861732 (+208972us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:52.861779 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:52.881983 (+ 20204us) log.cc:842] Serialized 26 byte log entry
0809 23:20:53.231184 (+349201us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:53.231238 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.693302 (+1462064us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:54.693352 (+    50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:54.830548 (+137196us) raft_consensus.cc:1671] finished
0809 23:20:54.830579 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:54.836873 (+  6294us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1253,"raft.queue_time_us":1994,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":476666240,"thread_start_us":273979,"threads_started":1,"wal-append.queue_time_us":349465}
I20260809 23:20:54.895959 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.856900  1248 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 12 FOLLOWER]: Advancing to term 13
I20260809 23:20:53.548856  1220 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:54.897878 24309 rpcz_store.cc:269] 0809 23:20:53.193876 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.736248 (+1542372us) service_pool.cc:227] Handling call
0809 23:20:54.897451 (+161203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.906440 24425 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
W20260809 23:20:54.906553 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.357s	user 0.030s	sys 0.071s
W20260809 23:20:54.908396 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1378) took 2030 ms (client timeout 2533 ms). Trace:
I20260809 23:20:54.862643 24312 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:20:54.875303  1036 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.871274 24429 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:54.936900 24429 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:54.936616 24541 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260809 23:20:54.817102  1270 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.872614 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3053) took 1597 ms. Trace:
I20260809 23:20:54.915716 24424 rpcz_store.cc:276] 0809 23:20:52.696795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.854016 (+2157221us) service_pool.cc:227] Handling call
0809 23:20:54.879330 (+ 25314us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.879569 (+   239us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:54.879606 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:54.879661 (+    55us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:54.879715 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.879782 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:54.879963 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.908188  1255 pending_rounds.cc:68] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting op as it isn't in flight: id { term: 7 index: 6 } timestamp: 7316756756685172736 op_type: NO_OP noop_request { }
I20260809 23:20:54.920797 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3151) took 1992 ms. Trace:
I20260809 23:20:54.898885  1289 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.922696 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.925587 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3046) took 2179 ms. Trace:
I20260809 23:20:54.925565 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:54.945644 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1492) took 1751 ms. Trace:
I20260809 23:20:54.945899 24321 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:54.946141 24323 rpcz_store.cc:276] 0809 23:20:53.194239 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.911576 (+1717337us) service_pool.cc:227] Handling call
0809 23:20:54.945573 (+ 33997us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.945480 24544 rpcz_store.cc:276] 0809 23:20:52.746078 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.867732 (+2121654us) service_pool.cc:227] Handling call
0809 23:20:54.925539 (+ 57807us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.905913  1295 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:54.934393 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3216) took 5866 ms. Trace:
I20260809 23:20:54.947964 24425 rpcz_store.cc:276] 0809 23:20:49.067654 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.862345 (+3794691us) service_pool.cc:227] Handling call
0809 23:20:52.911549 (+ 49204us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.902197 (+1990648us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b540035df20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:20:54.934128 (+ 31931us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.934326 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4962688}
I20260809 23:20:54.856382  1049 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.952293 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2730) took 5004 ms. Trace:
I20260809 23:20:54.956422 24423 rpcz_store.cc:276] 0809 23:20:49.947467 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.270870 (+3323403us) service_pool.cc:227] Handling call
0809 23:20:53.271253 (+   383us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:53.281262 (+ 10009us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:53.281368 (+   106us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:53.289779 (+  8411us) log.cc:842] Serialized 26 byte log entry
0809 23:20:54.805231 (+1515452us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:54.805271 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:54.805336 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:54.952068 (+146732us) raft_consensus.cc:1671] finished
0809 23:20:54.952091 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:54.952230 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":324,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":26005,"thread_start_us":1306979,"threads_started":1,"wal-append.queue_time_us":1586297}
I20260809 23:20:54.913460  1198 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:54.934867 24543 rpcz_store.cc:269] 0809 23:20:52.878208 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.755204 (+1876996us) service_pool.cc:227] Handling call
0809 23:20:54.908343 (+153139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.912034 24534 rpcz_store.cc:276] 0809 23:20:52.880759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.832112 (+1951353us) service_pool.cc:227] Handling call
0809 23:20:54.911377 (+ 79265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.960005 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.960785 24534 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:54.962553 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3161) took 2182 ms. Trace:
W20260809 23:20:54.935611 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.386s	user 0.031s	sys 0.071s
I20260809 23:20:54.938884 24545 rpcz_store.cc:276] 0809 23:20:53.275398 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.861459 (+1586061us) service_pool.cc:227] Handling call
0809 23:20:54.872532 (+ 11073us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.919523 24540 rpcz_store.cc:276] 0809 23:20:52.781227 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.859282 (+2078055us) service_pool.cc:227] Handling call
0809 23:20:54.919128 (+ 59846us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.889624  1267 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.922711  1159 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:54.918542  1303 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:54.981460  1303 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:54.928512   717 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.704s	user 0.024s	sys 0.021s
I20260809 23:20:54.914397   877 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:54.983633   877 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:54.984664  1270 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:54.986783 24317 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:54.937770 24429 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:54.988950 24429 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:54.941392 24311 rpcz_store.cc:276] 0809 23:20:52.928057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.861428 (+1933371us) service_pool.cc:227] Handling call
0809 23:20:54.920721 (+ 59293us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:54.914373   984 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.159s	user 0.025s	sys 0.053s
I20260809 23:20:54.941745 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.992830 24309 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:54.993296 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3145) took 2271 ms. Trace:
I20260809 23:20:54.993566 24309 rpcz_store.cc:276] 0809 23:20:52.722255 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.929634 (+2207379us) service_pool.cc:227] Handling call
0809 23:20:54.993236 (+ 63602us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.943614 24314 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:54.921285  1275 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:54.996237 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:54.996945 24311 raft_consensus.cc:595] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:54.997246 24311 raft_consensus.cc:603] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:54.941342  1255 raft_consensus.cc:2866] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:20:54.967350 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:54.962889 24534 rpcz_store.cc:276] 0809 23:20:52.780563 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.959651 (+2179088us) service_pool.cc:227] Handling call
0809 23:20:54.962487 (+  2836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.965405 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.999033 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:55.000157 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3172) took 1773 ms. Trace:
I20260809 23:20:54.974177 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:54.977100  1065 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:55.000572 24544 rpcz_store.cc:276] 0809 23:20:53.227004 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.947126 (+1720122us) service_pool.cc:227] Handling call
0809 23:20:55.000103 (+ 52977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.977793 24542 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:54.999521 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3146) took 2276 ms. Trace:
I20260809 23:20:54.977245 24539 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:20:55.002007 24323 rpcz_store.cc:276] 0809 23:20:52.722676 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.947153 (+2224477us) service_pool.cc:227] Handling call
0809 23:20:54.999446 (+ 52293us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:55.002401 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1371) took 2127 ms (client timeout 1942 ms). Trace:
W20260809 23:20:55.002880 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1384) took 1784 ms (client timeout 1927 ms). Trace:
W20260809 23:20:55.003064 24539 rpcz_store.cc:269] 0809 23:20:52.875115 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.265309 (+390194us) service_pool.cc:227] Handling call
0809 23:20:55.002307 (+1736998us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:55.003376 24542 rpcz_store.cc:269] 0809 23:20:53.218342 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.723916 (+1505574us) service_pool.cc:227] Handling call
0809 23:20:55.002826 (+278910us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:54.975818 24321 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:55.001542 24534 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:55.005007 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3148) took 2246 ms. Trace:
I20260809 23:20:55.005525 24321 rpcz_store.cc:276] 0809 23:20:52.782237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.925258 (+2143021us) service_pool.cc:227] Handling call
0809 23:20:55.004948 (+ 79690us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.005772 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1365) took 2220 ms. Trace:
W20260809 23:20:54.992403   984 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.275s	user 0.025s	sys 0.053s
I20260809 23:20:55.011581 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3160) took 2231 ms. Trace:
I20260809 23:20:55.006503 24534 rpcz_store.cc:276] 0809 23:20:52.785965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.000974 (+2215009us) service_pool.cc:227] Handling call
0809 23:20:55.001322 (+   348us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.005431 (+  4109us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.005706 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.007911 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 41) took 1725 ms. Trace:
I20260809 23:20:55.012003 24543 rpcz_store.cc:276] 0809 23:20:52.780288 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.959084 (+2178796us) service_pool.cc:227] Handling call
0809 23:20:55.011519 (+ 52435us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:55.015524  1370 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260809 23:20:55.015655  1222 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:55.016299  1222 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:55.017771  1365 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260809 23:20:55.000543  1255 pending_rounds.cc:68] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting op as it isn't in flight: id { term: 5 index: 5 } timestamp: 7316756707937681408 op_type: NO_OP noop_request { }
I20260809 23:20:55.019326  1255 raft_consensus.cc:2866] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:20:55.019753  1255 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Raft consensus is shut down!
I20260809 23:20:55.010040  1065 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:54.925117  1352 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:55.021099  1065 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.012856 24311 rpcz_store.cc:276] 0809 23:20:53.272463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.989819 (+1717356us) service_pool.cc:227] Handling call
0809 23:20:54.997628 (+  7809us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.009359 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1368) took 2148 ms. Trace:
W20260809 23:20:55.023165 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.020s	sys 0.019s
I20260809 23:20:55.024770 24429 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 14. (term mismatch)
W20260809 23:20:55.025611  1369 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260809 23:20:54.961665  1295 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.025723 24539 rpcz_store.cc:276] 0809 23:20:52.860499 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.004209 (+2143710us) service_pool.cc:227] Handling call
0809 23:20:55.004575 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.004901 (+   326us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.004942 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.004999 (+    57us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:55.005041 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.008726 (+  3685us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.009228 (+   502us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.028071 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1366) took 2236 ms. Trace:
I20260809 23:20:55.026590 24323 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:55.034662 24544 rpcz_store.cc:276] 0809 23:20:52.791918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.001692 (+2209774us) service_pool.cc:227] Handling call
0809 23:20:55.027128 (+ 25436us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.027490 (+   362us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.027526 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.027578 (+    52us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:55.027621 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.027687 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.027807 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.025728   682 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.036495 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3143) took 2337 ms. Trace:
I20260809 23:20:55.037642 24543 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:20:55.039470 24323 rpcz_store.cc:276] 0809 23:20:52.699509 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.002699 (+2303190us) service_pool.cc:227] Handling call
0809 23:20:55.026400 (+ 23701us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.036081 (+  9681us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.036445 (+   364us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.040432 24429 pending_rounds.cc:76] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:20:55.041930  1036 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.039955 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1369) took 2165 ms. Trace:
I20260809 23:20:55.044142 24543 rpcz_store.cc:276] 0809 23:20:52.874379 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.013720 (+2139341us) service_pool.cc:227] Handling call
0809 23:20:55.035108 (+ 21388us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.039658 (+  4550us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.039893 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.045877  1049 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:55.048451 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.75533e+06us
I20260809 23:20:55.040895 24429 pending_rounds.cc:97] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 15.14
W20260809 23:20:55.050329 25323 connection.cc:232] RPC call timeout handler was delayed by -1.2542611199896783s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
W20260809 23:20:55.052356 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.023s	sys 0.020s
I20260809 23:20:55.068228 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:55.053059  1374 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260809 23:20:55.057771 24535 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260809 23:20:55.058849 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 369766us
I20260809 23:20:55.039206  1065 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:55.060189 24543 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:55.061645 24429 raft_consensus.cc:2866] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:55.057376 24323 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:55.071245 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:55.080201 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1211) took 1991 ms. Trace:
I20260809 23:20:55.049291   562 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.069078 24423 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:55.080653 24429 rpcz_store.cc:276] 0809 23:20:53.088351 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.860464 (+1772113us) service_pool.cc:227] Handling call
0809 23:20:54.871163 (+ 10699us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.079959 (+208796us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.080147 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.080971 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1482) took 1956 ms. Trace:
I20260809 23:20:55.081878 24323 rpcz_store.cc:276] 0809 23:20:53.124722 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.040868 (+1916146us) service_pool.cc:227] Handling call
0809 23:20:55.057108 (+ 16240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.080735 (+ 23627us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.080926 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.083823 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1381) took 1978 ms. Trace:
I20260809 23:20:55.084251 24543 rpcz_store.cc:276] 0809 23:20:53.105823 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.045308 (+1939485us) service_pool.cc:227] Handling call
0809 23:20:55.055523 (+ 10215us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.083537 (+ 28014us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.083764 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:55.068974  1017 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.797s	user 0.012s	sys 0.006s
I20260809 23:20:55.086864 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.091704 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.098385 24543 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260809 23:20:55.099520 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1383) took 1884 ms. Trace:
I20260809 23:20:55.100006 24543 rpcz_store.cc:276] 0809 23:20:53.214803 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.084952 (+1870149us) service_pool.cc:227] Handling call
0809 23:20:55.096633 (+ 11681us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.099095 (+  2462us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.099438 (+   343us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.104694 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.102677  1119 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:55.108752 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3052) took 1833 ms. Trace:
I20260809 23:20:55.112849 24543 rpcz_store.cc:276] 0809 23:20:53.275085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.101259 (+1826174us) service_pool.cc:227] Handling call
0809 23:20:55.103513 (+  2254us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.108296 (+  4783us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.108334 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.108402 (+    68us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:55.108446 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.108518 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.108671 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1995,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100}
I20260809 23:20:55.097216   562 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:55.094776   999 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:55.117594  1119 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.147221  1176 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:55.149520  1176 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:55.150285 24313 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:55.159247 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3221) took 5258 ms. Trace:
I20260809 23:20:55.159665 24428 rpcz_store.cc:276] 0809 23:20:49.900685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.267380 (+3366695us) service_pool.cc:227] Handling call
0809 23:20:53.284956 (+ 17576us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:53.378059 (+ 93103us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:53.378102 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:53.410363 (+ 32261us) log.cc:842] Serialized 26 byte log entry
0809 23:20:55.011660 (+1601297us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:55.011696 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.011760 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:55.159008 (+147248us) raft_consensus.cc:1671] finished
0809 23:20:55.159030 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.159180 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30175,"raft.queue_time_us":30794,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":29450,"thread_start_us":1461856,"threads_started":1,"wal-append.queue_time_us":1645702}
W20260809 23:20:55.136731  1287 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
W20260809 23:20:55.168545  1384 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260809 23:20:55.111838   717 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:55.134943   562 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:55.216588 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.024s	sys 0.031s
W20260809 23:20:55.218017 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.024s	sys 0.031s
W20260809 23:20:55.219347 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 527517us
I20260809 23:20:55.224148 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:55.229902 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:55.173274  1289 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.222576   562 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:55.249876 25371 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.001s	sys 0.006s
W20260809 23:20:55.257431 25371 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.002s	sys 0.006s
I20260809 23:20:55.267747  1275 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.273403  1390 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:55.273857  1390 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:55.274163  1390 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:20:55.274879 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3042) took 2534 ms (client timeout 2256 ms). Trace:
W20260809 23:20:55.275290 24546 rpcz_store.cc:269] 0809 23:20:52.740397 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.301203 (+560806us) service_pool.cc:227] Handling call
0809 23:20:55.274649 (+1973446us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.274792 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":509998464}
I20260809 23:20:55.251966  1287 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:55.287294 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.281193  1392 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:55.252959  1389 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:55.291781  1389 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.294340 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.287176  1267 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.271660 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.288399 24428 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:55.290076 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.296312 24431 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:55.213058   717 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.299994 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3175) took 1740 ms. Trace:
I20260809 23:20:55.300336 24546 rpcz_store.cc:276] 0809 23:20:53.559542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.276479 (+1716937us) service_pool.cc:227] Handling call
0809 23:20:55.299311 (+ 22832us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.299660 (+   349us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.299690 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.299736 (+    46us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:55.299781 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.299843 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.299950 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.312500 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.318491 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.320117 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.327401 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.294353  1389 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.315128  1295 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.336139 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:55.323091  1275 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:55.323661  1267 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:55.335094  1065 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:55.414857 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:55.415266 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.006s	sys 0.009s
W20260809 23:20:55.415573 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.006s	sys 0.009s
W20260809 23:20:55.426252 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.002s	sys 0.019s
W20260809 23:20:55.426904 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.007s	sys 0.007s
W20260809 23:20:55.430212 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.002s	sys 0.019s
W20260809 23:20:55.430368 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.008s	sys 0.008s
W20260809 23:20:55.429666 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.053s	sys 0.147s
W20260809 23:20:55.431628 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.053s	sys 0.147s
W20260809 23:20:55.432314 24498 connection.cc:232] RPC call timeout handler was delayed by -4.2862044039648026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260809 23:20:55.432804 24498 connection.cc:232] RPC call timeout handler was delayed by -3.662504404084757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22060000000000002s for a response
W20260809 23:20:55.433277 24498 connection.cc:232] RPC call timeout handler was delayed by -3.3502044039778411s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2553s for a response
I20260809 23:20:55.434053 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:55.340648  1275 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:20:55.435271 24427 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.009s	sys 0.013s
W20260809 23:20:55.439980 24427 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.010s	sys 0.013s
I20260809 23:20:55.435987 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:55.411387  1017 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:55.441794 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:55.437970 24270 leader_election.cc:398] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:20:55.411998  1267 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:55.443609 24270 leader_election.cc:401] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260809 23:20:55.438457 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.008s	sys 0.015s
I20260809 23:20:55.413527  1065 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:55.444607 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.009s	sys 0.015s
W20260809 23:20:55.438273  1098 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.001s
W20260809 23:20:55.452225 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.013s	sys 0.017s
W20260809 23:20:55.428100  1081 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.007s	sys 0.013s
W20260809 23:20:55.453966 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.971s	user 0.004s	sys 0.012s
W20260809 23:20:55.457980 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.004s	sys 0.012s
I20260809 23:20:55.451400 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3218) took 5843 ms. Trace:
I20260809 23:20:55.438607  1295 raft_consensus.cc:2741] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260809 23:20:55.441695  1275 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:55.456259 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.013s	sys 0.017s
I20260809 23:20:55.441831  1207 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:55.445910  1267 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:55.452122  1065 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:55.452632  1279 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:55.457847  1081 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.008s	sys 0.014s
W20260809 23:20:55.461453 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 383329us
I20260809 23:20:55.454946  1363 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:55.454262  1017 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:55.427501  1382 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.043s
I20260809 23:20:55.470413 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:55.480651 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:55.456131  1098 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:55.501760  1361 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.004s	user 0.004s	sys 0.012s
I20260809 23:20:55.492883 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3158) took 3995 ms. Trace:
I20260809 23:20:55.511889 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:55.466774  1255 ts_tablet_manager.cc:1905] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260809 23:20:55.503398 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1460) took 3050 ms (client timeout 1809 ms). Trace:
W20260809 23:20:55.519142 25370 rpcz_store.cc:269] 0809 23:20:52.453067 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.461826 (+  8759us) service_pool.cc:227] Handling call
0809 23:20:55.503199 (+3041373us) spinlock_profiling.cc:243] Waited 3.01 s on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.503331 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6918366976}
I20260809 23:20:55.499380 32640 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:55.527489  1386 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:55.528038  1386 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:55.529080  1386 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.532675  1017 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:55.508543   717 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:55.535156 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1480) took 2236 ms (client timeout 2181 ms). Trace:
I20260809 23:20:55.535583   717 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:55.536190   717 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:55.536676   717 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:55.540472  1098 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:55.535699 25368 rpcz_store.cc:269] 0809 23:20:53.298620 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.299261 (+   641us) service_pool.cc:227] Handling call
0809 23:20:55.534866 (+2235605us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.535071 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199345536}
W20260809 23:20:55.483023  1302 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.619s	user 0.006s	sys 0.006s
W20260809 23:20:55.544744  1248 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.610s	user 0.005s	sys 0.006s
I20260809 23:20:55.487919  1081 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:55.471442  1303 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.511799 24541 rpcz_store.cc:276] 0809 23:20:51.497125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.688012 (+1190887us) service_pool.cc:227] Handling call
0809 23:20:52.701696 (+ 13684us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:54.939394 (+2237698us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.492599 (+553205us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:55.492818 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5139174144}
I20260809 23:20:55.502633  1119 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.471261  1295 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260809 23:20:55.472500  1279 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.476922  1389 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.077s
I20260809 23:20:55.494767  1222 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.530719  1386 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:55.558776  1081 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:55.594285  1198 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.004s
I20260809 23:20:55.584066  1389 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:55.582005  1248 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:55.554519  1302 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:55.621086  1405 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260809 23:20:55.605099  1390 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:55.636137  1390 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:55.667548 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.960s	user 0.006s	sys 0.020s
W20260809 23:20:55.668871 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.006s	sys 0.020s
I20260809 23:20:55.671090 24309 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:20:55.675875 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.002s	sys 0.010s
W20260809 23:20:55.676416 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.002s	sys 0.010s
W20260809 23:20:55.676244 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 245241us
I20260809 23:20:55.676673 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:55.679049 24541 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:55.678750 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1494) took 2380 ms. Trace:
I20260809 23:20:55.681615 24541 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260809 23:20:55.679454 24323 rpcz_store.cc:276] 0809 23:20:53.297831 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.082461 (+1784630us) service_pool.cc:227] Handling call
0809 23:20:55.096812 (+ 14351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.097017 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:55.102376 (+  5359us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.677640 (+575264us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.677922 (+   282us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:55.678088 (+   166us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.678220 (+   132us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.678586 (+   366us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2114,"prepare.queue_time_us":5725,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":434,"thread_start_us":574327,"threads_started":1,"wal-append.queue_time_us":502250}
I20260809 23:20:55.682365 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3177) took 1975 ms. Trace:
W20260809 23:20:55.459189 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 976089us
I20260809 23:20:55.682830 24541 rpcz_store.cc:276] 0809 23:20:53.706665 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.680492 (+1973827us) service_pool.cc:227] Handling call
0809 23:20:55.681186 (+   694us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:55.682051 (+   865us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.682270 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.686513 24541 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260809 23:20:55.459252 24422 rpcz_store.cc:276] 0809 23:20:49.607872 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.247569 (+3639697us) service_pool.cc:227] Handling call
0809 23:20:53.269532 (+ 21963us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.269766 (+   234us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:20:54.694119 (+1424353us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.424175 (+730056us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.424345 (+   170us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.424420 (+    75us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:20:55.451086 (+ 26666us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.451182 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.451331 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7963,"prepare.queue_time_us":367663,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":19282,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":10406,"spinlock_wait_cycles":13104384,"thread_start_us":2095348,"threads_started":2}
I20260809 23:20:55.687899 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3170) took 2600 ms. Trace:
I20260809 23:20:55.671168 24321 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:55.693272 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:55.695214 24425 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260809 23:20:55.702131 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1279) took 2503 ms. Trace:
W20260809 23:20:55.704871 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 483229us
I20260809 23:20:55.705628  1222 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:20:55.606344  1248 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:55.709363  1279 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.591861  1081 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:55.715001  1302 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.716567  1303 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:55.743158 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3178) took 1863 ms. Trace:
I20260809 23:20:55.786054 24541 rpcz_store.cc:276] 0809 23:20:53.862813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.685178 (+1822365us) service_pool.cc:227] Handling call
0809 23:20:55.685911 (+   733us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:55.725842 (+ 39931us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.726155 (+   313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:20:55.746371 24323 rpcz_store.cc:276] 0809 23:20:53.198578 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.683137 (+2484559us) service_pool.cc:227] Handling call
0809 23:20:55.702019 (+ 18882us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:20:55.800673 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 2.51899e+06us
I20260809 23:20:55.867475 24541 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260809 23:20:55.733068 24544 rpcz_store.cc:276] 0809 23:20:53.087036 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.037064 (+1950028us) service_pool.cc:227] Handling call
0809 23:20:55.057905 (+ 20841us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:55.058923 (+  1018us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.058991 (+    68us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:55.059579 (+   588us) log.cc:842] Serialized 26 byte log entry
0809 23:20:55.459690 (+400111us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:55.459717 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.459802 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:55.687265 (+227463us) raft_consensus.cc:1671] finished
0809 23:20:55.687315 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.687655 (+   340us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3138,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9167,"thread_start_us":363216,"threads_started":1,"wal-append.queue_time_us":492250}
I20260809 23:20:55.888696 24431 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:55.903337 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.007s	sys 0.015s
W20260809 23:20:55.903990 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.007s	sys 0.016s
I20260809 23:20:55.905781 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3176) took 2327 ms. Trace:
W20260809 23:20:55.907011  1352 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.008s	sys 0.035s
W20260809 23:20:55.907470  1352 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.008s	sys 0.036s
W20260809 23:20:55.908165 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.005s	sys 0.017s
W20260809 23:20:55.908555 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.006s	sys 0.017s
I20260809 23:20:55.909744 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3045) took 3165 ms. Trace:
W20260809 23:20:55.910987 25371 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.002s	sys 0.013s
W20260809 23:20:55.911381 25371 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.002s	sys 0.014s
I20260809 23:20:55.913053 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1379) took 2991 ms. Trace:
I20260809 23:20:55.894141  1248 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:55.917069 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3174) took 2638 ms. Trace:
I20260809 23:20:55.917438 24543 rpcz_store.cc:276] 0809 23:20:53.278421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.114441 (+1836020us) service_pool.cc:227] Handling call
0809 23:20:55.117737 (+  3296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.117971 (+   234us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:55.462728 (+344757us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.916330 (+453602us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.916432 (+   102us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:55.916557 (+   125us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.916688 (+   131us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.916886 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":167470,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":32,"thread_start_us":753024,"threads_started":2,"wal-append.queue_time_us":452344}
W20260809 23:20:55.925374 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 413307us
I20260809 23:20:55.927083 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:55.928298  1295 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:55.930454  1442 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:55.932190  1363 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:55.913529 24539 rpcz_store.cc:276] 0809 23:20:52.921969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.034647 (+2112678us) service_pool.cc:227] Handling call
0809 23:20:55.036307 (+  1660us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.036499 (+   192us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:55.456072 (+419573us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.912219 (+456147us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.912337 (+   118us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:55.912479 (+   142us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.912620 (+   141us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.912862 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13908,"prepare.queue_time_us":248626,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"thread_start_us":811364,"threads_started":2}
I20260809 23:20:55.906414 24546 rpcz_store.cc:276] 0809 23:20:53.578817 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.301058 (+1722241us) service_pool.cc:227] Handling call
0809 23:20:55.321915 (+ 20857us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.322147 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:55.334756 (+ 12609us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.904941 (+570185us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.905126 (+   185us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:55.905277 (+   151us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.905410 (+   133us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.905685 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":12254,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":1091,"raft.queue_time_us":2613,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":309,"thread_start_us":568034,"threads_started":1,"wal-append.queue_time_us":588380}
I20260809 23:20:55.936985 25365 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:55.938742   999 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:55.943913  1295 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.985734  1295 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:55.951494  1303 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:55.951826  1429 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:55.967172 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3166) took 3171 ms (client timeout 3077 ms). Trace:
W20260809 23:20:55.952319 25325 connection.cc:232] RPC call timeout handler was delayed by -1.3145837639458477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
I20260809 23:20:55.952455 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3139) took 3305 ms. Trace:
W20260809 23:20:55.995692 25325 connection.cc:232] RPC call timeout handler was delayed by -1.1622494529001415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
I20260809 23:20:55.953966 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1478) took 2691 ms. Trace:
I20260809 23:20:55.996582  1429 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.997455 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:55.954308 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3133) took 3320 ms (client timeout 2066 ms). Trace:
I20260809 23:20:55.996412 25371 rpcz_store.cc:276] 0809 23:20:53.262484 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.262641 (+   157us) service_pool.cc:227] Handling call
0809 23:20:53.277860 (+ 15219us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.278062 (+   202us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:53.340306 (+ 62244us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.911875 (+2571569us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.953085 (+ 41210us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.953211 (+   126us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:55.953372 (+   161us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.953503 (+   131us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.953820 (+   317us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40651,"prepare.queue_time_us":724512,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":136,"raft.queue_time_us":201126,"raft.run_cpu_time_us":478,"raft.run_wall_time_us":7522,"thread_start_us":2570660,"threads_started":3,"wal-append.queue_time_us":1581728}
I20260809 23:20:56.001202  1429 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:55.958218 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3134) took 3323 ms (client timeout 1876 ms). Trace:
I20260809 23:20:56.001962 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3229) took 3589 ms. Trace:
W20260809 23:20:56.002097 24320 rpcz_store.cc:269] 0809 23:20:52.634374 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.952158 (+317784us) service_pool.cc:227] Handling call
0809 23:20:55.957787 (+3005629us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54003854d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.957983 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3964052352}
I20260809 23:20:55.960036 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3129) took 3389 ms. Trace:
I20260809 23:20:56.003578 24314 rpcz_store.cc:276] 0809 23:20:52.570697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.570823 (+   126us) service_pool.cc:227] Handling call
0809 23:20:52.631278 (+ 60455us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.995732 (+2364454us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.959527 (+963795us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:55.959839 (+   312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5317447040}
W20260809 23:20:56.006022 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1483) took 2877 ms (client timeout 2066 ms). Trace:
W20260809 23:20:56.006520 24312 rpcz_store.cc:269] 0809 23:20:53.128658 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.760113 (+1631455us) service_pool.cc:227] Handling call
0809 23:20:56.005589 (+1245476us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b5400253ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:56.005781 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93376384}
I20260809 23:20:56.006031 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3168) took 3145 ms. Trace:
I20260809 23:20:55.960714  1442 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.963388 24311 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 12. Current term is 13. Ops: []
I20260809 23:20:56.015508  1335 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:20:56.016657 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1484) took 2886 ms. Trace:
I20260809 23:20:56.017231  1335 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:56.017957  1335 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:55.967172  1036 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:56.020478   960 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:56.020758 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.022059 24422 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:55.967990  1279 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260809 23:20:55.969812  1363 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:55.896422  1389 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.027370  1389 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.028345 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.002502 24427 rpcz_store.cc:276] 0809 23:20:52.404517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.734574 (+2330057us) service_pool.cc:227] Handling call
0809 23:20:54.746806 (+ 12232us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.747311 (+   505us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:55.440460 (+693149us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.937978 (+497518us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.938068 (+    90us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:55.938171 (+   103us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.941030 (+  2859us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b18002884c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:20:55.942148 (+  1118us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.993714 (+ 51566us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":225618,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":1723,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":249,"spinlock_wait_cycles":34994432,"thread_start_us":1150510,"threads_started":2,"wal-append.queue_time_us":481759}
W20260809 23:20:55.979799 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.000s	sys 0.015s
W20260809 23:20:56.031162 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.000s	sys 0.016s
W20260809 23:20:56.032641 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 617850us
I20260809 23:20:55.910127 24545 rpcz_store.cc:276] 0809 23:20:52.744295 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.978436 (+2234141us) service_pool.cc:227] Handling call
0809 23:20:54.978755 (+   319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.979025 (+   270us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:55.319670 (+340645us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.909173 (+589503us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.909268 (+    95us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:55.909390 (+   122us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.909499 (+   109us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.909696 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":305762,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"thread_start_us":929169,"threads_started":2}
W20260809 23:20:55.999123 24316 rpcz_store.cc:269] 0809 23:20:52.634060 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.216660 (+582600us) service_pool.cc:227] Handling call
0809 23:20:55.953663 (+2737003us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.953958 (+   295us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3343432064}
I20260809 23:20:55.956882 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3119) took 8171 ms. Trace:
I20260809 23:20:55.995786 24309 rpcz_store.cc:276] 0809 23:20:52.647209 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.994370 (+2347161us) service_pool.cc:227] Handling call
0809 23:20:55.010908 (+ 16538us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.671975 (+661067us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.952173 (+280198us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:55.952394 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1516468608}
W20260809 23:20:55.994544 24535 rpcz_store.cc:269] 0809 23:20:52.795317 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.804922 (+2009605us) service_pool.cc:227] Handling call
0809 23:20:55.966585 (+1161663us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:55.966844 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":547367680}
I20260809 23:20:56.007447 24542 rpcz_store.cc:276] 0809 23:20:52.860941 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.004336 (+2143395us) service_pool.cc:227] Handling call
0809 23:20:55.004615 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.058464 (+ 53849us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.058502 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.058583 (+    81us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:55.058632 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.005540 (+946908us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:56.005620 (+    80us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:56.005832 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123361536}
W20260809 23:20:56.011747  1362 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.258s	user 0.004s	sys 0.000s
W20260809 23:20:56.012123 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85445.6us
I20260809 23:20:56.043800 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:56.044114 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:56.045459 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:56.048761 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.026911  1363 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.049865   999 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:56.050173 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:56.050515   999 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:56.052302 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.053875 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.033989 24317 rpcz_store.cc:276] 0809 23:20:47.785477 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:47.785630 (+   153us) service_pool.cc:227] Handling call
0809 23:20:47.810315 (+ 24685us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:53.050257 (+5239942us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b1800205500. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2389b856 00007fac237edf7e 00007fac23857367 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:20:54.986546 (+1936289us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b58006ba2b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:20:55.006456 (+ 19910us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:55.006483 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:55.011200 (+  4717us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:55.011258 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:55.955885 (+944627us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b1800321100. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:20:55.956388 (+   503us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b540019cdd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:55.956415 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:55.956680 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1275,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":13386,"spinlock_wait_cycles":11274569344}
W20260809 23:20:56.057297  1065 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.583s	user 0.020s	sys 0.003s
I20260809 23:20:56.057785  1065 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:56.058404 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1486) took 2457 ms (client timeout 2072 ms). Trace:
I20260809 23:20:56.059214 24428 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
W20260809 23:20:56.059968 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3142) took 3360 ms (client timeout 1965 ms). Trace:
W20260809 23:20:56.060388 24313 rpcz_store.cc:269] 0809 23:20:52.699064 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.212432 (+513368us) service_pool.cc:227] Handling call
0809 23:20:56.059495 (+2847063us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:56.059746 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4432878336}
I20260809 23:20:56.060838 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.061719 24431 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:56.017104 24311 rpcz_store.cc:276] 0809 23:20:53.130261 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.048116 (+1917855us) service_pool.cc:227] Handling call
0809 23:20:55.958759 (+910643us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:56.015217 (+ 56458us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.016392 (+  1175us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400225e50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:56.016579 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2068320000}
I20260809 23:20:56.064136  1393 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:56.064651  1393 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.065383  1429 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:56.065840  1429 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:56.066184  1429 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.066103 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.030614  1389 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:56.040433 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:20:56.041302 24430 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.003s	sys 0.014s
I20260809 23:20:56.046780 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:56.058837 25365 rpcz_store.cc:269] 0809 23:20:53.601265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.739151 (+1137886us) service_pool.cc:227] Handling call
0809 23:20:56.058130 (+1318979us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:56.058301 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2130369408}
I20260809 23:20:56.071681 24545 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:56.067560  1389 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:20:56.098847  1442 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.099118  1403 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:56.099596  1389 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:20:56.099859  1430 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:56.100191 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1481) took 2975 ms. Trace:
I20260809 23:20:56.100418  1430 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:56.100898  1430 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.100654 24321 rpcz_store.cc:276] 0809 23:20:53.124423 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.006553 (+1882130us) service_pool.cc:227] Handling call
0809 23:20:55.009257 (+  2704us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.801618 (+792361us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.068553 (+266935us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:56.100107 (+ 31554us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1520460032}
I20260809 23:20:56.100929 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:56.107239 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68650.1us
I20260809 23:20:56.110273 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.084074  1363 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:56.121187 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3226) took 5928 ms. Trace:
I20260809 23:20:56.121549  1176 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:56.121678 24430 rpcz_store.cc:276] 0809 23:20:50.192324 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:53.298821 (+3106497us) service_pool.cc:227] Handling call
0809 23:20:54.685735 (+1386914us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.685954 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:54.746023 (+ 60069us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.464875 (+718852us) log.cc:842] Serialized 14 byte log entry
0809 23:20:56.103298 (+638423us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:56.118832 (+ 15534us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:56.120812 (+  1980us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.120962 (+   150us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:56.121108 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6726,"prepare.queue_time_us":242274,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":3774,"raft.queue_time_us":78798,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":1319610,"threads_started":2,"wal-append.queue_time_us":845900}
W20260809 23:20:56.123731 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 113330us
I20260809 23:20:56.124056  1287 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:56.125064 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.125015  1153 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:56.126040 24427 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:56.126232 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.128826  1248 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.131253  1363 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:56.131462 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 91237.3us
I20260809 23:20:56.121340  1335 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.131752  1363 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.132915  1335 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.139501  1448 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:56.139994  1448 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:56.140421  1448 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.140462 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.141062 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.141845 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.142568 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:56.143361 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.143075  1303 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.143586  1382 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:56.144773  1001 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:56.142871  1352 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:56.145211  1001 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.145787  1352 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.146076 24535 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260809 23:20:56.147692   877 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.045s	user 0.010s	sys 0.010s
I20260809 23:20:56.147279  1352 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.148289   877 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.149559 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.150355 24422 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:56.142599  1440 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:56.152263  1440 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:56.152750  1440 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:56.153626 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.152143 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.143841  1435 raft_consensus.cc:2741] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260809 23:20:56.155424 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.152261 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:56.155974 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.156906 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:56.122658 24539 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.156766 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.154906  1429 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.147213  1001 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.173877  1001 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:56.175681  1386 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.004s
I20260809 23:20:56.177103 24427 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:56.180095  1065 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.185953  1440 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:56.188813   562 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.917s	user 0.022s	sys 0.002s
I20260809 23:20:56.188683 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:56.189338   562 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.191633  1390 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:56.191933  1297 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:56.192936  1297 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:56.193388  1297 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:56.193785  1297 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:56.171424 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.191601 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:56.192092  1390 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:56.195849 24425 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260809 23:20:56.196297  1390 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:56.196746  1431 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:56.200416 25365 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260809 23:20:56.189630 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.169703  1429 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:56.177147  1248 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:56.065855  1393 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:56.208597 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 262743us
W20260809 23:20:56.208963 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52208.4us
I20260809 23:20:56.220391 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.217608 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.227428 25365 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:56.210827 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:56.221930 24428 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:56.224802 24539 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:56.220752 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:56.224931 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 160232us
I20260809 23:20:56.221287 25371 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:56.213002 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.223218 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1177) took 7530 ms. Trace:
I20260809 23:20:56.224385 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.171269  1392 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.086s
I20260809 23:20:56.235792 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.234057 24435 rpcz_store.cc:276] 0809 23:20:48.692994 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.805443 (+4112449us) service_pool.cc:227] Handling call
0809 23:20:52.805800 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:54.918253 (+2112453us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.222722 (+1304469us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54003903d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:56.223049 (+   327us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4751326080}
W20260809 23:20:56.211992  1275 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.745s	user 0.012s	sys 0.005s
W20260809 23:20:56.224188  1423 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.299s	user 0.003s	sys 0.000s
W20260809 23:20:56.237057 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 120278us
W20260809 23:20:56.210549  1444 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.286s	user 0.002s	sys 0.000s
I20260809 23:20:56.210681  1198 raft_consensus.cc:979] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:20:56.238559 24321 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260809 23:20:56.242051 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.242748 24430 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:20:56.236451 24427 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:56.224087  1393 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:56.212975  1036 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:56.243891  1438 consensus_queue.cc:1159] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 17 status { last_received { term: 17 index: 11 } last_committed_idx: 9 last_received_current_leader { term: 17 index: 11 } }
I20260809 23:20:56.244171  1275 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.209616   877 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.213032   562 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:56.249157 24269 connection.cc:232] RPC call timeout handler was delayed by -1.4829205428250134s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260809 23:20:56.229116  1176 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:56.250620  1437 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.303s	user 0.002s	sys 0.002s
I20260809 23:20:56.250828 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.251464 24429 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260809 23:20:56.222172  1287 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:56.255622 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.251569 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:56.257174 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0896205429453403s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
W20260809 23:20:56.262470  1433 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.337s	user 0.000s	sys 0.004s
W20260809 23:20:56.263622 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0080601868685335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
I20260809 23:20:56.256934 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.260861 24429 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 6
I20260809 23:20:56.267373  1386 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:20:56.253510  1428 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.286s	user 0.003s	sys 0.001s
I20260809 23:20:56.214975   999 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.274037 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.268710 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.266594 24429 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260809 23:20:56.253813  1036 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:56.276335 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.277136 25365 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:56.265954  1287 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:56.282919  1421 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.314s	user 0.004s	sys 0.000s
W20260809 23:20:56.283658 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 123312us
I20260809 23:20:56.280077 24425 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:56.278539  1303 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.285068 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.286634 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:56.287086 24428 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260809 23:20:56.288084 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.264199  1017 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:56.279789 24429 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:56.287248 24317 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:56.276039  1205 raft_consensus.cc:1067] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:20:56.293972 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3237) took 1094 ms. Trace:
I20260809 23:20:56.300347 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.307006 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.312455 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:56.314002  1303 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:56.315026 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1374) took 3360 ms. Trace:
I20260809 23:20:56.314893  1176 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.315629 24534 rpcz_store.cc:276] 0809 23:20:52.876705 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.014117 (+2137412us) service_pool.cc:227] Handling call
0809 23:20:55.014557 (+   440us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.014817 (+   260us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:55.447059 (+432242us) log.cc:842] Serialized 14 byte log entry
0809 23:20:56.083659 (+636600us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:56.111466 (+ 27807us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:56.118162 (+  6696us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.203170 (+ 85008us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:56.237462 (+ 34292us) inbound_call.cc:173] Queueing success response
0809 23:20:56.271166 (+ 33704us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
Metrics: {"prepare.queue_time_us":270197,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":23577,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":196,"spinlock_wait_cycles":483776896,"thread_start_us":930955,"threads_started":2,"wal-append.queue_time_us":484161}
I20260809 23:20:56.317875  1176 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.318601  1036 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.320509  1036 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:56.324196 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59401.2us
I20260809 23:20:56.327330  1440 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.327553  1017 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:20:56.329170 24323 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:56.330921 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.332726  1275 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.332768 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.333724 24422 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:56.334985 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.335997 24323 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:56.339125  1440 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:56.339630  1440 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.340314   999 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.340543  1440 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.341326 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.342288 24320 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:20:56.343339  1287 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.343623 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.344403 24320 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:56.345171  1207 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.879s	user 0.021s	sys 0.002s
I20260809 23:20:56.345461  1287 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.345590  1207 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:56.342344 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.346576 24323 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:56.347540 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.347106  1081 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.348841 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.349589 24323 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:20:56.350472 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.355049  1335 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 23:20:56.283460   562 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260809 23:20:56.356299   562 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:56.356777   562 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:56.358137 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.359369   999 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:56.359824   999 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:56.360212   999 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.341655  1440 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:56.367329 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.368314 24314 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:56.369849 24314 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 20. Ops: []
W20260809 23:20:56.371199  1424 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.370s	user 0.004s	sys 0.000s
I20260809 23:20:56.373981 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.375020 24429 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:56.375844  1427 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.455s	user 0.004s	sys 0.000s
I20260809 23:20:56.376639  1255 ts_tablet_manager.cc:1918] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20260809 23:20:56.377435  1255 log.cc:1196] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: Deleting WAL directory at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/wals/e4ed6a400f2643da95b8f744ebab6f4c
W20260809 23:20:56.387593 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83884.3us
I20260809 23:20:56.394485 24312 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:56.395605 24435 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.398370 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:56.399134 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86826.3us
I20260809 23:20:56.400280  1207 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:56.400363 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.401336  1431 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:56.401940  1431 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.403090  1202 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:20:56.403159  1431 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.403798  1202 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:56.403720  1389 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:56.405951  1303 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:56.407331  1207 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260809 23:20:56.411394  1422 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.489s	user 0.001s	sys 0.004s
I20260809 23:20:56.412503 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.413321 25371 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:56.415773 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.417062 25370 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:56.418455 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.419646 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.420258 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.420707 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.421383 24321 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260809 23:20:56.421794 24321 pending_rounds.cc:76] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 9
I20260809 23:20:56.422046 24321 pending_rounds.cc:97] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260809 23:20:56.422310 24321 raft_consensus.cc:2866] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:56.422194 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.421274 25369 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:56.422637 25363 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:56.298686 24424 rpcz_store.cc:276] 0809 23:20:55.070308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:55.070447 (+   139us) service_pool.cc:227] Handling call
0809 23:20:55.084259 (+ 13812us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:55.084480 (+   221us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:55.099551 (+ 15071us) log.cc:842] Serialized 14 byte log entry
0809 23:20:55.942334 (+842783us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:56.009338 (+ 67004us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:56.098435 (+ 89097us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.111569 (+ 13134us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:56.164447 (+ 52878us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15267,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":38012,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"thread_start_us":789407,"threads_started":2,"wal-append.queue_time_us":514777}
I20260809 23:20:56.350664  1081 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.405016  1202 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.418992 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.423957  1386 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.433784 24428 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:56.431510 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.433238 24429 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:56.406500  1389 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260809 23:20:56.442078 24543 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:56.439116 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.358299   562 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.445196 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:56.429783   717 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.892s	user 0.015s	sys 0.005s
I20260809 23:20:56.453522 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.454432 25373 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:56.455442   717 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.456691 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:56.458331 25365 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:56.461766 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.462792 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.464316 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.467034 24323 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:56.427395  1198 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.303486  1248 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.130s
I20260809 23:20:56.427740  1205 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.477667 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.481200 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.474165 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.484579 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:56.487411  1267 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.967s	user 0.021s	sys 0.000s
I20260809 23:20:56.487794 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.489604 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.490190 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.502534 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.498864 24321 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:56.505367 24378 connection.cc:232] RPC call timeout handler was delayed by -2.090802859980613s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1876s for a response
W20260809 23:20:56.505769 24378 connection.cc:232] RPC call timeout handler was delayed by -2.0107028600759804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
I20260809 23:20:56.424702  1363 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:56.506085 24378 connection.cc:232] RPC call timeout handler was delayed by -1.9198028601240367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
I20260809 23:20:56.506517 24313 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:56.463652  1429 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:56.505936 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44424.9us
I20260809 23:20:56.514549 25371 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:56.517288 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.519224 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.516330 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.554205 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.558897 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:56.552834 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:56.564462 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.553923 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.561834 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.553798 24425 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:56.566730 25363 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:20:56.560118 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49191.1us
I20260809 23:20:56.574458 25368 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:56.554294 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:56.459864  1448 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.582178 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:56.582428 25369 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:56.583776 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.593132 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.590575 24317 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:20:56.501550  1335 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.473582  1386 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.601641 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:56.603530 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 147001us
I20260809 23:20:56.606297 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.515060  1429 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:56.461549  1303 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.611644 24430 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:20:56.519438  1392 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.076s
I20260809 23:20:56.617692 24427 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:56.585566   562 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.514485   717 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.565501  1275 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260809 23:20:56.635030 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.614231 24323 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:56.555732  1248 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.054s
I20260809 23:20:56.643623 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.491577  1205 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.631006 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:56.555809   999 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.642439 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.646061 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.647185 25375 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:56.627967  1207 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:56.617098  1442 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:56.648556 24312 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:56.520255  1267 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:56.553900  1198 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.650804  1403 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:56.629084  1255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42134 (request call id 0) took 6014 ms. Trace:
I20260809 23:20:56.653323  1386 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.663972 24422 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:20:56.665666 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49482.8us
I20260809 23:20:56.668628 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:56.669589  1445 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.455s	user 0.003s	sys 0.000s
I20260809 23:20:56.673125 24321 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:20:56.699445 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 129142us
I20260809 23:20:56.692893 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.701460 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:56.718786 24427 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: [15.12-17.13]
I20260809 23:20:56.722257 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.723121 24323 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:56.719825 25373 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:20:56.632180  1297 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.724596 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.728590 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.728837 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.735386 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:56.736392  1198 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.737639 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:56.737593  1434 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:20:56.738569  1449 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.738966  1352 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260809 23:20:56.739365  1449 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.739748  1352 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:20:56.740279 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:56.741806  1449 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.742256  1440 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.743043  1267 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:56.743875 24540 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:56.745851 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3047) took 3999 ms. Trace:
I20260809 23:20:56.746199 24540 rpcz_store.cc:276] 0809 23:20:52.746396 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:54.979886 (+2233490us) service_pool.cc:227] Handling call
0809 23:20:55.968621 (+988735us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:56.744279 (+775658us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.744995 (+   716us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:56.745676 (+   681us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400115b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:56.745796 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4054918272}
I20260809 23:20:56.747206 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:56.747694 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 134461us
I20260809 23:20:56.748731 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.749886 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.747248  1438 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:20:56.751410  1438 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:56.751736  1438 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.751818  1392 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:56.752323  1392 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:56.752151  1386 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:20:56.752842  1392 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.755133  1081 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:56.755656 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.759215 25372 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.762544 24430 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:56.763684 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.750814 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:56.767452 24425 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:56.756810   999 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.771667 24423 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260809 23:20:56.773178 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 155993us
I20260809 23:20:56.782173 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.787598 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:56.791790  1438 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.792868 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:56.794169  1438 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:56.724115  1255 rpcz_store.cc:276] 0809 23:20:50.588689 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:50.590766 (+  2077us) service_pool.cc:227] Handling call
0809 23:20:51.467638 (+876872us) ts_tablet_manager.cc:1145] Deleting tablet e4ed6a400f2643da95b8f744ebab6f4c
0809 23:20:51.516526 (+ 48888us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0809 23:20:53.647003 (+2130477us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540029d954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23868352 00007fac2723026c 00007fac287f0cfa 00007fac28820e10 00007fac28807e0d 00007fac28807db9 00007fac28807d49 00007fac28807d11 00007fac28806f3d 00007fac29b9c204
0809 23:20:56.130232 (+2483229us) tablet_metadata.cc:641] Metadata flushed
0809 23:20:56.376138 (+245906us) tablet_metadata.cc:641] Metadata flushed
0809 23:20:56.598758 (+222620us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":746062,"spinlock_wait_cycles":285052672,"tablet-delete.queue_time_us":804596,"thread_start_us":285197,"threads_started":1}
I20260809 23:20:56.796608  1198 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:56.755857  1081 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.797469  1198 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:20:56.797977  1198 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:56.798409  1198 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:56.799075  1081 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:56.760309  1207 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:56.809670 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.818557 24317 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:56.807071 24321 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:56.765911 24314 raft_consensus.cc:2362] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16. Re-sending same reply.
I20260809 23:20:56.821297 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.822350 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:56.824630 25371 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:20:56.813221 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.779817  1295 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:20:56.834133 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:56.835836 24313 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:56.780360  1442 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:56.840061 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:56.840559 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:56.841387 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57547.4us
I20260809 23:20:56.842397 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:56.845453 24323 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:56.877717 24323 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:20:56.887642 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.893980 24323 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:56.900875 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.902164 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.833770  1295 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:56.819922  1382 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.026s	sys 0.030s
I20260809 23:20:56.920599 25372 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:56.826530  1363 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:56.927176  1382 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.026s	sys 0.030s
W20260809 23:20:56.947235  1372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.019s	sys 0.032s
I20260809 23:20:56.947716  1207 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:56.960615 24314 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:20:56.970721 24320 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.626s	user 0.012s	sys 0.001s
I20260809 23:20:56.971168 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
W20260809 23:20:56.936851 24431 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.018s	sys 0.053s
I20260809 23:20:56.967471  1297 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:56.967952 24316 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.009s	sys 0.020s
I20260809 23:20:56.962687 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:56.972304 24431 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.018s	sys 0.053s
I20260809 23:20:56.972451 24430 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:20:56.972754 24316 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.009s	sys 0.020s
I20260809 23:20:56.976250 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.977028 24321 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:56.977258 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:56.981271 24321 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:20:56.972977  1176 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:56.969208  1372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.019s	sys 0.033s
I20260809 23:20:56.997479 24323 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 16. Ops: [14.10-14.10]
W20260809 23:20:57.023854 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.010s	sys 0.027s
W20260809 23:20:57.024396 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.010s	sys 0.027s
I20260809 23:20:57.027767 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:57.029049 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.017s	sys 0.017s
W20260809 23:20:57.029376 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.017s	sys 0.017s
I20260809 23:20:56.984134  1448 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:57.032853 24423 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:20:57.033648 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.010s	sys 0.019s
W20260809 23:20:57.034175 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.011s	sys 0.019s
W20260809 23:20:57.034569 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 632249us
W20260809 23:20:57.035844  1390 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.839s	user 0.013s	sys 0.000s
I20260809 23:20:57.037158  1390 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:57.037734 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:57.038086 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:57.039237 24428 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
W20260809 23:20:57.053947 24435 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.630s	user 0.018s	sys 0.000s
W20260809 23:20:57.011178  1431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.004s	sys 0.027s
I20260809 23:20:57.059836 24435 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:20:56.986064 24320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:57.070868 24320 raft_consensus.cc:595] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:57.071842 24320 raft_consensus.cc:603] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:57.073365 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:57.075130 24423 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.075062 24323 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260809 23:20:57.067411  1431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.004s	sys 0.028s
I20260809 23:20:56.925912  1429 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.080960 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:20:57.055838 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.014s	sys 0.018s
W20260809 23:20:57.082064 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.014s	sys 0.018s
W20260809 23:20:57.085304 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.04587e+06us
I20260809 23:20:57.083283 24323 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260809 23:20:57.084375 24430 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:57.108593 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:57.100091 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 479066us
I20260809 23:20:57.040557  1295 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:57.080678  1205 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:57.114530 25370 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.010s	sys 0.027s
I20260809 23:20:57.089335  1335 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:57.120849 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:57.121752 25370 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.010s	sys 0.027s
I20260809 23:20:57.091899  1440 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:57.122035 24424 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.000s	sys 0.041s
W20260809 23:20:57.205124 24424 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.000s	sys 0.042s
I20260809 23:20:57.204268 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:57.206418 24427 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.078013  1442 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.206086 24323 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:57.207123 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:57.208868 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
W20260809 23:20:57.210346  1454 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260809 23:20:57.215392 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:57.205750  1390 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:57.225189 24430 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260809 23:20:57.225813 24422 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:57.241638 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:57.264801 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51194.4us
W20260809 23:20:57.239055 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.029s	sys 0.047s
W20260809 23:20:57.273985 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.034s	sys 0.049s
W20260809 23:20:57.265762 24152 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57064.4us
I20260809 23:20:57.275173 25371 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:57.277304 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.014s	sys 0.022s
W20260809 23:20:57.284489 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.014s	sys 0.022s
I20260809 23:20:57.239118 25369 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:57.286638 24317 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:20:57.293856 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.014s	sys 0.039s
W20260809 23:20:57.295360 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.014s	sys 0.040s
W20260809 23:20:57.295655 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 534980us
I20260809 23:20:57.303083 24317 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:20:57.303543 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:57.273597  1363 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:57.261289  1295 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:20:57.287987   877 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.028s	sys 0.032s
I20260809 23:20:57.322000 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:57.330267 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.004s	sys 0.012s
W20260809 23:20:57.341333  1430 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.240s	user 0.010s	sys 0.005s
W20260809 23:20:57.335625   877 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.029s	sys 0.033s
I20260809 23:20:57.345398 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:57.340214  1036 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.019s	user 0.015s	sys 0.000s
I20260809 23:20:57.365636 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:57.381654 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59613.1us
W20260809 23:20:57.344129 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.004s	sys 0.012s
I20260809 23:20:57.404891 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:57.383018 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1496) took 1104 ms. Trace:
W20260809 23:20:57.405668 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 634999us
W20260809 23:20:57.414377 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65830.7us
I20260809 23:20:57.305064  1153 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.087s
I20260809 23:20:57.428893 24430 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.349484  1363 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:57.411351 24316 rpcz_store.cc:276] 0809 23:20:56.278911 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.280911 (+  2000us) service_pool.cc:227] Handling call
0809 23:20:56.281237 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:56.281592 (+   355us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:20:56.975885 (+694293us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.379715 (+403830us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:57.381564 (+  1849us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:57.382657 (+  1093us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.382768 (+   111us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.382944 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1049849,"threads_started":2}
I20260809 23:20:57.380378  1390 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:57.380831  1389 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:57.446107 25370 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
W20260809 23:20:57.345857  1392 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.592s	user 0.014s	sys 0.000s
I20260809 23:20:57.470152 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:57.492412 24316 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:20:57.351814  1430 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.524041 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:57.459451  1389 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:57.496091 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 91301.7us
I20260809 23:20:57.502884 24428 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260809 23:20:57.526561 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 742948us
W20260809 23:20:57.479842  1446 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.328s	user 0.006s	sys 0.006s
I20260809 23:20:57.547765 24316 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:20:57.535156  1297 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260809 23:20:57.556555 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86470.5us
I20260809 23:20:57.565352 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:57.479874  1248 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:57.580299 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:20:57.573608 24314 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.587s	user 0.009s	sys 0.003s
I20260809 23:20:57.581081 24314 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:57.582569 24313 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260809 23:20:57.587533 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51766.7us
I20260809 23:20:57.587890 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:57.587908 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:57.589982 25370 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:20:57.591003 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67085.4us
I20260809 23:20:57.588763 24428 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:20:57.597534 25368 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
W20260809 23:20:57.607339 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45912.9us
I20260809 23:20:57.472800  1392 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.595839  1303 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:57.596041  1363 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:57.607306 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54574.7us
I20260809 23:20:57.607759 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.608161 24428 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:57.608669 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:57.630944 24545 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:57.608623  1248 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.595988  1207 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20260809 23:20:57.480810  1390 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:57.629774 24427 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:57.603608  1297 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:57.644276 25365 raft_consensus.cc:1799] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Handling vote request from an unknown peer 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:57.650560 25365 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 17.
I20260809 23:20:57.647470 24428 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260809 23:20:57.643139  1440 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:57.657189 24427 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:57.656903  1389 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.628435  1303 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:57.658506  1303 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.659859 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2675) took 1287 ms. Trace:
I20260809 23:20:57.660034 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:57.660264 24313 rpcz_store.cc:276] 0809 23:20:56.372919 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.987148 (+614229us) service_pool.cc:227] Handling call
0809 23:20:57.582264 (+595116us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:57.589218 (+  6954us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.659632 (+ 70414us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54003c32d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:57.659789 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1311143296}
I20260809 23:20:57.661530 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:57.662736 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3156) took 1387 ms. Trace:
I20260809 23:20:57.663380 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3071) took 1221 ms. Trace:
I20260809 23:20:57.663100 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:57.671430 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 321415us
I20260809 23:20:57.673107 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.678359 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3193) took 1168 ms. Trace:
I20260809 23:20:57.667364 24542 rpcz_store.cc:276] 0809 23:20:56.442327 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.442485 (+   158us) service_pool.cc:227] Handling call
0809 23:20:56.461106 (+ 18621us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.487226 (+ 26120us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.487267 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.487888 (+   621us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.029713 (+541825us) raft_consensus.cc:1631] Marking committed up to 6
0809 23:20:57.029753 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.029821 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:57.663172 (+633351us) raft_consensus.cc:1671] finished
0809 23:20:57.663193 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.663326 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1917,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":85,"thread_start_us":541435,"threads_started":1,"wal-append.queue_time_us":1171197}
I20260809 23:20:57.684024 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:57.682859 24543 rpcz_store.cc:276] 0809 23:20:56.509818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.519801 (+  9983us) service_pool.cc:227] Handling call
0809 23:20:56.569817 (+ 50016us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:56.679156 (+109339us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:57.126132 (+446976us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.677890 (+551758us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:57.677949 (+    59us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:57.677999 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.678065 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.678183 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16168,"prepare.queue_time_us":793017,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":10617,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":882706,"threads_started":2}
I20260809 23:20:57.685374 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.686704 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.688014 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.685447 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:57.686738 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3250) took 1621 ms. Trace:
I20260809 23:20:57.693846 24534 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:57.663111 24309 rpcz_store.cc:276] 0809 23:20:56.275068 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.275237 (+   169us) service_pool.cc:227] Handling call
0809 23:20:56.280990 (+  5753us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.281496 (+   506us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.281523 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.282045 (+   522us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.024827 (+742782us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:20:57.024866 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.024939 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:57.662515 (+637576us) raft_consensus.cc:1671] finished
0809 23:20:57.662547 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.662681 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1243,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":742384,"threads_started":1,"wal-append.queue_time_us":1344409}
I20260809 23:20:57.693998 24535 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:57.744760 24544 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:57.747385 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2772) took 1288 ms. Trace:
W20260809 23:20:57.748024 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69706.7us
I20260809 23:20:57.747816 24424 rpcz_store.cc:276] 0809 23:20:56.458862 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.459005 (+   143us) service_pool.cc:227] Handling call
0809 23:20:56.482218 (+ 23213us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:56.510996 (+ 28778us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:57.207968 (+696972us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.746494 (+538526us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.746659 (+   165us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:57.746715 (+    56us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:57.746761 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.746821 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.747296 (+   475us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":557795,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":54,"thread_start_us":1088960,"threads_started":2}
W20260809 23:20:57.755194 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 163271us
I20260809 23:20:57.747197 24539 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.764593 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3163) took 1187 ms. Trace:
I20260809 23:20:57.765069 24317 rpcz_store.cc:276] 0809 23:20:56.566291 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.321692 (+755401us) service_pool.cc:227] Handling call
0809 23:20:57.348316 (+ 26624us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:57.348548 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:20:57.565481 (+216933us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.752877 (+187396us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:57.752956 (+    79us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:57.753011 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.753533 (+   522us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.753672 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":283017,"threads_started":2}
I20260809 23:20:57.656867  1440 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:57.749732  1386 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:57.676527  1297 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:57.705865 24431 rpcz_store.cc:276] 0809 23:20:56.065728 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.080352 (+ 14624us) service_pool.cc:227] Handling call
0809 23:20:56.108011 (+ 27659us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.204720 (+ 96709us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:20:56.975641 (+770921us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.345904 (+370263us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:57.461007 (+115103us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.461185 (+   178us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:57.461230 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.461318 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:57.686518 (+225200us) raft_consensus.cc:1671] finished
0809 23:20:57.686538 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.686682 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56478,"prepare.queue_time_us":831804,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":3643,"raft.queue_time_us":39355,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":56947,"thread_start_us":1053449,"threads_started":2,"wal-append.queue_time_us":673736}
I20260809 23:20:57.668828  1275 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.091s
I20260809 23:20:57.753063  1207 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:57.746621  1176 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:57.706633  1403 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
W20260809 23:20:57.799762 24423 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.720s	user 0.012s	sys 0.001s
I20260809 23:20:57.795135 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2770) took 1337 ms. Trace:
I20260809 23:20:57.793674 24534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:46304
I20260809 23:20:57.798513 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:57.815544 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:57.819989 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.812656 24429 rpcz_store.cc:276] 0809 23:20:56.458090 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.458224 (+   134us) service_pool.cc:227] Handling call
0809 23:20:56.488417 (+ 30193us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.488922 (+   505us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.488952 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.489428 (+   476us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.285821 (+796393us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:57.285872 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.285961 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:57.794609 (+508648us) raft_consensus.cc:1671] finished
0809 23:20:57.794635 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:57.795034 (+   399us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2156,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":795925,"threads_started":1,"wal-append.queue_time_us":1298763}
I20260809 23:20:57.805320  1176 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:57.830909 24542 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:20:57.820112  1449 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:20:57.832733  1176 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:57.894814 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:57.825227  1431 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:57.897392 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:57.835022  1479 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260809 23:20:57.828651  1390 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:57.807510  1386 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:57.899118 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 210521us
W20260809 23:20:57.894140  1460 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260809 23:20:57.830444  1430 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:57.994690 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 92616.2us
W20260809 23:20:57.905297 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 296673us
W20260809 23:20:57.971884 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.927s (SENT)
I20260809 23:20:57.905098 24542 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:57.900509 24543 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260809 23:20:57.829039 24430 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:57.835427  1485 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:57.830179  1153 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:20:58.001438 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 340002us
I20260809 23:20:58.018200 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.013144 24539 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:57.835281  1481 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:58.019734 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:57.834254  1490 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:57.918730  1392 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:57.918552  1476 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.447s	user 0.000s	sys 0.002s
W20260809 23:20:58.025388 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53589.7us
W20260809 23:20:58.025070 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 228205us
I20260809 23:20:57.833244  1491 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:58.036602 24542 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260809 23:20:58.033463 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.037431 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:57.835516  1484 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.203s
W20260809 23:20:58.040184 24320 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.005s	sys 0.018s
I20260809 23:20:58.024950 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:57.904762  1431 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:58.079417 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:57.903656  1498 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:20:58.084652 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47775.8us
W20260809 23:20:58.082088 24320 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.005s	sys 0.019s
I20260809 23:20:58.020438  1036 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.081452 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.101437 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:57.981529  1449 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260809 23:20:58.102880 24316 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:58.037226  1490 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:58.109620 24542 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260809 23:20:58.129384 24312 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.000s	user 0.010s	sys 0.003s
I20260809 23:20:57.833418  1501 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.495s
I20260809 23:20:58.158576 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.043226  1248 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:20:58.084456   877 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.019s	sys 0.020s
I20260809 23:20:58.158188 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:58.142041   562 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.500s	user 0.012s	sys 0.004s
I20260809 23:20:58.157645 24312 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:58.158497 24535 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:20:58.104532  1198 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.306s	user 0.016s	sys 0.000s
W20260809 23:20:58.168418  1506 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.250s	user 0.000s	sys 0.000s
W20260809 23:20:58.169356  1507 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
I20260809 23:20:58.171002 24317 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:20:57.917990  1516 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.188s
W20260809 23:20:58.177932 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 146147us
I20260809 23:20:58.180069 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.104844  1036 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.157713  1390 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:58.215516  1434 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260809 23:20:58.184056 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:58.248472  1531 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260809 23:20:58.247272  1506 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.054s	user 0.007s	sys 0.007s
I20260809 23:20:58.249588 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:20:58.250550  1535 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260809 23:20:58.251600  1504 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260809 23:20:58.251736 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:58.252671  1508 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.357s	user 0.001s	sys 0.001s
I20260809 23:20:58.253120 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:58.160722  1490 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:58.203898 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3154) took 1988 ms. Trace:
I20260809 23:20:58.271795 24314 raft_consensus.cc:595] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.213441 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.173370  1440 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:58.171806  1541 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:58.233285 24534 raft_consensus.cc:465] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Not starting forced leader election -- already a leader
W20260809 23:20:58.191936 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 173166us
I20260809 23:20:58.272609 24314 raft_consensus.cc:603] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:20:58.175215   877 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.022s	sys 0.022s
I20260809 23:20:58.215626 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:58.215445 24542 raft_consensus.cc:2396] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260809 23:20:58.317993 24425 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:20:58.307987 24540 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:20:58.220779 24313 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:58.335489 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.166764  1491 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:58.334578 24539 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:20:58.261665 24545 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.000s
I20260809 23:20:58.362164 24545 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:20:58.316708 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2763) took 2110 ms (client timeout 2038 ms). Trace:
W20260809 23:20:58.362871 24435 rpcz_store.cc:269] 0809 23:20:56.206098 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.249701 (+ 43603us) service_pool.cc:227] Handling call
0809 23:20:58.316434 (+2066733us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b5400332cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:58.316601 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74982784}
W20260809 23:20:58.299582 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 502571us
W20260809 23:20:58.364216 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 148672us
I20260809 23:20:58.172102  1449 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:58.157811  1392 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:20:58.299640  1550 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
W20260809 23:20:58.314986  1477 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.106s	user 0.001s	sys 0.001s
I20260809 23:20:58.272498 24311 rpcz_store.cc:276] 0809 23:20:56.215025 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.215167 (+   142us) service_pool.cc:227] Handling call
0809 23:20:56.215613 (+   446us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.216046 (+   433us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.216079 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.221121 (+  5042us) log.cc:842] Serialized 26 byte log entry
0809 23:20:56.620583 (+399462us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:56.620630 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:56.620718 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:58.203564 (+1582846us) raft_consensus.cc:1671] finished
0809 23:20:58.203585 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:58.203797 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":399072,"threads_started":1,"wal-append.queue_time_us":756478}
I20260809 23:20:58.207216  1297 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:58.378384 24542 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:20:58.388435  1176 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.389401 24435 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:20:58.389681 24309 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:20:58.390558  1176 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:58.390851  1390 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.391544  1390 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.393955  1390 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:58.383174  1541 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:58.398020  1541 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.301931  1431 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.403410  1431 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:58.378620 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:58.407564  1491 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260809 23:20:58.407765  1546 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260809 23:20:58.408099  1491 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:58.408613  1491 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:58.407585  1487 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.936s	user 0.002s	sys 0.002s
I20260809 23:20:58.416451  1363 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:58.416996  1363 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:20:58.419072  1542 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260809 23:20:58.220333 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1479) took 1602 ms. Trace:
I20260809 23:20:58.318740  1352 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:20:58.420382  1352 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:58.420713   877 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:20:58.422427  1352 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.424126  1352 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:20:58.425981 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190962us
I20260809 23:20:58.428287 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:58.207525  1386 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:58.188177  1198 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.214704  1516 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:58.382858 24430 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:20:58.215965  1480 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.385131 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.431612  1480 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.385373  1539 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.432417 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:58.432677 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54137.6us
I20260809 23:20:58.433195 24313 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:58.432896  1539 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.433838 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.435096  1501 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:20:58.435678  1539 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:58.385764  1440 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:58.386650 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:58.386564  1392 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.387089 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.436558 25366 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:58.437407 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.438328 24314 raft_consensus.cc:2396] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260809 23:20:58.438463  1392 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:58.435954 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.439131 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.378561  1153 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.441105  1153 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:58.443099 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80455.6us
I20260809 23:20:58.299885  1434 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.432099  1516 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:20:58.451577 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 293180us
I20260809 23:20:58.456540 25366 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:20:58.457074 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66170.2us
I20260809 23:20:58.431754  1198 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:20:58.462169 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:58.427845 25363 rpcz_store.cc:276] 0809 23:20:56.583025 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.583174 (+   149us) service_pool.cc:227] Handling call
0809 23:20:56.602064 (+ 18890us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.614508 (+ 12444us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.614537 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.615024 (+   487us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.070422 (+455398us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:20:57.070469 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.070549 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:58.185528 (+1114979us) raft_consensus.cc:1671] finished
0809 23:20:58.185559 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:58.185729 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3530,"raft.queue_time_us":89709,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":6925,"thread_start_us":454944,"threads_started":1,"wal-append.queue_time_us":875250}
I20260809 23:20:58.462303 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.464630 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:58.466704 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 387246us
I20260809 23:20:58.467844 24311 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260809 23:20:58.458619  1335 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.045s	sys 0.072s
I20260809 23:20:58.466894 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3160) took 2075 ms. Trace:
I20260809 23:20:58.418115  1363 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.468832 25366 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:58.490012 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:20:58.465286 25323 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.681s (SENT) [suppressed 47 similar messages]
I20260809 23:20:58.491101 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.476292 24320 rpcz_store.cc:276] 0809 23:20:56.391726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.091710 (+699984us) service_pool.cc:227] Handling call
0809 23:20:57.112349 (+ 20639us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:57.129622 (+ 17273us) raft_consensus.cc:1520] Early marking committed up to index 8
0809 23:20:58.094514 (+964892us) log.cc:842] Serialized 14 byte log entry
0809 23:20:58.466018 (+371504us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:58.466068 (+    50us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:58.466112 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.466706 (+   594us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:58.466828 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74937,"prepare.queue_time_us":620337,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":31,"raft.queue_time_us":164,"thread_start_us":1060368,"threads_started":2}
I20260809 23:20:58.456830  1440 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:58.493294 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.478471 24430 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:20:58.446101   877 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:58.477735 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:58.501776 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:20:58.502703 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.030s	sys 0.069s
W20260809 23:20:58.503052 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.030s	sys 0.069s
W20260809 23:20:58.498898 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45711.5us
I20260809 23:20:58.496778 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:20:58.505080 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 622672us
I20260809 23:20:58.505991 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.506337 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:58.467434  1413 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:58.509990 24320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:58.510637 24320 raft_consensus.cc:595] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:58.511019 24320 raft_consensus.cc:603] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:58.509529  1528 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:20:58.482661  1335 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.046s	sys 0.072s
I20260809 23:20:58.511220 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:58.467483  1481 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:58.511930 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.523082 25363 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:20:58.537837 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.546773 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.548697 25366 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:58.541904 24542 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:58.512357  1198 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:58.468695  1539 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260809 23:20:58.553280 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:58.510177  1413 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:20:58.558178 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51947.7us
I20260809 23:20:58.559268 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.559608  1527 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:58.535879  1440 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.512751  1390 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:58.466832  1434 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:58.590772  1295 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.247s	user 0.016s	sys 0.000s
I20260809 23:20:58.481181  1548 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.073s
I20260809 23:20:58.602371 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.614964 24545 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:20:58.600951 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.507257  1522 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.088s
I20260809 23:20:58.604277 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:58.491014  1297 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260809 23:20:58.550364  1153 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:58.603468 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102874us
I20260809 23:20:58.515592  1386 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.094s
I20260809 23:20:58.591120  1541 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:20:58.603103  1413 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:58.617684 24542 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:20:58.557304  1442 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260809 23:20:58.543414  1528 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.616439  1548 consensus_queue.cc:1225] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:20:58.646544 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.633100 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.649629 24542 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:20:58.651099 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:20:58.651521 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 135076us
I20260809 23:20:58.652518 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:20:58.657577 25323 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.975s (SENT)
I20260809 23:20:58.658707 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.658859 24311 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
W20260809 23:20:58.660022 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 169013us
I20260809 23:20:58.661509 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.663735 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:58.608009  1481 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.670320 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:58.670346 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.669010 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:58.673522  1555 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260809 23:20:58.676010 24311 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:20:58.676191  1468 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:58.676579  1468 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:58.676918  1468 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:58.677448 24311 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260809 23:20:58.677789 24311 pending_rounds.cc:76] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 8
I20260809 23:20:58.678032 24311 pending_rounds.cc:97] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260809 23:20:58.678249 24311 raft_consensus.cc:2866] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:58.678579 24311 pending_rounds.cc:97] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 14.10
W20260809 23:20:58.679239  1510 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.777s	user 0.001s	sys 0.001s
I20260809 23:20:58.679541  1502 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
W20260809 23:20:58.679661  1551 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260809 23:20:58.679996  1502 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:58.681088  1545 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:58.680987  1502 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.681562  1545 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:58.681968  1545 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:58.677804 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3194) took 1219 ms. Trace:
I20260809 23:20:58.683166 24541 rpcz_store.cc:276] 0809 23:20:57.458379 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.458554 (+   175us) service_pool.cc:227] Handling call
0809 23:20:57.479896 (+ 21342us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:57.492668 (+ 12772us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:20:57.898783 (+406115us) log.cc:842] Serialized 14 byte log entry
0809 23:20:58.462874 (+564091us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:58.482638 (+ 19764us) log.cc:842] Serialized 26 byte log entry
0809 23:20:58.482867 (+   229us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:58.482912 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.483561 (+   649us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:58.677572 (+194011us) raft_consensus.cc:1671] finished
0809 23:20:58.677597 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:58.677741 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4806,"prepare.queue_time_us":95214,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":57,"raft.queue_time_us":8429,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":38218,"thread_start_us":818042,"threads_started":2,"wal-append.queue_time_us":504780}
I20260809 23:20:58.617774   877 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.687487   877 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:58.687179  1516 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.688400  1516 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:58.684523 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1235) took 1325 ms. Trace:
I20260809 23:20:58.678824 24311 raft_consensus.cc:2866] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:58.689775 24428 rpcz_store.cc:276] 0809 23:20:57.358892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.633713 (+274821us) service_pool.cc:227] Handling call
0809 23:20:57.646648 (+ 12935us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:57.662859 (+ 16211us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.684301 (+1021442us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b54002f8fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:58.684458 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28721536}
I20260809 23:20:58.691354  1569 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:58.690068 24541 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260809 23:20:58.695782 24311 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260809 23:20:58.697240 24311 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:20:58.698374 24311 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:58.661116  1275 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.663215  1561 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:58.682092  1502 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:58.773630 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1483) took 2118 ms. Trace:
W20260809 23:20:58.776182 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 114709us
I20260809 23:20:58.767373  1390 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:58.671021  1442 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:20:58.795933 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3188) took 2353 ms (client timeout 1966 ms). Trace:
W20260809 23:20:58.794844 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 124563us
I20260809 23:20:58.798164 24311 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260809 23:20:58.776978 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80620us
I20260809 23:20:58.804497 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.761484  1539 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:20:58.658596  1554 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.278s
W20260809 23:20:58.802099 24544 rpcz_store.cc:269] 0809 23:20:56.419202 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.419344 (+   142us) service_pool.cc:227] Handling call
0809 23:20:58.772813 (+2353469us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400116554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:58.772961 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2304576384}
I20260809 23:20:58.796695 25368 rpcz_store.cc:276] 0809 23:20:56.655284 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.658400 (+  3116us) service_pool.cc:227] Handling call
0809 23:20:56.658837 (+   437us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:57.623694 (+964857us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.773348 (+1149654us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:58.773555 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2158234368}
I20260809 23:20:58.810384 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2676) took 2437 ms. Trace:
I20260809 23:20:58.812419 24430 pending_rounds.cc:76] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 11
I20260809 23:20:58.812716 24430 pending_rounds.cc:97] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260809 23:20:58.812959 24430 raft_consensus.cc:2866] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:20:58.776717  1386 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260809 23:20:58.807387  1303 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.140s	user 0.016s	sys 0.000s
I20260809 23:20:58.801524  1434 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:58.794219  1556 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.776158  1553 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:58.665977  1491 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.807699   682 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:20:58.826860 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:58.838397 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:20:58.839066 24313 raft_consensus.cc:595] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.839573 24313 raft_consensus.cc:603] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:20:58.810842 24321 rpcz_store.cc:276] 0809 23:20:56.373245 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.985625 (+612380us) service_pool.cc:227] Handling call
0809 23:20:57.209536 (+223911us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b54003c2228. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac287822dc 00007fac287b637c 00007fac287a180b 00007fac287a12a2 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c
0809 23:20:57.209668 (+   132us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:57.210111 (+   443us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:20:57.706683 (+496572us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.988836 (+282153us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:58.039786 (+ 50950us) log.cc:842] Serialized 26 byte log entry
0809 23:20:58.051738 (+ 11952us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:58.052278 (+   540us) log.cc:842] Serialized 14 byte log entry
0809 23:20:58.322607 (+270329us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b1800364840. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:20:58.810061 (+487454us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.810138 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:58.810183 (+    45us) raft_consensus.cc:1671] finished
0809 23:20:58.810196 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:58.810318 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54078,"prepare.queue_time_us":617429,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4677,"raft.queue_time_us":507042,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":66850,"spinlock_wait_cycles":127830912,"thread_start_us":1118745,"threads_started":3,"wal-append.queue_time_us":446127}
I20260809 23:20:58.801955  1442 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:58.807175  1541 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:58.769984  1275 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.813441  1568 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:58.848726 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 336841us
I20260809 23:20:58.850124 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:58.844549 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 347859us
I20260809 23:20:58.831168  1153 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:58.855248 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50856.9us
I20260809 23:20:58.856176 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:58.833770  1528 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:58.860019 24314 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
W20260809 23:20:58.864135 24316 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.750s	user 0.011s	sys 0.000s
I20260809 23:20:58.822880  1295 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260809 23:20:58.839114  1539 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.836910  1440 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:58.864632 24316 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:58.868554 24311 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:20:58.845307  1553 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:58.838594   682 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:20:58.876780 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1227) took 1908 ms. Trace:
I20260809 23:20:58.845899  1569 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:58.891542 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91249.3us
I20260809 23:20:58.833544  1303 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:58.898865 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 294645us
I20260809 23:20:58.843381  1434 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260809 23:20:58.899858 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:58.880458 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:58.797399  1561 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.847291  1198 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:58.881771 24425 rpcz_store.cc:276] 0809 23:20:56.968276 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.968862 (+   586us) service_pool.cc:227] Handling call
0809 23:20:58.317653 (+1348791us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:58.337504 (+ 19851us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:58.876417 (+538913us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400332cd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:58.876683 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3085312384}
I20260809 23:20:58.887285  1541 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260809 23:20:58.907845 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51779.3us
I20260809 23:20:58.913591 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:58.914625  1557 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:58.915045  1571 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:58.915606  1585 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:20:58.916002  1570 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.16, Next index: 17, Last known committed idx: 13, Time since last communication: 0.081s
I20260809 23:20:58.916671 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260809 23:20:58.919343  1482 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.021s	user 0.001s	sys 0.001s
I20260809 23:20:58.921727  1583 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:20:58.836881  1556 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.923197  1582 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:58.923869  1491 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:20:58.953154 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55828.3us
I20260809 23:20:58.905473  1553 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:58.878849  1539 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:58.959656 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59908.3us
W20260809 23:20:58.928000  1564 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.269s	user 0.002s	sys 0.000s
I20260809 23:20:58.875509  1153 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:58.962517 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.928294  1440 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:58.984607 24541 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:58.985370 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:58.948874  1557 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:58.946981  1568 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:58.969476 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:58.955987  1585 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:58.996609 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:58.955359  1527 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:58.959507  1571 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:58.980018   682 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.000623 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.016865 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3070) took 2587 ms. Trace:
I20260809 23:20:59.014021  1576 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:20:59.023736  1571 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.023480  1576 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.024981  1556 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.025665  1576 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:59.015107  1589 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.027976  1589 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:59.028764  1500 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.219s	user 0.003s	sys 0.000s
I20260809 23:20:59.030129  1589 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.030485   682 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:59.015677  1468 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.030799  1198 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.032894  1468 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.020224 24546 rpcz_store.cc:276] 0809 23:20:56.429345 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.429477 (+   132us) service_pool.cc:227] Handling call
0809 23:20:56.429889 (+   412us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.507972 (+ 78083us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.508014 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.597607 (+ 89593us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.108764 (+511157us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:57.108806 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.108892 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:58.839091 (+1730199us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:20:59.016534 (+177443us) raft_consensus.cc:1671] finished
0809 23:20:59.016561 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:59.016779 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9829632,"thread_start_us":425671,"threads_started":1,"wal-append.queue_time_us":1164334}
I20260809 23:20:58.922749  1586 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260809 23:20:58.896049  1390 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:59.036263 24498 connection.cc:232] RPC call timeout handler was delayed by -2.7298713540658355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31970000000000004s for a response
I20260809 23:20:59.014555  1577 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.119s
I20260809 23:20:59.045897 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:59.051433 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:59.052192 24435 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:59.055492 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:58.991577  1352 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:20:58.995010  1557 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:58.999073  1491 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:58.992296  1582 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:59.056026 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1486) took 2399 ms. Trace:
I20260809 23:20:59.056257 24425 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:59.062608 24423 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:20:59.001564  1585 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:20:59.041543  1431 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
I20260809 23:20:59.063795 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.096988 24535 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260809 23:20:58.994176  1528 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.051s
W20260809 23:20:59.074144 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 160667us
W20260809 23:20:59.074687 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 160019us
W20260809 23:20:59.078238 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 420687us
I20260809 23:20:59.044730  1434 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:20:59.080808 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.081084 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:58.994295  1522 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260809 23:20:59.092698  1352 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:59.093788  1545 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.075323  1468 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:20:59.095636  1303 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.000197  1568 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.107297 24429 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:20:58.995178  1553 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260809 23:20:59.055545  1571 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260809 23:20:58.999260  1541 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
W20260809 23:20:59.123247 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71763.9us
I20260809 23:20:59.077776 25375 rpcz_store.cc:276] 0809 23:20:56.656632 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.785298 (+128666us) service_pool.cc:227] Handling call
0809 23:20:56.785652 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:56.795224 (+  9572us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:56.795264 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:56.795863 (+   599us) log.cc:842] Serialized 26 byte log entry
0809 23:20:57.282069 (+486206us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:20:57.505021 (+222952us) log.cc:842] Serialized 14 byte log entry
0809 23:20:57.539536 (+ 34515us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:57.539640 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:20:59.028301 (+1488661us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b240036d418. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:20:59.040022 (+ 11721us) raft_consensus.cc:1671] finished
0809 23:20:59.055827 (+ 15805us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:20:59.055967 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50801,"prepare.queue_time_us":479667,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":64941,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":8386,"spinlock_wait_cycles":4398464,"thread_start_us":677703,"threads_started":2,"wal-append.queue_time_us":919996}
I20260809 23:20:59.015538  1610 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.089s
I20260809 23:20:59.079282  1491 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:59.128423 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:59.133651 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1481) took 2479 ms. Trace:
W20260809 23:20:59.137637 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 137084us
I20260809 23:20:59.121677  1275 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:59.147753 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.174181 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:59.202123 24435 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:20:59.175040 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.133882  1583 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
W20260809 23:20:59.178503 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53877.3us
I20260809 23:20:59.179397 24431 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:20:59.180347 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.181880 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.113121  1431 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.134589  1429 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:59.233461 24546 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:20:59.171648  1413 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.000s
I20260809 23:20:59.211131 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:20:59.177534 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 120656us
I20260809 23:20:59.202054 25369 rpcz_store.cc:276] 0809 23:20:56.654233 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.657919 (+  3686us) service_pool.cc:227] Handling call
0809 23:20:57.234576 (+576657us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:57.296356 (+ 61780us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.133362 (+1837006us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b540065d6d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:20:59.133570 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1325290240}
I20260809 23:20:59.226816  1468 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:59.227620  1604 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:20:59.228096  1586 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.229988  1537 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:59.210341  1491 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:59.200100  1490 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.897s	user 0.006s	sys 0.007s
I20260809 23:20:59.124125  1434 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:20:59.241293 24498 connection.cc:232] RPC call timeout handler was delayed by -2.6353713541757315s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33020000000000005s for a response
I20260809 23:20:59.243219 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1400) took 1210 ms. Trace:
I20260809 23:20:59.243868 24428 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18.
I20260809 23:20:59.251947 24429 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:20:59.218211  1583 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:59.257576 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:59.221810 24425 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 13.
I20260809 23:20:59.270371 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.227612  1352 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.274086 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.276340 24539 rpcz_store.cc:276] 0809 23:20:58.032991 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.189163 (+156172us) service_pool.cc:227] Handling call
0809 23:20:58.292311 (+103148us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:58.365210 (+ 72899us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.242957 (+877747us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:59.243158 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96704384}
I20260809 23:20:59.280706 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.280946 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.282393 25375 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:20:59.263931  1275 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.258354  1561 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.122069  1527 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.308342 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.231902  1198 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.056s
I20260809 23:20:59.240036  1431 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:20:59.328775 24498 connection.cc:232] RPC call timeout handler was delayed by -2.0192267009988427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
W20260809 23:20:59.329314 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9616267010569572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260809 23:20:59.329653 24498 connection.cc:232] RPC call timeout handler was delayed by -1.8892713540699333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4131s for a response
I20260809 23:20:59.325410 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:59.329996 24498 connection.cc:232] RPC call timeout handler was delayed by -1.8892713540699333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4131s for a response
I20260809 23:20:59.330628 25375 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:59.295918  1539 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.343083 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:59.245272  1413 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.314603  1153 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.352857 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:20:59.259689  1604 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.357088  1584 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:20:59.357568  1584 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:59.357935  1584 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:20:59.354681  1539 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260809 23:20:59.362813  1598 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260809 23:20:59.364463 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.358739  1604 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.365736 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.366344 24539 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:59.274857  1490 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.368732  1490 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.369958  1480 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:59.371232 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.371654  1568 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.371234 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:59.376955   717 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:20:59.377389   717 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:20:59.383158  1545 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.383021  1611 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.180s
I20260809 23:20:59.384002  1604 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:20:59.384378  1604 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:20:59.385102  1625 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260809 23:20:59.385320  1604 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.387075  1604 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:59.387281  1583 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.388669  1583 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:20:59.353485   877 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.665s	user 0.007s	sys 0.009s
I20260809 23:20:59.378268   717 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.323995  1522 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.286017  1582 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.262327  1586 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.394907 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.421051 25372 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260809 23:20:59.351599 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70739us
I20260809 23:20:59.407394  1431 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.397099  1637 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:59.424454 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:59.407637  1434 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.424858  1637 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.428036  1637 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.428186  1413 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.313753  1275 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.428687  1392 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.429898 25369 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
I20260809 23:20:59.430222  1440 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:59.430377  1586 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:59.432801  1516 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.744s	user 0.009s	sys 0.002s
W20260809 23:20:59.433840  1595 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.314s	user 0.000s	sys 0.000s
I20260809 23:20:59.434139  1557 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.438212  1491 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.397493  1629 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:59.525166 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.442543  1431 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.398629  1468 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.16, Last appended by leader: 16, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.403541   717 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:59.407946  1522 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:59.455132 24498 connection.cc:232] RPC call timeout handler was delayed by -1.7708267010748386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260809 23:20:59.535001 24498 connection.cc:232] RPC call timeout handler was delayed by -1.4909267011098564s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
W20260809 23:20:59.535334 24498 connection.cc:232] RPC call timeout handler was delayed by -1.0054068039171398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
I20260809 23:20:59.455590  1481 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.398360  1634 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:20:59.461707 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49469.8us
I20260809 23:20:59.500895 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.423667 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:20:59.508913 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 396240us
I20260809 23:20:59.397945  1624 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.521598 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.540735 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1391) took 1833 ms. Trace:
I20260809 23:20:59.541118 24544 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:20:59.527038 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:59.524192 24546 raft_consensus.cc:2362] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17. Re-sending same reply.
W20260809 23:20:59.537949 24430 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.008s	sys 0.003s
W20260809 23:20:59.538810 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 115243us
I20260809 23:20:59.541205 24543 rpcz_store.cc:276] 0809 23:20:57.707052 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.753759 (+ 46707us) service_pool.cc:227] Handling call
0809 23:20:57.780498 (+ 26739us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:58.014387 (+233889us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.537171 (+1522784us) spinlock_profiling.cc:243] Waited 53.3 ms on lock 0x7b54003dca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:20:59.540673 (+  3502us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":122643328}
I20260809 23:20:59.542414 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.542989 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:59.543243 24430 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.008s	sys 0.004s
I20260809 23:20:59.543308 24539 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:59.529918  1545 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:20:59.547886 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.499737  1392 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.547201  1537 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:59.552454   682 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260809 23:20:59.552335  1440 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:59.532662  1431 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.537660  1629 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.550496  1481 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.550537  1624 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:20:59.556785  1642 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.022s
I20260809 23:20:59.572242  1629 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.573433  1624 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:20:59.552243   877 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:20:59.574640  1036 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.577137 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.578172 24435 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260809 23:20:59.578769 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.552246 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:20:59.579921 24428 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:59.582883  1275 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:20:59.584874 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.585335 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.585950 24435 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:59.586685 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.586831  1645 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:20:59.587807 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:59.588120  1645 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:59.588244 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.589352 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:59.589430 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:59.527240  1637 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260809 23:20:59.589911  1645 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:20:59.590420 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.552142  1634 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.592485  1629 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:20:59.592630 24428 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:20:59.592942  1629 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:20:59.594431  1557 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.556840 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.591049 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:59.587121  1449 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:20:59.591432  1275 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:20:59.580008 24543 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:20:59.576805 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:59.595748 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:20:59.596009 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:59.568758 25368 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260809 23:20:59.598511 25366 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:20:59.599520 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:20:59.564934  1558 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.931s	user 0.000s	sys 0.009s
I20260809 23:20:59.599704 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.603488 25366 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:20:59.584095  1502 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.784s	user 0.012s	sys 0.000s
I20260809 23:20:59.624464 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:20:59.652990 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53551.4us
I20260809 23:20:59.653003 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:20:59.584133  1585 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:20:59.658164 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62203.7us
I20260809 23:20:59.571933  1481 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.659155 25366 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:20:59.662010 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.601460  1645 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:59.607291  1036 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:20:59.680096 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 103462us
I20260809 23:20:59.652526  1650 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:20:59.561067  1522 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:59.680157 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100496us
I20260809 23:20:59.647984  1557 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:59.552403  1352 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.111s
I20260809 23:20:59.685635 24423 raft_consensus.cc:1203] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Deduplicated request from leader. Original: 13.11->[15.12-17.13]   Dedup: 17.13->[]
I20260809 23:20:59.685797 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:59.683758 25321 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18.
I20260809 23:20:59.570428  1431 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.615232  1434 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.071s
W20260809 23:20:59.679421  1658 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260809 23:20:59.552620  1545 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.695979 24423 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: [15.12-17.13]
I20260809 23:20:59.700824 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:59.696213 25321 leader_election.cc:401] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260809 23:20:59.687378 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:20:59.703615 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 114331us
I20260809 23:20:59.703805 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:20:59.708074 24543 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:59.708516 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:59.678542  1585 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:20:59.709259 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52628.3us
I20260809 23:20:59.713527 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:20:59.709874 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:59.714666 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:59.714771 24313 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:20:59.686499  1650 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:59.719051 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:59.660895  1449 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:20:59.681936  1645 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:59.677850  1634 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:20:59.663574  1413 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260809 23:20:59.663725  1429 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.045s
I20260809 23:20:59.715550 24320 raft_consensus.cc:2362] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15. Re-sending same reply.
I20260809 23:20:59.716166 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:59.680779  1638 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:20:59.720445 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:20:59.720794 24430 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:20:59.700459   877 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:20:59.721618 24425 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:59.694525  1528 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:20:59.712122 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:59.722266 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:59.722241  1650 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:20:59.559381  1537 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.712507 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:59.751654 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:20:59.701184  1583 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:20:59.702013  1557 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:20:59.694607  1649 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:20:59.731141 24313 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:20:59.732131  1634 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:20:59.715139  1644 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:20:59.715458  1582 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:20:59.736918 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:20:59.776181 24435 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260809 23:20:59.824842 24430 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
W20260809 23:20:59.759541  1653 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260809 23:20:59.742385 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.020071 24320 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:20:59.722779 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:59.746335 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.727762  1645 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:20:59.727567  1198 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:20:59.760671  1557 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.066406 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:00.069090 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 346808us
I20260809 23:20:59.775749  1413 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:59.731956  1429 consensus_queue.cc:1225] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260809 23:20:59.757928 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:20:59.655807  1275 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:20:59.736248  1449 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:20:59.733752  1614 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260809 23:20:59.675980  1629 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.125308 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:20:59.780417  1667 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.356s
I20260809 23:20:59.749944  1528 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:20:59.780161 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:20:59.734062 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:00.012743 24430 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:21:00.034732 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3256) took 2915 ms. Trace:
I20260809 23:21:00.070397 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:20:59.749063 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:00.075966 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 375940us
I20260809 23:20:59.896533  1644 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:00.079133 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:00.149493 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 369390us
I20260809 23:20:59.896127  1611 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.193s
I20260809 23:21:00.079972 24539 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:00.158136 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:20:59.780649  1491 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:00.065953  1595 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.307s	user 0.001s	sys 0.000s
I20260809 23:20:59.896662  1480 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:00.099467 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.087419 24313 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17. Re-sending same reply.
I20260809 23:21:00.140030 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.143155 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.148854 25366 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:00.147727 24422 rpcz_store.cc:276] 0809 23:20:57.119718 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.119979 (+   261us) service_pool.cc:227] Handling call
0809 23:20:57.120322 (+   343us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:57.280221 (+159899us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.034295 (+2754074us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:00.034625 (+   330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239925632}
I20260809 23:21:00.069083  1571 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.172813 24435 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:00.084487 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.152652 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:00.152908  1522 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:00.079926  1413 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:00.091789  1275 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:00.161774 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91489.3us
I20260809 23:21:00.174146 24320 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:20:59.737999  1584 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.182909 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.183022 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:20:59.788564   877 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.145243  1638 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.155166  1644 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:00.212859 24313 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:00.212771 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3217) took 1193 ms. Trace:
W20260809 23:21:00.221935 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 487906us
I20260809 23:21:00.011868  1649 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.229233 24541 rpcz_store.cc:276] 0809 23:20:59.019550 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.026001 (+  6451us) service_pool.cc:227] Handling call
0809 23:20:59.026424 (+   423us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:59.026883 (+   459us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:59.026921 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:59.043809 (+ 16888us) log.cc:842] Serialized 26 byte log entry
0809 23:20:59.507473 (+463664us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:20:59.507520 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.507814 (+   294us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:00.212470 (+704656us) raft_consensus.cc:1671] finished
0809 23:21:00.212496 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.212689 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9856,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6142,"thread_start_us":403174,"threads_started":1,"wal-append.queue_time_us":469663}
I20260809 23:21:00.241556 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.229246 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.167452  1480 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:00.250481 24313 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:21:00.258010 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:00.259177 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.074s	sys 0.124s
W20260809 23:21:00.259895 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.074s	sys 0.125s
I20260809 23:21:00.254104 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:00.203521  1413 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.183990  1579 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:21:00.260810 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 525597us
I20260809 23:21:00.254902  1390 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:21:00.213233  1275 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.275796 24424 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260809 23:21:00.251158 24316 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:21:00.269116 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 556648us
I20260809 23:21:00.228760  1449 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:21:00.238741  1614 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:00.184851  1491 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
W20260809 23:21:00.241019  1654 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.557s	user 0.001s	sys 0.001s
I20260809 23:21:00.274641  1198 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:00.291296 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:00.292482 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.258709 24313 raft_consensus.cc:2362] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17. Re-sending same reply.
I20260809 23:21:00.300885 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:21:00.264815  1480 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:00.217824  1440 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
W20260809 23:21:00.306526 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 123594us
I20260809 23:21:00.301651 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.305001 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:00.200484  1571 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.308835 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.251992 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.309397 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.319141 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:00.320003 24541 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:00.320518 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:00.317628 25366 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:00.292402  1545 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.327744 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.329576  1528 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:00.338836 24269 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.682s (SENT)
I20260809 23:21:00.340310 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.341004 24313 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:00.235989  1638 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.346306 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.330374  1429 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:00.330235  1449 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:00.351011 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.352988 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:00.366639 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1488) took 3329 ms (client timeout 4131 ms). Trace:
W20260809 23:21:00.377416 24425 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.639s	user 0.006s	sys 0.006s
I20260809 23:21:00.371410 25371 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
W20260809 23:21:00.377657 25365 rpcz_store.cc:269] 0809 23:20:57.037563 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.037709 (+   146us) service_pool.cc:227] Handling call
0809 23:21:00.366385 (+3328676us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b540071dc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.366543 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1392230272}
I20260809 23:21:00.381225 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:00.382161 24320 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:00.377960 24425 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:00.397354 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:00.400400 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.345451  1677 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:00.352617  1429 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:00.375316  1650 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.577s	user 0.008s	sys 0.000s
I20260809 23:21:00.401278 24316 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 13.
I20260809 23:21:00.430050 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:00.388139  1528 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.439620 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.440804 25365 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260809 23:21:00.408811  1390 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:00.406596  1434 raft_consensus.cc:2741] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260809 23:21:00.410470  1579 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:00.430686  1585 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:00.429450  1429 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.457193 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:00.458076 24543 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.741s	user 0.008s	sys 0.003s
I20260809 23:21:00.463033 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1242) took 1317 ms. Trace:
I20260809 23:21:00.436045  1545 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:21:00.461975 24543 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:00.463511 24427 rpcz_store.cc:276] 0809 23:20:59.145475 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.145671 (+   196us) service_pool.cc:227] Handling call
0809 23:20:59.192728 (+ 47057us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:59.243057 (+ 50329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:59.243090 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:20:59.268179 (+ 25089us) log.cc:842] Serialized 52 byte log entry
0809 23:20:59.550884 (+282705us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:20:59.550942 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.551014 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:00.462696 (+911682us) raft_consensus.cc:1671] finished
0809 23:21:00.462726 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.462959 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25204,"raft.queue_time_us":50005,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10780,"thread_start_us":237454,"threads_started":1,"wal-append.queue_time_us":400013}
I20260809 23:21:00.466712 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.454254  1579 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:00.468835  1579 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.469523 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.445758  1528 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.471916  1392 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.444674  1390 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:00.472971  1390 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.454277  1675 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:00.474030  1675 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:00.475236  1675 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.476624  1675 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:21:00.438810  1635 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.209s	user 0.038s	sys 0.043s
I20260809 23:21:00.467291 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.461345  1649 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:00.483141 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 327069us
I20260809 23:21:00.431834  1650 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.468011  1545 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:00.494261 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.495770 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.480985 25366 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:00.500481 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:00.504737  1678 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260809 23:21:00.505179  1571 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:00.480199  1635 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.251s	user 0.038s	sys 0.044s
I20260809 23:21:00.502213 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:00.508702 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:00.511512 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.512404 24316 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:21:00.513494 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:00.471912  1528 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.479007  1585 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:00.524844 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.522174  1571 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.528107  1485 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:00.531746  1485 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:00.532279 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:46304 (request call id 1389) took 2869 ms. Trace:
I20260809 23:21:00.531724  1571 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.531937  1583 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:00.533713  1583 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:00.534138 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.534840 24316 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:00.535275  1621 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.179s	user 0.002s	sys 0.012s
I20260809 23:21:00.535356  1691 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.532714  1485 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:00.536491  1573 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:00.536096  1691 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.535147  1583 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.532733 24534 rpcz_store.cc:276] 0809 23:20:57.662580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.722660 (+ 60080us) service_pool.cc:227] Handling call
0809 23:21:00.532079 (+2809419us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23857da7 00007fac287a3ee0 00007fac21e95784 00007fac21e95712 00007fac21e95641 00007fac21e955bc 00007fac21e94872 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.532170 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":963686784}
I20260809 23:21:00.540453  1390 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.522066  1556 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:00.437284  1198 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.544713  1638 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.551515  1485 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.580614  1390 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.581105 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1522) took 2100 ms. Trace:
I20260809 23:21:00.555281  1688 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:00.502679  1545 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:00.559110  1573 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:00.561074 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.596315  1573 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.597033 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.597910  1573 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:00.598093 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1503) took 1031 ms. Trace:
I20260809 23:21:00.590665  1638 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260809 23:21:00.598662 25365 rpcz_store.cc:276] 0809 23:20:59.566451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.596686 (+1030235us) service_pool.cc:227] Handling call
0809 23:21:00.598022 (+  1336us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:00.567651 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 162060us
I20260809 23:21:00.451607  1677 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.604178 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.604929 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1497) took 1075 ms. Trace:
I20260809 23:21:00.571707  1694 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:21:00.460386  1449 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.612829  1413 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.507994  1647 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.069s
I20260809 23:21:00.581993 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:00.582073 24269 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.924s (SENT)
I20260809 23:21:00.619117  1571 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.619256 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:00.620193  1571 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:00.620664  1571 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:00.621048  1571 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:00.621941  1649 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:00.581475 24311 rpcz_store.cc:276] 0809 23:20:58.481067 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.803822 (+322755us) service_pool.cc:227] Handling call
0809 23:20:58.815609 (+ 11787us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:58.877422 (+ 61813us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.580866 (+1703444us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:00.581045 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":114394880}
I20260809 23:21:00.583985 24316 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:00.581562  1485 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.502638  1522 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:00.623594 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.624275 24311 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:00.624034  1522 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.557343  1584 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.606302  1677 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.570660 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:00.628178 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:00.628473 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.608613 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.631280 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:00.623670 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.632629  1675 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:00.632829 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.634083  1584 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:00.634761  1584 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:00.635000 24316 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:21:00.635643  1584 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.636268 24316 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:00.638916  1681 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260809 23:21:00.639983  1595 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260809 23:21:00.584946  1202 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:00.640933 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:00.530414  1680 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:00.605049  1691 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.648659 24313 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:00.649601 24311 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:00.646293 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.611966 25365 rpcz_store.cc:276] 0809 23:20:59.529428 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.603911 (+1074483us) service_pool.cc:227] Handling call
0809 23:21:00.604834 (+   923us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.651714 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:00.657801 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.659694 24424 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260809 23:21:00.641289  1202 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.665199 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.664165 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.664553 24431 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:00.666327 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:00.671057 24428 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260809 23:21:00.651340   717 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.114s	user 0.012s	sys 0.001s
I20260809 23:21:00.594084  1585 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:00.652518  1645 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.572s	user 0.012s	sys 0.001s
I20260809 23:21:00.683583 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.709049 25365 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:00.688759 24428 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:21:00.646307  1604 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:00.647718  1680 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:00.709956 24424 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
W20260809 23:21:00.710748 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49606.8us
I20260809 23:21:00.664281  1485 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:00.655265  1691 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:00.717036 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1498) took 1169 ms. Trace:
I20260809 23:21:00.685149  1635 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:00.720314 25365 rpcz_store.cc:276] 0809 23:20:59.547967 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.650751 (+1102784us) service_pool.cc:227] Handling call
0809 23:21:00.716942 (+ 66191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.726030 25363 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:00.661023  1484 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:21:00.727830 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.708096  1667 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.727891 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3192) took 1029 ms. Trace:
I20260809 23:21:00.655313  1275 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.719492  1352 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.746552 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.713128  1680 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:00.737108 24313 rpcz_store.cc:276] 0809 23:20:59.698593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.649324 (+950731us) service_pool.cc:227] Handling call
0809 23:21:00.649613 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:00.649800 (+   187us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:00.673244 (+ 23444us) log.cc:842] Serialized 14 byte log entry
0809 23:21:00.727592 (+ 54348us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:00.727641 (+    49us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:00.727681 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.727739 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.727838 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23302,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":54034,"threads_started":1}
I20260809 23:21:00.683868  1649 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.043s
I20260809 23:21:00.684646  1537 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.042s
I20260809 23:21:00.672741  1647 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260809 23:21:00.655130  1683 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.071s
I20260809 23:21:00.755955  1683 consensus_queue.cc:1225] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:21:00.747587 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1499) took 1182 ms. Trace:
I20260809 23:21:00.756588 25365 rpcz_store.cc:276] 0809 23:20:59.565142 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.727220 (+1162078us) service_pool.cc:227] Handling call
0809 23:21:00.747486 (+ 20266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.750901  1352 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.757630 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.759924  1202 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.759984  1352 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.726028  1485 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:00.767448 24541 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:00.738194   717 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.725353  1635 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:00.773470 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61521.4us
I20260809 23:21:00.774456 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:00.775519 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:00.775605 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:00.720028  1698 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.715050  1645 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.782503 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:00.782998 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.754552  1480 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.769986  1202 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.740569  1691 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.769522  1413 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.774410  1650 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.781278  1537 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:00.804095 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:00.742813  1667 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.805226  1635 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.805611  1545 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.786420  1335 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:00.818348  1275 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.819420 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1397) took 2953 ms. Trace:
I20260809 23:21:00.820124  1688 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:00.808149 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1411) took 1102 ms. Trace:
I20260809 23:21:00.815862  1528 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:00.807802  1557 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.713s	user 0.013s	sys 0.001s
W20260809 23:21:00.807420  1644 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.552s	user 0.009s	sys 0.000s
I20260809 23:21:00.819772 24535 rpcz_store.cc:276] 0809 23:20:57.866411 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.187295 (+320884us) service_pool.cc:227] Handling call
0809 23:20:58.291371 (+104076us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:59.102170 (+810799us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.819166 (+1716996us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b5400147154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:00.819358 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1827800832}
I20260809 23:21:00.831187  1335 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.864807  1429 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.832580  1545 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:00.833153 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:00.860584  1528 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.871224 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.871672 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.872713  1528 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.836019 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.867390  1557 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.833878  1485 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.873888 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.875227  1557 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.877067 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:00.877926 25365 raft_consensus.cc:2354] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18.
I20260809 23:21:00.878367 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:00.879361  1501 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:00.879979  1501 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:00.881490  1429 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:00.881939 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77929us
I20260809 23:21:00.880000 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1510) took 1224 ms. Trace:
I20260809 23:21:00.883958 25365 rpcz_store.cc:276] 0809 23:20:59.655517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.757301 (+1101784us) service_pool.cc:227] Handling call
0809 23:21:00.879802 (+122501us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.879942 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273217920}
I20260809 23:21:00.885005 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.836439  1645 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.840378  1667 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.886435 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3077) took 2407 ms. Trace:
I20260809 23:21:00.886830 24540 rpcz_store.cc:276] 0809 23:20:58.479026 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.495804 (+ 16778us) service_pool.cc:227] Handling call
0809 23:20:58.515257 (+ 19453us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:58.957070 (+441813us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:58.957119 (+    49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:20:58.957173 (+    54us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:20:58.957218 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.886178 (+1928960us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54002706d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:00.886231 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.886375 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1015440256}
I20260809 23:21:00.888331 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:00.888960 24540 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.842487 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1247) took 1266 ms. Trace:
I20260809 23:21:00.889945 24425 rpcz_store.cc:276] 0809 23:20:59.576374 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.576510 (+   136us) service_pool.cc:227] Handling call
0809 23:21:00.842250 (+1265740us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.842409 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":283917696}
I20260809 23:21:00.843918  1480 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.778656   717 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.892120  1553 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:00.892655  1553 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:00.893390   717 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.845498 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.847450 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1480) took 2476 ms. Trace:
I20260809 23:21:00.848405 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:00.849639  1634 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.953s	user 0.012s	sys 0.005s
I20260809 23:21:00.894831  1634 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:00.895113 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3219) took 1792 ms (client timeout 1966 ms). Trace:
I20260809 23:21:00.895525 24321 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:00.822535  1390 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.894419 24429 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260809 23:21:00.896625  1390 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:00.897563 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1507) took 2817 ms (client timeout 1681 ms). Trace:
W20260809 23:21:00.898003 24309 rpcz_store.cc:269] 0809 23:20:58.080164 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.080302 (+   138us) service_pool.cc:227] Handling call
0809 23:21:00.897385 (+2817083us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54002530d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.897498 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":491935104}
I20260809 23:21:00.899217  1501 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:00.899533 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.898079  1335 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:00.900252 24309 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:00.901422 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.809796  1413 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.902201  1413 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.904803  1413 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.788055  1698 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.856666  1691 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.859337 24431 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:00.894116 25371 rpcz_store.cc:276] 0809 23:20:58.370711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.382983 (+ 12272us) service_pool.cc:227] Handling call
0809 23:21:00.371206 (+1988223us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:00.378697 (+  7491us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.847249 (+468552us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b540071dc50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.847394 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4544017280}
I20260809 23:21:00.907438 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1243) took 1737 ms. Trace:
I20260809 23:21:00.907873 24423 rpcz_store.cc:276] 0809 23:20:59.169909 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.241859 (+ 71950us) service_pool.cc:227] Handling call
0809 23:20:59.685517 (+443658us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:20:59.709759 (+ 24242us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.907243 (+1197484us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b5400061b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.907378 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":971167104}
I20260809 23:21:00.908457 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.909925 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1511) took 1248 ms. Trace:
W20260809 23:21:00.825655  1582 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.908s	user 0.005s	sys 0.007s
W20260809 23:21:00.857543  1618 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.499s	user 0.001s	sys 0.009s
I20260809 23:21:00.809439  1537 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:00.799594  1202 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.872566 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.894164  1498 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:00.845007 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:00.886773 25365 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:21:00.932425 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:00.932988 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1508) took 1365 ms. Trace:
I20260809 23:21:00.933435 25365 rpcz_store.cc:276] 0809 23:20:59.567955 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.884729 (+1316774us) service_pool.cc:227] Handling call
0809 23:21:00.932892 (+ 48163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.897032 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1533) took 1205 ms. Trace:
I20260809 23:21:00.934612 24321 rpcz_store.cc:276] 0809 23:20:59.691893 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.722883 (+ 30990us) service_pool.cc:227] Handling call
0809 23:21:00.896883 (+1174000us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.896979 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2295639424}
I20260809 23:21:00.821235 24544 rpcz_store.cc:276] 0809 23:20:59.705969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.471904 (+765935us) service_pool.cc:227] Handling call
0809 23:21:00.472331 (+   427us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:00.653968 (+181637us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:00.654007 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:00.657243 (+  3236us) log.cc:842] Serialized 26 byte log entry
0809 23:21:00.777545 (+120302us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:00.777593 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.777672 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:00.807285 (+ 29613us) raft_consensus.cc:1671] finished
0809 23:21:00.807374 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.808049 (+   675us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":107027,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"spinlock_wait_cycles":35712,"thread_start_us":274136,"threads_started":2,"wal-append.queue_time_us":120515}
W20260809 23:21:00.895538 24546 rpcz_store.cc:269] 0809 23:20:59.102612 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.242732 (+140120us) service_pool.cc:227] Handling call
0809 23:21:00.894861 (+1652129us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b5400116554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:00.895028 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":565617408}
I20260809 23:21:00.855645  1579 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.910682 25371 rpcz_store.cc:276] 0809 23:20:59.661545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.908146 (+1246601us) service_pool.cc:227] Handling call
0809 23:21:00.909817 (+  1671us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.901911 24309 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:00.907467 24540 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:00.923019 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.924489  1545 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:00.925953  1442 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 16.16, Last appended by leader: 16, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.927870 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.947886 24430 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:00.949189 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:00.949421 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.929405  1202 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.930197  1537 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:00.931396 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3211) took 2037 ms. Trace:
I20260809 23:21:00.954952 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.955454 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.939914 24535 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:00.956411 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.957401 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1513) took 1279 ms. Trace:
I20260809 23:21:00.957993 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:00.923904  1528 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:00.893666  1553 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.964671  1528 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.953059 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.966547 24321 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:00.955996 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1512) took 1278 ms. Trace:
I20260809 23:21:00.969638 25365 rpcz_store.cc:276] 0809 23:20:59.677670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.934315 (+1256645us) service_pool.cc:227] Handling call
0809 23:21:00.955940 (+ 21625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.970062  1545 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:00.970511  1545 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:00.970623 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:00.971503 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1494) took 1240 ms. Trace:
I20260809 23:21:00.971853 25365 rpcz_store.cc:276] 0809 23:20:59.731268 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.970332 (+1239064us) service_pool.cc:227] Handling call
0809 23:21:00.971447 (+  1115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.971613  1545 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.972780 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:00.973834  1585 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:00.974395  1585 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:00.975433  1585 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.977322 24430 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:21:00.931699  1582 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18.
I20260809 23:21:00.950219 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:00.947525  1434 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:00.932027  1498 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:00.961259 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.987804 25366 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:00.961515  1442 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:00.943343  1571 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:00.988670 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.989843 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.946475  1698 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:00.990387 24535 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:00.955108 24545 rpcz_store.cc:276] 0809 23:20:58.894190 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.894340 (+   150us) service_pool.cc:227] Handling call
0809 23:20:58.907301 (+ 12961us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:58.946445 (+ 39144us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:58.946476 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:20:58.982429 (+ 35953us) log.cc:842] Serialized 26 byte log entry
0809 23:20:59.077596 (+ 95167us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:20:59.077646 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:20:59.077732 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:00.850377 (+1772645us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:00.931020 (+ 80643us) raft_consensus.cc:1671] finished
0809 23:21:00.931050 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:00.931309 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13968,"raft.queue_time_us":35901,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":8315,"spinlock_wait_cycles":8736640,"thread_start_us":42394,"threads_started":1,"wal-append.queue_time_us":347287}
I20260809 23:21:00.991140  1571 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.987622 24309 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
I20260809 23:21:00.947149 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:00.962008 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:00.963819  1579 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.951311  1390 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:00.965441  1553 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:21:00.945483  1704 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:21:00.910853  1644 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:00.978250  1638 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:01.001616 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:01.002723 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55682.7us
I20260809 23:21:00.973522 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1490) took 1277 ms. Trace:
I20260809 23:21:01.003722 25365 rpcz_store.cc:276] 0809 23:20:59.696437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.972518 (+1276081us) service_pool.cc:227] Handling call
0809 23:21:00.973475 (+   957us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.957825 25371 rpcz_store.cc:276] 0809 23:20:59.678274 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.944080 (+1265806us) service_pool.cc:227] Handling call
0809 23:21:00.957340 (+ 13260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:00.993553 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:00.960852 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:00.994803 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:00.991693 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.007486 24545 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:00.973255  1545 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:01.005098 24309 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:01.006801 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:01.010579 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:01.010851 24430 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:01.014357  1644 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:01.015362 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1501) took 1709 ms. Trace:
I20260809 23:21:01.015705 25368 rpcz_store.cc:276] 0809 23:20:59.306283 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.306435 (+   152us) service_pool.cc:227] Handling call
0809 23:21:01.015162 (+1708727us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b540073f5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.015300 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":500819712}
I20260809 23:21:01.016639 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1504) took 1530 ms. Trace:
I20260809 23:21:01.016816 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:01.017027 25363 rpcz_store.cc:276] 0809 23:20:59.486803 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.486985 (+   182us) service_pool.cc:227] Handling call
0809 23:20:59.519576 (+ 32591us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:00.726671 (+1207095us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.016383 (+289712us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.016559 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2745369984}
I20260809 23:21:01.017676 25368 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:21:01.018051 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1500) took 1452 ms. Trace:
I20260809 23:21:01.018350 25368 rpcz_store.cc:276] 0809 23:20:59.565520 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.016503 (+1450983us) service_pool.cc:227] Handling call
0809 23:21:01.018003 (+  1500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.019264 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.020433 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.021595 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.022368 24546 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260809 23:21:01.022696  1694 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:01.023252  1694 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.023903  1519 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:01.024370  1519 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:01.024413  1691 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:21:01.024873  1519 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:01.025920  1202 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:01.026408 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2702) took 1286 ms. Trace:
I20260809 23:21:01.026679  1202 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:01.027107 24320 rpcz_store.cc:276] 0809 23:20:59.740463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.215937 (+475474us) service_pool.cc:227] Handling call
0809 23:21:01.026212 (+810275us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.026347 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":165303552}
I20260809 23:21:01.027948  1202 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:01.029136 25369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1503) took 1607 ms (client timeout 1946 ms). Trace:
W20260809 23:21:01.029683 25369 rpcz_store.cc:269] 0809 23:20:59.421987 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.422163 (+   176us) service_pool.cc:227] Handling call
0809 23:21:01.028883 (+1606720us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.029066 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10177408}
I20260809 23:21:01.031651 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.033627  1667 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.034626  1628 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:01.035076  1628 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:01.028559 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:00.984668  1434 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:01.002084 24321 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:00.996405 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:01.034637 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3190) took 1336 ms. Trace:
I20260809 23:21:01.036707 24435 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260809 23:21:01.036118 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:01.030830 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.059590 24309 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:01.038908 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:01.018074 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.060211 24423 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:01.060470 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1495) took 1314 ms. Trace:
W20260809 23:21:01.050482 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1487) took 4013 ms (client timeout 1882 ms). Trace:
I20260809 23:21:01.061141 25363 rpcz_store.cc:276] 0809 23:20:59.745747 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.017842 (+1272095us) service_pool.cc:227] Handling call
0809 23:21:01.060397 (+ 42555us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.051265 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:01.061484 25372 rpcz_store.cc:269] 0809 23:20:57.037130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.037349 (+   219us) service_pool.cc:227] Handling call
0809 23:21:01.050226 (+4012877us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.050406 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5478490368}
I20260809 23:21:01.036074 24546 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260809 23:21:01.052009 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3193) took 1338 ms. Trace:
I20260809 23:21:01.065141 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.065052 24311 rpcz_store.cc:276] 0809 23:20:59.713472 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.652776 (+939304us) service_pool.cc:227] Handling call
0809 23:21:00.653182 (+   406us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:00.653448 (+   266us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:00.653483 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:00.657657 (+  4174us) log.cc:842] Serialized 26 byte log entry
0809 23:21:00.918886 (+261229us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:00.918922 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.919013 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.051685 (+132672us) raft_consensus.cc:1671] finished
0809 23:21:01.051707 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.051929 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":193934,"threads_started":1,"wal-append.queue_time_us":359334}
I20260809 23:21:01.057885 24316 rpcz_store.cc:276] 0809 23:20:59.697827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.636979 (+939152us) service_pool.cc:227] Handling call
0809 23:21:00.637430 (+   451us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:00.638086 (+   656us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:00.639640 (+  1554us) log.cc:842] Serialized 14 byte log entry
0809 23:21:00.777860 (+138220us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:00.791738 (+ 13878us) log.cc:842] Serialized 26 byte log entry
0809 23:21:00.791913 (+   175us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:00.791959 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:00.792049 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.034430 (+242381us) raft_consensus.cc:1671] finished
0809 23:21:01.034452 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.034578 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":157,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":6531,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":10900,"thread_start_us":137701,"threads_started":2,"wal-append.queue_time_us":123549}
I20260809 23:21:01.067567 24546 pending_rounds.cc:76] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:21:01.067958 24546 pending_rounds.cc:97] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260809 23:21:01.048738 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.068109 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:01.069703 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:01.057165 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60868.3us
I20260809 23:21:01.037170 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.068277 24546 raft_consensus.cc:2866] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:01.075176 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.076884 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.077023 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:01.077818 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.078231 24321 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 14: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:01.079710 25371 raft_consensus.cc:2396] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:21:01.080235 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:01.079569 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.080554 25368 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:01.083096 24316 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:00.951681 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.085125 24540 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:01.051517  1176 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:01.088786 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.089715 25373 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
W20260809 23:21:01.090337 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58763.8us
I20260809 23:21:01.091188 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1489) took 1401 ms. Trace:
I20260809 23:21:01.091264 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1501) took 1525 ms. Trace:
I20260809 23:21:01.119714 24429 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260809 23:21:01.133937 25368 rpcz_store.cc:276] 0809 23:20:59.565838 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.036903 (+1471065us) service_pool.cc:227] Handling call
0809 23:21:01.091183 (+ 54280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.120743 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:01.135236 24311 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:01.113363 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.093943 24544 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260809 23:21:01.129961 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:01.136536 25371 rpcz_store.cc:276] 0809 23:20:59.689372 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.005030 (+1315658us) service_pool.cc:227] Handling call
0809 23:21:01.091129 (+ 86099us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.130405 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:01.133250 25373 pending_rounds.cc:76] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:21:01.137914 25373 pending_rounds.cc:97] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260809 23:21:01.121191 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1514) took 1442 ms. Trace:
I20260809 23:21:01.141198 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.141067 25365 rpcz_store.cc:276] 0809 23:20:59.678783 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.005491 (+1326708us) service_pool.cc:227] Handling call
0809 23:21:01.121124 (+115633us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:01.132851 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53401.5us
I20260809 23:21:01.095675 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:01.143146 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.144064 24309 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:01.114517 24545 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:01.090390 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:01.138363 25373 raft_consensus.cc:2866] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:01.146639 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:01.143460 24321 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:01.147343 25369 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:01.148541 24321 pending_rounds.cc:76] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 9
I20260809 23:21:01.057714  1553 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:01.064440  1634 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:01.064059  1434 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:01.164647 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.151401 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:01.185587 25368 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 14: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:01.186344 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1492) took 1473 ms. Trace:
I20260809 23:21:01.152376 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.001384  1571 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.120s
I20260809 23:21:01.155172 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.079609  1644 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:21:01.187944 25371 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:01.188657 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1504) took 1621 ms. Trace:
I20260809 23:21:01.189047 25371 rpcz_store.cc:276] 0809 23:20:59.566761 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.137582 (+1570821us) service_pool.cc:227] Handling call
0809 23:21:01.188583 (+ 51001us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.186818 25368 rpcz_store.cc:276] 0809 23:20:59.713279 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.135134 (+1421855us) service_pool.cc:227] Handling call
0809 23:21:01.186287 (+ 51153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.155560 24321 pending_rounds.cc:97] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260809 23:21:01.079020  1413 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:01.176151 24423 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:01.093452  1176 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:01.089049  1667 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:01.177057 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.163856 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:01.088889  1691 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:01.165632 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.089334  1491 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.154503  1569 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:01.184322  1553 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.177803 24320 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:21:01.195216 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:01.141470  1528 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:01.093688   877 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:01.157866  1539 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.000s	sys 0.011s
I20260809 23:21:01.139082  1680 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:01.217957 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.191323 24321 raft_consensus.cc:2866] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:01.203644 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:01.168639  1556 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.003s	sys 0.008s
I20260809 23:21:01.215523 24311 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:01.159229  1649 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:01.233261 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 67741us
I20260809 23:21:01.231463  1644 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:01.270270 24540 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:01.270678 24431 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 16. Ops: []
I20260809 23:21:01.242836 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1536) took 1039 ms. Trace:
I20260809 23:21:01.244275 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:01.154627  1586 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:01.245354 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 94726.2us
I20260809 23:21:01.179721  1634 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:01.178725  1647 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:01.242872 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47760.4us
W20260809 23:21:01.226383 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62605.6us
I20260809 23:21:01.256937 24427 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260809 23:21:01.257450 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 936973us
I20260809 23:21:01.265414 24311 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260809 23:21:01.266597 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.241232  1579 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:01.268244 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.236730 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.198068  1413 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.291123 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1502) took 1725 ms. Trace:
I20260809 23:21:01.304193 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.241436  1248 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
W20260809 23:21:01.269577 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3169) took 4607 ms (client timeout 1966 ms). Trace:
W20260809 23:21:01.307930 24312 rpcz_store.cc:269] 0809 23:20:56.656610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:56.710429 (+ 53819us) service_pool.cc:227] Handling call
0809 23:21:01.264325 (+4553896us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540032fad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.264441 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":915798400}
W20260809 23:21:01.269222 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 178899us
I20260809 23:21:01.273094 24320 rpcz_store.cc:276] 0809 23:21:00.203106 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.141644 (+938538us) service_pool.cc:227] Handling call
0809 23:21:01.177480 (+ 35836us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.242506 (+ 65026us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.242769 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:01.256609  1539 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.001s	sys 0.011s
I20260809 23:21:01.248582  1691 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.249358  1491 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:01.264453  1556 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.003s	sys 0.014s
I20260809 23:21:01.324669  1556 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:01.324777 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1481) took 2942 ms. Trace:
I20260809 23:21:01.325147 25373 rpcz_store.cc:276] 0809 23:20:58.382851 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.383172 (+   321us) service_pool.cc:227] Handling call
0809 23:21:01.051650 (+2668478us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.146646 (+ 94996us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.324580 (+177934us) spinlock_profiling.cc:243] Waited 2.66 s on lock 0x7b5400708fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.324720 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6126679808}
I20260809 23:21:01.325569  1556 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:01.326584 25373 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:21:01.327855  1553 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:01.328058 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1488) took 1639 ms (client timeout 1861 ms). Trace:
I20260809 23:21:01.284497  1176 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:01.328457 25366 rpcz_store.cc:269] 0809 23:20:59.688886 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.524545 (+835659us) service_pool.cc:227] Handling call
0809 23:21:01.327890 (+803345us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.328001 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1062264704}
I20260809 23:21:01.285743 24534 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:01.286736 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:01.331499 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.305980 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1505) took 1738 ms. Trace:
I20260809 23:21:01.301290  1579 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:01.332913  1579 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:01.332729 25363 rpcz_store.cc:276] 0809 23:20:59.567073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.305200 (+1738127us) service_pool.cc:227] Handling call
0809 23:21:01.305515 (+   315us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.305694 (+   179us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.305715 (+    21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:01.305752 (+    37us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:01.305784 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.305827 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.305930 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.334645 25363 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.336591  1389 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:01.336872 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1510) took 1768 ms. Trace:
I20260809 23:21:01.283897 25369 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:01.337298 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1512) took 3174 ms. Trace:
I20260809 23:21:01.337745 24314 rpcz_store.cc:276] 0809 23:20:58.162709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.440221 (+277512us) service_pool.cc:227] Handling call
0809 23:20:58.440547 (+   326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:20:58.860579 (+420032us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.337081 (+2476502us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.337242 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":963230464}
I20260809 23:21:01.338227 25369 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:01.339087 24314 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:01.339825  1722 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260809 23:21:01.339867 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1511) took 1770 ms. Trace:
I20260809 23:21:01.340458 25369 rpcz_store.cc:276] 0809 23:20:59.569046 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.337757 (+1768711us) service_pool.cc:227] Handling call
0809 23:21:01.338045 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.339628 (+  1583us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.339810 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.341636 25369 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:01.341794 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3194) took 1612 ms. Trace:
I20260809 23:21:01.342212 24313 rpcz_store.cc:276] 0809 23:20:59.729504 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.764935 (+1035431us) service_pool.cc:227] Handling call
0809 23:21:00.765393 (+   458us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:00.765872 (+   479us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:00.785837 (+ 19965us) log.cc:842] Serialized 14 byte log entry
0809 23:21:01.246789 (+460952us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:01.247368 (+   579us) log.cc:842] Serialized 26 byte log entry
0809 23:21:01.247535 (+   167us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:01.247896 (+   361us) log.cc:842] Serialized 14 byte log entry
0809 23:21:01.255871 (+  7975us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.255957 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.341581 (+ 85624us) raft_consensus.cc:1671] finished
0809 23:21:01.341601 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.341735 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13849,"prepare.queue_time_us":28185,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":96677,"raft.run_cpu_time_us":448,"raft.run_wall_time_us":25826,"spinlock_wait_cycles":16352768,"thread_start_us":417195,"threads_started":2,"wal-append.queue_time_us":408925}
I20260809 23:21:01.345525 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1506) took 1778 ms. Trace:
I20260809 23:21:01.304097 25368 rpcz_store.cc:276] 0809 23:20:59.566140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.289102 (+1722962us) service_pool.cc:227] Handling call
0809 23:21:01.289495 (+   393us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.289700 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.289737 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:01.289790 (+    53us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:01.289833 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.290042 (+   209us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.291009 (+   967us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.265192  1571 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
W20260809 23:21:01.255475  1468 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.360s	user 0.012s	sys 0.021s
W20260809 23:21:01.359417  1468 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.598s	user 0.012s	sys 0.021s
I20260809 23:21:01.360139  1694 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:01.360263 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.311766 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:01.313148 24311 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260809 23:21:01.363301 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3199) took 1028 ms. Trace:
I20260809 23:21:01.363621 24311 rpcz_store.cc:276] 0809 23:21:00.334877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.302761 (+967884us) service_pool.cc:227] Handling call
0809 23:21:01.307352 (+  4591us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.363060 (+ 55708us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.363251 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.364283  1680 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:01.364573  1248 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:01.313203  1647 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:01.365623  1248 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.366279  1677 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:01.366834  1647 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:01.367045  1502 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.367446  1739 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:01.368593 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.369460  1468 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:21:01.370973 24427 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.371688  1583 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.319872 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:01.289309  1634 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:01.372769  1571 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:01.373764  1571 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:01.321240  1706 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.500s	user 0.000s	sys 0.002s
I20260809 23:21:01.374812  1697 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:01.375284  1697 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:01.322767 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1416) took 1028 ms. Trace:
W20260809 23:21:01.375941 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64255.1us
I20260809 23:21:01.376227 24541 rpcz_store.cc:276] 0809 23:21:00.294402 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.560396 (+265994us) service_pool.cc:227] Handling call
0809 23:21:00.560651 (+   255us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:00.767942 (+207291us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.322504 (+554562us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.322702 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475179392}
I20260809 23:21:01.376392  1697 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.335601 25372 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
I20260809 23:21:01.377302   717 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:01.377518  1697 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:01.342202 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1512) took 1772 ms. Trace:
I20260809 23:21:01.383519 25369 rpcz_store.cc:276] 0809 23:20:59.569315 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.341193 (+1771878us) service_pool.cc:227] Handling call
0809 23:21:01.341481 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.341991 (+   510us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.342158 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.347762 25373 rpcz_store.cc:276] 0809 23:20:59.567378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.326055 (+1758677us) service_pool.cc:227] Handling call
0809 23:21:01.326375 (+   320us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:01.339895 (+ 13520us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.345471 (+  5576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.384158 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.385737 24314 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:01.337318 25363 rpcz_store.cc:276] 0809 23:20:59.568729 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.334004 (+1765275us) service_pool.cc:227] Handling call
0809 23:21:01.334295 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.336590 (+  2295us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.336788 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.382907 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3201) took 1030 ms. Trace:
I20260809 23:21:01.336652 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1418) took 1025 ms. Trace:
I20260809 23:21:01.387318 24320 rpcz_store.cc:276] 0809 23:21:00.352281 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.309630 (+957349us) service_pool.cc:227] Handling call
0809 23:21:01.382333 (+ 72703us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.382535 (+   202us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.382572 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:01.382623 (+    51us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:01.382665 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.382728 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.382843 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.310335 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:01.392445 25366 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:01.394557 25363 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:01.387602 24543 rpcz_store.cc:276] 0809 23:21:00.311616 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.570336 (+258720us) service_pool.cc:227] Handling call
0809 23:21:00.584727 (+ 14391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.038208 (+453481us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.038237 (+    29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:01.038292 (+    55us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:01.038344 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.336443 (+298099us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.336486 (+    43us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.336599 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2105,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":27543,"spinlock_wait_cycles":1042006912}
I20260809 23:21:01.398579 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:01.395839 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76062.1us
I20260809 23:21:01.303357 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.280022  1363 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:01.357169 25365 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260809 23:21:01.358471 24544 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
W20260809 23:21:01.358551 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54527.9us
I20260809 23:21:01.036125  1628 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:01.377779   717 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:01.346511 24540 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260809 23:21:01.379365  1725 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:21:01.380406 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:01.425138 24423 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:21:01.425015 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.307418  1569 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:01.381661 24430 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:01.410099 24498 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:21:01.445575 24498 leader_election.cc:401] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
W20260809 23:21:01.412040 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108776us
I20260809 23:21:01.416256 25369 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.421942 25363 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.288249  1675 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260809 23:21:01.430276 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:01.423787   717 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:01.386608  1638 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.443300 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:01.444056 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.449100 24311 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.449059 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1498) took 1097 ms. Trace:
I20260809 23:21:01.449852 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:01.366835  1677 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:01.288342  1586 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.413242  1691 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.074s
I20260809 23:21:01.453303 25363 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260809 23:21:01.453954  1633 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260809 23:21:01.455251 24543 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260809 23:21:01.455783 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.455830 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.457824 24496 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:21:01.520036 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1491) took 1814 ms. Trace:
W20260809 23:21:01.496404 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45299.2us
I20260809 23:21:01.527782 25365 rpcz_store.cc:276] 0809 23:20:59.705653 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.142683 (+1437030us) service_pool.cc:227] Handling call
0809 23:21:01.519692 (+377009us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540073f5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.519944 (+   252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":362389248}
I20260809 23:21:01.525600 24323 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 18. Ops: [16.10-16.10]
I20260809 23:21:01.528677 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.523305 25368 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260809 23:21:01.529444 25368 pending_rounds.cc:76] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 13
I20260809 23:21:01.526685 25363 pending_rounds.cc:76] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 10
I20260809 23:21:01.529742 25368 pending_rounds.cc:97] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 17.14
W20260809 23:21:01.507117 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59913.8us
I20260809 23:21:01.529954 25363 pending_rounds.cc:97] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 15.11
W20260809 23:21:01.527577 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71829.7us
I20260809 23:21:01.529300 24543 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:01.530130 24496 leader_election.cc:401] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260809 23:21:01.530108 25368 raft_consensus.cc:2866] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:01.417964  1617 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:01.540642 25363 raft_consensus.cc:2866] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:01.501845 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3195) took 1751 ms. Trace:
W20260809 23:21:01.512148 24498 raft_consensus.cc:2670] T e4ed6a400f2643da95b8f744ebab6f4c P 808fed3ed3f04882ad757a3db9abfdd9: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260809 23:21:01.554615 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 165902us
I20260809 23:21:01.459717  1468 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.097s
I20260809 23:21:01.411353  1667 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260809 23:21:01.513494 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57727.1us
I20260809 23:21:01.562175 25363 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:01.563323 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:01.454839  1736 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:01.555284 24316 rpcz_store.cc:276] 0809 23:20:59.749954 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.130588 (+1380634us) service_pool.cc:227] Handling call
0809 23:21:01.131027 (+   439us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:01.157453 (+ 26426us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.157485 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:01.177298 (+ 19813us) log.cc:842] Serialized 52 byte log entry
0809 23:21:01.271265 (+ 93967us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:01.321463 (+ 50198us) log.cc:842] Serialized 14 byte log entry
0809 23:21:01.362199 (+ 40736us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.362277 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.501567 (+139290us) raft_consensus.cc:1671] finished
0809 23:21:01.501596 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.501774 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40455,"prepare.queue_time_us":49443,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":43077,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":6629,"thread_start_us":52760,"threads_started":1,"wal-append.queue_time_us":218911}
I20260809 23:21:01.551307 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:01.552011 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:01.537297 25369 rpcz_store.cc:276] 0809 23:21:00.351333 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.384264 (+1032931us) service_pool.cc:227] Handling call
0809 23:21:01.415818 (+ 31554us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.448726 (+ 32908us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.448993 (+   267us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:01.559176 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:01.560398 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 113782us
I20260809 23:21:01.412094  1582 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.136s
I20260809 23:21:01.426982  1623 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
I20260809 23:21:01.572736 24544 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:01.573379 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:01.576736 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.589682 24540 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260809 23:21:01.545845  1553 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:01.517707  1649 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:01.560506  1303 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:01.605858  1303 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.586213 24320 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:01.608294 24312 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:01.609431 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.611869  1649 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:01.586897 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53796.8us
I20260809 23:21:01.588593 25365 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:01.564738 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.615556 24545 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:01.617153 24422 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260809 23:21:01.591676 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:01.618297 24546 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:01.457338  1502 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.109s
I20260809 23:21:01.621961 25375 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:01.622310  1649 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:01.622771  1649 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:01.623823  1649 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:01.595093 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48336.3us
I20260809 23:21:01.626361 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:01.599287  1677 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:01.629740 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:01.410718  1741 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.217s
I20260809 23:21:01.634035 24311 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:01.445829  1714 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.126s
I20260809 23:21:01.570204  1694 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:01.627534  1677 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:01.513756  1431 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.065s
W20260809 23:21:01.676115 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49879.7us
I20260809 23:21:01.646876 24544 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:01.631577  1582 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:21:01.450400  1644 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:01.680485 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71070us
I20260809 23:21:01.633728  1736 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:01.638072  1537 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.654s	user 0.004s	sys 0.012s
I20260809 23:21:01.592232  1680 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260809 23:21:01.719359  1586 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:01.578130  1737 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.113s
W20260809 23:21:01.691928  1712 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260809 23:21:01.679199  1584 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:01.635133  1502 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260809 23:21:01.635465  1638 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.063s
I20260809 23:21:01.680244  1645 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:01.736372 24422 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260809 23:21:01.736953 25373 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260809 23:21:01.637871  1585 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:01.642736  1468 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:01.691155  1633 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260809 23:21:01.721629  1413 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:01.745092  1740 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260809 23:21:01.713784  1537 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.734813  1629 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:01.705358  1202 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
I20260809 23:21:01.699905  1591 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260809 23:21:01.746564  1468 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:01.778047 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 321947us
I20260809 23:21:01.785698 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:01.791167 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:01.784137  1468 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:01.831152 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 269595us
I20260809 23:21:01.789525  1413 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:01.798156  1736 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:01.861656  1756 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260809 23:21:01.873831 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1250) took 1599 ms. Trace:
I20260809 23:21:01.876153 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1507) took 2308 ms. Trace:
I20260809 23:21:01.877306 25366 rpcz_store.cc:276] 0809 23:20:59.567675 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.329252 (+1761577us) service_pool.cc:227] Handling call
0809 23:21:01.392222 (+ 62970us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.395215 (+  2993us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.875911 (+480696us) spinlock_profiling.cc:243] Waited 62.7 ms on lock 0x7b54006027d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.876084 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":144133504}
I20260809 23:21:01.879837 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3205) took 3375 ms. Trace:
I20260809 23:21:01.880290 24542 rpcz_store.cc:276] 0809 23:20:58.503906 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:58.653696 (+149790us) service_pool.cc:227] Handling call
0809 23:20:58.660518 (+  6822us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:20:58.660785 (+   267us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:20:58.660822 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:01.346050 (+2685228us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:01.346442 (+   392us) log.cc:842] Serialized 26 byte log entry
0809 23:21:01.597592 (+251150us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:01.597649 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.597740 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.863657 (+265917us) raft_consensus.cc:1671] finished
0809 23:21:01.863688 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.879750 (+ 16062us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3648896,"thread_start_us":250763,"threads_started":1,"wal-append.queue_time_us":404063}
I20260809 23:21:01.874352 24435 rpcz_store.cc:276] 0809 23:21:00.274104 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.281616 (+  7512us) service_pool.cc:227] Handling call
0809 23:21:00.300892 (+ 19276us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:01.059468 (+758576us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.873411 (+813943us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.873754 (+   343us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1691190656}
I20260809 23:21:01.810822  1582 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260809 23:21:01.890178 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1515) took 2186 ms. Trace:
I20260809 23:21:01.890663 25368 rpcz_store.cc:276] 0809 23:20:59.703610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.361741 (+1658131us) service_pool.cc:227] Handling call
0809 23:21:01.522752 (+161011us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.557284 (+ 34532us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.879339 (+322055us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540073f5d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.890098 (+ 10759us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":369549696}
W20260809 23:21:01.865012  1519 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.835s	user 0.013s	sys 0.004s
I20260809 23:21:01.838006  1491 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260809 23:21:01.909963 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1516) took 1260 ms. Trace:
W20260809 23:21:01.899665  1753 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260809 23:21:01.912618 25371 rpcz_store.cc:276] 0809 23:21:00.649497 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.392109 (+742612us) service_pool.cc:227] Handling call
0809 23:21:01.420105 (+ 27996us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.567745 (+147640us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.567793 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:01.567849 (+    56us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:01.567892 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.909700 (+341808us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.909744 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.909881 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":338907392}
I20260809 23:21:01.840292  1541 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
W20260809 23:21:01.909857 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 336574us
I20260809 23:21:01.892798  1553 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:01.856823  1569 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
W20260809 23:21:01.929092 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 247392us
I20260809 23:21:01.908921  1519 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.895637  1691 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
W20260809 23:21:01.936683  1751 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260809 23:21:01.940200  1757 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260809 23:21:01.941872  1747 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260809 23:21:01.941903 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1538) took 1658 ms. Trace:
I20260809 23:21:01.942554 24321 rpcz_store.cc:276] 0809 23:21:00.283326 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.147840 (+864514us) service_pool.cc:227] Handling call
0809 23:21:01.148292 (+   452us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:01.287927 (+139635us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.287961 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:01.302514 (+ 14553us) log.cc:842] Serialized 52 byte log entry
0809 23:21:01.747097 (+444583us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:01.747140 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.747207 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.941624 (+194417us) raft_consensus.cc:1671] finished
0809 23:21:01.941653 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.941835 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44480,"raft.queue_time_us":19830,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":27512,"thread_start_us":333805,"threads_started":1,"wal-append.queue_time_us":456603}
I20260809 23:21:01.948616  1633 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:01.949707  1754 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260809 23:21:01.949252  1633 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:01.951683 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3197) took 1617 ms. Trace:
I20260809 23:21:01.953114  1633 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:01.952100 24309 rpcz_store.cc:276] 0809 23:21:00.334306 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.161842 (+827536us) service_pool.cc:227] Handling call
0809 23:21:01.198817 (+ 36975us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:01.235088 (+ 36271us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:01.235123 (+    35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:01.235650 (+   527us) log.cc:842] Serialized 52 byte log entry
0809 23:21:01.593208 (+357558us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:21:01.593263 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.872760 (+279497us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.872824 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:01.951357 (+ 78533us) raft_consensus.cc:1671] finished
0809 23:21:01.951383 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:01.951619 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19723,"raft.queue_time_us":19798,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3544,"spinlock_wait_cycles":82461440,"thread_start_us":318762,"threads_started":1,"wal-append.queue_time_us":510083}
I20260809 23:21:01.953971  1537 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:01.956367  1498 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:01.958846  1741 consensus_queue.cc:1159] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 17 status { last_received { term: 16 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:01.962826 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3202) took 1586 ms. Trace:
I20260809 23:21:01.965147 24316 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:01.977525 24312 rpcz_store.cc:276] 0809 23:21:00.376343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.312154 (+935811us) service_pool.cc:227] Handling call
0809 23:21:01.318035 (+  5881us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.608848 (+290813us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.962592 (+353744us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.962765 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":667036928}
W20260809 23:21:01.966281 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1492) took 2465 ms (client timeout 1682 ms). Trace:
W20260809 23:21:01.979789 25375 rpcz_store.cc:269] 0809 23:20:59.500781 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.537302 (+ 36521us) service_pool.cc:227] Handling call
0809 23:21:01.966115 (+2428813us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.966216 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4673657088}
I20260809 23:21:01.965409 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2686) took 5552 ms. Trace:
I20260809 23:21:01.981575 24323 rpcz_store.cc:276] 0809 23:20:56.412743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.217101 (+804358us) service_pool.cc:227] Handling call
0809 23:21:01.264544 (+4047443us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:01.529011 (+264467us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:01.964682 (+435671us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540032fad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:01.965233 (+   551us) spinlock_profiling.cc:243] Waited 4.05 s on lock 0x7b540032fad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:01.965355 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9907084800}
W20260809 23:21:01.968201  1697 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.571s	user 0.011s	sys 0.000s
W20260809 23:21:01.964717  1634 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.000s
W20260809 23:21:01.933910  1434 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.709s	user 0.005s	sys 0.009s
W20260809 23:21:02.006635  1440 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.543s	user 0.021s	sys 0.049s
I20260809 23:21:01.967667  1303 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.016084  1434 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:02.053505   717 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.563s	user 0.006s	sys 0.006s
W20260809 23:21:02.050432  1440 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.589s	user 0.022s	sys 0.050s
I20260809 23:21:02.054165   717 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.055224  1434 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:02.058212  1767 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
W20260809 23:21:02.059089  1363 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.001s
I20260809 23:21:01.969730  1468 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.060348 25372 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:02.014114  1634 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.060869 24539 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:01.932222  1553 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.080644 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 216700us
W20260809 23:21:02.082548 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 233640us
I20260809 23:21:02.086863 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:02.087441 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:02.060083  1519 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:02.087692 25368 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.088094 25366 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.064540   717 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:02.069983  1036 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.098098 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:02.099043 24321 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:02.100250 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:02.105229 24323 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:02.109522 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.109668 25366 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:02.111896 24545 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:02.112066 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:02.112722 24309 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:02.116384 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.119529 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.120110 24544 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:21:02.123497 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.106616  1645 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:02.139680 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56678.3us
I20260809 23:21:02.107890  1036 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.148841 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:02.150019 24545 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.149827 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.158970  1553 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:02.154660  1579 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.821s	user 0.015s	sys 0.006s
I20260809 23:21:02.176841 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1497) took 1829 ms. Trace:
I20260809 23:21:02.177376  1579 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:02.177496 25372 rpcz_store.cc:276] 0809 23:21:00.347633 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.377870 (+1030237us) service_pool.cc:227] Handling call
0809 23:21:01.378226 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:02.064956 (+686730us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:02.176530 (+111574us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:02.176759 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568148736}
I20260809 23:21:02.192281 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:02.193315 24544 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.198863 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:02.203256 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:02.180455  1579 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.138224  1645 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.215162 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67800.2us
I20260809 23:21:02.203689 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:02.217391 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.215517 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:02.221906 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:02.181923  1781 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:02.213716  1389 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:02.235266  1389 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:02.235889  1389 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:02.234738  1468 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.233959  1788 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:02.204404  1697 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260809 23:21:02.255589 24544 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:02.256435  1628 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:02.257830  1468 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.258318  1468 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.236256  1634 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:02.234763  1645 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.267196 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:02.265906 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48578.1us
W20260809 23:21:02.273411  1629 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.501s	user 0.009s	sys 0.002s
I20260809 23:21:02.274410 24317 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260809 23:21:02.276906  1694 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.002s
I20260809 23:21:02.277093 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3278) took 1362 ms. Trace:
I20260809 23:21:02.277341  1694 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16.
I20260809 23:21:02.277645 24431 rpcz_store.cc:276] 0809 23:21:00.914556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.917427 (+  2871us) service_pool.cc:227] Handling call
0809 23:21:00.943088 (+ 25661us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:01.279075 (+335987us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:02.276792 (+997717us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:02.277027 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":752656640}
I20260809 23:21:02.278564  1468 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:02.283740  1198 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:02.284255  1198 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:02.285372  1198 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.289484 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 450977us
W20260809 23:21:02.289866 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 367168us
I20260809 23:21:02.290762 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:02.295710 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:02.288234  1742 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.299211  1776 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:02.300258  1771 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260809 23:21:02.303237 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 81442.9us
I20260809 23:21:02.289422  1623 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.325510 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:02.329694 25368 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:02.337438 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:02.338338 24316 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:02.327282 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:02.303705 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:02.342119 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:02.268232 24544 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 18
I20260809 23:21:02.399437  1248 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:02.325525  1553 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.078s
I20260809 23:21:02.340895  1742 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.416635 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3094) took 1583 ms (client timeout 1606 ms). Trace:
I20260809 23:21:02.416929 24545 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:02.445729 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:02.448511 24534 rpcz_store.cc:269] 0809 23:21:00.833630 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.873643 (+ 40013us) service_pool.cc:227] Handling call
0809 23:21:02.371736 (+1498093us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:02.416557 (+ 44821us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":945720704}
I20260809 23:21:02.449273 24542 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:21:02.417155 25368 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:02.304208  1645 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:02.444136  1768 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.528s	user 0.004s	sys 0.003s
I20260809 23:21:02.318470  1794 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
W20260809 23:21:02.495848  1799 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260809 23:21:02.432021  1248 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:02.497081  1248 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:02.500942 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203241us
I20260809 23:21:02.501901 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:02.509172  1582 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.509754  1582 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.512270  1582 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:02.516110 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 220452us
I20260809 23:21:02.517858 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:02.521347 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:02.526942 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:02.502751 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:02.543545  1176 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:02.519740 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:02.547930 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:02.551875 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:02.552718 24429 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.557685 24431 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:21:02.558828  1694 raft_consensus.cc:2741] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260809 23:21:02.559998  1673 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:21:02.561115 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.567159  1527 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:02.567716  1527 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:02.568122  1527 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:02.571379 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 279732us
I20260809 23:21:02.571853  1591 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:02.572391 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:02.573525  1434 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:02.574025  1434 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:02.575958  1803 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260809 23:21:02.576603  1579 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:02.577018 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 253604us
I20260809 23:21:02.577495  1579 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:21:02.575323  1434 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.579119 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 157835us
W20260809 23:21:02.599397 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 72570.2us
W20260809 23:21:02.601187 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 297531us
I20260809 23:21:02.445338  1647 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:02.608659 24544 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:02.609236 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80762us
I20260809 23:21:02.610505 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:02.582551  1757 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260809 23:21:02.304692  1774 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.304s
I20260809 23:21:02.620741 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:02.621732 24314 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.594127  1781 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:02.528815  1644 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:02.606493 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.618188 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:02.623205 24323 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:02.529027  1556 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:02.625149 24542 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:21:02.654665 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:02.655407 24311 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:02.372947  1623 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.627796 24309 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260809 23:21:02.602685  1794 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:02.662369 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55943.4us
I20260809 23:21:02.662432 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:02.663249 24541 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.605903  1485 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:02.630429 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:02.664258 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3185) took 2972 ms (client timeout 2138 ms). Trace:
W20260809 23:21:02.664891 24317 rpcz_store.cc:269] 0809 23:20:59.691614 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:59.691743 (+   129us) service_pool.cc:227] Handling call
0809 23:21:02.664060 (+2972317us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:02.664187 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5817164672}
I20260809 23:21:02.667065 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:02.667445 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57012.9us
I20260809 23:21:02.668442 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.669711 24314 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:02.670799 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:02.676975  1784 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:21:02.678014  1797 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.382s	user 0.000s	sys 0.001s
I20260809 23:21:02.604089  1645 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.679387  1537 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:02.679241  1746 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.156s
I20260809 23:21:02.681357  1645 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.683125  1623 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:02.683924  1644 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260809 23:21:02.688020  1491 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
W20260809 23:21:02.690610 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112249us
I20260809 23:21:02.694098 24311 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:02.695624 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:02.652781  1434 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:02.658192  1781 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:02.606643  1737 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260809 23:21:02.691884 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:02.694648 24541 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:02.620734  1771 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
I20260809 23:21:02.625988  1647 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:02.653247   877 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.056s
I20260809 23:21:02.699369  1582 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:02.698851  1491 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:02.725544 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.629783  1539 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
I20260809 23:21:02.664502  1430 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260809 23:21:02.662381  1519 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260809 23:21:02.700686  1537 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:02.711292  1781 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:02.717073  1623 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.711112  1784 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:02.736958  1577 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:02.715920  1644 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.034s
I20260809 23:21:02.782172 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33980 (request call id 8) took 5044 ms. Trace:
I20260809 23:21:02.744414  1746 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.706755  1485 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260809 23:21:02.785959 25370 rpcz_store.cc:276] 0809 23:20:57.749969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:57.762049 (+ 12080us) service_pool.cc:227] Handling call
0809 23:21:02.750907 (+4988858us) spinlock_profiling.cc:243] Waited 3.76 s on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:02.760514 (+  9607us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8641210624}
W20260809 23:21:02.809810 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87723.9us
I20260809 23:21:02.745739  1081 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:02.820791 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 121883us
I20260809 23:21:02.785621  1577 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:02.763180  1430 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:02.827826 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.906683 24540 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:02.740383  1556 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.069s
W20260809 23:21:02.824038 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83336.6us
I20260809 23:21:02.833493 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.833986 24541 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: [15.11-17.12]
W20260809 23:21:02.823293  1788 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.560s	user 0.005s	sys 0.006s
I20260809 23:21:02.837513  1579 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.785406  1623 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.840217  1352 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.840250  1537 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.841195  1815 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260809 23:21:02.842487  1649 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:02.905623  1577 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:02.843219  1491 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.834599  1431 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:02.846292  1519 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.827081  1539 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:02.851575  1808 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260809 23:21:02.855913  1742 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:02.857947  1809 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
I20260809 23:21:02.860395  1561 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:02.861555  1248 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.864984  1647 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:02.789465  1794 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.867391 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1412) took 3161 ms (client timeout 1958 ms). Trace:
W20260809 23:21:02.971592 24539 rpcz_store.cc:269] 0809 23:20:59.706239 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:00.157852 (+451613us) service_pool.cc:227] Handling call
0809 23:21:02.867201 (+2709349us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:02.867311 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4373869824}
I20260809 23:21:02.874490  1731 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:02.823094  1746 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:02.875780  1389 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.623s	user 0.010s	sys 0.000s
I20260809 23:21:02.823710  1694 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:02.822500  1737 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:02.876746  1498 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:02.879359  1584 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:02.879961  1628 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.623s	user 0.010s	sys 0.000s
I20260809 23:21:02.903662 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:02.825614  1081 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.936884 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.905876  1430 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:02.985229 24540 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:02.950584 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:02.926823  1788 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:02.970098 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:02.974042 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:02.918540  1585 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.043s
I20260809 23:21:02.947710  1431 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:02.920112  1527 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:02.962499  1629 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.959438  1739 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:02.955396  1519 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:02.923307  1352 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:02.983031 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79496.8us
I20260809 23:21:02.922765  1579 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.973250  1731 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:02.992980 24546 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:02.958519  1539 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:02.993432 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:02.986066 24541 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
W20260809 23:21:02.980832 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 271504us
I20260809 23:21:03.034004 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:03.007510 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.040608 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:03.043716 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:02.988109  1501 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:03.059894 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 66670.7us
I20260809 23:21:02.969861  1561 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:02.994582  1545 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:03.082129 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:03.087889  1694 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:03.089294  1527 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.089681  1579 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.090272  1689 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.983189  1389 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.092150 24422 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:03.084813  1788 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:02.987628  1584 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.070835  1528 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:03.117615  1714 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:03.117753  1827 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:03.118573  1846 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:02.996304  1737 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.115187  1689 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.119374  1561 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:03.120816 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.031111  1739 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.121125  1582 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:03.031497  1781 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.082633  1485 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:03.118639  1788 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:03.070684  1501 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:03.126749 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57102.4us
W20260809 23:21:03.127749 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57730.4us
I20260809 23:21:03.154112 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.166320 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:03.012228  1629 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.178767  1849 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.118624  1519 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.121021  1389 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:03.180896 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.180923 24541 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:03.125911  1675 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.236845  1036 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.239171  1352 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.243530  1392 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.244263 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.251811 24311 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:03.156769  1528 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:03.272606 24535 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:21:03.109858  1431 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.284276 24428 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:03.275795 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1501) took 2878 ms. Trace:
I20260809 23:21:03.289528 25373 rpcz_store.cc:276] 0809 23:21:00.397162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:01.384528 (+987366us) service_pool.cc:227] Handling call
0809 23:21:01.412862 (+ 28334us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:01.737445 (+324583us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:03.275470 (+1538025us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:03.275718 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":735811840}
I20260809 23:21:03.292096 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:03.292940 24424 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:03.300525 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.301215 24423 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:03.294153 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.245914   877 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:03.314363 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.155821  1714 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.303661 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:03.303719 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:03.110037  1430 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.256198  1248 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:03.253377  1485 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:03.341677 24539 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:03.342273 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.258949  1623 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.343677 24427 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:03.346997 24542 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:03.342633 24311 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260809 23:21:03.350790 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.351482 25368 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:03.350837 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.352773 24423 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:03.352943 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.353525 25368 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:03.354637 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.355808 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.356933 25368 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:03.358798 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:03.360409 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:03.364672 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:03.365311 24423 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:03.367607  1628 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:21:03.341914  1796 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260809 23:21:03.367547 25370 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:21:03.378393 24540 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:03.364310  1485 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.382679 24542 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:03.342351  1561 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:03.422676 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.183121  1846 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.358667  1844 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.352136  1248 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.438831 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.465224 24539 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:03.311954  1392 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.380345  1628 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:03.442409 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.288242  1675 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.463706 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.457008 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:03.477706 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 230427us
I20260809 23:21:03.478808 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:03.376595  1746 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.485198 24539 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 21
I20260809 23:21:03.335095   877 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:03.489123 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 285308us
I20260809 23:21:03.489388 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.136034  1731 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.441726  1561 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:03.404838  1637 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:03.414651  1623 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:03.496259 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.273408  1849 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.478893  1539 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.492681 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:03.540795 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195773us
W20260809 23:21:03.541093 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48499.8us
I20260809 23:21:03.488622  1628 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:03.504400 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.544679 24541 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:03.497366  1746 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:03.529451 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 4.131s (SENT) [suppressed 8 similar messages]
I20260809 23:21:03.577813 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.530539 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:03.579849 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49381.1us
I20260809 23:21:03.461258  1584 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.494216 24311 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:03.529346  1561 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:03.566326 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.539271 25375 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:03.600500 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:03.481709  1844 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.608036 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.609166 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.609766  1844 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.610208 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:03.610570  1863 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:03.611500  1863 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.613574  1863 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.487401  1248 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.593163 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.535737  1623 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:03.615242  1843 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.335s	user 0.001s	sys 0.000s
I20260809 23:21:03.446969  1788 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.617553  1303 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.687350 24539 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:03.524025  1545 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.080s
I20260809 23:21:03.539037  1637 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.639894 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.620029   877 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.619334  1737 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:03.622438  1848 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260809 23:21:03.648717  1392 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:03.726065 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.018s	sys 0.023s
W20260809 23:21:03.652488  1577 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.701s	user 0.007s	sys 0.003s
I20260809 23:21:03.466303  1528 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.658342  1863 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.565760  1527 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.093s
I20260809 23:21:03.735882  1863 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:03.661092  1675 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.620261  1739 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.737277  1739 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:03.738052  1863 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:03.737751  1739 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.661715  1827 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:03.739959  1739 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.605335  1584 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:03.663355 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.666535 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1560) took 1330 ms. Trace:
I20260809 23:21:03.742045 24321 rpcz_store.cc:276] 0809 23:21:02.336557 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.336722 (+   165us) service_pool.cc:227] Handling call
0809 23:21:02.337182 (+   460us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:02.519338 (+182156us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:02.519369 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:02.519817 (+   448us) log.cc:842] Serialized 26 byte log entry
0809 23:21:03.282263 (+762446us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b1800449080. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:03.435782 (+153519us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:03.435839 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:03.435944 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:03.666175 (+230231us) raft_consensus.cc:1671] finished
0809 23:21:03.666198 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:03.666471 (+   273us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":228754,"raft.queue_time_us":191186,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":19023,"spinlock_wait_cycles":76074240,"thread_start_us":153263,"threads_started":1,"wal-append.queue_time_us":1112858}
I20260809 23:21:03.744187 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:03.745338 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.745954 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:03.598800  1746 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.748548  1746 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.654695  1248 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:03.667340 24313 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260809 23:21:03.668324  1491 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.749805  1491 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.753108  1491 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.636123  1629 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.618885  1714 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:03.673612  1850 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260809 23:21:03.673557 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:03.756189 24430 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:03.676986 24311 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:03.678124 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.680794 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:03.758597 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:03.686482 24424 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:03.760601 24424 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260809 23:21:03.746721 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.761631 24321 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260809 23:21:03.762138 24424 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:03.762439 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.763285 24321 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:03.486089 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:03.764055 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.764729 24321 raft_consensus.cc:2362] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18. Re-sending same reply.
I20260809 23:21:03.765573 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.766876 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:03.595510 24546 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.000s
I20260809 23:21:03.521610  1776 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.116s
I20260809 23:21:03.701930 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:03.708060 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.769251 25375 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:03.769335 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67529.7us
I20260809 23:21:03.628435  1781 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.628629  1611 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:03.675000  1623 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:03.621917  1689 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:03.636128  1649 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.000s
W20260809 23:21:03.726426 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.018s	sys 0.023s
I20260809 23:21:03.722345 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:03.733880 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.659905  1846 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:03.767613 24321 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:03.868980 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:03.772008 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91293.6us
I20260809 23:21:03.768724 24546 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:03.700261  1585 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.066s
W20260809 23:21:03.849262 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65133.6us
I20260809 23:21:03.889976 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.705308  1844 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
I20260809 23:21:03.860989 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.901530 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.903510 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.904891 24539 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:03.780190  1584 raft_consensus.cc:2741] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260809 23:21:03.779641  1714 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:03.906037 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.862421 25372 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:21:03.906811 24321 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:03.864420 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 334970us
I20260809 23:21:03.705117  1303 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:03.863133 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:03.906302 24311 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260809 23:21:03.907240 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:03.904268 24541 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:03.919585 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:03.892606 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:03.974642 25365 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.003s	sys 0.019s
W20260809 23:21:03.975240 25365 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.003s	sys 0.019s
W20260809 23:21:03.907642 24498 connection.cc:232] RPC call timeout handler was delayed by -3.0849739809054881s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
W20260809 23:21:03.976982 24498 connection.cc:232] RPC call timeout handler was delayed by -2.9526739809662104s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20260809 23:21:03.977701 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8617739810142666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260809 23:21:03.866642  1776 consensus_queue.cc:1225] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:21:03.867865  1739 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:03.978248 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8338739809114486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
I20260809 23:21:03.979291 24539 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260809 23:21:03.979712 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:03.764104  1577 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:03.981776 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89255.4us
W20260809 23:21:03.994917 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75435.6us
I20260809 23:21:03.866113  1746 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:03.899525  1827 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:03.998833 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:03.912581  1498 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.926s	user 0.004s	sys 0.008s
I20260809 23:21:04.000029 24539 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260809 23:21:04.001085 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8248739810660481s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260809 23:21:04.005700 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:04.043359 24498 connection.cc:232] RPC call timeout handler was delayed by -2.7681739809922874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
I20260809 23:21:04.010560 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:04.011492 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.002s	sys 0.014s
I20260809 23:21:03.983016  1656 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:03.912232  1714 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:04.018249 24317 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.009s	sys 0.009s
W20260809 23:21:04.048277 24317 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.009s	sys 0.009s
I20260809 23:21:03.979907  1176 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:03.868749  1781 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.128s
I20260809 23:21:04.009984 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:04.044097 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.002s	sys 0.014s
W20260809 23:21:04.044906 24498 connection.cc:232] RPC call timeout handler was delayed by -2.7105739810504019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260809 23:21:03.860324  1611 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.044255 24321 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:04.053311 25371 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [13.11-15.12]
W20260809 23:21:04.051270 24498 connection.cc:232] RPC call timeout handler was delayed by -2.7060739810112864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19510000000000002s for a response
I20260809 23:21:04.040534  1746 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:04.056202 24498 connection.cc:232] RPC call timeout handler was delayed by -2.6538739809766412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2009s for a response
I20260809 23:21:04.068905 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:04.069684 24498 connection.cc:232] RPC call timeout handler was delayed by -2.5971739809028804s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
I20260809 23:21:04.069090 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.066344 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.071314 24321 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:21:04.072117 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:04.065394 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.001s	sys 0.026s
I20260809 23:21:04.072677 24321 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:04.072852 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.001s	sys 0.026s
W20260809 23:21:04.003401  1537 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.017s	sys 0.038s
I20260809 23:21:04.054252 24314 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:21:04.072117 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.066975 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:04.070827 24312 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:04.054103  1870 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:21:04.079438 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.500s	user 0.005s	sys 0.024s
I20260809 23:21:04.049865  1714 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.048086  1656 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:04.047258  1849 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:04.099679 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 236397us
I20260809 23:21:04.004294  1739 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:04.070737 24498 connection.cc:232] RPC call timeout handler was delayed by -2.5665739809628576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260809 23:21:04.111023 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 568498us
I20260809 23:21:04.054165  1844 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:04.147364 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.047710  1827 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.149556 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.165383 25368 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:04.146036 24498 connection.cc:232] RPC call timeout handler was delayed by -2.5458739809691906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
I20260809 23:21:04.151031 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.070322  1629 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:04.153120 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.167218 25363 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
W20260809 23:21:04.100235  1537 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.018s	sys 0.038s
I20260809 23:21:04.166368 25366 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:04.153707 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:04.158241 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.005s	sys 0.017s
W20260809 23:21:04.179039 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.005s	sys 0.017s
I20260809 23:21:04.159376   877 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:04.114234 24321 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:04.166297 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:04.168540 24498 connection.cc:232] RPC call timeout handler was delayed by -2.5458739809691906s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
I20260809 23:21:04.180951 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:04.181353 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4990739810746163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260809 23:21:04.181829 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4801739810500294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260809 23:21:04.183097 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:04.183771 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.184099  1781 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:04.184764  1781 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:04.185204  1781 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:04.101887  1634 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.192698 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.053162  1176 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.186923 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.193321  1634 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.195258  1637 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.195535 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.196223 24424 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:04.078286  1485 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20260809 23:21:04.198365  1746 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.165591  1584 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:04.200088 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:04.110335  1870 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.200649  1577 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.200975  1870 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.202001  1392 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:04.202764  1392 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.205317  1392 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.107774  1527 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260809 23:21:04.200685 24312 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:04.197074 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.189090 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.145938  1846 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:04.226109 24312 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:04.156452  1844 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:04.177585  1691 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.235656 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:04.240461 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.020s	sys 0.044s
I20260809 23:21:04.228380  1036 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:04.224846 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.187467  1629 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:04.246407  1875 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260809 23:21:04.253906  1611 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.255162 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1454) took 1107 ms. Trace:
I20260809 23:21:04.295033 24535 rpcz_store.cc:276] 0809 23:21:03.148287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.163736 (+ 15449us) service_pool.cc:227] Handling call
0809 23:21:03.254294 (+ 90558us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:03.302004 (+ 47710us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.254890 (+952886us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b54000b5c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:04.255100 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12724608}
I20260809 23:21:04.243518  1691 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.264254  1623 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.241626  1844 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:04.269101  1739 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.277278  1780 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:04.274834  1831 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.184s	user 0.009s	sys 0.006s
I20260809 23:21:04.290401  1859 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:04.281702  1303 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:04.284878 24312 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260809 23:21:04.245385  1561 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.000s
W20260809 23:21:04.293941 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69155.3us
I20260809 23:21:04.299652  1784 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260809 23:21:04.308686  1628 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.726s	user 0.012s	sys 0.000s
I20260809 23:21:04.280795  1827 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.321199 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:04.290975 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 173134us
I20260809 23:21:04.322075 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.304557 24317 raft_consensus.cc:1203] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Deduplicated request from leader. Original: 12.11->[14.12-14.12]   Dedup: 14.12->[]
I20260809 23:21:04.192754   877 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.100s
I20260809 23:21:04.313277 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.313921 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:04.316802 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.308938  1637 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.316752 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1282) took 1011 ms. Trace:
I20260809 23:21:04.318017 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.318533 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.322033 25373 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:04.326478 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:04.309070  1780 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:04.327770 24317 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260809 23:21:04.328173 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.331313 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.331079 24535 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:04.328119 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:04.338372 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:04.330513 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:04.328716 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:04.347289  1198 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.348575  1781 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.349714  1846 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:04.318745  1561 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.323586  1628 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.351759  1584 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:04.352242  1584 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.352300  1691 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.352854  1561 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.334815  1637 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.353343  1628 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.355208  1637 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:04.356105  1890 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
W20260809 23:21:04.356884  1834 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.237s	user 0.011s	sys 0.007s
I20260809 23:21:04.309146  1886 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260809 23:21:04.359509  1656 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.360211 24498 leader_election.cc:398] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18.
I20260809 23:21:04.360562 24498 leader_election.cc:401] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260809 23:21:04.328387  1827 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.361161  1827 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.361601 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.361900 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3230) took 1109 ms. Trace:
I20260809 23:21:04.324679  1739 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.362778 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2809) took 1675 ms. Trace:
I20260809 23:21:04.363909  1827 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.364344 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3251) took 1746 ms. Trace:
I20260809 23:21:04.363360 24429 rpcz_store.cc:276] 0809 23:21:02.686987 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.687125 (+   138us) service_pool.cc:227] Handling call
0809 23:21:02.687480 (+   355us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:02.687809 (+   329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:02.687837 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:02.688275 (+   438us) log.cc:842] Serialized 26 byte log entry
0809 23:21:02.937243 (+248968us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:02.937294 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:02.937384 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.274240 (+1336856us) spinlock_profiling.cc:243] Waited 33.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:04.362480 (+ 88240us) raft_consensus.cc:1671] finished
0809 23:21:04.362502 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.362717 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8496,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":8355,"spinlock_wait_cycles":77118336,"thread_start_us":248538,"threads_started":1,"wal-append.queue_time_us":347263}
I20260809 23:21:04.364962 24543 rpcz_store.cc:276] 0809 23:21:02.618287 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.619688 (+  1401us) service_pool.cc:227] Handling call
0809 23:21:02.659226 (+ 39538us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:02.659579 (+   353us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:02.659605 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:02.671336 (+ 11731us) log.cc:842] Serialized 26 byte log entry
0809 23:21:02.944401 (+273065us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:02.944457 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:02.944553 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.355835 (+1411282us) spinlock_profiling.cc:243] Waited 7.75 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:04.364033 (+  8198us) raft_consensus.cc:1671] finished
0809 23:21:04.364060 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.364279 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9676,"raft.queue_time_us":4597,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":23464,"spinlock_wait_cycles":17829632,"thread_start_us":220697,"threads_started":1,"wal-append.queue_time_us":376690}
I20260809 23:21:04.366065 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.367164  1036 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18.
I20260809 23:21:04.367959  1036 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:04.368396  1879 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:04.368868  1879 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:04.369235  1879 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:21:04.369513  1842 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.284s	user 0.004s	sys 0.006s
I20260809 23:21:04.332414  1434 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:04.335585  1611 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:04.371883  1611 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:04.372283  1611 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.362344 24309 rpcz_store.cc:276] 0809 23:21:03.252548 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.252898 (+   350us) service_pool.cc:227] Handling call
0809 23:21:03.253338 (+   440us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.257994 (+  4656us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:21:03.258293 (+   299us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.076492 (+818199us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:04.086527 (+ 10035us) log.cc:842] Serialized 26 byte log entry
0809 23:21:04.086700 (+   173us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:04.086747 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.086834 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.361466 (+274632us) raft_consensus.cc:1671] finished
0809 23:21:04.361648 (+   182us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.361819 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14649,"prepare.queue_time_us":748,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":95702,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":25509,"thread_start_us":814604,"threads_started":1,"wal-append.queue_time_us":970296}
I20260809 23:21:04.374261 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.374667  1081 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:04.376113 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3225) took 1727 ms. Trace:
I20260809 23:21:04.328994  1784 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:04.376503 24323 rpcz_store.cc:276] 0809 23:21:02.649232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.659268 (+ 10036us) service_pool.cc:227] Handling call
0809 23:21:02.659631 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:02.748569 (+ 88938us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:02.748599 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:02.748641 (+    42us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:02.748680 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.375889 (+1627209us) spinlock_profiling.cc:243] Waited 88.7 ms on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:04.375940 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.376060 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204049664}
I20260809 23:21:04.378074 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.378314 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1271) took 1103 ms. Trace:
I20260809 23:21:04.380972 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:04.291162 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.020s	sys 0.044s
I20260809 23:21:04.370833   717 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:04.378798 24427 rpcz_store.cc:276] 0809 23:21:03.274556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.730510 (+455954us) service_pool.cc:227] Handling call
0809 23:21:03.769500 (+ 38990us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.769906 (+   406us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:03.769943 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:03.866769 (+ 96826us) log.cc:842] Serialized 26 byte log entry
0809 23:21:04.236018 (+369249us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:04.236073 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.236173 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.378030 (+141857us) raft_consensus.cc:1671] finished
0809 23:21:04.378060 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.378244 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":512,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":95025,"thread_start_us":257010,"threads_started":1,"wal-append.queue_time_us":437785}
I20260809 23:21:04.339766 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:04.384797 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 268563us
I20260809 23:21:04.386541 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3224) took 1759 ms. Trace:
I20260809 23:21:04.394835 24313 rpcz_store.cc:276] 0809 23:21:02.628334 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.650496 (+ 22162us) service_pool.cc:227] Handling call
0809 23:21:03.666987 (+1016491us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.749346 (+ 82359us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.386370 (+637024us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003854d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:04.386488 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2336651520}
I20260809 23:21:04.387492 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3114) took 2148 ms. Trace:
I20260809 23:21:04.396833 24545 rpcz_store.cc:276] 0809 23:21:02.239261 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.239476 (+   215us) service_pool.cc:227] Handling call
0809 23:21:02.416584 (+177108us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:02.448687 (+ 32103us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.387269 (+1938582us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54002710d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:04.387423 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":400322944}
I20260809 23:21:04.388123 24542 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:04.400312 24309 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 17. Ops: []
I20260809 23:21:04.360859  1656 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.350732  1781 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.407960 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.408780 24317 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:21:04.408874 24539 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 13.
I20260809 23:21:04.416679 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.411103 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3227) took 1160 ms. Trace:
W20260809 23:21:04.410790 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71126.6us
I20260809 23:21:04.415871 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:04.422231 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:04.429114 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:04.430505 24427 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.432369 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.388604   717 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:04.406597  1680 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.661s	user 0.025s	sys 0.013s
I20260809 23:21:04.438413 24321 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:04.363379  1739 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:04.448001  1680 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.705s	user 0.025s	sys 0.013s
W20260809 23:21:04.451396 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89853.6us
I20260809 23:21:04.428931  1303 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:04.452286 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121890us
I20260809 23:21:04.457816 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:04.439700  1081 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:04.454859 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.456580 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.443250   717 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:04.535181 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 79489.8us
I20260809 23:21:04.536792 25373 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:04.537519 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 79782.6us
I20260809 23:21:04.538869 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:04.539387 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:04.459342 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2717) took 1053 ms. Trace:
I20260809 23:21:04.504868 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.540093 24311 rpcz_store.cc:276] 0809 23:21:03.405942 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.758302 (+352360us) service_pool.cc:227] Handling call
0809 23:21:04.459158 (+700856us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:04.459286 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":325125376}
I20260809 23:21:04.540943 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.414275  1870 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.545935 25363 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:04.556941 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.554400 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.560346 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.457476  1637 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:04.562899 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.566617 25363 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:04.567507 24311 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:04.569110 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.571029 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:04.570485 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:04.570613 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.572105 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:04.571734 24535 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:04.543804  1303 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:04.554883  1081 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:04.544227  1633 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:04.503677  1667 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260809 23:21:04.577345 25373 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:04.581250 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.585832 24427 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:04.566159  1637 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:04.553743  1923 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:04.587105 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:04.588073 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131580us
I20260809 23:21:04.590721 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.553918  1403 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260809 23:21:04.574244  1629 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.566182  1906 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:04.576619  1827 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:04.595082  1839 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:04.594909  1081 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.575057  1623 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.604743 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.648437 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.457710  1780 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.591459  1637 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.665375  1810 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:04.667400  1780 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.669665  1431 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.605729  1906 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:04.672209  1906 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:04.673071 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:04.673559  1739 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.673700 24543 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.604554  1680 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260809 23:21:04.676285 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85676.1us
I20260809 23:21:04.677639 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:04.587863  1714 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.678236 24539 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:04.638584  1839 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:04.679214  1839 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:04.679772  1714 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.588581  1633 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.681941  1633 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.642170 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:04.590142  1691 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:04.682785 24427 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:04.683233  1635 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:04.683120  1923 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:04.683590 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.591296  1502 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:04.686115  1870 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.634953 24535 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:04.688040 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:04.678776 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:04.689805 24313 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: [17.9-17.9]
W20260809 23:21:04.655882  1910 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260809 23:21:04.657099  1913 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260809 23:21:04.659371  1584 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.660070  1859 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.686051 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.654333  1876 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:21:04.689922 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.692270 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:04.709995 24539 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:04.692641 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:04.696290 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1515) took 1346 ms. Trace:
I20260809 23:21:04.712149 25371 rpcz_store.cc:276] 0809 23:21:03.350108 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.351376 (+  1268us) service_pool.cc:227] Handling call
0809 23:21:03.351725 (+   349us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:04.065600 (+713875us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.696005 (+630405us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:04.696225 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1609318144}
W20260809 23:21:04.698115 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3250) took 2080 ms (client timeout 2073 ms). Trace:
W20260809 23:21:04.714010 24546 rpcz_store.cc:269] 0809 23:21:02.617985 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.629462 (+ 11477us) service_pool.cc:227] Handling call
0809 23:21:04.697939 (+2068477us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:04.698047 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":747418368}
I20260809 23:21:04.698127 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:04.701900  1935 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:04.652011  1827 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:04.711187 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:04.708792  1859 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:04.732252 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46292us
I20260809 23:21:04.733076 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:04.689697  1680 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.639941  1522 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260809 23:21:04.740722 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:04.742112 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88063.5us
I20260809 23:21:04.744444 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:04.745764 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.749609 25371 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:04.725059  1869 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:04.757726 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 192440us
I20260809 23:21:04.709280  1876 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.758759 24539 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:04.734375  1403 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:21:04.759646 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80744.3us
I20260809 23:21:04.762975 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:04.759526 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:04.768412 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:04.757120  1772 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.000s	sys 0.006s
I20260809 23:21:04.779716 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1513) took 1439 ms. Trace:
I20260809 23:21:04.753916  1629 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:04.746908  1691 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:04.785548 25365 rpcz_store.cc:276] 0809 23:21:03.340296 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.340406 (+   110us) service_pool.cc:227] Handling call
0809 23:21:03.367486 (+ 27080us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.975684 (+608198us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:03.975743 (+    59us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:03.976338 (+   595us) log.cc:842] Serialized 26 byte log entry
0809 23:21:04.285284 (+308946us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:04.285341 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.285762 (+   421us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.779523 (+493761us) raft_consensus.cc:1671] finished
0809 23:21:04.779544 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.779672 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":248295,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":813357,"threads_started":2,"wal-append.queue_time_us":408448}
I20260809 23:21:04.792392 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.765277  1876 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:04.794762 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:04.762740  1739 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:04.779237  1541 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.753259  1714 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260809 23:21:04.784390  1794 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:04.752648  1680 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.827597 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1562) took 2141 ms. Trace:
I20260809 23:21:04.735632  1303 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.779381  1788 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:21:04.842979 24314 rpcz_store.cc:276] 0809 23:21:02.686093 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.695365 (+  9272us) service_pool.cc:227] Handling call
0809 23:21:04.827397 (+2132032us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:04.827536 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3119346560}
I20260809 23:21:04.805003  1036 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:04.835990 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58555us
I20260809 23:21:04.787928  1611 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.779089  1781 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260809 23:21:04.808647  1584 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.858003 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.854329 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3239) took 1372 ms. Trace:
I20260809 23:21:04.824764  1770 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:04.873374  1938 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
W20260809 23:21:04.874222  1924 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
W20260809 23:21:04.880668  1879 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.000s
I20260809 23:21:04.881915  1637 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.765228  1869 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.884521  1680 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.858736 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:04.872025 24312 rpcz_store.cc:276] 0809 23:21:03.482122 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.382277 (+900155us) service_pool.cc:227] Handling call
0809 23:21:04.386688 (+  4411us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:04.451990 (+ 65302us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:04.452035 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:04.544970 (+ 92935us) log.cc:842] Serialized 52 byte log entry
0809 23:21:04.611692 (+ 66722us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:04.611747 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.611832 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.854021 (+242189us) raft_consensus.cc:1671] finished
0809 23:21:04.854052 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.854249 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64766,"raft.queue_time_us":66439,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":53326,"thread_start_us":37861,"threads_started":1,"wal-append.queue_time_us":111929}
I20260809 23:21:04.905046 25366 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 17. Ops: []
I20260809 23:21:04.909793 24320 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260809 23:21:04.912189 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54267.9us
I20260809 23:21:04.913774 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:04.853144  1849 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260809 23:21:04.918728 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 150442us
I20260809 23:21:04.889021  1784 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.943058 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:04.944705 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:04.918116 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59514.5us
I20260809 23:21:04.886425  1906 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.828112  1541 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.914769  1680 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.903563  1794 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.951709 24539 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:04.952010 24543 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:21:04.911335  1781 consensus_queue.cc:1225] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:04.955579 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3238) took 1515 ms. Trace:
I20260809 23:21:04.915922  1638 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:04.861194  1036 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:04.915305  1770 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:04.959298 24323 rpcz_store.cc:276] 0809 23:21:03.440352 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.380057 (+939705us) service_pool.cc:227] Handling call
0809 23:21:04.380455 (+   398us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:04.451106 (+ 70651us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:04.451140 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:04.451711 (+   571us) log.cc:842] Serialized 52 byte log entry
0809 23:21:04.644814 (+193103us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:04.645154 (+   340us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.691915 (+ 46761us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.692075 (+   160us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.955245 (+263170us) raft_consensus.cc:1671] finished
0809 23:21:04.955278 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.955499 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157153,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":116585,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":174204672,"thread_start_us":85232,"threads_started":1,"wal-append.queue_time_us":186679}
W20260809 23:21:04.949785  1813 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.668s	user 0.011s	sys 0.014s
I20260809 23:21:04.977303 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:04.978510 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:04.960953  1794 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:04.971894  1813 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.881s	user 0.012s	sys 0.015s
I20260809 23:21:04.949748  1784 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.001672  1784 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:04.987963  1920 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260809 23:21:05.008167  1541 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:04.987677  1629 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:04.992561 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85405.1us
I20260809 23:21:04.994127  1945 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:04.994778  1907 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:04.995337  1950 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:04.909786  1879 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.017364  1611 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.017808  1879 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.021564  1794 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:04.996696  1940 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260809 23:21:05.022707  1950 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:05.023267  1950 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.997496  1906 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.997681  1955 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:04.956321  1680 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:04.998889  1637 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:04.968917  1781 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:05.008796  1541 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:05.022334  1943 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260809 23:21:05.021584 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1268) took 1754 ms. Trace:
I20260809 23:21:04.968328  1959 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:05.031131 24323 pending_rounds.cc:76] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 12
I20260809 23:21:04.967125  1638 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.037652 24323 pending_rounds.cc:97] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260809 23:21:05.038087 24323 raft_consensus.cc:2866] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:05.037139 24425 rpcz_store.cc:276] 0809 23:21:03.267441 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.267616 (+   175us) service_pool.cc:227] Handling call
0809 23:21:03.299895 (+ 32279us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.300305 (+   410us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:03.300333 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:04.694788 (+1394455us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:04.703136 (+  8348us) log.cc:842] Serialized 26 byte log entry
0809 23:21:04.836583 (+133447us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:04.836729 (+   146us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.838547 (+  1818us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:05.009585 (+171038us) raft_consensus.cc:1671] finished
0809 23:21:05.009616 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:05.021512 (+ 11896us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80308096,"thread_start_us":89333,"threads_started":1,"wal-append.queue_time_us":226340}
I20260809 23:21:05.012984  1948 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.242s
I20260809 23:21:05.031890  1781 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:05.025319  1945 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.039016  1770 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.027216  1629 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:05.037585  1955 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260809 23:21:05.029721  1907 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:05.041934 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61597.5us
W20260809 23:21:05.065065 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 305586us
I20260809 23:21:05.021492  1876 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.068534 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:05.140532 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:05.048044  1847 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.926s	user 0.003s	sys 0.019s
I20260809 23:21:05.061934 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1266) took 2058 ms. Trace:
I20260809 23:21:05.063066 24534 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:05.058063 24311 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:05.158808 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 90346.5us
I20260809 23:21:05.173470 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:05.183528 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:05.184424 24534 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:05.042376 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:05.045747 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:05.182648 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 116084us
I20260809 23:21:05.044878  1680 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.190056 25365 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 15: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:05.192169 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:05.191041 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 175384us
I20260809 23:21:05.199935 25368 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:05.155081 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:05.182950 24435 rpcz_store.cc:276] 0809 23:21:03.003625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.004353 (+   728us) service_pool.cc:227] Handling call
0809 23:21:03.068508 (+ 64155us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:03.250153 (+181645us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:03.537278 (+287125us) log.cc:842] Serialized 14 byte log entry
0809 23:21:03.722478 (+185200us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.147040 (+424562us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:04.147104 (+    64us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:04.147145 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.251578 (+104433us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b180054d960. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:05.061699 (+810121us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540018cc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:05.061752 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:05.061877 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":995499,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":84,"raft.queue_time_us":47144,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":37278,"spinlock_wait_cycles":452032128,"thread_start_us":873967,"threads_started":3,"wal-append.queue_time_us":368067}
W20260809 23:21:05.045514  1919 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260809 23:21:05.196763 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:05.061023  1844 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.751s	user 0.006s	sys 0.006s
I20260809 23:21:05.202839 24539 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:05.043457  1959 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Stepping down as leader of term 14
W20260809 23:21:05.193606 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53162.2us
I20260809 23:21:05.064487  1501 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.207099 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:05.208595 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:05.209932 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:05.204308 24534 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.046613  1036 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:05.178622  1176 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:05.208880 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:05.210332 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:05.056046  1945 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.215540 25369 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:05.234344 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.234861 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.235028 24541 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:05.236104 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.236212 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.236683 24541 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:05.236733 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:05.237671 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.238276 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:05.215902 24534 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:05.238883 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.213039 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:05.241267 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.242436 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.214521 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.246101 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:05.231220 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:05.212534 24539 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:05.220395  1827 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:05.216207  1036 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:05.243122  1965 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:05.060115  1907 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.243618  1691 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:05.239566  1673 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:05.063582  1950 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.210840  1976 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:05.211409  1884 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:05.244549  1955 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:05.245119  1770 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:05.160454  1303 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:05.247092 24539 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:05.049104  1963 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.197s
W20260809 23:21:05.224637  1780 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.557s	user 0.009s	sys 0.002s
I20260809 23:21:05.249505 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.285919 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.290338 24429 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:05.242194  1859 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.251919 25369 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:05.207620  1959 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:05.047724   717 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.052743  1839 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.258631 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:05.293479 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.282841  1884 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:05.046105  1781 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.268388  1673 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.255721  1965 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:05.227314  1844 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.057062  1629 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.264410  1691 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:05.263536  1680 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:05.286623 24539 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.279727  1975 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:05.270484  1955 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:05.274390  1770 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:05.328440 24545 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:05.255374 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:05.303958 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:05.292528  1976 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:05.292763  1923 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:05.315443  1884 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.317116  1965 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:05.317818 25373 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:05.319998 24323 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:05.324198 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:05.345880 24543 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:05.329978 24539 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:05.307137  1561 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:05.337296 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1453) took 2506 ms. Trace:
I20260809 23:21:05.342659  1673 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.343649   717 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.347738 24544 rpcz_store.cc:276] 0809 23:21:02.830742 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.834721 (+  3979us) service_pool.cc:227] Handling call
0809 23:21:02.839650 (+  4929us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:02.840049 (+   399us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:02.876044 (+ 35995us) log.cc:842] Serialized 14 byte log entry
0809 23:21:03.069223 (+193179us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:03.124629 (+ 55406us) log.cc:842] Serialized 26 byte log entry
0809 23:21:03.124812 (+   183us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:03.124855 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:03.124946 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:05.047410 (+1922464us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:05.325612 (+278202us) raft_consensus.cc:1671] finished
0809 23:21:05.337028 (+ 11416us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:05.337215 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96720,"prepare.queue_time_us":36575,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":70209,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":32467,"spinlock_wait_cycles":5609856,"thread_start_us":131310,"threads_started":1,"wal-append.queue_time_us":245320}
I20260809 23:21:05.347666 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.325981  1691 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.323474  1876 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.318277  1959 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.331451  1770 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:05.331565  1955 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:05.330989  1923 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:05.373713 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1519) took 1901 ms. Trace:
I20260809 23:21:05.351028  1561 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:05.357010  1680 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:05.356559  1976 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:05.440608  1923 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:05.356801  1963 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260809 23:21:04.326370 24430 rpcz_store.cc:276] 0809 23:21:03.304891 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.756854 (+451963us) service_pool.cc:227] Handling call
0809 23:21:03.757132 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:03.757326 (+   194us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:04.116275 (+358949us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.315716 (+199441us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.316437 (+   721us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:04.316485 (+    48us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:04.316534 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.316598 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.316700 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10,"prepare.queue_time_us":254082,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":3626,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":13287,"spinlock_wait_cycles":21338624,"thread_start_us":556850,"threads_started":2,"wal-append.queue_time_us":255477}
W20260809 23:21:05.353511 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49619.8us
I20260809 23:21:05.442399 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.443593  1788 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:05.443940  1788 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:05.444240  1788 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:05.444310 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:05.425390 24539 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:05.427913 24543 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:21:05.447263 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.448114  1963 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:21:05.448014  1413 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:05.448707  1963 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:05.449580  1963 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.447081  1839 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:05.453234  1731 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.453791  1859 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.429314 24312 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:05.432240 25370 rpcz_store.cc:276] 0809 23:21:03.472091 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.472265 (+   174us) service_pool.cc:227] Handling call
0809 23:21:05.373388 (+1901123us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:05.373494 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2685036928}
I20260809 23:21:05.434296  1561 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.434785  1430 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.435156  1959 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:05.440058  1976 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.372335 24534 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:05.456326 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:05.461404  1961 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:05.465395  1961 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:05.465775  1961 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:05.462917 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3253) took 2844 ms. Trace:
I20260809 23:21:05.467592 24534 rpcz_store.cc:276] 0809 23:21:02.619028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.619146 (+   118us) service_pool.cc:227] Handling call
0809 23:21:02.619425 (+   279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:05.062959 (+2443534us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b540021c8a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:21:05.462668 (+399709us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:05.462868 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4553723136}
I20260809 23:21:05.468233  1844 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:05.463486 24312 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:05.464948 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:05.470069 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:05.470387 25370 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:05.470775 24312 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:05.471537 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:05.472892 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:05.476740 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:05.477599 24539 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:05.493039 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:05.476693  1859 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:05.532344  1965 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.535902  1981 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:05.536412  1981 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.538110  1198 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:05.539636 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:05.540241 24539 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:05.540692  1981 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.528172  1961 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.539990 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:05.544339 24534 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:05.496465 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:05.545226 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:05.545838 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49573.1us
W20260809 23:21:05.495146 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3264) took 2097 ms (client timeout 2269 ms). Trace:
I20260809 23:21:05.549401 24539 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:05.551620 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94367.8us
I20260809 23:21:05.551672 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.552841 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:05.549401 24542 rpcz_store.cc:269] 0809 23:21:03.397726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.415546 (+ 17820us) service_pool.cc:227] Handling call
0809 23:21:05.494852 (+2079306us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:05.495066 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2115366272}
W20260809 23:21:05.480600  1677 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
W20260809 23:21:05.558439 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94772.6us
I20260809 23:21:05.559528 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:05.560573  1680 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:21:05.561170  1680 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:05.562194  1680 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.562754 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:05.563405 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:05.542997  1961 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.534847  1965 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.555195  1176 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:05.554836  1430 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:05.571745 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3261) took 2217 ms. Trace:
I20260809 23:21:05.557456  1981 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:05.538690  1198 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.576232 24540 rpcz_store.cc:276] 0809 23:21:03.354008 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.484446 (+130438us) service_pool.cc:227] Handling call
0809 23:21:03.527736 (+ 43290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:03.528032 (+   296us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:03.567103 (+ 39071us) log.cc:842] Serialized 14 byte log entry
0809 23:21:05.446994 (+1879891us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b1800532d80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:21:05.571300 (+124306us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:05.571374 (+    74us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:05.571435 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:05.571523 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:05.571670 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":45493,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":4390,"spinlock_wait_cycles":6970112,"thread_start_us":124061,"threads_started":1}
I20260809 23:21:05.577133 24428 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260809 23:21:05.588989 24428 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 14
I20260809 23:21:05.589357 24428 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260809 23:21:05.589069 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:05.589704 24428 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:05.560212  1976 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:05.617412  1036 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:05.573377  1176 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:05.618537  1176 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.623200  1981 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:05.624142  1981 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.555698  1788 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.627751  1955 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.682169  1955 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.685460 24540 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:05.629514  1561 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.688325  1561 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.632519  1788 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:05.640214  1975 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:05.586553  1430 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:05.690768  1569 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:05.691591  1430 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.626909  1691 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.699122  1691 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:05.705314 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:05.706079  1770 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.648691  1413 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:05.657719  1923 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.830915  1502 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:05.856964  1502 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:05.833077  1413 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:05.909019 24542 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:05.970814  1788 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:06.022372  1998 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.164s	user 0.000s	sys 0.003s
I20260809 23:21:06.044201  1923 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:06.048138  1413 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.048664  1788 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.065430  1770 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.070391 24542 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:06.158452 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2811) took 3124 ms (client timeout 1847 ms). Trace:
W20260809 23:21:06.159276  1907 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.001s	sys 0.019s
W20260809 23:21:06.095090  1680 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.019s
I20260809 23:21:06.066126  1502 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:06.163968  1884 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.000s	sys 0.022s
W20260809 23:21:06.177789  1680 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.008s	sys 0.019s
W20260809 23:21:06.164935 24422 rpcz_store.cc:269] 0809 23:21:02.961090 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.961280 (+   190us) service_pool.cc:227] Handling call
0809 23:21:06.085096 (+3123816us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.085245 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236747648}
W20260809 23:21:06.209776  1959 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.732s	user 0.007s	sys 0.006s
W20260809 23:21:06.179168  1907 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.001s	sys 0.019s
W20260809 23:21:06.188669  1884 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.000s	sys 0.022s
W20260809 23:21:06.211351  1176 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.003s
I20260809 23:21:06.211862  1176 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.213244  1923 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.213784  1923 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:06.214684  2010 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
W20260809 23:21:06.226557 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2826) took 1840 ms (client timeout 1767 ms). Trace:
W20260809 23:21:06.227061 24429 rpcz_store.cc:269] 0809 23:21:04.385788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.385911 (+   123us) service_pool.cc:227] Handling call
0809 23:21:06.226338 (+1840427us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.226474 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2007412992}
W20260809 23:21:06.232282 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.004s	sys 0.045s
W20260809 23:21:06.232657 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.004s	sys 0.045s
W20260809 23:21:06.232934 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 653321us
I20260809 23:21:06.235329 24541 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:06.235785 24541 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:06.243217 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.247953 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.249998 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.250831 24539 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:06.246275  1176 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.264714 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:06.265540 24539 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:06.268556  1923 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.255271  2006 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:06.252285  1502 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:06.276726 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.012s	sys 0.029s
I20260809 23:21:06.269951 24546 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
W20260809 23:21:06.403545 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.012s	sys 0.029s
I20260809 23:21:06.361536 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:06.358105 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 214344us
W20260809 23:21:06.405700 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44604.8us
I20260809 23:21:06.410456 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:06.412086 24540 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:06.418411 24546 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:06.299090  1788 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.279876  1923 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:06.317430  2009 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.109s	user 0.002s	sys 0.000s
W20260809 23:21:06.460644 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3235) took 3105 ms (client timeout 3128 ms). Trace:
I20260809 23:21:06.461199 24539 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:06.461912  1981 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:06.317596  2011 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260809 23:21:06.463406 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3244) took 2781 ms. Trace:
I20260809 23:21:06.463258  1981 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.464170  1519 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:06.436534  2006 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:06.466086  2006 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.469436 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1275) took 3193 ms. Trace:
I20260809 23:21:06.440150  1955 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.441354  1964 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:06.470297  1964 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:06.470042  2006 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:06.470884  1964 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.280970  1502 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.459308  2027 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:06.314429  2014 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.106s	user 0.000s	sys 0.002s
I20260809 23:21:06.438999  1827 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:06.463958 24309 rpcz_store.cc:276] 0809 23:21:03.681860 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.401157 (+719297us) service_pool.cc:227] Handling call
0809 23:21:04.401433 (+   276us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:04.554292 (+152859us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:04.554323 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:04.554375 (+    52us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:04.554406 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.449720 (+1895314us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.449860 (+   140us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.463102 (+ 13242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":351117056}
I20260809 23:21:06.483434  1955 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.483976  1827 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.484922 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.485975 24309 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:06.486330  1955 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.488435  1955 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.478143  1981 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.491503 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1516) took 2044 ms. Trace:
I20260809 23:21:06.492144 25366 rpcz_store.cc:276] 0809 23:21:04.446414 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.446549 (+   135us) service_pool.cc:227] Handling call
0809 23:21:04.446843 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:04.911071 (+464228us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.490741 (+1579670us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b54007053d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.491163 (+   422us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1053185920}
I20260809 23:21:06.469951 24428 rpcz_store.cc:276] 0809 23:21:03.275758 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.741432 (+465674us) service_pool.cc:227] Handling call
0809 23:21:03.741786 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:05.590273 (+1848487us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.468763 (+878490us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.469130 (+   367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4219921664}
I20260809 23:21:06.489439  1955 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:06.469861 24321 rpcz_store.cc:269] 0809 23:21:03.354738 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.180619 (+825881us) service_pool.cc:227] Handling call
0809 23:21:06.460047 (+2279428us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.460300 (+   253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":365506816}
W20260809 23:21:06.511482 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 263491us
I20260809 23:21:06.513075  1955 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.513082 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.528615 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.530233  1788 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:06.532196 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.533069 24321 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:06.534461 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:06.535264 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:06.542346  2033 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:06.544644  2027 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.546370  1491 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260809 23:21:06.543650 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:06.536196 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.556806  1502 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.562506 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.516479  1923 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:06.563771  1827 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.564536  1638 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.567730 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:06.569209  1667 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:06.580518 24321 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.583295  2033 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.594650  1634 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.594491  1667 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.584472  2027 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:06.597738 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62130.1us
I20260809 23:21:06.599948 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:06.601301  1680 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:06.602077  1680 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:06.604275 24544 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:06.603870  1680 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.608784   877 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:06.609663 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 2.28068e+06us
I20260809 23:21:06.611586 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.612699 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.613902 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.615132  1352 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:06.616662 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.621273  1434 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:06.624225  1485 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:06.624600  1434 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:06.626516  1931 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.391s	user 0.002s	sys 0.000s
I20260809 23:21:06.615175 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.596318  1923 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:21:06.648677 24422 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:06.648561  1352 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.649269  1539 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:06.650053  1539 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.621277   877 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.652956  1539 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.653766   877 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.659613 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.659943  1434 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.659643 24540 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:06.661181 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.662098 24424 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:06.663542 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.664294  2033 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.664489 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.664808 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.665189 24427 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:06.666359 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122935us
I20260809 23:21:06.669713 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.670644 24435 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:06.671334 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.672087 24429 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:06.672667 24435 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:06.673166 24435 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:06.674780 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.676154 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.677330 24539 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:06.678495 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.679543 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.679778 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.680555 24543 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:06.680651 24539 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:06.682227 24425 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:06.683905  1352 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.684700  1958 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:06.685236 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.685642  1958 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.686761 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.687597  1667 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.688095  1958 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.688563  1667 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:06.665908 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.687650 24546 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:06.674435 24435 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.704941 24321 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:06.710839 24430 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:06.711071 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:06.715149  1923 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:06.715560  1923 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:06.715984  1923 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.721488 24534 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:06.179373 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:06.724927 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.725715 24424 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:06.696419  1667 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:06.728906  2001 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.683s	user 0.000s	sys 0.003s
W20260809 23:21:06.729955 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 174858us
W20260809 23:21:06.730151  2008 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.666s	user 0.004s	sys 0.000s
I20260809 23:21:06.730459  1485 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:06.731769 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1532) took 1168 ms. Trace:
I20260809 23:21:06.648447  1953 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:06.737206 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3248) took 2682 ms. Trace:
I20260809 23:21:06.739368 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3250) took 2343 ms. Trace:
I20260809 23:21:06.740720 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3249) took 2345 ms. Trace:
I20260809 23:21:04.418855 24316 rpcz_store.cc:276] 0809 23:21:03.250361 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.250508 (+   147us) service_pool.cc:227] Handling call
0809 23:21:03.250915 (+   407us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:03.493086 (+242171us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:03.493120 (+    34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:03.493786 (+   666us) log.cc:842] Serialized 52 byte log entry
0809 23:21:04.383669 (+889883us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:04.410530 (+ 26861us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.410772 (+   242us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:04.410861 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:04.410906 (+    45us) raft_consensus.cc:1671] finished
0809 23:21:04.410917 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:04.411050 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":30257,"raft.queue_time_us":48172,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":25164,"spinlock_wait_cycles":21184768,"thread_start_us":1050302,"threads_started":2,"wal-append.queue_time_us":792554}
I20260809 23:21:06.754719 24427 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:06.761013 24311 rpcz_store.cc:276] 0809 23:21:04.395577 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.803970 (+408393us) service_pool.cc:227] Handling call
0809 23:21:04.804305 (+   335us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:05.189335 (+385030us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.740228 (+1550893us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.740559 (+   331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":583120640}
I20260809 23:21:06.743800 24435 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.752393 24544 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:06.756667 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2733) took 1132 ms. Trace:
I20260809 23:21:06.759543 24313 rpcz_store.cc:276] 0809 23:21:04.054425 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.594586 (+540161us) service_pool.cc:227] Handling call
0809 23:21:04.594904 (+   318us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:04.691794 (+ 96890us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.736529 (+2044735us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b54003859d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.736975 (+   446us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":211759872}
I20260809 23:21:06.771456 24321 rpcz_store.cc:276] 0809 23:21:05.578770 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.533928 (+955158us) service_pool.cc:227] Handling call
0809 23:21:06.708596 (+174668us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.776198 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3165) took 1086 ms. Trace:
I20260809 23:21:06.767107 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1584) took 1705 ms. Trace:
I20260809 23:21:06.776762 24544 rpcz_store.cc:276] 0809 23:21:05.684116 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.684693 (+   577us) service_pool.cc:227] Handling call
0809 23:21:06.510501 (+825808us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900b7c 00007fac2090073b 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:21:06.770523 (+260022us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4100480}
I20260809 23:21:06.783981 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.776000 24430 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:06.784884 24430 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:06.704221  1964 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.778824 24323 rpcz_store.cc:276] 0809 23:21:05.041047 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.258399 (+217352us) service_pool.cc:227] Handling call
0809 23:21:06.746226 (+1487827us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.746450 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":44555136}
I20260809 23:21:06.762916  1953 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:06.804898 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.805351 24309 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:06.797345 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:06.757558 25365 rpcz_store.cc:276] 0809 23:21:05.563068 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.563219 (+   151us) service_pool.cc:227] Handling call
0809 23:21:05.563671 (+   452us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:05.564298 (+   627us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:05.564334 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:05.564856 (+   522us) log.cc:842] Serialized 26 byte log entry
0809 23:21:05.839661 (+274805us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:05.901038 (+ 61377us) log.cc:842] Serialized 14 byte log entry
0809 23:21:06.489430 (+588392us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b1800320860. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:06.704142 (+214712us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.730895 (+ 26753us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:06.731022 (+   127us) raft_consensus.cc:1671] finished
0809 23:21:06.731129 (+   107us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.731403 (+   274us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46498,"prepare.queue_time_us":60946,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":638895,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":35214,"spinlock_wait_cycles":8381312,"thread_start_us":370311,"threads_started":2,"wal-append.queue_time_us":353004}
I20260809 23:21:06.807957 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.813459 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:06.815014 24422 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:06.817000 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:06.816082  1528 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:06.818799  1784 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.819191  1528 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.819602  1485 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.819833  1784 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.817565 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:06.824970 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68325.3us
I20260809 23:21:06.825255  1403 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:06.825790  2040 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:06.826124  1403 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.826542  2040 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.828909  2040 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.829730  2030 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:06.830590  2030 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.825323 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1560) took 1367 ms. Trace:
I20260809 23:21:06.830987  1714 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:06.831954  1788 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.833158  1788 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.834324 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.835136 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:06.838322 24321 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:06.839102 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:06.839778 24323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:06.831881 25372 rpcz_store.cc:276] 0809 23:21:05.458063 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.458235 (+   172us) service_pool.cc:227] Handling call
0809 23:21:05.485474 (+ 27239us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:05.486337 (+   863us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:05.486364 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:05.523237 (+ 36873us) log.cc:842] Serialized 26 byte log entry
0809 23:21:05.813580 (+290343us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:05.813643 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:05.813748 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:06.729095 (+915347us) spinlock_profiling.cc:243] Waited 6.94 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:06.823010 (+ 93915us) raft_consensus.cc:1671] finished
0809 23:21:06.823035 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.825133 (+  2098us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":402,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":29552,"spinlock_wait_cycles":15972736,"thread_start_us":289895,"threads_started":1,"wal-append.queue_time_us":444060}
I20260809 23:21:06.843466  1746 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:06.844201  1746 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.846096 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1279) took 3559 ms. Trace:
I20260809 23:21:06.837971 24430 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.831303  2033 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:06.851406  2043 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:06.865842  2033 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:06.850466 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.866361 24430 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.831956  1667 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.851449 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:06.760582 24314 rpcz_store.cc:276] 0809 23:21:04.395896 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.843902 (+448006us) service_pool.cc:227] Handling call
0809 23:21:04.844280 (+   378us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:04.899293 (+ 55013us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:04.915164 (+ 15871us) log.cc:842] Serialized 14 byte log entry
0809 23:21:04.972057 (+ 56893us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:05.011870 (+ 39813us) log.cc:842] Serialized 52 byte log entry
0809 23:21:05.012013 (+   143us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:05.012204 (+   191us) log.cc:842] Serialized 14 byte log entry
0809 23:21:05.168458 (+156254us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b18004c23a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:05.642459 (+474001us) spinlock_profiling.cc:243] Waited 5.41 ms on lock 0x7b18004c23a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7
0809 23:21:05.642640 (+   181us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.738664 (+1096024us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b54003c32d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.738753 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:06.738843 (+    90us) raft_consensus.cc:1671] finished
0809 23:21:06.738857 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.739126 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":138904,"prepare.queue_time_us":174923,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":92049,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":10958,"spinlock_wait_cycles":51567104,"thread_start_us":47801,"threads_started":2,"wal-append.queue_time_us":139537,"wal-append.run_cpu_time_us":2715,"wal-append.run_wall_time_us":1123254}
I20260809 23:21:06.857506 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3218) took 4244 ms. Trace:
I20260809 23:21:06.874069 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3247) took 2823 ms. Trace:
I20260809 23:21:06.873843 24320 rpcz_store.cc:276] 0809 23:21:02.612729 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:02.614070 (+  1341us) service_pool.cc:227] Handling call
0809 23:21:02.653693 (+ 39623us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:04.910730 (+2257037us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.856727 (+1945997us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.857170 (+   443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5187769600}
I20260809 23:21:06.870492 24424 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:06.876353 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49975.2us
I20260809 23:21:06.868319 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.862775 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3161) took 1484 ms. Trace:
I20260809 23:21:06.865386 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3168) took 1161 ms. Trace:
I20260809 23:21:06.885267 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1587) took 1686 ms. Trace:
I20260809 23:21:06.885694 24320 rpcz_store.cc:276] 0809 23:21:05.198385 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.884194 (+1685809us) service_pool.cc:227] Handling call
0809 23:21:06.884657 (+   463us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.884912 (+   255us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:06.884935 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:06.884997 (+    62us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:06.885048 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.885098 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.885192 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.855571  1869 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.882382 24542 rpcz_store.cc:276] 0809 23:21:05.704136 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.705012 (+   876us) service_pool.cc:227] Handling call
0809 23:21:06.864990 (+1159978us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.865175 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":383837568}
I20260809 23:21:06.887647 24316 raft_consensus.cc:2354] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:06.881906 24545 rpcz_store.cc:276] 0809 23:21:05.378442 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.424311 (+ 45869us) service_pool.cc:227] Handling call
0809 23:21:05.424627 (+   316us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:05.439592 (+ 14965us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:05.439908 (+   316us) log.cc:842] Serialized 14 byte log entry
0809 23:21:05.440139 (+   231us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:05.440181 (+    42us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:05.440218 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.862201 (+1421983us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b54002706d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:06.862290 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.862507 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":679,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":37,"raft.queue_time_us":3245,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":33854208}
I20260809 23:21:06.891137 24423 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
W20260809 23:21:06.853160  2013 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.636s	user 0.003s	sys 0.000s
I20260809 23:21:06.891570 24423 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:06.892565 24423 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.894097 24423 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:06.893577  1953 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:06.870267  1784 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:06.895278  1953 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:06.874902 24317 rpcz_store.cc:276] 0809 23:21:03.980957 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.593474 (+612517us) service_pool.cc:227] Handling call
0809 23:21:04.603515 (+ 10041us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:04.635754 (+ 32239us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:04.635787 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:04.636451 (+   664us) log.cc:842] Serialized 52 byte log entry
0809 23:21:04.733314 (+ 96863us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:04.784769 (+ 51455us) log.cc:842] Serialized 14 byte log entry
0809 23:21:05.219271 (+434502us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b18003e5240. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:05.224461 (+  5190us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b18003e5240. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:06.744787 (+1520326us) spinlock_profiling.cc:243] Waited 7.37 ms on lock 0x7b18003e5240. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:06.803536 (+ 58749us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.803655 (+   119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:06.803722 (+    67us) raft_consensus.cc:1671] finished
0809 23:21:06.803734 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.803913 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72936,"prepare.queue_time_us":57174,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":48627,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":10176,"spinlock_wait_cycles":32680960,"thread_start_us":145524,"threads_started":2,"wal-append.queue_time_us":147988,"wal-append.run_cpu_time_us":3775,"wal-append.run_wall_time_us":1234326}
I20260809 23:21:06.899999 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.850257 24312 raft_consensus.cc:1203] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Deduplicated request from leader. Original: 15.9->[17.10-17.10]   Dedup: 17.10->[]
I20260809 23:21:06.901194 24312 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260809 23:21:06.902134 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:06.908326 24323 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260809 23:21:06.894646  1788 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:06.910944 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:06.846624  1746 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:06.913865 24435 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:06.914904 24321 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260809 23:21:06.916123 24424 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:06.912997 24316 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:06.920508 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.913851 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2729) took 1352 ms. Trace:
I20260809 23:21:06.921187 24435 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:06.921329 24311 rpcz_store.cc:276] 0809 23:21:05.561491 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.795302 (+1233811us) service_pool.cc:227] Handling call
0809 23:21:06.913784 (+118482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.923254 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.931159 24311 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 14. Current term is 15. Ops: []
I20260809 23:21:06.931928 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1595) took 1460 ms. Trace:
I20260809 23:21:06.932305 24311 rpcz_store.cc:276] 0809 23:21:05.471518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.930351 (+1458833us) service_pool.cc:227] Handling call
0809 23:21:06.930915 (+   564us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.931671 (+   756us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.931872 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.934656 24311 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:06.936537 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1589) took 1661 ms. Trace:
I20260809 23:21:06.937783  1844 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:06.939280 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.900107  1964 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:06.944648 24430 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260809 23:21:06.937924 24316 rpcz_store.cc:276] 0809 23:21:05.260371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.890034 (+1629663us) service_pool.cc:227] Handling call
0809 23:21:06.907888 (+ 17854us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.920884 (+ 12996us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.921143 (+   259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.944376 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.944311 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1590) took 1658 ms. Trace:
I20260809 23:21:06.947041 24317 rpcz_store.cc:276] 0809 23:21:05.263285 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.897070 (+1633785us) service_pool.cc:227] Handling call
0809 23:21:06.902113 (+  5043us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.917630 (+ 15517us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:06.917656 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:06.921300 (+  3644us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:06.921344 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.921398 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.921490 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.952831 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1596) took 1413 ms. Trace:
I20260809 23:21:06.953168 24311 rpcz_store.cc:276] 0809 23:21:05.539214 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.933644 (+1394430us) service_pool.cc:227] Handling call
0809 23:21:06.934210 (+   566us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.946227 (+ 12017us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.952786 (+  6559us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:06.945221 24430 pending_rounds.cc:76] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 8
I20260809 23:21:06.846462 24431 rpcz_store.cc:276] 0809 23:21:03.286833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:03.755556 (+468723us) service_pool.cc:227] Handling call
0809 23:21:03.755949 (+   393us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:03.861768 (+105819us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:04.317926 (+456158us) log.cc:842] Serialized 14 byte log entry
0809 23:21:06.606893 (+2288967us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:06.607775 (+   882us) log.cc:842] Serialized 26 byte log entry
0809 23:21:06.747268 (+139493us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:06.747351 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.747519 (+   168us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:06.845756 (+ 98237us) raft_consensus.cc:1671] finished
0809 23:21:06.845776 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.845936 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2350980,"prepare.queue_time_us":207954,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2409,"raft.queue_time_us":2291648,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":18251,"thread_start_us":638489,"threads_started":3,"wal-append.queue_time_us":475770,"wal-append.run_cpu_time_us":2365,"wal-append.run_wall_time_us":1272576}
I20260809 23:21:06.958552 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.961709 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1591) took 1697 ms. Trace:
I20260809 23:21:06.961449 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:06.962810 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:06.963330 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:06.962227 24323 rpcz_store.cc:276] 0809 23:21:05.264388 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.935289 (+1670901us) service_pool.cc:227] Handling call
0809 23:21:06.935703 (+   414us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.936002 (+   299us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:06.936389 (+   387us) log.cc:842] Serialized 14 byte log entry
0809 23:21:06.961309 (+ 24920us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:06.961379 (+    70us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:06.961431 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.961501 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.961622 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":219,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":134,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":117,"thread_start_us":24103,"threads_started":1}
I20260809 23:21:06.964468  1794 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:06.964535 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:06.958042 24430 pending_rounds.cc:97] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260809 23:21:06.966370 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.976922  1856 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:06.978338  1389 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:06.979473  1794 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:06.956051  1788 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:06.980798  1794 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.981650  1788 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.971282 24430 raft_consensus.cc:2866] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:21:06.983415  2051 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260809 23:21:06.984766 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:06.986685 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:06.980798 24422 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:06.988130 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1553) took 1956 ms (client timeout 2074 ms). Trace:
W20260809 23:21:06.990302 25373 rpcz_store.cc:269] 0809 23:21:05.031304 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.031568 (+   264us) service_pool.cc:227] Handling call
0809 23:21:06.987684 (+1956116us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:06.987923 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291995520}
I20260809 23:21:06.991089 24541 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.992290 24541 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:06.993273 24314 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:06.993964 24545 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:21:06.959897  2033 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:06.995844 24542 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260809 23:21:06.989388 24423 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:06.997287 24534 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260809 23:21:06.960009  2043 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.018954 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1588) took 1718 ms. Trace:
I20260809 23:21:07.019616 24320 rpcz_store.cc:276] 0809 23:21:05.255108 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.886637 (+1631529us) service_pool.cc:227] Handling call
0809 23:21:06.901720 (+ 15083us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.972959 (+ 71239us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:06.972985 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:06.973023 (+    38us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:06.973054 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:06.973116 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:06.973215 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:07.023195 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:06.996824  1528 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.024106 24423 raft_consensus.cc:2362] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20. Re-sending same reply.
W20260809 23:21:07.030514 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59936.8us
W20260809 23:21:07.030748 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66381.1us
I20260809 23:21:07.026126 24321 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:07.039300 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1594) took 1609 ms. Trace:
I20260809 23:21:07.044071 24534 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260809 23:21:07.019129  1788 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:07.039886 24313 rpcz_store.cc:276] 0809 23:21:05.429806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.941811 (+1512005us) service_pool.cc:227] Handling call
0809 23:21:06.942319 (+   508us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.942757 (+   438us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:06.943517 (+   760us) log.cc:842] Serialized 14 byte log entry
0809 23:21:07.038968 (+ 95451us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:07.039018 (+    50us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:07.039057 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:07.039115 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:07.039233 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":212,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":87962,"threads_started":1}
W20260809 23:21:07.059588 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 165469us
I20260809 23:21:07.060308 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.062878  2052 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:07.051234 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.060341 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1281) took 1748 ms. Trace:
I20260809 23:21:07.047067  1680 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:07.064529  1964 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.066299 24311 rpcz_store.cc:276] 0809 23:21:05.311567 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.958140 (+1646573us) service_pool.cc:227] Handling call
0809 23:21:07.060234 (+102094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:07.072518 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:07.076957 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 92361.9us
I20260809 23:21:07.082335 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:07.066511  2043 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.071439  1677 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:07.085722  1677 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.087719 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:07.087934  1198 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.088655  2052 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:07.088992  1677 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:07.090063  2052 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.091113 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:07.091847  1352 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
W20260809 23:21:07.093551 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2731) took 1519 ms (client timeout 1858 ms). Trace:
W20260809 23:21:07.104774 24309 rpcz_store.cc:269] 0809 23:21:05.573604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.510021 (+936417us) service_pool.cc:227] Handling call
0809 23:21:07.093234 (+583213us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:07.093421 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":503701760}
I20260809 23:21:07.095022  1736 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:07.083902 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:07.097041 24540 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
W20260809 23:21:07.115957 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63182.9us
I20260809 23:21:07.114441 24424 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:07.126196 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.126606 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.126878 24321 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:07.127264 25370 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:07.128481 25371 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: []
I20260809 23:21:07.129407 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.130192 25368 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:07.072739  1680 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:07.152251 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.122452  1352 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:07.124384 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:07.133359 25373 raft_consensus.cc:2362] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17. Re-sending same reply.
I20260809 23:21:07.116279  1736 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:07.177414 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93636.6us
I20260809 23:21:07.190752 24544 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [13.11-15.12]
I20260809 23:21:07.194360 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:07.196038 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.195196 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.165732  1680 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:07.116770  1964 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.105171   877 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
I20260809 23:21:07.166716  1352 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:07.263718 24544 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260809 23:21:07.264395  1352 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.264958 24378 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:07.265390 24378 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260809 23:21:07.265820  1352 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:07.266202  1844 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:07.195371  1539 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:07.267612  1539 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.212445 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.176240  2027 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:07.227298  1781 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.231712  2030 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:07.235404 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64002.5us
I20260809 23:21:07.238045  1403 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.241945  1780 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:07.242554  1485 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:07.270526  1539 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.243177  1556 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.243734 24542 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
W20260809 23:21:07.243315 24380 connection.cc:232] RPC call timeout handler was delayed by -2.1744385289493948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
W20260809 23:21:07.310844 24380 connection.cc:232] RPC call timeout handler was delayed by -1.2768598350230604s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22690000000000002s for a response
I20260809 23:21:07.247195  1823 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:07.251740 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90253us
I20260809 23:21:07.256871  1964 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.195140  2040 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:07.257215 24539 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.259793  1917 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:07.260670  1737 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260809 23:21:07.213920 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45827.8us
W20260809 23:21:07.283156 24266 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.847s (SENT)
I20260809 23:21:07.287459 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.287868 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3146) took 2595 ms. Trace:
I20260809 23:21:07.338610 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.288074 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.195668  1736 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.293861 25370 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:07.343225 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.299604 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.344842 24534 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:07.243163  1677 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:07.312084 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.311157 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.320662  1556 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.321040  1485 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.340375 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.333352  1917 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.347337 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.365477 24498 leader_election.cc:398] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:07.365844 24498 leader_election.cc:401] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260809 23:21:07.328737  2040 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:07.346920  1737 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:07.354696 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.347451  1964 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:07.355106 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.355460 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:07.355707 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 153213us
I20260809 23:21:07.356216 25369 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:07.356491 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.328119  1823 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.363257 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:07.346158  1923 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.008s
I20260809 23:21:07.366621 24435 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:07.348866 25370 raft_consensus.cc:2396] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:21:07.339005 24535 rpcz_store.cc:276] 0809 23:21:04.692796 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:04.692961 (+   165us) service_pool.cc:227] Handling call
0809 23:21:04.731217 (+ 38256us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:04.731622 (+   405us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:04.731654 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:04.732257 (+   603us) log.cc:842] Serialized 26 byte log entry
0809 23:21:06.861286 (+2129029us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b1800367c00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:06.985977 (+124691us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:07.025772 (+ 39795us) log.cc:842] Serialized 14 byte log entry
0809 23:21:07.038271 (+ 12499us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:07.038359 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:07.280445 (+242086us) raft_consensus.cc:1671] finished
0809 23:21:07.287561 (+  7116us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:07.287688 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5714,"prepare.queue_time_us":24777,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":15280,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":20752,"spinlock_wait_cycles":7744256,"thread_start_us":84961,"threads_started":1,"wal-append.queue_time_us":2386227}
I20260809 23:21:07.391427 24429 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:07.371433 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.376232 24423 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:07.376464 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.378571 24428 raft_consensus.cc:2362] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18. Re-sending same reply.
I20260809 23:21:07.384960 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.385774 24425 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:07.394448 24431 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260809 23:21:07.393015 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.398231 24430 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 16: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:07.395334 24545 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:07.401414 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.402171 24543 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:07.403152 25369 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:07.404209 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.404857 24540 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:07.405887 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.406556 24435 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:07.407564 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.408574 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.409709 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.414269 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.416369 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.419734 24427 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260809 23:21:07.423375 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.424237  1917 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.424126 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.424692 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:07.425231 24425 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:07.399734  1923 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.428601  1923 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.429389 24546 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:07.432147 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:07.433296 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:07.435832  1205 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:07.436060 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.437067  1389 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:07.437645  1389 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:07.437795 24430 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:07.438380  2069 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.118s	user 0.002s	sys 0.000s
I20260809 23:21:07.438696  1556 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.402185  1823 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.343037  1780 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.443101 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.373947  1677 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:07.397806 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45318.3us
I20260809 23:21:07.436880  1485 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.368779  1977 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.347904  2027 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.373744  2040 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.451268 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.384194 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.372758  1737 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:07.458168 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.452095 24423 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:07.458770 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.457407 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.471458 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.463140 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.464618 24534 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:07.436374  1205 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:07.442684  1389 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:07.473716 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.468528  1977 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.469202 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.475919 24378 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:07.479030  1205 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:07.452828  1403 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:07.449869   877 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260809 23:21:07.483426 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:07.485834 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.521314 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.484093 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.486121 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.491623 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.493114 24321 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:07.494410 24320 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:07.496963 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.506129 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.492268 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.514842 25370 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
W20260809 23:21:07.498368  2061 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260809 23:21:07.533562 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.530025 24311 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:07.538901 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:07.537467 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 80914.5us
I20260809 23:21:07.540462 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.541847 24317 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:07.544039 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:07.546100 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.550503 25372 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260809 23:21:07.552215 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.500114  1780 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.561874 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.565609 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.567783 25370 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:07.568827 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 107345us
I20260809 23:21:07.570765 24321 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:07.571290 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.572263 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.570518 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.521790  1539 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260809 23:21:07.525784   877 consensus_queue.cc:1225] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:07.581171 25369 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:21:07.589226 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.580145 25372 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260809 23:21:07.558736 25321 connection.cc:232] RPC call timeout handler was delayed by -1.5838586050085723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
I20260809 23:21:07.563913  1429 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:07.565398  1352 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:07.592779 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:07.594511 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.587772 24320 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:07.596824 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50807.1us
I20260809 23:21:07.575976  1737 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.604054  1737 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:07.498240  1966 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.128s
I20260809 23:21:07.612092  1977 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.560036  1556 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:07.612785 24317 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260809 23:21:07.613895 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.596130 25371 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:07.615715 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.616832 25369 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:07.619238 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.620462 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.624167 24309 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:07.624713 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:07.612135 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.621362 24428 raft_consensus.cc:2362] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18. Re-sending same reply.
I20260809 23:21:07.683210 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:07.684244 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 107341us
W20260809 23:21:07.681069 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69040.7us
W20260809 23:21:07.676621 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74426.3us
I20260809 23:21:07.692413 24539 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:07.695256 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72486us
W20260809 23:21:07.710863 25321 connection.cc:232] RPC call timeout handler was delayed by -1.5379586049821228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1896s for a response
W20260809 23:21:07.711287 25321 connection.cc:232] RPC call timeout handler was delayed by -1.4687306638807058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3718s for a response
I20260809 23:21:07.721192 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.726511 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:07.727608 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.730283 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.730686 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.736408 24535 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:21:07.737365 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:07.738662 24425 raft_consensus.cc:2354] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:07.738416 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.743252 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:07.744037 25373 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.720610 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2728) took 2549 ms. Trace:
I20260809 23:21:07.751718 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.741381 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:07.579990  2056 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.183s
I20260809 23:21:07.755524 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.750162 24312 rpcz_store.cc:276] 0809 23:21:05.155680 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:05.473808 (+318128us) service_pool.cc:227] Handling call
0809 23:21:06.811391 (+1337583us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:06.920495 (+109104us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:07.566110 (+645615us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b540019d7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:07.691044 (+124934us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540019d7d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:07.704786 (+ 13742us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3146418432}
I20260809 23:21:07.759907 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.707024  1977 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:07.765522 24435 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:07.774120 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:07.769481 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.060s	sys 0.078s
W20260809 23:21:07.837944 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.061s	sys 0.079s
I20260809 23:21:07.781409 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.785321  1780 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.754567  2006 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.786677 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.788450 24435 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:07.789134 24423 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260809 23:21:07.789636  1923 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:21:07.788257 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47060us
I20260809 23:21:07.790640 24427 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260809 23:21:07.800366  2074 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.509s	user 0.003s	sys 0.005s
I20260809 23:21:07.805751 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:07.726428  1746 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260809 23:21:07.816327 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.817238   877 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:21:07.855779 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.818166  1667 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.719702 24163 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:07.824085 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.829564 25371 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:07.786440 24430 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260809 23:21:07.880728 24430 pending_rounds.cc:76] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:21:07.842234 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:07.845678 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.015s	sys 0.097s
W20260809 23:21:07.882129 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.015s	sys 0.097s
I20260809 23:21:07.616891  1736 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.844838 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.712170  1556 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260809 23:21:07.816195  1429 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:07.886272 24429 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:07.886709 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:07.887333 24430 pending_rounds.cc:97] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260809 23:21:07.756474  1205 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.887661 24430 raft_consensus.cc:2866] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:07.772485  1977 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:07.685040  1677 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.873718 24435 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:07.873292  1737 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260809 23:21:07.850910  1923 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:07.848551  1794 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.060s	sys 0.089s
I20260809 23:21:07.756127  1539 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
W20260809 23:21:07.881558 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 65336us
I20260809 23:21:07.850869  1856 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:07.890440 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 103878us
I20260809 23:21:07.729844  1485 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.120s
I20260809 23:21:07.893666 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.876130  1823 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:21:07.890420  1429 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:07.898622  1794 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.061s	sys 0.089s
I20260809 23:21:07.899108 24428 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:07.902439 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52107.6us
I20260809 23:21:07.904896 24427 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:07.905768 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.897200 25368 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:07.887741  1746 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:07.915318 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.883946  2027 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.892041  1403 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:07.919164  1856 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.920058 24534 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:07.920524  1675 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:07.921049 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.921518  1856 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:07.921681 24540 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:07.924650  1736 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:07.911669  2056 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:07.925774 24314 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:07.925663  2056 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.873555  1780 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.924721  1498 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:07.917383  1352 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:07.927958  2056 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:07.928509  2056 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:07.929119  2056 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.931689  1917 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:07.899950  1923 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:07.853216  2006 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:07.936385 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.933883 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.934088 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:07.927340  1498 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:07.892808  1977 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:07.928373  1780 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:07.947206 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 209960us
I20260809 23:21:07.948757 24542 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:07.948961 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:07.958444 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:07.926589  1691 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260809 23:21:07.967929 24540 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:07.942564  1485 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:07.890321  1667 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.995312 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.936012  1917 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:07.995965 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:07.996042 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:07.941479  1788 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:08.003997 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:08.003901 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.002368 24428 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:07.951745  1675 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260809 23:21:07.953588  1539 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:08.004631 24430 raft_consensus.cc:2362] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19. Re-sending same reply.
I20260809 23:21:07.960639  1403 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:07.950079  2056 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:08.011435 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48832.5us
I20260809 23:21:08.017558 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.017607 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.012221 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:08.020498 24431 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:21:08.016848 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:08.021587  1485 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:08.021730  1556 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:07.988935  2030 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:08.022775 24431 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:08.023499 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89724.9us
I20260809 23:21:08.019516 24429 raft_consensus.cc:2362] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20. Re-sending same reply.
I20260809 23:21:07.974251  1746 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:07.947048  1736 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.067s
I20260809 23:21:08.000224  1869 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:08.004546  1352 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:08.042687 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:08.043351 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.044049  2032 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:08.044462  2032 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:08.044812  2032 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:08.047835  2040 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.568s	user 0.012s	sys 0.000s
I20260809 23:21:08.048317  2040 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.051364 25366 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:08.051433  1917 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.052901  1917 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:08.013541  1539 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:08.032891 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.004118  1788 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:08.014183  1403 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:08.061954 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:08.062961 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.040395  1746 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:07.991950  1780 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260809 23:21:08.057219 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:08.031996 24431 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.060964  1498 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.064754  1869 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.065316  2017 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:08.066179  1205 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.067590  1770 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:08.067601  2040 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.068773  1569 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:08.069667  1667 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.070750 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.095669 24534 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260809 23:21:08.061331  1691 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:08.071264  1403 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:08.093798  1528 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.008s	sys 0.011s
I20260809 23:21:08.071780 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.074916 24314 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260809 23:21:08.029148  2030 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:08.075928 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58439.8us
I20260809 23:21:08.062357 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.090098 24431 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:08.061563  1788 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:08.062203  1539 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:08.091944 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.003s	sys 0.010s
I20260809 23:21:08.077937  1556 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:08.136085  1527 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:08.135958  1556 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.138387  1556 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.028015  1794 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:21:08.053786  1966 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
W20260809 23:21:08.092142  1680 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.830s	user 0.011s	sys 0.003s
W20260809 23:21:08.140409 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 118112us
I20260809 23:21:08.140260  1680 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.141199 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.142421  1680 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.144479  2006 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.145386  2006 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:08.146014  2006 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:08.135011  2030 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:08.147724  1869 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:08.148283  1389 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.004s
I20260809 23:21:08.148730  1389 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.151234  1403 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.153316  1403 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:08.154635 24321 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 13. Current term is 14. Ops: []
I20260809 23:21:08.157306 25368 raft_consensus.cc:1203] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Deduplicated request from leader. Original: 16.12->[18.13-18.13]   Dedup: 18.13->[]
I20260809 23:21:08.157766 25368 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260809 23:21:08.116659 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1601) took 1036 ms. Trace:
I20260809 23:21:08.080657  1429 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:08.166255 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.003s	sys 0.010s
I20260809 23:21:08.109525  1498 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:08.168840 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:08.171804  1966 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:08.171739 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:08.172472  1966 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:08.173272  1389 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.173575  1966 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.174605  1923 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:08.175932 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106379us
I20260809 23:21:08.172663 25371 raft_consensus.cc:2354] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:08.177860  1923 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.179029  1780 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:08.180137  1780 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.182396  1780 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.184967  1788 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.187628  1780 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:08.187974  1736 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:08.188823  1736 consensus_queue.cc:1225] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:08.189184  1485 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:08.189586  1485 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:08.190227 24378 leader_election.cc:398] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
I20260809 23:21:08.190832 24378 leader_election.cc:401] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260809 23:21:08.190523  1485 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.192118  1485 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:08.192229  1736 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:08.170311 25373 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:08.179267 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.199605  1691 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:08.179560  1429 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:08.203104  1731 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.004s	sys 0.011s
W20260809 23:21:08.203639  1731 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.004s	sys 0.011s
I20260809 23:21:08.203294 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:08.204819 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:08.205394 24424 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:08.213423 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:08.131893  1528 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.008s	sys 0.011s
I20260809 23:21:08.219272 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:08.116478  1886 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:08.232003 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.229519 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3170) took 1996 ms. Trace:
I20260809 23:21:08.162493 24313 rpcz_store.cc:276] 0809 23:21:07.077652 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.108816 (+ 31164us) service_pool.cc:227] Handling call
0809 23:21:07.125013 (+ 16197us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:07.183145 (+ 58132us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:07.183191 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:07.220115 (+ 36924us) log.cc:842] Serialized 26 byte log entry
0809 23:21:07.886707 (+666592us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:07.916144 (+ 29437us) log.cc:842] Serialized 14 byte log entry
0809 23:21:07.916341 (+   197us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:07.916443 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:08.109484 (+193041us) raft_consensus.cc:1671] finished
0809 23:21:08.109526 (+    42us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.114294 (+  4768us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32503,"prepare.queue_time_us":26695,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":57981,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":10221,"thread_start_us":661936,"threads_started":1,"wal-append.queue_time_us":844752}
I20260809 23:21:08.165257  1680 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:08.233768 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:08.233726 24541 rpcz_store.cc:276] 0809 23:21:06.233309 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:06.233472 (+   163us) service_pool.cc:227] Handling call
0809 23:21:06.235204 (+  1732us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:06.990418 (+755214us) spinlock_profiling.cc:243] Waited 1.42 ms on lock 0x7b54000ce2a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f6951 00007fac23857321 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612
0809 23:21:07.336698 (+346280us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:07.354008 (+ 17310us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.167117 (+813109us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.167392 (+   275us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.167442 (+    50us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:08.167512 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.218709 (+ 51197us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.229452 (+ 10743us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6653,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":87,"raft.queue_time_us":24569,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":13752,"spinlock_wait_cycles":110537856,"thread_start_us":812382,"threads_started":1}
I20260809 23:21:08.253163 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.278594 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:08.233310  1964 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:08.277073 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:08.300762 24165 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 342) took 57453 ms (client timeout 15000 ms). Trace:
I20260809 23:21:08.303447  1788 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:08.303750  1569 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:08.304539  1569 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:08.301265 24165 rpcz_store.cc:269] 0809 23:20:10.847501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:10.847700 (+   199us) service_pool.cc:227] Handling call
0809 23:21:08.300466 (+57452766us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:20:50.597077 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:20:50.754242 (+157165us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:20:50.781470 (+ 27228us) write_op.cc:435] Acquired schema lock
0809 23:20:50.781489 (+    19us) tablet.cc:660] Decoding operations
0809 23:20:51.713794 (+932305us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:20:51.713913 (+   119us) write_op.cc:641] Partition lock acquired for write op
0809 23:20:51.713956 (+    43us) tablet.cc:683] Acquiring locks for 17 operations
0809 23:20:51.715326 (+  1370us) tablet.cc:699] Row locks acquired
0809 23:20:51.715364 (+    38us) write_op.cc:260] PREPARE: finished
0809 23:20:51.715563 (+   199us) write_op.cc:270] Start()
0809 23:20:51.715713 (+   150us) write_op.cc:276] Timestamp: P: 1786317651715480 usec, L: 0
0809 23:20:51.715750 (+    37us) op_driver.cc:331] REPLICATION: starting
0809 23:20:51.915863 (+200113us) log.cc:842] Serialized 6665 byte log entry
0809 23:20:53.197904 (+1282041us) op_driver.cc:444] REPLICATION: finished
0809 23:20:54.942477 (+1744573us) write_op.cc:301] APPLY: starting
0809 23:20:54.961772 (+ 19295us) tablet.cc:1365] starting BulkCheckPresence
0809 23:20:55.621558 (+659786us) tablet.cc:1368] finished BulkCheckPresence
0809 23:20:55.846070 (+224512us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:20:56.145872 (+299802us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:20:56.146340 (+   468us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0809 23:20:56.146411 (+    71us) write_op.cc:312] APPLY: finished
0809 23:20:56.157831 (+ 11420us) log.cc:842] Serialized 156 byte log entry
0809 23:20:56.997457 (+839626us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:57.144537 (+147080us) write_op.cc:454] Released schema lock
0809 23:20:57.217227 (+ 72690us) write_op.cc:341] FINISH: Updating metrics
0809 23:20:57.346569 (+129342us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:20:57.349128 (+  2559us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2034996,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":1715233,"apply.run_cpu_time_us":91782,"apply.run_wall_time_us":2609126,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75119,"num_ops":17,"prepare.queue_time_us":185125,"prepare.run_cpu_time_us":56094,"prepare.run_wall_time_us":2175234,"raft.queue_time_us":261330,"raft.run_cpu_time_us":10363,"raft.run_wall_time_us":141179,"replication_time_us":1475039,"thread_start_us":1996228,"threads_started":5,"wal-append.queue_time_us":1633246,"wal-append.run_cpu_time_us":17292,"wal-append.run_wall_time_us":2409621}]]}
I20260809 23:21:08.313062 24163 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 10 to 14, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 14 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:08.315207  1352 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:08.317044 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:08.319746 24431 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260809 23:21:08.270588  1794 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:08.249461  1680 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:08.227514  1691 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260809 23:21:08.289166  1528 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:08.331329  1352 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:08.337997 24313 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:08.293226  1964 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:08.351950  1964 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:08.363812  1736 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19.
W20260809 23:21:08.331890   877 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
W20260809 23:21:08.326541 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73514.9us
I20260809 23:21:08.335290  1680 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:08.399708 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:08.350421  1528 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.389580  1569 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:08.402542 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.008s	sys 0.043s
I20260809 23:21:08.394402  1667 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:08.406497  1915 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.007s	sys 0.022s
W20260809 23:21:08.420553 24311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.021s	sys 0.016s
W20260809 23:21:08.421013 24311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.021s	sys 0.018s
W20260809 23:21:08.405674 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s	user 0.009s	sys 0.042s
W20260809 23:21:08.423859 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.009s	sys 0.042s
W20260809 23:21:08.426712 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.008s	sys 0.043s
W20260809 23:21:08.443835 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44332.2us
I20260809 23:21:08.408723  1977 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:08.412386  1915 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.008s	sys 0.023s
W20260809 23:21:08.457286 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.871s	user 0.010s	sys 0.042s
I20260809 23:21:08.457625 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:08.418892  2033 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.019s	sys 0.018s
W20260809 23:21:08.460557 25375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.015s	sys 0.008s
W20260809 23:21:08.461084 25375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.016s	sys 0.008s
I20260809 23:21:08.461568  1977 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:08.468544  1774 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.252s	user 0.013s	sys 0.022s
W20260809 23:21:08.469175  1774 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.373s	user 0.013s	sys 0.023s
I20260809 23:21:08.405061  1667 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:08.484594 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.007s	sys 0.020s
W20260809 23:21:08.485136 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.007s	sys 0.021s
W20260809 23:21:08.457959 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.010s	sys 0.042s
W20260809 23:21:08.480060  2033 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.020s	sys 0.018s
I20260809 23:21:08.486824  1485 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260809 23:21:08.503607  1647 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:08.503319  1539 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.504035  1527 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:08.483839  1886 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:08.444765  2030 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.510506  1647 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:08.503281 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3297) took 1078 ms. Trace:
I20260809 23:21:08.514655 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3258) took 1035 ms. Trace:
I20260809 23:21:08.501989  1556 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:08.537451 24422 rpcz_store.cc:276] 0809 23:21:07.407674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.781218 (+373544us) service_pool.cc:227] Handling call
0809 23:21:07.785366 (+  4148us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:07.785639 (+   273us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:07.786010 (+   371us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.485691 (+699681us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.485765 (+    74us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:08.485836 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.485937 (+   101us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.486079 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2737,"prepare.queue_time_us":9232,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":117,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":17269,"thread_start_us":699207,"threads_started":1}
I20260809 23:21:08.563601 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:08.566048 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 121211us
I20260809 23:21:08.549072  1647 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:08.406265 24163 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 8 to 13, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 13 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:21:08.597944 25321 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.013s	sys 0.024s
I20260809 23:21:08.512825  1731 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:08.514066  1527 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:08.515743  1886 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:08.562073  1539 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.507325  2032 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:08.609050 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.040s	sys 0.033s
W20260809 23:21:08.654943 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.041s	sys 0.034s
I20260809 23:21:08.643193  1886 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:08.661405 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.026s	sys 0.027s
W20260809 23:21:08.661931 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.026s	sys 0.027s
W20260809 23:21:08.663702 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 777388us
I20260809 23:21:08.664557 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:08.667857 25373 raft_consensus.cc:1203] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Deduplicated request from leader. Original: 16.12->[18.13-18.13]   Dedup: 18.13->[]
I20260809 23:21:08.668542 25373 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [18.13-18.13]
W20260809 23:21:08.600860 25321 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.013s	sys 0.025s
I20260809 23:21:08.670574 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:08.606644  1731 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:08.739981 25365 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
W20260809 23:21:08.740875 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.012s	sys 0.027s
W20260809 23:21:08.741351 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.012s	sys 0.027s
I20260809 23:21:08.753686 24313 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:08.760720 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3264) took 1152 ms. Trace:
I20260809 23:21:08.659981  1528 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:08.786341 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 219030us
I20260809 23:21:08.674984  1352 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:08.673614 24312 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
W20260809 23:21:08.764963 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.006s	sys 0.017s
W20260809 23:21:08.793941 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.006s	sys 0.018s
I20260809 23:21:08.792403 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:08.668995 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 914784us
I20260809 23:21:08.763636  1539 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:08.844554  1431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.001s	sys 0.033s
W20260809 23:21:08.845078  1431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.029s	user 0.001s	sys 0.033s
I20260809 23:21:08.845587  1906 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.256s
I20260809 23:21:08.795027  1352 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:08.847651  1352 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:08.850973  1569 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260809 23:21:08.791798  2006 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:08.853955  2083 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260809 23:21:08.855377 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46627.5us
W20260809 23:21:08.864068 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70996.5us
I20260809 23:21:08.858297 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.785180   877 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260809 23:21:08.875252 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.876822 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.540300 24323 rpcz_store.cc:276] 0809 23:21:07.479293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.479474 (+   181us) service_pool.cc:227] Handling call
0809 23:21:07.479767 (+   293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:07.479966 (+   199us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:07.522441 (+ 42475us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.508582 (+986141us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.514248 (+  5666us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.514301 (+    53us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:08.514363 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.514430 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.514551 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15378,"prepare.queue_time_us":46216,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":8072,"raft.queue_time_us":15912,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":20504,"spinlock_wait_cycles":3376384,"thread_start_us":911745,"threads_started":1}
I20260809 23:21:08.878062 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.879284   877 consensus_queue.cc:1225] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 19.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260809 23:21:08.886440 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.006s	sys 0.022s
W20260809 23:21:08.886902 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.006s	sys 0.022s
I20260809 23:21:08.761320 24309 rpcz_store.cc:276] 0809 23:21:07.608329 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.690224 (+ 81895us) service_pool.cc:227] Handling call
0809 23:21:07.726952 (+ 36728us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:07.738079 (+ 11127us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:07.845281 (+107202us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.760023 (+914742us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.760142 (+   119us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:08.760259 (+   117us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.760398 (+   139us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.760603 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":88736,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":18210,"thread_start_us":838020,"threads_started":1}
W20260809 23:21:08.889333 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.006s	sys 0.022s
W20260809 23:21:08.889890 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.006s	sys 0.022s
W20260809 23:21:08.863282  1776 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.031s	sys 0.034s
W20260809 23:21:08.893664 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.020s	sys 0.028s
W20260809 23:21:08.916921 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.020s	sys 0.029s
W20260809 23:21:08.920114 25321 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.658s (SENT)
I20260809 23:21:08.921520 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.922183 24425 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:08.922598 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:08.923676 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.852735  2006 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.924510  1647 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.925830 24424 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260809 23:21:08.925365 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3208) took 1006 ms. Trace:
W20260809 23:21:08.892208  2052 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.604s	user 0.015s	sys 0.048s
W20260809 23:21:08.926963  2084 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260809 23:21:08.927172 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.931274 24424 pending_rounds.cc:76] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:21:08.951162 24424 pending_rounds.cc:97] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260809 23:21:08.951519 24424 raft_consensus.cc:2866] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:08.952104 24424 pending_rounds.cc:97] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260809 23:21:08.951668 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:08.952471 24424 raft_consensus.cc:2866] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:08.933423  1647 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.888432   877 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260809 23:21:08.889963  1569 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:08.936296  2006 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.935956 24163 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 14 to 16, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:08.936987 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1603) took 1547 ms. Trace:
I20260809 23:21:08.922122 25365 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260809 23:21:08.787199  1527 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:08.941666 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.888561  1794 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:08.966557 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.009s	sys 0.010s
W20260809 23:21:08.942564 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.012s	sys 0.032s
I20260809 23:21:08.937381 24543 rpcz_store.cc:276] 0809 23:21:07.919338 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.931254 (+ 11916us) service_pool.cc:227] Handling call
0809 23:21:07.931578 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:07.931782 (+   204us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:07.941234 (+  9452us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.917716 (+976482us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.917787 (+    71us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:08.917941 (+   154us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.918054 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.925167 (+  7113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13597,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"thread_start_us":909758,"threads_started":1}
I20260809 23:21:08.952832 24422 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
W20260809 23:21:08.969007 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.012s	sys 0.032s
I20260809 23:21:08.955991 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:08.969458 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 912404us
I20260809 23:21:08.954834  1667 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:08.958109 24423 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:08.963997  1917 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.911s	user 0.013s	sys 0.003s
W20260809 23:21:08.959594 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.000s	sys 0.057s
W20260809 23:21:08.977187 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.001s	sys 0.057s
W20260809 23:21:08.961269 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 77192.3us
W20260809 23:21:08.942586  2052 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.654s	user 0.015s	sys 0.049s
I20260809 23:21:08.924299 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:08.963117  2032 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:08.884797  1539 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.926203  1556 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:08.995309 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:08.962556 24316 rpcz_store.cc:276] 0809 23:21:07.389602 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.397330 (+  7728us) service_pool.cc:227] Handling call
0809 23:21:07.400994 (+  3664us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:07.523761 (+122767us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:07.523806 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:07.559919 (+ 36113us) log.cc:842] Serialized 26 byte log entry
0809 23:21:08.427374 (+867455us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:08.427428 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.538354 (+110926us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:08.538431 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:08.936690 (+398259us) raft_consensus.cc:1671] finished
0809 23:21:08.936712 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.936925 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":540,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9407,"spinlock_wait_cycles":281379200,"thread_start_us":826787,"threads_started":1,"wal-append.queue_time_us":1363691}
I20260809 23:21:08.996907 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1493) took 1634 ms. Trace:
I20260809 23:21:08.887465  1431 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.063s
W20260809 23:21:08.893771  1776 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.052s	user 0.032s	sys 0.035s
I20260809 23:21:08.997264 24545 rpcz_store.cc:276] 0809 23:21:07.362050 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.443930 (+ 81880us) service_pool.cc:227] Handling call
0809 23:21:07.469898 (+ 25968us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:07.481775 (+ 11877us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:07.481812 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:07.536068 (+ 54256us) log.cc:842] Serialized 26 byte log entry
0809 23:21:08.424345 (+888277us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:21:08.424401 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.424495 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:08.996694 (+572199us) raft_consensus.cc:1671] finished
0809 23:21:08.996716 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:08.996850 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":833548,"threads_started":1,"wal-append.queue_time_us":1378992}
W20260809 23:21:08.999344 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87020.8us
W20260809 23:21:09.000339 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.781s (SENT)
I20260809 23:21:09.000819 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.002780  1780 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:09.003196  1780 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:09.004024  1780 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.005218  1780 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:08.979974  1429 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:09.006604  1429 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:08.995325  1556 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:09.007326 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.008306 24545 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
I20260809 23:21:08.866421  1731 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.011785 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2875) took 1016 ms. Trace:
I20260809 23:21:09.012176 24427 rpcz_store.cc:276] 0809 23:21:07.995270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.995803 (+   533us) service_pool.cc:227] Handling call
0809 23:21:08.011947 (+ 16144us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:08.982866 (+970919us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.982903 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:08.982961 (+    58us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:08.983001 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.011543 (+ 28542us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:09.011589 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.011723 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2830,"raft.queue_time_us":8692,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":2225534336}
W20260809 23:21:08.957479 25366 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.018s	sys 0.027s
W20260809 23:21:08.970516 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.010s	sys 0.010s
W20260809 23:21:08.963510  2017 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.006s	sys 0.013s
W20260809 23:21:08.959004 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.005s	sys 0.014s
W20260809 23:21:08.971571 24535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.886s	user 0.015s	sys 0.030s
I20260809 23:21:08.960181  2030 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:08.978502 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:08.953840  1677 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.032s	user 0.024s	sys 0.036s
W20260809 23:21:09.015789  2017 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.007s	sys 0.013s
W20260809 23:21:09.020674  1677 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s	user 0.024s	sys 0.037s
I20260809 23:21:09.021315  2030 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:09.021907  2030 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:08.983072 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:08.987108 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:08.966392  2006 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:08.991619  1964 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.639s	user 0.010s	sys 0.002s
I20260809 23:21:08.993235  1886 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.027873  2017 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:09.027959  2006 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.029758  1886 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:09.014351 25366 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.019s	sys 0.027s
W20260809 23:21:09.015329 24535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.015s	sys 0.031s
I20260809 23:21:09.030499  2006 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:09.014976 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.005s	sys 0.014s
I20260809 23:21:09.014510 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:08.982846 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:09.023851 24544 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:09.031422 24545 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:09.034082 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.034674 24313 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:09.035626 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:09.036392  2030 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.036978  2030 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:09.037449 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55107.8us
I20260809 23:21:09.037529  2030 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.024514 24543 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:08.985918 24163 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:21:09.022267  1677 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:09.038363 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:09.041911  1539 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.042346 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:09.043557 24163 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 13 to 16, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:09.045343  2017 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:09.045820  2017 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.046193  2017 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:09.037012 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54258.6us
I20260809 23:21:09.037006 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:09.025966  1680 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.628s	user 0.010s	sys 0.001s
I20260809 23:21:09.047237 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.045508 24163 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:09.050921 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:08.983716 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.012s	sys 0.012s
W20260809 23:21:09.053297 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.012s	sys 0.012s
I20260809 23:21:09.027179  1429 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260809 23:21:09.046415  1839 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.020s	sys 0.045s
I20260809 23:21:09.053922 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:09.052188  1539 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:09.055931 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.040938  1964 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.041422  2006 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.059365 24427 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260809 23:21:09.063452 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:09.063889 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:09.055032  1746 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.009s	sys 0.017s
I20260809 23:21:09.049211  2052 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:09.060451 24430 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260809 23:21:09.058732 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.010s	sys 0.017s
W20260809 23:21:09.058018  1839 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.020s	sys 0.046s
I20260809 23:21:09.057551 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:09.068384 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:09.067843 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.011s	sys 0.017s
I20260809 23:21:09.059332 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.062086 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:09.078080 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 936862us
I20260809 23:21:09.056535  1429 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:09.060196  2106 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:09.060729  1915 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:09.079865 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.085820 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3277) took 1820 ms. Trace:
W20260809 23:21:09.067452  1746 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.009s	sys 0.017s
I20260809 23:21:09.080945 24323 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:09.091038 24544 rpcz_store.cc:276] 0809 23:21:07.265106 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.284472 (+ 19366us) service_pool.cc:227] Handling call
0809 23:21:07.313660 (+ 29188us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:09.031465 (+1717805us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.085542 (+ 54077us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:09.085755 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3932488064}
I20260809 23:21:09.094484 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.100380 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.103371 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.103766 24425 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:09.103963 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.104194 24425 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:09.106709 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:09.083041  2056 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.974s	user 0.012s	sys 0.012s
I20260809 23:21:09.107858 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.052189  1680 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.110411 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:09.107504 24309 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:09.114485 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:09.065302  1352 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.124593 24542 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:09.125267  1352 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:09.127043 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.127800 24427 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:09.127910  1964 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.128342 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3202) took 1419 ms. Trace:
I20260809 23:21:09.087342  1429 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:09.130267 24496 leader_election.cc:398] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:09.056690  1886 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
W20260809 23:21:09.084969  1966 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.006s	sys 0.019s
I20260809 23:21:09.136363 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:09.137456 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.986s	user 0.012s	sys 0.041s
I20260809 23:21:09.141144 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:09.142695 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.991s	user 0.012s	sys 0.041s
I20260809 23:21:09.145314 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:09.145920 24314 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:09.140291 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.087754  1539 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:09.149816 24541 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260809 23:21:09.151217 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:09.153784 24543 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:09.112125  2056 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.012s	sys 0.012s
I20260809 23:21:09.148718 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.157577 24423 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:09.105387  2052 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:09.128757 24546 rpcz_store.cc:276] 0809 23:21:07.708501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.708687 (+   186us) service_pool.cc:227] Handling call
0809 23:21:07.737314 (+ 28627us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:07.768518 (+ 31204us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:07.768561 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:07.847577 (+ 79016us) log.cc:842] Serialized 52 byte log entry
0809 23:21:08.662291 (+814714us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:21:08.662330 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.662400 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.125911 (+463511us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b24003c0158. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:21:09.128000 (+  2089us) raft_consensus.cc:1671] finished
0809 23:21:09.128023 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.128272 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6156,"raft.queue_time_us":37834,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":5992,"spinlock_wait_cycles":65911168,"thread_start_us":758791,"threads_started":1,"wal-append.queue_time_us":1094729}
I20260809 23:21:09.069554  2006 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.155092 24425 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.159427 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:09.148083  1966 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.006s	sys 0.019s
I20260809 23:21:09.150175  1770 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
W20260809 23:21:09.160346 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46622.9us
I20260809 23:21:09.175230 24543 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:09.142102  1776 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:09.169905 24425 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:09.180461 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:09.170077 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.002s	sys 0.026s
I20260809 23:21:09.156780 24309 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:09.173398 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.148862  2030 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:09.182411 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.002s	sys 0.026s
I20260809 23:21:09.183765 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.184072 24544 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260809 23:21:09.161468  1677 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:09.186383 24422 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:09.190110 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.191721  1677 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:09.192229  2030 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.192699  1677 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.192785  2030 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.205883 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:09.205953 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.207466 24541 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:09.198837  1731 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:09.209000 24541 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:09.210577 24541 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
W20260809 23:21:09.194897  2126 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260809 23:21:09.214602 24545 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:09.215533 24544 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:09.169922  2056 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:09.219048 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.023s	sys 0.025s
I20260809 23:21:09.174361  1770 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:09.219267 24423 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:09.226285 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.024s	sys 0.025s
I20260809 23:21:09.228565 24546 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:09.229003 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:09.231808 24543 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260809 23:21:09.196570  2030 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.235562 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3215) took 1074 ms. Trace:
I20260809 23:21:09.103288 24163 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:21:09.224134 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.740s (SENT) [suppressed 28 similar messages]
I20260809 23:21:09.238116 24539 rpcz_store.cc:276] 0809 23:21:08.160719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.160853 (+   134us) service_pool.cc:227] Handling call
0809 23:21:08.172436 (+ 11583us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:08.173152 (+   716us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.173202 (+    50us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:08.173928 (+   726us) log.cc:842] Serialized 52 byte log entry
0809 23:21:08.795546 (+621618us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:08.795642 (+    96us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.795772 (+   130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.194105 (+398333us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:09.218156 (+ 24051us) raft_consensus.cc:1671] finished
0809 23:21:09.218195 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.235473 (+ 17278us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3548,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":61,"spinlock_wait_cycles":10172800,"thread_start_us":620129,"threads_started":1,"wal-append.queue_time_us":889445}
I20260809 23:21:09.218513  1527 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:09.249469 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3267) took 1640 ms. Trace:
I20260809 23:21:09.249872 24317 rpcz_store.cc:276] 0809 23:21:07.609176 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.764266 (+155090us) service_pool.cc:227] Handling call
0809 23:21:07.843295 (+ 79029us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:07.897779 (+ 54484us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:07.897832 (+    53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:07.911854 (+ 14022us) log.cc:842] Serialized 26 byte log entry
0809 23:21:08.887440 (+975586us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:08.887496 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.887675 (+   179us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.249233 (+361558us) raft_consensus.cc:1671] finished
0809 23:21:09.249265 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.249405 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15452,"raft.queue_time_us":9787,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7248,"thread_start_us":894960,"threads_started":1,"wal-append.queue_time_us":1214782}
I20260809 23:21:09.251835  2106 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:09.252274  2052 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:09.254226  2104 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260809 23:21:09.228237  1886 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.033s
W20260809 23:21:09.263067 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62799.9us
I20260809 23:21:09.263909 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.265815 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.274304 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:09.278092 24423 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:09.276312  1736 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.279722  1736 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.280092  2030 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:09.280503  2030 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.280835  2030 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:09.281993  1736 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:09.282823 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96791.3us
I20260809 23:21:09.283560 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:09.204519  1556 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.245414  1680 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.267247  1794 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:21:09.243116  2056 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260809 23:21:09.329720 24423 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
W20260809 23:21:09.334915 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.041s	user 0.014s	sys 0.017s
I20260809 23:21:09.320192  1966 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:09.294090  1776 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260809 23:21:09.299770 24314 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:09.300028  1774 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.006s	sys 0.031s
W20260809 23:21:09.349313 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120372us
W20260809 23:21:09.349254  1774 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.007s	sys 0.033s
I20260809 23:21:09.357867  1556 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.358671  1776 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:09.362664  1915 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:09.363124 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.364046  1915 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:09.364970  1915 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:09.365311  1915 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:09.367188 24431 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:09.367569 24431 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:09.368387 24431 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:09.369962 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.080s	user 0.014s	sys 0.017s
I20260809 23:21:09.313097  1431 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:09.330322  2106 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.366139  1915 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.389127 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.389914  1915 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:09.368479  1556 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:09.355261  2006 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.396425 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.407336 24545 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:09.297601  1770 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.405959 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.413038 24541 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:09.410336 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.415763 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:09.416735 24546 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:09.419268 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67654.9us
I20260809 23:21:09.420424 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:09.414335 24543 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:09.384792  2115 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:09.434049 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.446722 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:09.442279 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.013s	sys 0.016s
I20260809 23:21:09.448489 24314 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:21:09.454533 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.013s	sys 0.016s
W20260809 23:21:09.454905 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 541595us
I20260809 23:21:09.387085  2118 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:09.458402 24545 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
I20260809 23:21:09.459038 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.460948 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:09.462446  1977 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.415261  1431 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.466770 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3272) took 1310 ms. Trace:
I20260809 23:21:09.468474 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:09.392445  1556 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:09.471877 24313 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:09.483047 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.445966  1794 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:09.495648 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:09.496964  2123 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:21:09.497900 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.473683  2056 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:09.352270 24163 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:21:09.500761  1429 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.473914  2118 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.494563  1569 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:09.495086 24316 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:09.508500 24321 rpcz_store.cc:276] 0809 23:21:08.156795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.156949 (+   154us) service_pool.cc:227] Handling call
0809 23:21:08.157266 (+   317us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:08.157609 (+   343us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.157636 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:08.158223 (+   587us) log.cc:842] Serialized 52 byte log entry
0809 23:21:08.368020 (+209797us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b180027df00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:09.229050 (+861030us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.245656 (+ 16606us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.319317 (+ 73661us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b180027df00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:09.320017 (+   700us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.320953 (+   936us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.466511 (+145558us) raft_consensus.cc:1671] finished
0809 23:21:09.466534 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.466714 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14822,"prepare.queue_time_us":16353,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":75304,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":48985,"spinlock_wait_cycles":69145216,"thread_start_us":860743,"threads_started":1,"wal-append.queue_time_us":1304739}
I20260809 23:21:09.517606 24431 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:09.535197 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:09.505848 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.004s	sys 0.015s
I20260809 23:21:09.519556 25368 raft_consensus.cc:1203] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Deduplicated request from leader. Original: 16.12->[18.13-18.13]   Dedup: 18.13->[]
I20260809 23:21:09.391052  2106 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.520123 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:09.470542   877 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.033s	sys 0.055s
I20260809 23:21:09.555958 25371 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:09.527649 24541 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:09.519404 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51061.1us
I20260809 23:21:09.537149 25368 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260809 23:21:09.519769 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:09.560320 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.009s	sys 0.016s
I20260809 23:21:09.536043 25373 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
W20260809 23:21:09.562140 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 296361us
W20260809 23:21:09.537634 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.004s	sys 0.016s
I20260809 23:21:09.559223 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:09.515581  1539 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:09.575114 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.010s	sys 0.016s
W20260809 23:21:09.575618 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 548299us
I20260809 23:21:09.576858 24314 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:09.572893 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:09.567576 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.008s	sys 0.032s
W20260809 23:21:09.589875 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.008s	sys 0.032s
W20260809 23:21:09.590252 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 539598us
I20260809 23:21:09.601166 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.558454  2134 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:09.519151  2006 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.534435  2127 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:09.555895  1794 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:09.556152  1774 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:09.558271  2056 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:09.571674  1839 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:09.626652 24431 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260809 23:21:09.513728  1977 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.602716  1667 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:09.669185 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.664407 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:09.523018  1431 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.559648  1569 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.606376  2144 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:09.667121 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:09.611850  2134 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:09.687662  2134 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:09.675143 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3217) took 1513 ms. Trace:
I20260809 23:21:09.646540  2168 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260809 23:21:09.522205 24312 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:09.686439 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:09.659042  2121 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260809 23:21:09.601014  1539 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.709882 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:09.675894  1667 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:09.692123 24534 rpcz_store.cc:276] 0809 23:21:08.161661 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.193631 (+ 31970us) service_pool.cc:227] Handling call
0809 23:21:08.194071 (+   440us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:08.213499 (+ 19428us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.213565 (+    66us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:21:08.278183 (+ 64618us) log.cc:842] Serialized 78 byte log entry
0809 23:21:09.069240 (+791057us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:09.069296 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.069396 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.674600 (+605204us) raft_consensus.cc:1671] finished
0809 23:21:09.674622 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.675039 (+   417us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":18846,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":8291,"thread_start_us":769881,"threads_started":1,"wal-append.queue_time_us":1100502}
I20260809 23:21:09.676939   877 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.611577  1352 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
W20260809 23:21:09.711796 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 487658us
I20260809 23:21:09.602264  2115 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.105s
I20260809 23:21:09.737814 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:21:09.747836 24269 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.074s (SENT)
I20260809 23:21:09.755647 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:09.756683 25365 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:09.663166  2006 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.665508  1774 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.736500  1667 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:09.734651  1736 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:09.655465  1556 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:09.773454 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 166913us
I20260809 23:21:09.791615 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2748) took 1733 ms. Trace:
I20260809 23:21:09.791356 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:09.792039 24320 rpcz_store.cc:276] 0809 23:21:08.058149 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.074268 (+ 16119us) service_pool.cc:227] Handling call
0809 23:21:08.091602 (+ 17334us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:08.143106 (+ 51504us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.143137 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:08.143738 (+   601us) log.cc:842] Serialized 52 byte log entry
0809 23:21:08.977663 (+833925us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:08.988359 (+ 10696us) log.cc:842] Serialized 14 byte log entry
0809 23:21:08.988605 (+   246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:08.988730 (+   125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.791305 (+802575us) raft_consensus.cc:1671] finished
0809 23:21:09.791334 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.791538 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":422,"prepare.queue_time_us":10470,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":36,"raft.queue_time_us":29679,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":196,"thread_start_us":833521,"threads_started":1,"wal-append.queue_time_us":1178123}
I20260809 23:21:09.793306 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:21:09.803668 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 803341us
W20260809 23:21:09.804347 25321 connection.cc:232] RPC call timeout handler was delayed by -1.4229457280598581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260809 23:21:09.793246  1731 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.585s	user 0.007s	sys 0.004s
I20260809 23:21:09.735993  1677 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.060s
I20260809 23:21:09.817306 24312 raft_consensus.cc:595] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:09.769416  2146 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:09.765637  2116 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:09.752903   877 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.831076 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:09.838985 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 228268us
I20260809 23:21:09.845752 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.854001 24320 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:09.852099 25373 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:09.798802  1780 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260809 23:21:09.859706 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:09.844319  2133 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.327s	user 0.000s	sys 0.001s
I20260809 23:21:09.830150  1680 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:09.831180  2121 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:09.834024  1527 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:09.868640 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 199324us
I20260809 23:21:09.833601  1429 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:09.755412  1794 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.831323  2146 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:09.844028  2118 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:09.856660  2115 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:09.921237 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.772625  2056 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:09.866416  2124 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.300s	user 0.000s	sys 0.001s
W20260809 23:21:09.899055  2030 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.002s
I20260809 23:21:09.913000 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:09.908340 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:09.860323  1731 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:09.927706 25373 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 13.
I20260809 23:21:09.911855  2146 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:09.872778  2106 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.943368 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:09.935099  1569 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:09.930609  2118 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:09.898238  1527 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:09.883096  2121 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.977461 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:09.964807 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3212) took 1919 ms. Trace:
I20260809 23:21:09.979918 24311 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:09.982498 24540 rpcz_store.cc:276] 0809 23:21:08.045677 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.045833 (+   156us) service_pool.cc:227] Handling call
0809 23:21:08.046142 (+   309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:08.046369 (+   227us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:08.046399 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:08.115597 (+ 69198us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:08.151344 (+ 35747us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.143147 (+991803us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.143186 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.143246 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:09.964354 (+821108us) raft_consensus.cc:1671] finished
0809 23:21:09.964377 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:09.964711 (+   334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17850112,"thread_start_us":991200,"threads_started":1,"wal-append.queue_time_us":1317290}
W20260809 23:21:09.951311 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 125740us
W20260809 23:21:09.955960  2156 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260809 23:21:09.967489  1917 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:09.972151  1977 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:09.963527  2159 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
I20260809 23:21:09.989866  2121 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:09.992316 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:10.050230 25373 raft_consensus.cc:2354] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
I20260809 23:21:09.997918  2168 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:09.999735 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 169062us
I20260809 23:21:10.000732 25371 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
W20260809 23:21:09.965111  2155 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:21:10.060726 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:09.968793  2164 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260809 23:21:09.931252  2115 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:09.943902  2188 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:10.006537  1915 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.616s	user 0.006s	sys 0.006s
W20260809 23:21:10.075141 25321 connection.cc:232] RPC call timeout handler was delayed by -1.1894664678256959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20830000000000004s for a response
W20260809 23:21:10.005954  2170 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260809 23:21:10.076049 25321 connection.cc:232] RPC call timeout handler was delayed by -1.0638150130398571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.467s for a response
I20260809 23:21:10.053287 25368 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:09.931092  2030 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.055955 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 280104us
I20260809 23:21:09.932288  1731 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.057334 25366 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.005s	sys 0.001s
I20260809 23:21:10.059500 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:10.078027 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:09.969949  2162 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260809 23:21:10.062318 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.062562 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:10.040544  2153 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.282s	user 0.001s	sys 0.001s
I20260809 23:21:10.109805 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:09.969327  2152 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.199s	user 0.000s	sys 0.000s
I20260809 23:21:09.963775  2006 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.126314 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:09.978279  1527 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:09.943634  1431 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.072487 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1534) took 2788 ms. Trace:
I20260809 23:21:09.976656  1774 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.032918 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1307) took 1107 ms. Trace:
I20260809 23:21:10.159392 24431 rpcz_store.cc:276] 0809 23:21:08.894578 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.916707 (+ 22129us) service_pool.cc:227] Handling call
0809 23:21:08.917085 (+   378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:09.506503 (+589418us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5800b525b0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:21:09.662141 (+155638us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.001639 (+339498us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:10.001878 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1066140288}
I20260809 23:21:10.003399  1917 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:10.039749 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.084465 24321 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:10.006800  1539 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.000519  2194 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.088084 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:10.064618  1528 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.093384 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:10.093356 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.179724 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.180687 25371 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:10.089511 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 229953us
I20260809 23:21:10.099938 24316 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:10.117017 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.117694 25373 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:10.078739  1770 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:10.125537 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.112058  2188 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:10.137564 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2888) took 1063 ms. Trace:
I20260809 23:21:10.149794 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.082602  2121 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.148903 25375 rpcz_store.cc:276] 0809 23:21:07.271723 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:07.272641 (+   918us) service_pool.cc:227] Handling call
0809 23:21:07.682337 (+409696us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:21:07.704577 (+ 22240us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:07.741126 (+ 36549us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:08.461687 (+720561us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.606575 (+1144888us) log.cc:842] Serialized 14 byte log entry
0809 23:21:10.060072 (+453497us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:10.060150 (+    78us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:10.060216 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.060320 (+   104us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.060474 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20003,"prepare.queue_time_us":1176069,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":80,"raft.queue_time_us":643851,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":223,"spinlock_wait_cycles":14782848,"thread_start_us":2246137,"threads_started":4,"wal-append.queue_time_us":1005534}
I20260809 23:21:10.092113  2171 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.099072  1776 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:10.176177 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:10.249442  2151 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:10.250177  2151 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:09.861425 24163 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 11, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 11 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:10.163136   877 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.252312 24163 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:10.121551  2184 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:10.242766  2121 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.186550 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.180850 24431 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:10.261904 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2891) took 1186 ms. Trace:
I20260809 23:21:10.262058 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:10.127444 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.262806 24431 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:10.194427 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:10.263878 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.211136 24429 rpcz_store.cc:276] 0809 23:21:09.074006 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.074137 (+   131us) service_pool.cc:227] Handling call
0809 23:21:09.088449 (+ 14312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.088643 (+   194us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.088664 (+    21us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.112547 (+ 23883us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.573688 (+461141us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:09.573736 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.573845 (+   109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.137295 (+563450us) raft_consensus.cc:1671] finished
0809 23:21:10.137330 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.137489 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":439325,"threads_started":1,"wal-append.queue_time_us":536628}
I20260809 23:21:10.153966  2006 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.267735 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3287) took 1406 ms. Trace:
I20260809 23:21:10.194332  2194 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.195188 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.195113  1917 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.195212  1528 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.197395  2224 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260809 23:21:10.198948  2183 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260809 23:21:10.199215  1431 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
W20260809 23:21:10.199402 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111453us
I20260809 23:21:10.198628  2204 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.203112 24320 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:21:10.204813 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3211) took 2163 ms. Trace:
W20260809 23:21:10.207142 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 113832us
I20260809 23:21:10.213924 25363 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
W20260809 23:21:10.182458  2155 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260809 23:21:10.267272 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:10.264940 25370 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 15.
I20260809 23:21:10.069264  2168 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.218081  2107 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260809 23:21:10.311501 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:10.324613 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57439.3us
I20260809 23:21:10.262341 24428 rpcz_store.cc:276] 0809 23:21:09.075697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.075831 (+   134us) service_pool.cc:227] Handling call
0809 23:21:09.076178 (+   347us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.088307 (+ 12129us) raft_consensus.cc:1520] Early marking committed up to index 7
0809 23:21:09.088347 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.088872 (+   525us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.201096 (+112224us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b1800491fe0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:09.595204 (+394108us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:21:09.595271 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.595370 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.217315 (+621945us) raft_consensus.cc:1671] finished
0809 23:21:10.217346 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.261843 (+ 44497us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29811,"raft.queue_time_us":23337,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":8378,"spinlock_wait_cycles":9103744,"thread_start_us":363816,"threads_started":1,"wal-append.queue_time_us":538088}
I20260809 23:21:10.325587 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.268086 24542 rpcz_store.cc:276] 0809 23:21:08.861732 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.125367 (+263635us) service_pool.cc:227] Handling call
0809 23:21:09.125679 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.154381 (+ 28702us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.154408 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.154822 (+   414us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.607959 (+453137us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:09.823495 (+215536us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.826046 (+  2551us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.826121 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.267514 (+441393us) raft_consensus.cc:1671] finished
0809 23:21:10.267537 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.267675 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48759,"prepare.queue_time_us":356119,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":155466,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":62518,"thread_start_us":573954,"threads_started":2,"wal-append.queue_time_us":567396}
I20260809 23:21:10.319808 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.262192  2121 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:10.319403 24535 rpcz_store.cc:276] 0809 23:21:08.040932 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.053958 (+ 13026us) service_pool.cc:227] Handling call
0809 23:21:08.054409 (+   451us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.030733 (+976324us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.030770 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.041157 (+ 10387us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.538457 (+497300us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:21:09.538494 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.538555 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.204524 (+665969us) raft_consensus.cc:1671] finished
0809 23:21:10.204556 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.204736 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16270,"raft.queue_time_us":71185,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8800,"thread_start_us":1441395,"threads_started":2,"wal-append.queue_time_us":632695}
I20260809 23:21:10.245561  1776 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.341199 24541 pending_rounds.cc:76] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:21:10.341507 24541 pending_rounds.cc:97] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260809 23:21:10.341830 24541 raft_consensus.cc:2866] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:10.334878 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:10.343075 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:10.267560  2170 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260809 23:21:10.296270  1774 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.338404 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.344009 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.352517 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.357460 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.366498 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:10.367564 24429 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260809 23:21:10.370524 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:10.331403 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:10.371753 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 100005us
I20260809 23:21:10.373170 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:10.433287 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 89319.2us
I20260809 23:21:10.434554 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.347965  1776 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:10.438946 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:10.440562 25375 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:10.337724  1677 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:10.441556 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:10.346763  1485 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.345929  2144 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:10.454630 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123474us
I20260809 23:21:10.457996 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1541) took 2300 ms. Trace:
I20260809 23:21:10.325847  2116 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260809 23:21:10.468524 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:10.386762  2162 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:21:10.463680 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:10.267140  2171 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.464919 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.455997 25368 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:10.455192 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.386217  1917 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:10.556550 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 179397us
I20260809 23:21:10.558547 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:10.453645  2159 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260809 23:21:10.558859 24546 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260809 23:21:10.455420  2183 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260809 23:21:10.562526 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:10.571522 24534 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:10.567898 24429 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260809 23:21:10.462668  1677 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.581498  2216 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.581745  2237 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:10.292415  2006 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.504069   877 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:10.462741 25366 rpcz_store.cc:276] 0809 23:21:08.156409 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.157560 (+  1151us) service_pool.cc:227] Handling call
0809 23:21:08.157847 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:08.158038 (+   191us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:09.030565 (+872527us) log.cc:842] Serialized 14 byte log entry
0809 23:21:10.442764 (+1412199us) log.cc:842] Serialized 14 byte log entry
0809 23:21:10.455908 (+ 13144us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:10.455995 (+    87us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:10.456146 (+   151us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.456227 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.456364 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31736,"prepare.queue_time_us":813594,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":13322,"raft.queue_time_us":243552,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":28422,"thread_start_us":2240276,"threads_started":4,"wal-append.queue_time_us":645834}
W20260809 23:21:10.569095  2017 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.517s	user 0.009s	sys 0.012s
I20260809 23:21:10.375276  1774 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.560119  2189 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.552270  1539 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:10.335024  2204 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.613415 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 148560us
I20260809 23:21:10.472348  1485 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:10.582923  2155 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260809 23:21:10.621142 24429 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
W20260809 23:21:10.607049  2152 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260809 23:21:10.607630  1964 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:10.624833 24546 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260809 23:21:10.598449  1677 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:10.582450  2118 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:10.585454  2030 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.614703 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2894) took 1524 ms. Trace:
I20260809 23:21:10.612656  2115 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:10.600425  1731 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.610409  2237 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:10.667464  2237 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:10.667754  2210 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260809 23:21:10.668157  2171 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.639892 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:10.669368  2175 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.480s	user 0.003s	sys 0.000s
I20260809 23:21:10.649271  1680 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:10.670995  1736 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.671505  1736 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.673441  1736 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:10.673843  2146 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.000s
I20260809 23:21:10.674002  1736 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.680107  1736 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.679212 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2885) took 1613 ms. Trace:
I20260809 23:21:10.681942 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2900) took 1251 ms. Trace:
I20260809 23:21:10.674254  2146 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.682314 24425 rpcz_store.cc:276] 0809 23:21:09.430892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.431340 (+   448us) service_pool.cc:227] Handling call
0809 23:21:09.431766 (+   426us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.432186 (+   420us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.432223 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.454838 (+ 22615us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.763553 (+308715us) raft_consensus.cc:1631] Marking committed up to 8
0809 23:21:09.763599 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.763661 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.668982 (+905321us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b2400415b08. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:21:10.681663 (+ 12681us) raft_consensus.cc:1671] finished
0809 23:21:10.681684 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.681888 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15509,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":15509,"spinlock_wait_cycles":35706624,"thread_start_us":300174,"threads_started":1,"wal-append.queue_time_us":615989}
I20260809 23:21:10.683415  1528 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.627417  1539 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.651136 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.644731 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:10.693984 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54178us
I20260809 23:21:10.694942 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:10.471791  2144 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.657948  1774 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:10.659058  2177 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.405s	user 0.000s	sys 0.001s
I20260809 23:21:10.659219  2168 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.624770  1839 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:10.659914  2194 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:10.660104  2206 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260809 23:21:10.627975  2188 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.435423  2151 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.627215  2189 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.622385  2017 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.660851  2169 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
W20260809 23:21:10.349849  1527 consensus_queue.cc:1159] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 21 status { last_received { term: 20 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:10.659081  2180 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.769224  2180 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.693221  1964 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.771515  2180 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.772697 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2899) took 1499 ms. Trace:
I20260809 23:21:10.661845  2108 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
W20260809 23:21:10.663542  2134 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.952s	user 0.007s	sys 0.006s
I20260809 23:21:10.773464  1964 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.773895  2134 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.775492  2204 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.775993  2134 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.780658  2017 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.781548 24316 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:10.666448  1485 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.611057  2216 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.636631  1977 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:10.646138 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2902) took 1178 ms. Trace:
I20260809 23:21:10.785745 24429 rpcz_store.cc:276] 0809 23:21:09.467530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.599467 (+1131937us) service_pool.cc:227] Handling call
0809 23:21:10.618606 (+ 19139us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:10.645878 (+ 27272us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.646082 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:10.676844 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.679214 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2901) took 1233 ms. Trace:
I20260809 23:21:10.650269 24430 rpcz_store.cc:276] 0809 23:21:09.090081 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.110950 (+ 20869us) service_pool.cc:227] Handling call
0809 23:21:09.140320 (+ 29370us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.160892 (+ 20572us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.160953 (+    61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.161371 (+   418us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.564763 (+403392us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:09.606109 (+ 41346us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.649835 (+ 43726us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.649936 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.614436 (+964500us) raft_consensus.cc:1671] finished
0809 23:21:10.614463 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.614638 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":165783,"prepare.queue_time_us":44484,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":65099,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":13451,"thread_start_us":379803,"threads_started":1,"wal-append.queue_time_us":649394}
I20260809 23:21:10.694228 24314 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:10.655807  2176 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:10.611558  1431 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:10.690451 24534 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:21:10.647601 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 208788us
I20260809 23:21:10.697783 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.790951 24546 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:10.681525 24424 rpcz_store.cc:276] 0809 23:21:09.065687 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.065865 (+   178us) service_pool.cc:227] Handling call
0809 23:21:09.077892 (+ 12027us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.112541 (+ 34649us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:09.127173 (+ 14632us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.249028 (+121855us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b18004c64e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11
0809 23:21:09.250276 (+  1248us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b18004c64e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:09.251601 (+  1325us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b18004c64e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac27708bcf 00007fac209387b6 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d
0809 23:21:09.830383 (+578782us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.872673 (+ 42290us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.872837 (+   164us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.939993 (+ 67156us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.940182 (+   189us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.940257 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.678995 (+738738us) raft_consensus.cc:1671] finished
0809 23:21:10.679014 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.679152 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151523,"prepare.queue_time_us":223063,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":102,"raft.queue_time_us":99927,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":33632,"spinlock_wait_cycles":24926464,"thread_start_us":633266,"threads_started":2,"wal-append.queue_time_us":642587}
I20260809 23:21:10.787163 24423 rpcz_store.cc:276] 0809 23:21:09.446127 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.446387 (+   260us) service_pool.cc:227] Handling call
0809 23:21:09.446812 (+   425us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:09.470015 (+ 23203us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.470048 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:09.512777 (+ 42729us) log.cc:842] Serialized 52 byte log entry
0809 23:21:09.821548 (+308771us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:09.922826 (+101278us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b48001701d8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:10.088377 (+165551us) log.cc:842] Serialized 14 byte log entry
0809 23:21:10.088584 (+   207us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.088656 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.260756 (+172100us) spinlock_profiling.cc:243] Waited 9.91 ms on lock 0x7b180044ba80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:10.677632 (+416876us) spinlock_profiling.cc:243] Waited 8.94 ms on lock 0x7b2400415b08. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:21:10.678905 (+  1273us) raft_consensus.cc:1671] finished
0809 23:21:10.679046 (+   141us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.679165 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32736,"prepare.queue_time_us":162586,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":23777,"raft.run_cpu_time_us":6060,"raft.run_wall_time_us":477742,"spinlock_wait_cycles":50012672,"thread_start_us":364746,"threads_started":2,"wal-append.queue_time_us":484403}
I20260809 23:21:10.696168 24428 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:10.703763  2118 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:10.705696  1556 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:10.706478  2171 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.804476  2171 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.804680 24163 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 14, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 14 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:21:10.673379  1667 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.875s	user 0.007s	sys 0.002s
I20260809 23:21:10.807302  1667 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.808142  2189 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.808303  2216 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:10.809234  2006 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.809777  1667 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.707695 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:10.811635 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 104012us
I20260809 23:21:10.708298  1680 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:10.708637  2219 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:10.706768  2237 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.709702  2144 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:10.715029 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 257318us
W20260809 23:21:10.800436  2206 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260809 23:21:10.720775 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 97425.5us
I20260809 23:21:10.786835  2204 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.738369 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2903) took 1270 ms. Trace:
W20260809 23:21:10.736610 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 272938us
I20260809 23:21:10.831858 24425 rpcz_store.cc:276] 0809 23:21:09.467858 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.700310 (+1232452us) service_pool.cc:227] Handling call
0809 23:21:10.737310 (+ 37000us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:10.737544 (+   234us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:10.737578 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:10.737638 (+    60us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:10.737692 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:10.738120 (+   428us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.738297 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:10.698524  1539 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:10.711349  1731 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.766239  2190 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:10.766971  2192 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260809 23:21:10.784219 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.000s	sys 0.013s
W20260809 23:21:10.857354 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.000s	sys 0.013s
I20260809 23:21:10.773108 24427 rpcz_store.cc:276] 0809 23:21:09.273049 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.273313 (+   264us) service_pool.cc:227] Handling call
0809 23:21:09.291093 (+ 17780us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.291307 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.291330 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.291715 (+   385us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.716502 (+424787us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.716538 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.716774 (+   236us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.772455 (+1055681us) raft_consensus.cc:1671] finished
0809 23:21:10.772481 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.772632 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":391070,"threads_started":1,"wal-append.queue_time_us":842441}
I20260809 23:21:10.782024 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2762) took 1346 ms. Trace:
I20260809 23:21:10.823968  2006 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260809 23:21:10.795964 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:10.862174 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66325.2us
I20260809 23:21:10.796980 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.798874 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:10.821376  1680 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:10.863724 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64983.4us
W20260809 23:21:10.863941 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67081.9us
I20260809 23:21:10.799714 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:10.800073 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.864758 24430 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:10.800010 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.865448 24429 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:10.862527 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.801491  2118 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.802109  1431 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.871470 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.803030  2230 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.067s
I20260809 23:21:10.803107  1977 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.810818 24313 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:10.741338  1839 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.821352 24314 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:10.683738  2146 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.736470 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.834578 24312 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:10.837138 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.854127 24545 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:10.854671 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.847764 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.855901 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:10.858254 24309 rpcz_store.cc:276] 0809 23:21:09.435850 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.435979 (+   129us) service_pool.cc:227] Handling call
0809 23:21:09.436460 (+   481us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.465468 (+ 29008us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.465496 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.465967 (+   471us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.753825 (+287858us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.753886 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.753988 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.781815 (+1027827us) raft_consensus.cc:1671] finished
0809 23:21:10.781834 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:10.781971 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9885,"raft.queue_time_us":10944,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":89,"thread_start_us":287394,"threads_started":1,"wal-append.queue_time_us":628180}
I20260809 23:21:10.824199  1964 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260809 23:21:10.887439 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.888481 24309 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:10.794137 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:10.864938 24424 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:10.870637 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.888749 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.889590 25368 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:10.870807 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:10.892558 25363 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:10.891494 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.864491  2192 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.876387 24427 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:10.876729 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.878482 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.882175 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.893301 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.903460 24539 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:10.904265 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.873768  2176 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:10.895298 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.896673 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:10.898602 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:10.858567  1736 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:10.914870 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.904819 24539 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:10.875883 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84628us
I20260809 23:21:10.905015 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:10.861689  2006 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:10.868575  1680 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:10.873374  2118 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:10.906329 24425 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:10.909147 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.870875  2108 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:10.913715 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.917362 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.919239 24424 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:10.851257  2204 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.935520 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.940393 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.941320 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:10.935544 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.943338 24314 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260809 23:21:10.936479 24542 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:10.953147 24544 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:10.898288  2146 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:10.906253  2192 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:10.865203  1731 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:10.942569  1770 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.727s	user 0.007s	sys 0.006s
I20260809 23:21:10.964025 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:10.973241 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:10.960199 24498 leader_election.cc:398] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
I20260809 23:21:10.920374  2144 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:10.982090 24323 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:10.923599  1352 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:10.966701 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 98065.5us
I20260809 23:21:10.968685 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:10.968838 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102562us
I20260809 23:21:10.969599  2195 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:10.962549  2017 raft_consensus.cc:979] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:10.970331 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:10.825820  2219 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:10.974027 24498 leader_election.cc:401] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260809 23:21:10.974050 24317 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:10.976701 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:10.923710  2006 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:10.900110  2222 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
I20260809 23:21:10.918974  2240 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:10.984472 24430 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:10.986433  1352 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.960167  1736 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:10.986213 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:10.990151 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:10.864379  2190 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:10.926426  2176 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:10.962024  2146 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:10.960683  2250 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:10.961390  2168 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:10.923062  2134 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260809 23:21:10.963548  2108 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:10.986014 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:11.032550 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88615.1us
I20260809 23:21:11.093176 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:11.070124 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106503us
I20260809 23:21:10.981396  2144 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:11.072904 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102683us
I20260809 23:21:10.990358  2237 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.068806 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:11.078146 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87996.9us
I20260809 23:21:11.079149 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:10.989631  2171 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:11.103688 24542 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:11.026247  2106 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.104450 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.105499  2106 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:11.106012 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.031476  2188 raft_consensus.cc:1067] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:11.107084  2238 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:11.079304  1917 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.000s
I20260809 23:21:11.078797  2216 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:11.099272 24423 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260809 23:21:11.073573  1485 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.101444  2195 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:11.101833  2116 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:11.101636  2108 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.219384  1977 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:11.223785  2258 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260809 23:21:11.226989  2195 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:11.227406  2195 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:11.228299  2195 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.229609  2195 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:11.226241  1680 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.235169  2243 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.037s
I20260809 23:21:11.028023  1770 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:11.077845  1352 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:11.236282  1770 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.237201  1680 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:11.237705  1776 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.767s	user 0.009s	sys 0.002s
I20260809 23:21:11.072365  2194 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:11.120767 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.238301  1776 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.239075  1770 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:11.239706 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119041us
I20260809 23:21:11.240612  1776 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.081708  2121 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:11.140300 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106831us
I20260809 23:21:11.244206 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.244809 24312 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:11.245962 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:11.247895  2216 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:11.248754  2260 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260809 23:21:11.142138  2188 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:11.246811 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:11.142745  2168 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:11.254079 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.143725 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.144428  2214 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.147197  2237 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:11.148293  2106 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:11.148649  1839 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.149432  2238 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:11.139277 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 153450us
I20260809 23:21:11.164208 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2890) took 2089 ms. Trace:
I20260809 23:21:11.275383 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.169514 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3293) took 1077 ms. Trace:
I20260809 23:21:11.169514  2219 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.172003 25363 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:11.164502  2118 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:11.177419  2251 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260809 23:21:10.983189 24163 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 12, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:11.280539  2238 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.287756 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.274524  2249 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:11.289836  2249 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:11.186476  2176 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.290215  2249 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:11.087152  2146 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:11.291301  1731 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:11.292174 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 177181us
I20260809 23:21:11.292323 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1615) took 1163 ms. Trace:
I20260809 23:21:11.293022 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.293908  2204 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:11.294296 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:11.087646  2154 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:11.094449  2222 consensus_queue.cc:1225] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:11.280364  2151 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.085332  2250 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:11.280689  2245 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.296132  2250 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.297830  2250 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.101501  2233 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.119836 24431 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:11.299335 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3324) took 1324 ms. Trace:
I20260809 23:21:11.299723 24431 rpcz_store.cc:276] 0809 23:21:09.975006 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.264686 (+289680us) service_pool.cc:227] Handling call
0809 23:21:11.299178 (+1034492us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:11.299279 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1804002176}
I20260809 23:21:11.115099  2144 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.115561  2030 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:11.213434  2116 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.213831  1429 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:11.315469 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.235719 24539 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:11.316815  2224 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:11.317441  2224 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:11.318074  1677 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.002s
I20260809 23:21:11.318675  1539 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.247979  2056 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:11.319887  2056 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.321019  1539 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.248839 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.322093  2197 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.322326  1839 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:11.242980  2134 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:11.102947  2255 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260809 23:21:11.248636  2216 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.277252 24323 rpcz_store.cc:276] 0809 23:21:10.091655 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.091829 (+   174us) service_pool.cc:227] Handling call
0809 23:21:11.169339 (+1077510us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:11.169453 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1966249344}
W20260809 23:21:11.322094 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73337.5us
I20260809 23:21:11.280347 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:11.324920  2216 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:11.325430 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45167.1us
I20260809 23:21:11.273506  2237 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:11.300534 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53789.3us
W20260809 23:21:11.284387  2257 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
I20260809 23:21:11.281419 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.280972 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.292681 24316 rpcz_store.cc:276] 0809 23:21:10.129116 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.196750 (+ 67634us) service_pool.cc:227] Handling call
0809 23:21:10.197158 (+   408us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:10.785278 (+588120us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:11.292103 (+506825us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:11.292267 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1343477504}
I20260809 23:21:11.275240 24435 rpcz_store.cc:276] 0809 23:21:09.074813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:09.074964 (+   151us) service_pool.cc:227] Handling call
0809 23:21:09.191012 (+116048us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:09.282974 (+ 91962us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:09.283008 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:09.292239 (+  9231us) log.cc:842] Serialized 26 byte log entry
0809 23:21:09.708627 (+416388us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:09.766780 (+ 58153us) log.cc:842] Serialized 14 byte log entry
0809 23:21:09.766993 (+   213us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:09.767066 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:10.504820 (+737754us) raft_consensus.cc:1671] finished
0809 23:21:10.504849 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:11.164029 (+659180us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540023e450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:11.164143 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":216260,"prepare.queue_time_us":57958,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":162613,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":41895,"spinlock_wait_cycles":266077568,"thread_start_us":353703,"threads_started":1,"wal-append.queue_time_us":497622}
I20260809 23:21:11.338317 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:11.253278  2188 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:11.313803 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:11.341460 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.342238 24309 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:11.336668 24535 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:11.345547 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.346591 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.318500  1677 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.322611  2056 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.119258 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:11.325439 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.327288 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.430328 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.432880 24541 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:11.316836 24422 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:11.431653 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:11.433130 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 107773us
I20260809 23:21:11.280813 24430 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:11.327329  2154 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.331583 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:11.443059 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74550.2us
I20260809 23:21:11.326710  1839 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:11.349316 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55103.7us
I20260809 23:21:11.312507  2214 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:11.352943 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:11.326239  1429 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:11.446385 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.432052 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:11.354537 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.330681 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.449848 24323 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:11.450914 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:11.435537 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:11.286765  2238 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.436287 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.354238 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.465456 25369 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:11.293299  2190 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:11.348038 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:11.325477  2216 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:11.429201 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.450863 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.274187  2168 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.278191  2106 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:11.436895 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 317704us
I20260809 23:21:11.431209  1731 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:11.461195 24534 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:11.466151 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1555) took 1090 ms. Trace:
I20260809 23:21:11.344249  2188 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:11.445780 24535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:21:11.502964 24535 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:21:11.503719 24535 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:11.483114 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.436338  2242 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:11.455448  2214 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:11.435477  2118 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.515700 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.353643  2030 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:11.525655 24323 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:11.537451 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.496879 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.538473 24321 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:11.505100 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:11.507371 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58277.6us
I20260809 23:21:11.435738  1917 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:11.540365 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:11.523468 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:11.541424  1776 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:11.541795 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.542686  1776 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:11.435199  1527 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:11.326517  2219 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.118s
I20260809 23:21:11.516047 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.533052 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.465772  2204 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:11.516963 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:11.561733 25363 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:11.561834 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.506201  1731 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:11.502462 25370 rpcz_store.cc:276] 0809 23:21:10.375634 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.375776 (+   142us) service_pool.cc:227] Handling call
0809 23:21:10.435843 (+ 60067us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:10.613913 (+178070us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:10.613958 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:10.623796 (+  9838us) log.cc:842] Serialized 26 byte log entry
0809 23:21:10.831412 (+207616us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:10.831456 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:11.441940 (+610484us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:11.442012 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:11.442076 (+    64us) raft_consensus.cc:1671] finished
0809 23:21:11.442095 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:11.466074 (+ 23979us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7849,"raft.queue_time_us":4953,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":6653,"spinlock_wait_cycles":373779200,"thread_start_us":128302,"threads_started":1,"wal-append.queue_time_us":320412}
I20260809 23:21:11.536051 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.563910 25373 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:11.564687 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.473707 24435 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:11.505951  2216 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:11.539562 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:11.473166  2056 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:11.547986 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.545770 24312 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:11.344492  2233 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.176s
I20260809 23:21:11.552563 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.353806  2192 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.556841 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.521268 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.520877  2238 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:11.318499  2224 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.483340  2269 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:11.567276 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:11.531515  2214 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.512588  2240 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:11.541607 24316 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:11.568478 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:11.569407 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.592902 24316 raft_consensus.cc:603] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:11.571295 24323 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260809 23:21:11.544327  1917 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:11.583702 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:11.584816 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.561794  1776 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:11.559368  1527 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.596448 25366 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:11.591084  2056 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:11.597034 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.605547 25370 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.567845  1731 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:11.570444  2030 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:11.600274 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3329) took 1615 ms. Trace:
I20260809 23:21:11.596448 24317 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:11.609242 25373 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
W20260809 23:21:11.611470 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91231.1us
I20260809 23:21:11.580754  2243 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:11.614406 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.612931 24321 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:11.617293 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:11.591729  2198 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:11.609794 24429 rpcz_store.cc:276] 0809 23:21:09.984599 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.866220 (+881621us) service_pool.cc:227] Handling call
0809 23:21:10.884791 (+ 18571us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:10.885539 (+   748us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:10.885594 (+    55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:10.914175 (+ 28581us) log.cc:842] Serialized 26 byte log entry
0809 23:21:11.316866 (+402691us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:11.316911 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:11.316982 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:11.600069 (+283087us) raft_consensus.cc:1671] finished
0809 23:21:11.600090 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:11.600214 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":202,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":8520,"thread_start_us":349012,"threads_started":1,"wal-append.queue_time_us":665949}
I20260809 23:21:11.602133  2240 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.602334  1917 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:11.605244   877 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:11.608568  1964 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:11.648794  1776 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.644055  2116 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:11.659395 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:11.677280 24429 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:11.684370 24541 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:11.580112  1839 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.644834  2233 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:11.661185 25370 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:11.704269 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:11.674170  2275 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260809 23:21:11.710371 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74289.6us
I20260809 23:21:11.646234  1964 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:11.720811 24545 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:11.723588 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:11.650995  1677 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.674044  1774 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:11.675827  2237 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:11.693084   877 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:11.728681  1964 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:11.635538  2242 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:11.745500 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:11.746172 24321 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:11.747627 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 130405us
I20260809 23:21:11.721918  2154 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:11.722352  1569 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:11.635187  2056 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:11.754077  2108 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.008s	sys 0.023s
I20260809 23:21:11.655911  2216 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:11.762216  2281 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260809 23:21:11.751206  2168 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.747920  2237 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:11.832924 24269 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.740s (SENT)
I20260809 23:21:11.841193 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:11.795244  2108 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.009s	sys 0.024s
I20260809 23:21:11.744740  1774 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:11.847887 25372 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:11.856619 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:11.858966 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 135452us
I20260809 23:21:11.860951 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:11.861544 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:11.869688 25370 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 17
I20260809 23:21:11.793040  2106 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:11.883417 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 112328us
I20260809 23:21:11.888573 24321 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:11.816159  2192 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:11.894348 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.768136  2154 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:11.895236  2292 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260809 23:21:11.747821  2030 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:11.854378  2006 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.772s	user 0.009s	sys 0.003s
W20260809 23:21:11.862636  2225 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260809 23:21:11.772639  1569 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:11.905869 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73012.8us
I20260809 23:21:11.932456 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:11.884754  2237 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:11.904820 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:11.914812 24429 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:11.937002 25370 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:11.941250 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:11.913409  2283 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
W20260809 23:21:11.912691  2281 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260809 23:21:11.960806 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:11.890163  1527 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:11.963012 24429 raft_consensus.cc:2396] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260809 23:21:11.972345 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49053.6us
I20260809 23:21:11.974850 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:11.904330  1539 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:11.942108  2266 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260809 23:21:11.983867 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51486.7us
W20260809 23:21:11.984712 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 422928us
I20260809 23:21:11.999898 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.007381 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:12.017163 24269 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.791s (SENT)
I20260809 23:21:11.989746  1539 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.012610  1677 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260809 23:21:12.014200  2329 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:11.960709  2237 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.031039  2329 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:11.929880  2006 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.993228  1527 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:11.891229   877 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:11.963483  2106 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260809 23:21:12.044471 24544 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
I20260809 23:21:12.047762 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.063547 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:12.077976 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:12.079003 25370 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:12.032472  2243 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:12.077845  1677 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.078609 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:12.044827  2238 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:12.104664 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1614) took 1976 ms (client timeout 2509 ms). Trace:
W20260809 23:21:12.105386 24313 rpcz_store.cc:269] 0809 23:21:10.128733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.134054 (+  5321us) service_pool.cc:227] Handling call
0809 23:21:12.104437 (+1970383us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000dc5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:12.104605 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1297188096}
I20260809 23:21:12.091899  2217 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:12.110505  2217 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:12.047063  2298 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:12.091436  1677 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.102421  2195 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:12.124800 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53688.1us
W20260809 23:21:12.124517 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 150199us
W20260809 23:21:12.121932 24535 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.618s	user 0.006s	sys 0.013s
W20260809 23:21:12.124902 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 126612us
I20260809 23:21:12.102691  2243 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:12.131340  1527 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:12.133826 24269 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.000s (SENT)
I20260809 23:21:12.133751  1527 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.134495 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:12.138953 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:12.142882 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:12.130892 24535 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.122314  2298 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:12.162066 24535 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:12.159248 25369 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.670s	user 0.008s	sys 0.004s
I20260809 23:21:12.112564  2238 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.189121 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 256594us
W20260809 23:21:12.192445  2299 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260809 23:21:12.137975  2189 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:12.224611  2121 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:12.226948  2294 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.439s	user 0.000s	sys 0.002s
I20260809 23:21:12.233381  2237 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.149156  2168 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:12.151954  2217 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.236058  2188 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.700s	user 0.005s	sys 0.010s
W20260809 23:21:12.237339  2269 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.007s
W20260809 23:21:12.239414  2249 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.949s	user 0.012s	sys 0.005s
I20260809 23:21:12.239651  1539 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.240423  2154 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.240615 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3244) took 1412 ms. Trace:
I20260809 23:21:12.241943  1774 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.330103  2269 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:12.032150  2329 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.331099  2337 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:12.330700 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 183177us
I20260809 23:21:12.243835 25369 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
W20260809 23:21:12.252744 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109948us
I20260809 23:21:12.329728 24539 rpcz_store.cc:276] 0809 23:21:10.827788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.916574 (+ 88786us) service_pool.cc:227] Handling call
0809 23:21:10.916840 (+   266us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.316497 (+399657us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.240379 (+923882us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.240556 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":732927744}
I20260809 23:21:12.333614 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:12.333913 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 201796us
I20260809 23:21:12.159873  2106 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.135905  2195 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:12.249310  2146 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.957s	user 0.013s	sys 0.010s
I20260809 23:21:12.339398 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.294409 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:12.250140  2194 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.005s	user 0.013s	sys 0.004s
W20260809 23:21:12.256538  2144 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.921s	user 0.011s	sys 0.007s
I20260809 23:21:12.329864  2217 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:12.251235  2127 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.103s	user 0.007s	sys 0.009s
I20260809 23:21:12.244131  2006 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:12.304342 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 168417us
I20260809 23:21:12.325281 24312 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:12.259538  2116 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
W20260809 23:21:12.305238  1352 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.069s	user 0.015s	sys 0.002s
I20260809 23:21:12.193447  2200 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:12.332105 25375 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:12.244764 25370 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:21:12.256135 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.008s	sys 0.008s
I20260809 23:21:12.335493 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.340762 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.341637 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.340018 24535 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 14.
W20260809 23:21:12.247810  2250 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.950s	user 0.009s	sys 0.003s
W20260809 23:21:12.331887  2120 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.104s	user 0.025s	sys 0.057s
I20260809 23:21:12.331743  2337 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.347272  2194 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
I20260809 23:21:12.349881 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:12.350512 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88502.5us
I20260809 23:21:12.261778  1964 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.354176 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:12.356488  2300 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.406s	user 0.000s	sys 0.002s
I20260809 23:21:12.361943 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3245) took 1533 ms. Trace:
I20260809 23:21:12.347438  2146 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.364557  2312 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.345s	user 0.001s	sys 0.000s
W20260809 23:21:12.365988  1917 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.718s	user 0.007s	sys 0.008s
I20260809 23:21:12.350620  2144 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.366838 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:12.368297  2238 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.294646  2121 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.370621 24269 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.037s (SENT)
I20260809 23:21:12.372177 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.348598  2242 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:12.374699  2006 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.377588 24539 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:12.384649 24309 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:12.386161 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:12.354517  2127 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.206s	user 0.007s	sys 0.013s
W20260809 23:21:12.387377 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.000s	sys 0.026s
I20260809 23:21:12.443871 24428 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:12.444250 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.000s	sys 0.026s
I20260809 23:21:12.326253  2188 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.385842  2116 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:12.382480  2120 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.231s	user 0.025s	sys 0.058s
I20260809 23:21:12.332312  2249 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.396938 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.411166  1677 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:12.412891 24321 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:12.375389  1569 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:12.447089 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61005us
I20260809 23:21:12.348881  2106 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.465902 24320 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:12.389418 24544 rpcz_store.cc:276] 0809 23:21:10.828111 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.963660 (+135549us) service_pool.cc:227] Handling call
0809 23:21:10.989968 (+ 26308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.073343 (+1083375us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.361692 (+288349us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.361883 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2397079808}
I20260809 23:21:12.451570 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.332857  2243 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.376389 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.009s	sys 0.008s
I20260809 23:21:12.383816  1352 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.411461  1539 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.462560 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3336) took 1670 ms. Trace:
I20260809 23:21:12.472515 24430 rpcz_store.cc:276] 0809 23:21:10.792380 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.069578 (+277198us) service_pool.cc:227] Handling call
0809 23:21:11.071939 (+  2361us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.435299 (+363360us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.459895 (+1024596us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.462481 (+  2586us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":476664192}
I20260809 23:21:12.463263 24534 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260809 23:21:12.464864 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:12.461380 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2907) took 2241 ms. Trace:
I20260809 23:21:12.474041 24427 rpcz_store.cc:276] 0809 23:21:10.220348 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.914808 (+694460us) service_pool.cc:227] Handling call
0809 23:21:10.921452 (+  6644us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:10.973172 (+ 51720us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:11.097807 (+124635us) log.cc:842] Serialized 14 byte log entry
0809 23:21:11.361370 (+263563us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:11.431954 (+ 70584us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:11.432010 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.461068 (+1029058us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.461140 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.461303 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29095,"prepare.queue_time_us":106190,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":4415,"raft.queue_time_us":874,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":16877,"spinlock_wait_cycles":118179584,"thread_start_us":373740,"threads_started":2,"wal-append.queue_time_us":447228}
I20260809 23:21:12.446058  2030 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:12.479434 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3341) took 1229 ms. Trace:
I20260809 23:21:12.479534  2030 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:12.480046  2030 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:12.480139  2106 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.479853 24435 rpcz_store.cc:276] 0809 23:21:11.249791 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.566714 (+316923us) service_pool.cc:227] Handling call
0809 23:21:11.567049 (+   335us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:11.572074 (+  5025us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:11.572121 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:11.574984 (+  2863us) log.cc:842] Serialized 26 byte log entry
0809 23:21:11.765939 (+190955us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:11.820082 (+ 54143us) log.cc:842] Serialized 14 byte log entry
0809 23:21:11.831804 (+ 11722us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:11.831909 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:12.363885 (+531976us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:12.479154 (+115269us) raft_consensus.cc:1671] finished
0809 23:21:12.479256 (+   102us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.479376 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12864,"prepare.queue_time_us":33829,"prepare.run_cpu_time_us":161,"prepare.run_wall_time_us":162,"raft.queue_time_us":27339,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":19171,"spinlock_wait_cycles":69383936,"thread_start_us":168465,"threads_started":1,"wal-append.queue_time_us":325879}
I20260809 23:21:12.483186   877 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:12.484185  2214 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.887s	user 0.018s	sys 0.004s
I20260809 23:21:12.484723  2214 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.485257  2116 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.487092  2146 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.449397  2006 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.489830  2006 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:12.451866  2238 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:12.490710  2238 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.491212  2238 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:12.492152  2121 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.477835 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.478946 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:12.470675 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 796903us
I20260809 23:21:12.497138 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:12.499130  1966 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:12.500651 24427 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:12.501408  2192 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:12.501814   877 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.502315   877 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:12.502552  2240 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.006s
I20260809 23:21:12.519999 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.468274  2242 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:12.524927 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.520713 24430 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:12.520035 24425 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:12.527145 24269 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.565s (SENT)
I20260809 23:21:12.478344  2337 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.501833  2192 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.525588 24534 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:12.504676  2345 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.417105  1917 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.532639 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:12.526109  1731 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.908s	user 0.008s	sys 0.009s
I20260809 23:21:12.536814 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:12.537940 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 204410us
I20260809 23:21:12.502460  1429 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.363448  2298 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.551409 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:12.515480  2250 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260809 23:21:12.554916 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:12.570650 24313 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:12.571044 24313 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:12.555370  2032 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:12.556346  1917 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.557736 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.574335 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.548539  2195 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:12.552879  1429 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:12.582203  2056 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.582752  2056 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.563565 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2908) took 2001 ms. Trace:
I20260809 23:21:12.585700 24428 rpcz_store.cc:276] 0809 23:21:10.562578 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.905582 (+343004us) service_pool.cc:227] Handling call
0809 23:21:10.905868 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.450535 (+1544667us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.563278 (+112743us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54003903d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.563493 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3536454400}
I20260809 23:21:12.505203  1539 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.531697  2265 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260809 23:21:12.588732  1539 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.536499  2192 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:12.589175  2265 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:12.589572  2265 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:12.589802 24430 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:12.546825  2323 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:12.591145  2323 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:12.591511  2323 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:12.591684 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.592242 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.593060 24544 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:12.527096  2240 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.569665 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.558279  1794 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:12.595312  1794 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:12.595403  2240 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.574326 24534 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:12.596257  1794 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.597369  1794 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:12.598282 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 2914) took 1265 ms. Trace:
I20260809 23:21:12.571874 24313 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.598644 24425 rpcz_store.cc:276] 0809 23:21:11.332557 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.547760 (+215203us) service_pool.cc:227] Handling call
0809 23:21:12.598086 (+1050326us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:12.598204 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2126344064}
I20260809 23:21:12.599315 24313 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:12.599786 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:12.600344 24425 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:12.600852 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.602341  2190 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:12.602711  2190 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.603086  2190 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:12.603345 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.604871 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:12.606225  2314 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.712s	user 0.000s	sys 0.003s
I20260809 23:21:12.607300 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.596267 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1519) took 1476 ms. Trace:
I20260809 23:21:12.608209 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.608922 24544 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:12.608587 24542 rpcz_store.cc:276] 0809 23:21:11.119510 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.119725 (+   215us) service_pool.cc:227] Handling call
0809 23:21:11.143457 (+ 23732us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.547079 (+403622us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:11.547119 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:11.547162 (+    43us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:21:11.547202 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.596050 (+1048848us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.596095 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.596215 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":927723264}
I20260809 23:21:12.609584  1964 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.586498  2250 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:12.610133  2214 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.610447  2250 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:12.584965  2030 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.611416  2250 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.612509  2030 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.612646 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1566) took 1048 ms. Trace:
I20260809 23:21:12.613005 25375 rpcz_store.cc:276] 0809 23:21:11.564618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.565557 (+   939us) service_pool.cc:227] Handling call
0809 23:21:12.612484 (+1046927us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:12.612590 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1697969408}
I20260809 23:21:12.614169  2030 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:12.606092 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.545845  2345 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.616339 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.617015 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:21:12.617079 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3321) took 1141 ms. Trace:
I20260809 23:21:12.617334  2345 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.617803 24541 rpcz_store.cc:276] 0809 23:21:11.475332 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.475747 (+   415us) service_pool.cc:227] Handling call
0809 23:21:11.476107 (+   360us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.684906 (+208799us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.616821 (+931915us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54003dca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.616999 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":477833088}
I20260809 23:21:12.586750  2056 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:12.618571  2032 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.619184  2268 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:12.619568  2268 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:12.619874   877 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.620760  2032 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.620414  2268 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.621517  2268 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:12.614727 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3322) took 2640 ms (client timeout 1659 ms). Trace:
W20260809 23:21:12.625260 24422 rpcz_store.cc:269] 0809 23:21:09.974351 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.179219 (+204868us) service_pool.cc:227] Handling call
0809 23:21:12.614506 (+2435287us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b540018d154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:12.614633 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2225569024}
I20260809 23:21:12.626449 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:12.627848 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.629756 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.630400 24541 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:12.631572 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.592500 24430 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:12.634274 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:12.636811 24422 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:12.631212 24427 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:12.640796 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2921) took 1306 ms. Trace:
I20260809 23:21:12.648003 24534 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:12.660274  2006 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:12.660893  2006 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.663277 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.612782  2250 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:12.657526 24544 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:12.656141 24427 rpcz_store.cc:276] 0809 23:21:11.334022 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.639281 (+1305259us) service_pool.cc:227] Handling call
0809 23:21:12.639712 (+   431us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.639969 (+   257us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.640005 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:12.640059 (+    54us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:21:12.640102 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.640167 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.640282 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:12.676323  2345 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:12.676505  2144 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.681514 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.683107 24422 pending_rounds.cc:76] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:21:12.683476 24309 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:12.683601 24422 pending_rounds.cc:97] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260809 23:21:12.684582 24422 raft_consensus.cc:2866] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:12.685195 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.687057 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2923) took 1148 ms. Trace:
I20260809 23:21:12.687137 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.687700 24427 rpcz_store.cc:276] 0809 23:21:11.530306 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.678308 (+1148002us) service_pool.cc:227] Handling call
0809 23:21:12.678567 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.678751 (+   184us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.678774 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:12.678810 (+    36us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:12.678842 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.678906 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.679054 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:12.689136 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:12.689824 24427 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:12.691408 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:12.643100  2056 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:12.614617  2030 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.547557  1731 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.665057   877 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:12.679673  2362 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:12.590397  2265 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.724606 24313 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:12.725111 24425 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:12.656086  2237 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260809 23:21:12.727828 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.704325  2121 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:12.712749  2030 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:12.745250 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.710739  2245 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:12.682296  2192 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.710538  2115 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:12.680145  1977 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260809 23:21:12.743628 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.710842  2345 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:12.752167  2345 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.753818 24427 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:12.757154  2121 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:12.758339  2121 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.759622 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.760227 24534 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:12.722414  2362 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.746555 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2915) took 1413 ms. Trace:
I20260809 23:21:12.714461   877 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.719938  2188 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:12.745491  2224 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.008s	sys 0.012s
W20260809 23:21:12.795538  2224 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.009s	sys 0.012s
W20260809 23:21:12.796623  2296 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.143s	user 0.000s	sys 0.002s
I20260809 23:21:12.803231  2006 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:12.765311 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:12.676335  2242 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.808295 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.812381 24535 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:12.746805  1770 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:12.744570  2249 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.817860 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:12.824741 24313 raft_consensus.cc:595] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:12.818727 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3333) took 2815 ms. Trace:
I20260809 23:21:12.825096 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:12.767027  2254 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.231s	user 0.000s	sys 0.001s
W20260809 23:21:12.858558 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 458497us
W20260809 23:21:12.790153  2357 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260809 23:21:12.825408 24423 rpcz_store.cc:276] 0809 23:21:10.003686 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:10.892960 (+889274us) service_pool.cc:227] Handling call
0809 23:21:10.919559 (+ 26599us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:11.193142 (+273583us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.818384 (+1625242us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:12.818649 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":412567936}
I20260809 23:21:12.825124 24313 raft_consensus.cc:603] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:12.826921 24534 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:12.768795 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.831313 24427 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:12.749694  2115 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.832902 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.833612 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1524) took 1038 ms. Trace:
I20260809 23:21:12.764930 24424 rpcz_store.cc:276] 0809 23:21:11.332817 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.571473 (+238656us) service_pool.cc:227] Handling call
0809 23:21:11.583750 (+ 12277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.583975 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:21:11.598749 (+ 14774us) log.cc:842] Serialized 14 byte log entry
0809 23:21:12.742556 (+1143807us) log.cc:842] Serialized 14 byte log entry
0809 23:21:12.744196 (+  1640us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:12.744248 (+    52us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:12.746300 (+  2052us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.746389 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.746502 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25900,"prepare.queue_time_us":21104,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":443,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":2467,"thread_start_us":1142956,"threads_started":2,"wal-append.queue_time_us":966436}
I20260809 23:21:12.791378  1977 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:12.792608  2192 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.872990  1770 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:12.873463  2295 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.010s	user 0.003s	sys 0.000s
I20260809 23:21:12.874054  1770 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:12.875339 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 628465us
I20260809 23:21:12.876816 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:12.840031  1527 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.879005 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:12.879808 25370 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:12.880734 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:12.881361 25369 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:12.882207 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2772) took 1530 ms. Trace:
I20260809 23:21:12.792613  2245 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:12.767939  2362 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:12.842372  2188 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 23:21:12.843753  2359 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
W20260809 23:21:12.844200  2168 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.536s	user 0.005s	sys 0.004s
I20260809 23:21:12.853808 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1563) took 1322 ms. Trace:
I20260809 23:21:12.855084  2108 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:12.791139  2364 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260809 23:21:12.816274 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:12.861589 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:12.804831   877 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:12.909047  2217 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.000s
W20260809 23:21:12.871459 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 480711us
I20260809 23:21:12.792439  1731 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.872006  2154 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:12.872138 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.909502 24535 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:12.880868 24269 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.671s (SENT)
W20260809 23:21:12.940416 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59577.8us
I20260809 23:21:12.841609  2323 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:12.868260 24545 rpcz_store.cc:276] 0809 23:21:11.794828 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.795055 (+   227us) service_pool.cc:227] Handling call
0809 23:21:11.843955 (+ 48900us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.844201 (+   246us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:11.844475 (+   274us) log.cc:842] Serialized 14 byte log entry
0809 23:21:11.844815 (+   340us) log.cc:842] Serialized 14 byte log entry
0809 23:21:12.235285 (+390470us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b1800374740. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11
0809 23:21:12.237756 (+  2471us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b1800374740. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:12.833011 (+595255us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b1800374740. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac234fde10 00007fac234f1663 00007fac234e702a 00007fac234e757b 00007fac2386a88a 00007fac23879f0f 00007fac23879e88
0809 23:21:12.833244 (+   233us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:12.833298 (+    54us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:12.833353 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.833432 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.833552 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5577,"prepare.queue_time_us":393800,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":17302,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":17740,"spinlock_wait_cycles":93551360}
I20260809 23:21:12.886139 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.915331 24423 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
W20260809 23:21:12.921593 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60106.5us
I20260809 23:21:12.905824 24427 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:21:12.899791 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 131083us
I20260809 23:21:12.911975 25369 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 14.
I20260809 23:21:12.908898 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:12.882570 24311 rpcz_store.cc:276] 0809 23:21:11.352237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.352400 (+   163us) service_pool.cc:227] Handling call
0809 23:21:11.352810 (+   410us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:11.432062 (+ 79252us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:11.432092 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:11.447567 (+ 15475us) log.cc:842] Serialized 26 byte log entry
0809 23:21:11.617619 (+170052us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:11.617660 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:11.617728 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:12.872789 (+1255061us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:12.881903 (+  9114us) raft_consensus.cc:1671] finished
0809 23:21:12.881926 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.882146 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87597,"raft.queue_time_us":80139,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":20076,"spinlock_wait_cycles":329362304,"thread_start_us":169743,"threads_started":1,"wal-append.queue_time_us":325558}
I20260809 23:21:12.762844  1352 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:12.960119 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.805483  2265 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:12.872296  2115 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:12.972886 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.922401 24425 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:12.973891 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.947268 24546 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:12.928692 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:12.963573 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:12.966028 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:12.967504 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3344) took 1249 ms. Trace:
I20260809 23:21:12.975253 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:12.975571 24535 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:12.926591 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:12.884325  1977 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:12.973704 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3340) took 2099 ms. Trace:
I20260809 23:21:12.928102 24424 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:12.975639 24423 rpcz_store.cc:276] 0809 23:21:11.717971 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.863666 (+1145695us) service_pool.cc:227] Handling call
0809 23:21:12.911547 (+ 47881us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.967258 (+ 55711us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.967448 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:12.984347 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:12.984627 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:12.987267 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:12.977818 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91757.3us
I20260809 23:21:12.988510 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:12.976753 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:12.907809  2006 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.059s
I20260809 23:21:12.910111  2192 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:12.921849  2108 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.976919 25365 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:13.002701 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:13.019392  2194 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:12.977613  2144 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.018582 24321 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260809 23:21:13.032044  2144 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.022588  2216 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:12.999857 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:12.926033  2240 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:13.010535 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:12.922060  2315 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:13.035429  2192 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:12.910742  2056 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:12.994879  2217 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:13.036376  2056 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:13.036821  2056 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:13.032017  2194 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.023090 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:13.033811  2108 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:13.000108  1352 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:13.043977  1352 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:13.045056  1352 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.016345  2188 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:13.000257 24309 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:12.996579 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2924) took 1216 ms. Trace:
I20260809 23:21:12.986002 24431 rpcz_store.cc:276] 0809 23:21:10.874019 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.312742 (+438723us) service_pool.cc:227] Handling call
0809 23:21:12.614647 (+1301905us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.615156 (+   509us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.615191 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:12.615233 (+    42us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:12.615270 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.615326 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.973537 (+358211us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540018d150. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:12.973650 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":640,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":2992750976}
I20260809 23:21:13.033807 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.035858  2240 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:13.015127 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.000748  2127 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:12.994666  1527 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:12.919202  2168 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:13.006561 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:12.910859 25368 rpcz_store.cc:276] 0809 23:21:11.531587 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.531768 (+   181us) service_pool.cc:227] Handling call
0809 23:21:11.551979 (+ 20211us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:11.571015 (+ 19036us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:11.571052 (+    37us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:21:11.604235 (+ 33183us) log.cc:842] Serialized 78 byte log entry
0809 23:21:11.799085 (+194850us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:11.983622 (+184537us) log.cc:842] Serialized 14 byte log entry
0809 23:21:12.852753 (+869131us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b18004df6e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:12.853157 (+   404us) log.cc:842] Serialized 14 byte log entry
0809 23:21:12.853360 (+   203us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.853439 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:12.853579 (+   140us) raft_consensus.cc:1671] finished
0809 23:21:12.853594 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:12.853742 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":152787,"prepare.queue_time_us":281704,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":117,"raft.queue_time_us":892823,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":10158,"spinlock_wait_cycles":17134976,"thread_start_us":286838,"threads_started":2,"wal-append.queue_time_us":515485}
I20260809 23:21:13.038393 24545 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:13.038794  2315 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:13.040143  2192 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:13.051370 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 85441.5us
I20260809 23:21:13.055478 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:13.073616 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49913.6us
I20260809 23:21:13.190749 25370 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:13.191140 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:13.130353 24424 rpcz_store.cc:276] 0809 23:21:11.780107 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.869027 (+1088920us) service_pool.cc:227] Handling call
0809 23:21:12.927883 (+ 58856us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.996295 (+ 68412us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.996507 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.133746 24423 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260809 23:21:13.193954 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:13.188524 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:13.206638 25371 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:13.188913 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:13.231194 25371 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:13.231431 24542 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
W20260809 23:21:13.135135  2176 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.374s	user 0.025s	sys 0.006s
I20260809 23:21:13.187414 24314 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:13.201334 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.189266 25366 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:13.207093 25369 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:13.034765  2216 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.206530 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.219815 24541 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:13.200419 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2926) took 1186 ms. Trace:
I20260809 23:21:13.248529 24423 rpcz_store.cc:276] 0809 23:21:12.013857 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.999964 (+986107us) service_pool.cc:227] Handling call
0809 23:21:13.133482 (+133518us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:13.200151 (+ 66669us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.200361 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.233410 25369 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:13.182247 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.012s	sys 0.032s
W20260809 23:21:13.246470 24269 leader_election.cc:334] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.845s (SENT)
W20260809 23:21:13.251339 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.013s	sys 0.033s
W20260809 23:21:13.252745 24269 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.867s (SENT)
I20260809 23:21:13.253298 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:13.220428  1539 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.245961 25371 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.258515 25371 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:13.258134  1539 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.259441  2367 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Leader election won for term 14
I20260809 23:21:13.260499  1539 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:13.263545 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 290978us
W20260809 23:21:13.258800  2342 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260809 23:21:13.265563 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:13.042057  2194 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:13.271633 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:13.272550 25368 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:13.263466 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3349) took 1140 ms. Trace:
W20260809 23:21:13.220659  2238 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.728s	user 0.015s	sys 0.000s
W20260809 23:21:13.274631 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71987.9us
I20260809 23:21:13.042207  1886 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:13.247066 24424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:40954
W20260809 23:21:13.247848  2176 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.497s	user 0.025s	sys 0.006s
I20260809 23:21:13.203575  2240 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:13.306396 25370 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:21:13.284802 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 278334us
I20260809 23:21:13.273887 25369 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.046904  2171 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.195s
I20260809 23:21:13.235528  2192 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:13.280939  2230 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:13.313019  1746 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:13.310659  2382 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.006s
W20260809 23:21:13.316362  1794 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.002s
I20260809 23:21:13.136521  2237 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:13.136271  2222 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:13.324064 25366 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:21:13.347273 25365 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:21:13.287352 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.338559 25370 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:13.201756  2204 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.137387  2108 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:13.246150  2221 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.318027  2120 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:13.369542 25369 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:13.368221  2380 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
W20260809 23:21:13.338368  2250 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.655s	user 0.006s	sys 0.005s
W20260809 23:21:13.328840  2318 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
W20260809 23:21:13.348282  2180 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.587s	user 0.023s	sys 0.040s
I20260809 23:21:13.408376 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.310680  2238 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.433553 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.201421  1977 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.420693  2204 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.435843  2216 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:13.436388  2216 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:13.437175  2204 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:13.436877  2216 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:13.439613  2373 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.249s	user 0.000s	sys 0.000s
W20260809 23:21:13.440805  2364 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260809 23:21:13.443154 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79624.2us
I20260809 23:21:13.314983 25371 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260809 23:21:13.448414 25371 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 12
I20260809 23:21:13.448773 25371 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260809 23:21:13.449177 25371 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:13.457695 25369 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:13.457528 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:13.430003  2180 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.675s	user 0.023s	sys 0.041s
I20260809 23:21:13.468361 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:13.475658 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:13.480005 24429 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:13.484834 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:13.485633 24321 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:13.406332  2237 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.465816  2323 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:13.466292  2216 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:13.502530 24269 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.183s (SENT)
I20260809 23:21:13.503031 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:13.504668 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:13.481066  2382 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:13.491540  2204 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:13.412984  2222 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.219751  2127 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.419994  2221 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.537357 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1625) took 1971 ms. Trace:
I20260809 23:21:13.537897 24317 rpcz_store.cc:276] 0809 23:21:11.566341 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.580600 (+ 14259us) service_pool.cc:227] Handling call
0809 23:21:11.596256 (+ 15656us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:11.616200 (+ 19944us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.536976 (+1920776us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540045b620. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:13.537282 (+   306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311462784}
I20260809 23:21:13.510638  2216 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:13.554622 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.555764  1886 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:13.556325  2250 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:13.512669  2204 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:13.578260  2217 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.007s	sys 0.015s
W20260809 23:21:13.559418  2056 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.000s
I20260809 23:21:13.557119  2345 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:13.560135 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2917) took 2226 ms. Trace:
I20260809 23:21:13.583611 24428 rpcz_store.cc:276] 0809 23:21:11.333305 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.617844 (+1284539us) service_pool.cc:227] Handling call
0809 23:21:12.618142 (+   298us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.618328 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:12.943175 (+324847us) log.cc:842] Serialized 14 byte log entry
0809 23:21:13.559778 (+616603us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:13.559836 (+    58us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:13.559883 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.559947 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:13.560077 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":639800,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":56887,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":17381,"thread_start_us":940078,"threads_started":3}
I20260809 23:21:13.560434 24321 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:13.571866  2376 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.504s	user 0.000s	sys 0.001s
I20260809 23:21:13.260516  2315 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.553920  2322 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260809 23:21:13.586938  2384 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
W20260809 23:21:13.601764  2265 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.617s	user 0.008s	sys 0.000s
I20260809 23:21:13.602646  2238 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:13.603392  2238 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260809 23:21:13.605007  2115 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.595s	user 0.009s	sys 0.000s
I20260809 23:21:13.605499  2115 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:13.605994  2362 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.685s	user 0.007s	sys 0.000s
I20260809 23:21:13.606554  2362 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.607362 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2920) took 2273 ms. Trace:
I20260809 23:21:13.607793 24534 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
W20260809 23:21:13.513931  2268 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.892s	user 0.007s	sys 0.009s
I20260809 23:21:13.608245  2345 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
W20260809 23:21:13.572669  2372 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.351s	user 0.000s	sys 0.001s
I20260809 23:21:13.608842  2362 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:13.507658  2323 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:13.047947  2245 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.585880 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:13.587222 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.611312 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:13.612196 24321 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:13.613060 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:13.583293  2217 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.007s	sys 0.015s
I20260809 23:21:13.613777  2217 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:13.585868 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:13.589089 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:13.590624 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95956.5us
I20260809 23:21:13.615257 24428 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:13.604784  2238 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:13.577287 25368 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:13.607722 24422 rpcz_store.cc:276] 0809 23:21:11.333876 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.637384 (+1303508us) service_pool.cc:227] Handling call
0809 23:21:12.682868 (+ 45484us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:12.711098 (+ 28230us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.711144 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:12.711670 (+   526us) log.cc:842] Serialized 26 byte log entry
0809 23:21:12.956373 (+244703us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:12.956406 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:12.956479 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:13.607128 (+650649us) raft_consensus.cc:1671] finished
0809 23:21:13.607152 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:13.607293 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":189908,"threads_started":1,"wal-append.queue_time_us":754124}
W20260809 23:21:13.585234  2190 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.979s	user 0.010s	sys 0.005s
I20260809 23:21:13.622673  2115 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:13.622828  2217 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 17.17, Last appended by leader: 17, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.625687  2268 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:21:13.273597 24431 rpcz_store.cc:276] 0809 23:21:12.123107 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.262488 (+1139381us) service_pool.cc:227] Handling call
0809 23:21:13.262795 (+   307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.263136 (+   341us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:13.263167 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:13.263205 (+    38us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:13.263240 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.263294 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:13.263413 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3451}
I20260809 23:21:13.646006  2148 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:13.668092  2333 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Stepping down as leader of term 13
I20260809 23:21:13.671681 24314 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:13.676609  2154 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:13.680529  1770 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:13.681800  2377 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.479s	user 0.001s	sys 0.001s
W20260809 23:21:13.685160 24425 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.022s	sys 0.071s
W20260809 23:21:13.787067 24425 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.022s	sys 0.071s
I20260809 23:21:13.787108 24431 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:13.695533 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.003s	sys 0.029s
I20260809 23:21:13.520187  2382 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.588397  2204 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:13.591959  2188 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.584977  2216 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:13.771579 24422 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260809 23:21:13.785343 24314 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
W20260809 23:21:13.788352 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.004s	sys 0.029s
I20260809 23:21:13.798318 24314 pending_rounds.cc:76] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 13
I20260809 23:21:13.798553 24314 pending_rounds.cc:97] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260809 23:21:13.788127 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2932) took 1162 ms. Trace:
I20260809 23:21:13.798878 24431 rpcz_store.cc:276] 0809 23:21:12.626189 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.748433 (+1122244us) service_pool.cc:227] Handling call
0809 23:21:13.786942 (+ 38509us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.787883 (+   941us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.788077 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.583384  1886 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.585835  2250 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.792744 24543 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:13.795233 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1623) took 2599 ms. Trace:
I20260809 23:21:13.797794 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2931) took 1172 ms. Trace:
W20260809 23:21:13.762985  2030 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.000s	user 0.002s	sys 0.010s
I20260809 23:21:13.748067  2032 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:13.798785 24314 raft_consensus.cc:2866] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:13.595685  2056 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.806550 24320 rpcz_store.cc:276] 0809 23:21:11.196086 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:11.196409 (+   323us) service_pool.cc:227] Handling call
0809 23:21:11.196889 (+   480us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.473139 (+1276250us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.794843 (+1321704us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003854d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:13.795171 (+   328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2914283392}
I20260809 23:21:13.808511 24321 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260809 23:21:13.777449  1429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.029s	sys 0.043s
W20260809 23:21:13.797277  2006 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.006s	sys 0.047s
I20260809 23:21:13.810626 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1630) took 1783 ms. Trace:
I20260809 23:21:13.822192 24312 rpcz_store.cc:276] 0809 23:21:12.027140 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.027293 (+   153us) service_pool.cc:227] Handling call
0809 23:21:12.027564 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.369844 (+342280us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.810392 (+1440548us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:13.810563 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684167552}
I20260809 23:21:13.761279  2148 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:13.870764 25370 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:13.817148 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2783) took 1269 ms. Trace:
I20260809 23:21:13.871726 25369 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:13.817317 24431 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:13.872912 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2933) took 1235 ms. Trace:
I20260809 23:21:13.873253 24431 rpcz_store.cc:276] 0809 23:21:12.637250 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.799773 (+1162523us) service_pool.cc:227] Handling call
0809 23:21:13.817099 (+ 17326us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.872647 (+ 55548us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.872852 (+   205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.806958 24422 rpcz_store.cc:276] 0809 23:21:12.625892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.697566 (+1071674us) service_pool.cc:227] Handling call
0809 23:21:13.771137 (+ 73571us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.797515 (+ 26378us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.797730 (+   215us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.785526  1770 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:13.777251  2333 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:13.748082  1527 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:13.820019 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:13.879892 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.880703 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:13.716166  2190 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:13.871291 24323 rpcz_store.cc:276] 0809 23:21:12.547917 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.548123 (+   206us) service_pool.cc:227] Handling call
0809 23:21:12.556122 (+  7999us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:12.621561 (+ 65439us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.621594 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:12.622038 (+   444us) log.cc:842] Serialized 26 byte log entry
0809 23:21:13.251820 (+629782us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:13.251856 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.801609 (+549753us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b54001afc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:13.801682 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:13.801752 (+    70us) raft_consensus.cc:1671] finished
0809 23:21:13.801772 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:13.817065 (+ 15293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2463,"raft.queue_time_us":1542,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":28228,"spinlock_wait_cycles":149737344,"thread_start_us":629487,"threads_started":1,"wal-append.queue_time_us":860596}
I20260809 23:21:13.882165 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.886193 25370 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:13.886482 24539 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:13.887518 25363 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
W20260809 23:21:13.871069  1429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.029s	sys 0.044s
I20260809 23:21:13.897882 25369 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
W20260809 23:21:13.870638  2006 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.006s	sys 0.047s
I20260809 23:21:13.893045 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:13.904505 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.905987 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:13.895314 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:13.898950 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.908706 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2794) took 1218 ms. Trace:
I20260809 23:21:13.909126 24309 rpcz_store.cc:276] 0809 23:21:12.690088 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.690340 (+   252us) service_pool.cc:227] Handling call
0809 23:21:12.690695 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.143650 (+452955us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.908461 (+764811us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:13.908648 (+   187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":699454336}
W20260809 23:21:13.913796  2371 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.711s	user 0.000s	sys 0.002s
I20260809 23:21:13.926646 24430 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:13.928483 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.929313 24431 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:13.930123 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.931653 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.932365 24431 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:13.807029  2188 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.937412 24431 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:13.879832  2238 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:13.845677  2032 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.815846  2030 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:13.964007 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.014s	sys 0.052s
I20260809 23:21:13.965839 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2934) took 1309 ms. Trace:
I20260809 23:21:13.966619 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:13.969242 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.014s	sys 0.052s
I20260809 23:21:13.903251  2237 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:13.922433  2320 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:13.969628 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 835569us
I20260809 23:21:13.945076  2221 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:13.970155 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.970253 24431 rpcz_store.cc:276] 0809 23:21:12.656408 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.936812 (+1280404us) service_pool.cc:227] Handling call
0809 23:21:13.937092 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.952433 (+ 15341us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.965781 (+ 13348us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:13.972819 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.969671 25369 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:21:13.975149 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:13.991501 24422 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:13.996217 24535 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:13.996361 24431 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260809 23:21:13.997485 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:13.997612 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:13.964579  2222 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:14.002034 25366 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:14.007862 25366 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:14.002808 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.010344 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.013926 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.008889 25366 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.026721 25366 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:14.031597  1886 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:14.032090  2412 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260809 23:21:14.033766  2192 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.697s	user 0.010s	sys 0.002s
I20260809 23:21:14.034186  2192 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:14.034341  2056 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.036695  2032 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.001338  1527 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.001410  2127 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.040205 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1632) took 1098 ms. Trace:
I20260809 23:21:14.040661 24313 rpcz_store.cc:276] 0809 23:21:12.942183 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.943743 (+  1560us) service_pool.cc:227] Handling call
0809 23:21:12.944199 (+   456us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:12.944496 (+   297us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.944530 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:12.967092 (+ 22562us) log.cc:842] Serialized 26 byte log entry
0809 23:21:13.521204 (+554112us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:13.521255 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.521488 (+   233us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:14.039929 (+518441us) raft_consensus.cc:1671] finished
0809 23:21:14.039959 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.040131 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":469762,"threads_started":1,"wal-append.queue_time_us":888222}
W20260809 23:21:14.053790 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 497225us
I20260809 23:21:14.046320 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:14.000741  2371 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260809 23:21:14.069865 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:14.076002 25373 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 17. Ops: []
I20260809 23:21:14.005409  2250 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:13.971987  2238 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.084242 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.084722 25369 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:14.085364 25373 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260809 23:21:14.070290 24428 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
W20260809 23:21:14.057897   877 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.111s	user 0.006s	sys 0.007s
W20260809 23:21:14.062719  2242 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.034s	sys 0.096s
I20260809 23:21:14.054996  2222 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.108717 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:14.108932 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
W20260809 23:21:14.092021  2395 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260809 23:21:14.114199 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.114907 25373 raft_consensus.cc:2362] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16. Re-sending same reply.
I20260809 23:21:13.973819  2216 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.117231 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:13.972532  2148 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:13.989957  1770 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:14.100198  2242 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.034s	sys 0.097s
W20260809 23:21:14.099756 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.000s	sys 0.022s
I20260809 23:21:14.125429 24323 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260809 23:21:14.132120 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.000s	sys 0.023s
I20260809 23:21:14.126225 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:14.128340 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:14.128450 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:14.110337  2315 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.006s	sys 0.019s
W20260809 23:21:14.139114  2315 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.008s	sys 0.019s
I20260809 23:21:14.139729  2315 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
W20260809 23:21:14.140369  2417 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260809 23:21:14.142299 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:14.142964 24321 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
W20260809 23:21:14.144170  2399 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260809 23:21:14.144390 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:14.145416 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2925) took 2291 ms. Trace:
I20260809 23:21:14.099340  2250 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:14.110123  2386 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.055s	sys 0.067s
W20260809 23:21:14.132409 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 543683us
I20260809 23:21:14.115762  2382 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.118067  2245 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:14.118875  1352 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.011s	sys 0.015s
I20260809 23:21:13.988729  2333 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.147215 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.148026 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:14.095861   877 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.111230  2419 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:14.137109 25370 raft_consensus.cc:2396] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:14.160879 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:14.161809 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.162245 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:14.131491  2240 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.793s	user 0.013s	sys 0.001s
I20260809 23:21:14.110539  1429 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260809 23:21:14.145790 24427 rpcz_store.cc:276] 0809 23:21:11.853592 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.960052 (+1106460us) service_pool.cc:227] Handling call
0809 23:21:12.960397 (+   345us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:12.962181 (+  1784us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.962208 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:12.962637 (+   429us) log.cc:842] Serialized 26 byte log entry
0809 23:21:13.798514 (+835877us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:13.798552 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.798616 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:14.145193 (+346577us) raft_consensus.cc:1671] finished
0809 23:21:14.145217 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.145343 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":835581,"threads_started":1,"wal-append.queue_time_us":1006482}
W20260809 23:21:14.150079  2386 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.055s	sys 0.068s
W20260809 23:21:14.157822 25372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.023s	sys 0.023s
I20260809 23:21:14.118140  1776 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
W20260809 23:21:14.157227  1352 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.011s	sys 0.015s
I20260809 23:21:14.165427  2127 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:14.148823  2250 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.162405 25371 raft_consensus.cc:2362] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18. Re-sending same reply.
I20260809 23:21:14.184301 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.185119  2338 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:14.185412 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.165982  2382 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260809 23:21:14.166420  2192 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:14.187824 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3267) took 1361 ms. Trace:
I20260809 23:21:14.169036 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:14.189707 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:14.166150 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1577) took 1291 ms. Trace:
I20260809 23:21:14.118330  2320 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260809 23:21:14.153923  1431 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:14.188331 24535 rpcz_store.cc:276] 0809 23:21:12.826778 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.976915 (+150137us) service_pool.cc:227] Handling call
0809 23:21:12.977214 (+   299us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.996704 (+1019490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.187592 (+190888us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:14.187765 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2328875264}
W20260809 23:21:14.174836 25372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.023s	sys 0.023s
I20260809 23:21:14.152802  1569 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.198463 24424 raft_consensus.cc:595] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:14.198972 24424 raft_consensus.cc:603] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:14.199182 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:14.199795 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3347) took 2335 ms. Trace:
I20260809 23:21:14.200698 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:14.112016  2222 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.112987  2420 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260809 23:21:14.164933  2333 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 13 FOLLOWER]: Advancing to term 14
I20260809 23:21:14.178521 24423 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:14.176064  2171 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:14.159991  2148 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:14.190969 25375 rpcz_store.cc:276] 0809 23:21:12.874649 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.874846 (+   197us) service_pool.cc:227] Handling call
0809 23:21:12.875297 (+   451us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:12.875726 (+   429us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:12.875766 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:12.876219 (+   453us) log.cc:842] Serialized 26 byte log entry
0809 23:21:13.376520 (+500301us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:13.376566 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.376654 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:14.165935 (+789281us) raft_consensus.cc:1671] finished
0809 23:21:14.165958 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.166093 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2379,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":499937,"threads_started":1,"wal-append.queue_time_us":913920}
I20260809 23:21:14.176311  2052 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:14.208324 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1572) took 1330 ms. Trace:
I20260809 23:21:14.213476 25372 rpcz_store.cc:276] 0809 23:21:12.878146 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.906555 (+ 28409us) service_pool.cc:227] Handling call
0809 23:21:12.906911 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.907176 (+   265us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:12.922271 (+ 15095us) log.cc:842] Serialized 14 byte log entry
0809 23:21:13.639054 (+716783us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b1800442420. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:21:14.207901 (+568847us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:14.207970 (+    69us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:14.208038 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.208128 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.208258 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12083,"prepare.queue_time_us":14929,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":15839,"raft.queue_time_us":13391,"spinlock_wait_cycles":10123136,"thread_start_us":556450,"threads_started":1}
I20260809 23:21:14.182730  2190 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:14.187896  2030 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:14.225343 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.200263 24425 rpcz_store.cc:276] 0809 23:21:11.864432 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.973984 (+1109552us) service_pool.cc:227] Handling call
0809 23:21:12.974279 (+   295us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:12.974472 (+   193us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:13.787623 (+813151us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.198184 (+410561us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:14.198252 (+    68us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:14.198320 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.199582 (+  1262us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.199731 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":283878,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":95,"thread_start_us":1197962,"threads_started":2}
I20260809 23:21:14.227080 24425 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:14.159385  2245 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.229540 25372 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:14.228616  2416 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:14.196790  2268 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:14.238149 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.206310  2338 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:14.194201  2192 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:14.254797 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.588s (SENT) [suppressed 19 similar messages]
I20260809 23:21:14.349859 24425 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260809 23:21:14.442118 25371 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:14.436164 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.445212 24435 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 14. Current term is 15. Ops: []
I20260809 23:21:14.199462  1731 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:14.179304  2017 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.045s
I20260809 23:21:14.444303 25375 raft_consensus.cc:2362] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18. Re-sending same reply.
W20260809 23:21:14.449573 25368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.016s	sys 0.023s
W20260809 23:21:14.501072 25368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.016s	sys 0.023s
I20260809 23:21:14.487643 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1535) took 1222 ms. Trace:
I20260809 23:21:14.210858  2171 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:14.502583 24540 rpcz_store.cc:276] 0809 23:21:13.264780 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.283699 (+ 18919us) service_pool.cc:227] Handling call
0809 23:21:13.283976 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.284185 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:13.351321 (+ 67136us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.445883 (+1094562us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:14.445937 (+    54us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:14.445978 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.446051 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.487565 (+ 41514us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1097991,"prepare.queue_time_us":56103,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":1147948}
I20260809 23:21:14.445030  2192 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:14.448316  2431 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:14.212872  1431 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:14.498142 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:14.175173  2240 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.448505  2338 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:14.439049  2188 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.448899  1839 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.447532  2367 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:14.438323  2052 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:14.439271  2268 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:14.208063  1569 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.516155  1731 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.643046  2115 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:14.500061  2119 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:14.798957 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:14.652630 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.602532  2171 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:14.501883  1429 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:14.444521 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3351) took 2034 ms (client timeout 1870 ms). Trace:
W20260809 23:21:14.717582  2217 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.012s	sys 0.062s
I20260809 23:21:14.728482 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:14.729897 25366 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.703s	user 0.010s	sys 0.007s
I20260809 23:21:14.831092 24544 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:14.494225 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3272) took 1579 ms. Trace:
I20260809 23:21:14.618839  2238 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:14.488041  1485 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.202s
I20260809 23:21:14.448195  2416 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.889206 24544 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:14.737246 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3273) took 1417 ms. Trace:
I20260809 23:21:14.631415  2006 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:14.734189  2198 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.741957 24422 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18.
I20260809 23:21:14.645850  2221 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.650316  2056 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:14.651814  2431 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:14.740577  1839 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.645879  2249 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.101s
I20260809 23:21:14.616405  2436 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.244s
W20260809 23:21:14.794557 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.028s	sys 0.058s
W20260809 23:21:14.952760 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.028s	sys 0.058s
I20260809 23:21:14.953918 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1318) took 1079 ms. Trace:
W20260809 23:21:14.778378 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.007s	sys 0.003s
W20260809 23:21:14.955191 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.007s	sys 0.004s
W20260809 23:21:14.796659 24321 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.006s	sys 0.005s
W20260809 23:21:14.955749 24321 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.006s	sys 0.006s
W20260809 23:21:14.790203 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.000s	sys 0.011s
W20260809 23:21:14.956339 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.000s	sys 0.011s
I20260809 23:21:14.726054  2200 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20260809 23:21:14.741621  2243 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:14.821669 24429 rpcz_store.cc:269] 0809 23:21:12.410525 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.412240 (+  1715us) service_pool.cc:227] Handling call
0809 23:21:14.444113 (+2031873us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002f94d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:14.444451 (+   338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2328365312}
I20260809 23:21:14.833797 24539 rpcz_store.cc:276] 0809 23:21:12.914969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.998722 (+ 83753us) service_pool.cc:227] Handling call
0809 23:21:12.999110 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.891502 (+892392us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.484277 (+592775us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b54000b4d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:14.494146 (+  9869us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2040144640}
I20260809 23:21:14.890823 24542 rpcz_store.cc:276] 0809 23:21:13.319405 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.319559 (+   154us) service_pool.cc:227] Handling call
0809 23:21:13.648405 (+328846us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.648646 (+   241us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:13.648682 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:13.648737 (+    55us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:13.648778 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:13.648851 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.716801 (+1067950us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540029ca50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:14.737177 (+ 20376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":755339008}
I20260809 23:21:14.786149  2115 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:14.811467  2432 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.175s	user 0.002s	sys 0.003s
W20260809 23:21:14.889312 25365 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.000s	sys 0.020s
W20260809 23:21:14.967970 25365 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.001s	sys 0.025s
W20260809 23:21:14.912734 25373 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.007s	sys 0.011s
I20260809 23:21:14.918169 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:14.968668 25373 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.007s	sys 0.011s
W20260809 23:21:14.926679 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.005s	sys 0.018s
W20260809 23:21:14.969215 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.005s	sys 0.018s
W20260809 23:21:14.969702 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 789728us
W20260809 23:21:14.973505 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 469421us
I20260809 23:21:14.980443 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:14.827690  1776 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:14.957008 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:14.832535  2116 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:14.823212   877 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:14.966238  2250 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.965910  2428 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:14.914916  1527 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.010s	sys 0.010s
W20260809 23:21:14.825768  2217 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.012s	sys 0.062s
W20260809 23:21:14.985846 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.000s	sys 0.049s
I20260809 23:21:14.980638  2144 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.987802  2127 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:14.960143  2419 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:14.993008 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.033s	sys 0.037s
I20260809 23:21:15.061352 24534 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:14.993738 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.833359  1429 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.954311 24431 rpcz_store.cc:276] 0809 23:21:13.874672 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.006581 (+131909us) service_pool.cc:227] Handling call
0809 23:21:14.006896 (+   315us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:14.029454 (+ 22558us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:14.029862 (+   408us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.953413 (+923551us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.953559 (+   146us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:14.953603 (+    44us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:14.953650 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.953721 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.953856 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1529,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":922880,"threads_started":1}
W20260809 23:21:14.964214 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.024s	sys 0.024s
W20260809 23:21:15.065321 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.024s	sys 0.024s
I20260809 23:21:14.962525  2056 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.097232  2056 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:15.054468 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.000s	sys 0.049s
I20260809 23:21:14.972466  2243 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.098150 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.098992 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:14.987056 24429 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:15.099465  2243 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.099390 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.100126  2243 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.100850 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:14.955937  2221 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.992995  2188 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.102967  2298 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:15.103394  2298 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:15.103251  2188 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.104187  2168 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:15.104296  2298 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.105325  2298 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:14.914587  2006 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.105633  2188 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.107285  2006 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:14.991917 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1581) took 1171 ms. Trace:
W20260809 23:21:15.065661 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 902259us
I20260809 23:21:14.980615  2435 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:15.065810 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:14.949177  2198 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.077721  1429 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.060619  2127 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:15.080332  2115 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:15.086424  2204 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.276s	user 0.011s	sys 0.001s
W20260809 23:21:15.090590  2148 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.874s	user 0.008s	sys 0.000s
I20260809 23:21:15.006186  2222 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.092092  2416 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.093132  1569 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:14.976652  2367 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:15.135772  2315 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.008s	sys 0.006s
I20260809 23:21:15.100641  2243 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.157907  2243 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:15.158466 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.058s	sys 0.112s
W20260809 23:21:15.158794 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.058s	sys 0.112s
W20260809 23:21:15.158954  2315 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.009s	sys 0.006s
I20260809 23:21:15.161151 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.050534  2428 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.162376 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.163297 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.163796  2428 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.163985 24535 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:15.164444  2115 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:15.164888 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.165654 24535 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:15.166204 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33980 (request call id 16) took 1059 ms. Trace:
I20260809 23:21:15.166991 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3329) took 2117 ms. Trace:
I20260809 23:21:15.167433 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1597) took 1055 ms. Trace:
I20260809 23:21:15.168025 25365 rpcz_store.cc:276] 0809 23:21:14.111950 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.112224 (+   274us) service_pool.cc:227] Handling call
0809 23:21:14.112838 (+   614us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:14.135172 (+ 22334us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:14.135219 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:14.156750 (+ 21531us) log.cc:842] Serialized 26 byte log entry
0809 23:21:14.968324 (+811574us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:14.968370 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.968433 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:15.167183 (+198750us) raft_consensus.cc:1671] finished
0809 23:21:15.167202 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.167371 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28233,"raft.queue_time_us":41116,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":7530,"thread_start_us":791686,"threads_started":1,"wal-append.queue_time_us":992600}
I20260809 23:21:15.169812 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.170413  2190 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:14.820510  2338 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:15.109076  2433 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.370s	user 0.000s	sys 0.001s
W20260809 23:21:15.109340 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.003s	sys 0.028s
W20260809 23:21:15.176255 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.961s	user 0.004s	sys 0.028s
W20260809 23:21:15.111248  2423 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260809 23:21:15.176977 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40954 (request call id 6) took 2161 ms. Trace:
I20260809 23:21:15.112790 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:15.123103  2435 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:15.177433  1839 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.178182  1839 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.001653  2017 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.003129  2245 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.116508 25368 rpcz_store.cc:276] 0809 23:21:13.820777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.824505 (+  3728us) service_pool.cc:227] Handling call
0809 23:21:13.824961 (+   456us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.825225 (+   264us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:14.501529 (+676304us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.989604 (+488075us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:14.989677 (+    73us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:14.989740 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.991703 (+  1963us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:14.991858 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1127218,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"spinlock_wait_cycles":1713920,"thread_start_us":960008,"threads_started":2}
I20260809 23:21:14.958884  2052 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:15.135515 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.013s	sys 0.000s
I20260809 23:21:14.959681  2268 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.145321 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:14.889956  1431 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:14.998019  1776 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.065960  2144 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.153456 25363 raft_consensus.cc:2354] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:21:15.151062  2240 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.163394  2056 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.143240  2204 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.107614  2221 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:15.061512 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.034s	sys 0.038s
I20260809 23:21:15.164942  2115 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.132416 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:15.324862 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.252097  2419 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.172369 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.173491 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.172521 25370 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:15.170889 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.167373 24311 rpcz_store.cc:276] 0809 23:21:13.049667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.684151 (+634484us) service_pool.cc:227] Handling call
0809 23:21:13.684573 (+   422us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.811611 (+127038us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:13.811652 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:13.811697 (+    45us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:13.811728 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.166675 (+1354947us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540019d7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:15.166721 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.166891 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":291702016}
I20260809 23:21:15.176438  2198 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.176666  2127 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:15.056061  1527 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.912s	user 0.019s	sys 0.011s
W20260809 23:21:15.148780  2434 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.415s	user 0.000s	sys 0.001s
W20260809 23:21:15.116966  2168 consensus_queue.cc:1159] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 18 status { last_received { term: 18 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 18 index: 11 } }
I20260809 23:21:14.830781  2119 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.166592 25369 rpcz_store.cc:276] 0809 23:21:14.107147 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.117975 (+ 10828us) service_pool.cc:227] Handling call
0809 23:21:15.165646 (+1047671us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400860954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:15.166053 (+   407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2003415936}
I20260809 23:21:15.177323 24424 rpcz_store.cc:276] 0809 23:21:13.053767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.200368 (+146601us) service_pool.cc:227] Handling call
0809 23:21:15.176764 (+1976396us) spinlock_profiling.cc:243] Waited 891 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:15.176888 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2049297152}
W20260809 23:21:15.180847 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68147.7us
I20260809 23:21:15.345367 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:15.345508 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.346190 24429 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:15.200875 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.151337  2222 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.123351  2337 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260809 23:21:15.251677 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1321) took 1376 ms. Trace:
I20260809 23:21:15.176474  2171 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.179437  2006 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.277544 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.355547 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:15.173377 24546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.938s	user 0.023s	sys 0.062s
W20260809 23:21:15.356390 24546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.121s	user 0.023s	sys 0.062s
I20260809 23:21:15.355969 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.217962  2421 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:15.296873  2431 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.292294  2463 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.025s
I20260809 23:21:15.292840  2451 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:15.303577  2437 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.356s	user 0.000s	sys 0.001s
I20260809 23:21:15.309237  1776 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.310827  2428 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:15.292512  2056 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.256181  1539 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.262835  1429 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:15.200754  2030 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:15.377935 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.382143 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.382524 25368 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:15.339685 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.384266 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.001930  2116 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.276587  2268 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:15.352171 24539 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:15.353214 24427 rpcz_store.cc:276] 0809 23:21:13.875554 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.212039 (+336485us) service_pool.cc:227] Handling call
0809 23:21:14.212574 (+   535us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:14.235738 (+ 23164us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:14.491586 (+255848us) log.cc:842] Serialized 14 byte log entry
0809 23:21:15.251229 (+759643us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.251290 (+    61us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:15.251348 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.251434 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.251603 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45405,"prepare.queue_time_us":255811,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":8207,"raft.queue_time_us":58422,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":28107,"thread_start_us":589558,"threads_started":1,"wal-append.queue_time_us":798302}
I20260809 23:21:15.178350  2435 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.392148 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.219895  1839 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.241464 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:15.362679  2461 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:15.395239 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 153878us
I20260809 23:21:15.395612 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:15.395962 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:15.292269 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.03759e+06us
W20260809 23:21:15.398486 24269 connection.cc:232] RPC call timeout handler was delayed by -1.1304010020103306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260809 23:21:15.357620 24431 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:15.399833 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.358875 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.357193 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3271) took 2446 ms. Trace:
I20260809 23:21:15.400786 24546 rpcz_store.cc:276] 0809 23:21:12.911064 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.984809 (+ 73745us) service_pool.cc:227] Handling call
0809 23:21:13.015268 (+ 30459us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.015657 (+   389us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:13.189276 (+173619us) log.cc:842] Serialized 14 byte log entry
0809 23:21:14.234593 (+1045317us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b18002337a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:15.356786 (+1122193us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.356855 (+    69us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:15.356909 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.356996 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.357135 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":139770,"prepare.queue_time_us":173655,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":12604,"raft.queue_time_us":1256274,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":25019,"spinlock_wait_cycles":26611968,"thread_start_us":1121779,"threads_started":1}
W20260809 23:21:15.402622 24269 connection.cc:232] RPC call timeout handler was delayed by -1.1170149210374802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20260809 23:21:15.403079 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0863010019529611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260809 23:21:15.369874 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.272975  2144 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.404162 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:15.403609 24269 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.934s (SENT)
W20260809 23:21:15.355938  2192 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
I20260809 23:21:15.411226  2192 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.374522 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.376071 25365 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:15.351492  2393 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:15.358129  2242 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.389856 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:15.394188 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.361156 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.402801 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:15.414696 24546 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:15.377933  2217 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:15.360458  2006 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.414397 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:15.414428 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.392812  2405 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:15.343688  2017 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.334863  2419 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.418303 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.404160  2428 consensus_queue.cc:1225] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:21:15.417716 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.412343 25369 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:15.419458 24431 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:15.422412 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.417685 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2796) took 2426 ms. Trace:
I20260809 23:21:15.436163 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1326) took 1530 ms. Trace:
I20260809 23:21:15.422680 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:15.403844  2466 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260809 23:21:15.355437  2222 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.425141 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3359) took 1442 ms. Trace:
I20260809 23:21:15.436501 24431 rpcz_store.cc:276] 0809 23:21:13.905459 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.418848 (+1513389us) service_pool.cc:227] Handling call
0809 23:21:15.419240 (+   392us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.435926 (+ 16686us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.436112 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:15.439947 24427 rpcz_store.cc:276] 0809 23:21:13.982424 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.424358 (+1441934us) service_pool.cc:227] Handling call
0809 23:21:15.424603 (+   245us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.424789 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.424825 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.424877 (+    52us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:15.424925 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.424991 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.425092 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:15.441017 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2942) took 1288 ms. Trace:
I20260809 23:21:15.412369  2221 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:15.426573 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.441340 24431 rpcz_store.cc:276] 0809 23:21:14.152897 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.440273 (+1287376us) service_pool.cc:227] Handling call
0809 23:21:15.440560 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.440707 (+   147us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.440733 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.440770 (+    37us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:15.440798 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.440847 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.440971 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:15.444684 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:15.445441 24546 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:15.445644 24309 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:15.429981 25372 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:15.431988 24429 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:15.432324 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.432577 24425 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 17. Ops: []
I20260809 23:21:15.433750 24311 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:21:15.433894 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.413865 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.456943 24427 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:15.436393 25368 raft_consensus.cc:2396] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260809 23:21:15.436426 24317 rpcz_store.cc:276] 0809 23:21:12.991037 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.621964 (+630927us) service_pool.cc:227] Handling call
0809 23:21:13.622393 (+   429us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:13.622649 (+   256us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:13.622682 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:13.623229 (+   547us) log.cc:842] Serialized 26 byte log entry
0809 23:21:14.114657 (+491428us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:14.114715 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:14.114805 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:15.417463 (+1302658us) raft_consensus.cc:1671] finished
0809 23:21:15.417487 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.417615 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":491065,"threads_started":1,"wal-append.queue_time_us":1192334}
I20260809 23:21:15.407087  2198 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:15.436064 24535 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:15.459563 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.439091 24542 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260809 23:21:15.461149 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.428828 24424 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:15.463109 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3358) took 1481 ms. Trace:
I20260809 23:21:15.463502 24424 rpcz_store.cc:276] 0809 23:21:13.982080 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.422103 (+1440023us) service_pool.cc:227] Handling call
0809 23:21:15.422385 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.462800 (+ 40415us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.463044 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:15.387285  1539 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.442679 24541 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:15.420639  2217 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:15.420830  2435 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:15.479671  2217 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.480954  2217 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:15.466478 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.469794 24309 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:15.377625  2451 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.431151  2298 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:15.504145 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.009s	sys 0.064s
W20260809 23:21:15.653321 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.009s	sys 0.065s
I20260809 23:21:15.615573  2469 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:15.507771 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.465405  2468 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.392339  1429 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:15.484031 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79947.5us
I20260809 23:21:15.516396 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.520161 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.500878 24541 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:15.523893 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.665275 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:15.423542  2242 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:15.527304  2333 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.317s	user 0.009s	sys 0.003s
I20260809 23:21:15.503985  2198 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:15.412585  2459 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.110s
I20260809 23:21:15.628237 24314 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:15.430879  2393 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:15.643113 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 451116us
I20260809 23:21:15.643240 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:15.645744 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 388374us
I20260809 23:21:15.644759 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:15.653570  2115 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.392637  2030 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.684640 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.684841 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.685681 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:15.507352 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 307190us
I20260809 23:21:15.665789 24311 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:15.673979 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:15.661823  2463 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:15.677699 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.702579 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:15.655308  2119 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.676395  2313 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:15.682037 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 439597us
I20260809 23:21:15.683910 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.714886 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:15.657188 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.012s	sys 0.021s
I20260809 23:21:15.655566  2171 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.687577 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:15.688874 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.710091 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:15.715945 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.012s	sys 0.022s
I20260809 23:21:15.716388 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:15.428297  2405 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.736447 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:15.737298 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 586971us
I20260809 23:21:15.682960  2194 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.749516 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.662015  2298 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.729285 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:15.729444 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.729475 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.801400 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.708527  2461 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:15.802460 24311 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:15.707737  2056 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.795028 24545 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 13. Current term is 14. Ops: []
W20260809 23:21:15.795142 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58777.5us
W20260809 23:21:15.717396 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.003s	sys 0.022s
I20260809 23:21:15.706671  2116 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:15.791891 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 126695us
I20260809 23:21:15.798913 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1594) took 1826 ms. Trace:
I20260809 23:21:15.709976  2192 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.670223  2468 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.712265  2006 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.707948  2115 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:15.809284 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:15.811126 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1610) took 1167 ms. Trace:
I20260809 23:21:15.809183 25366 rpcz_store.cc:276] 0809 23:21:13.972621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.972743 (+   122us) service_pool.cc:227] Handling call
0809 23:21:14.001868 (+ 29125us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:14.888171 (+886303us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:15.413902 (+525731us) log.cc:842] Serialized 14 byte log entry
0809 23:21:15.796407 (+382505us) log.cc:842] Serialized 14 byte log entry
0809 23:21:15.796517 (+   110us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.796565 (+    48us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:15.796634 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.798697 (+  2063us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.798846 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44526,"prepare.queue_time_us":83042,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":55846,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":30161,"spinlock_wait_cycles":1155840,"thread_start_us":802533,"threads_started":2}
I20260809 23:21:15.821992 25369 rpcz_store.cc:276] 0809 23:21:14.643936 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.707352 (+1063416us) service_pool.cc:227] Handling call
0809 23:21:15.811055 (+103703us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:15.808884 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:15.814634 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.003s	sys 0.022s
I20260809 23:21:15.712831  2222 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:15.720359  1839 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.737097  2405 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:15.739205  2486 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:15.846300  2451 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.846748 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1325) took 1951 ms. Trace:
I20260809 23:21:15.847116 24425 rpcz_store.cc:276] 0809 23:21:13.895739 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.495150 (+599411us) service_pool.cc:227] Handling call
0809 23:21:14.495571 (+   421us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.464779 (+969208us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.846522 (+381743us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:15.846691 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2154421632}
I20260809 23:21:15.848367 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.849475 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.850118 24425 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:15.795583  1556 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:15.852142 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.854434  2200 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:15.842265 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1316) took 1972 ms (client timeout 1918 ms). Trace:
W20260809 23:21:15.855609 24428 rpcz_store.cc:269] 0809 23:21:13.870370 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.895060 (+ 24690us) service_pool.cc:227] Handling call
0809 23:21:15.842084 (+1947024us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:15.842186 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306765696}
I20260809 23:21:15.856566 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.850787 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.858018 25375 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:15.858729 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.674142  2469 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:15.735462  2144 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.879213 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:15.880492 24428 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260809 23:21:15.882058 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:15.817600 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.741645  2204 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.822160 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:15.830314 24535 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:15.823612 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:15.892274 25366 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:15.811728  2230 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260809 23:21:15.832859  2322 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:15.835427  2487 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260809 23:21:15.904207 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82111.4us
I20260809 23:21:15.835956  2393 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:15.837157  2473 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260809 23:21:15.905023 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:15.906021 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1598) took 1789 ms. Trace:
I20260809 23:21:15.837284  1429 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:15.838573  2221 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:15.839422  2298 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:15.728917  2313 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.906375 25373 rpcz_store.cc:276] 0809 23:21:14.117021 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.121992 (+  4971us) service_pool.cc:227] Handling call
0809 23:21:14.122370 (+   378us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:14.969082 (+846712us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:14.969119 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:14.976029 (+  6910us) log.cc:842] Serialized 26 byte log entry
0809 23:21:15.420560 (+444531us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:15.420621 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.420725 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:15.905814 (+485089us) raft_consensus.cc:1671] finished
0809 23:21:15.905834 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:15.905965 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18325,"raft.queue_time_us":40018,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":1231511,"threads_started":2,"wal-append.queue_time_us":669280}
I20260809 23:21:15.805886  2194 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.839953 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2918) took 4506 ms. Trace:
I20260809 23:21:15.926837 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.805927  2461 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:15.840026 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.940268 24311 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 22 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260809 23:21:15.840946  2419 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.841341  2116 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:15.841593 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1609) took 1198 ms. Trace:
I20260809 23:21:15.803036  2017 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:15.710736  2463 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:15.842103  2333 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:15.806236  2481 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260809 23:21:15.842855   877 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:15.845007 24539 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:15.831055 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:15.861510 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:15.862066 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 175210us
I20260809 23:21:15.877771 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:15.894582 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:15.818540 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.143504 24424 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:15.854884  2200 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:15.901571 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.910602  2119 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.172s
I20260809 23:21:15.902278 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.907094 24542 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:15.912969 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:15.919807 24312 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260809 23:21:15.919873 24323 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:15.829267  2115 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:15.926720 24430 rpcz_store.cc:276] 0809 23:21:11.333556 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.632979 (+1299423us) service_pool.cc:227] Handling call
0809 23:21:12.677783 (+ 44804us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.927345 (+1249562us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.839700 (+1912355us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:15.839895 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2871486848}
I20260809 23:21:15.948098 25366 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:15.907776  2493 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:15.919414  2144 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:15.909729  1539 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.161720 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:15.953713 25363 rpcz_store.cc:276] 0809 23:21:14.643670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.100606 (+456936us) service_pool.cc:227] Handling call
0809 23:21:15.841398 (+740792us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:15.841525 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100039040}
I20260809 23:21:15.958606 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:15.924751  2204 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:16.137966  2419 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.161273 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:16.249898 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3352) took 1050 ms. Trace:
W20260809 23:21:16.163331 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 250455us
W20260809 23:21:16.165521 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 334529us
I20260809 23:21:16.253116 24541 rpcz_store.cc:276] 0809 23:21:15.199206 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.905607 (+706401us) service_pool.cc:227] Handling call
0809 23:21:15.905920 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.906137 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:15.906432 (+   295us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.249444 (+343012us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:16.249512 (+    68us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:16.249584 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.249677 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:16.249815 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":4618,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":74,"thread_start_us":342603,"threads_started":1}
I20260809 23:21:16.255723 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:15.925465  2475 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:15.950459  2461 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:16.084139 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 288879us
I20260809 23:21:16.264820 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:15.954432  2052 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.000s	sys 0.014s
I20260809 23:21:16.156112  2115 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:16.144480 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:16.148236 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:16.153635 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 346535us
I20260809 23:21:16.153846 24539 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:21:16.086886  2315 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.156524 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:16.154847 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:16.160488 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 431110us
I20260809 23:21:16.088141  2032 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:16.088416  2333 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.162909 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1601) took 1041 ms. Trace:
I20260809 23:21:16.223361 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3264) took 3403 ms. Trace:
I20260809 23:21:16.255630 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:16.202031 25363 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:16.087860 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1538) took 1344 ms. Trace:
I20260809 23:21:16.261181 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2916) took 4928 ms. Trace:
W20260809 23:21:16.262203 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1612) took 1617 ms (client timeout 2011 ms). Trace:
I20260809 23:21:16.166610 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:16.252092  2499 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260809 23:21:16.137602 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:16.147504   877 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:16.357421 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:16.358142 25373 rpcz_store.cc:276] 0809 23:21:15.121329 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.951060 (+829731us) service_pool.cc:227] Handling call
0809 23:21:16.162828 (+211768us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.358489 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3353) took 1136 ms. Trace:
I20260809 23:21:16.204123  2322 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.294627 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:16.359548 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 211390us
I20260809 23:21:16.357841 24424 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 15: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:16.357434 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:16.167006  2222 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:16.373144 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 117672us
I20260809 23:21:16.362277 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:16.358922 24543 rpcz_store.cc:276] 0809 23:21:12.820417 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.820566 (+   149us) service_pool.cc:227] Handling call
0809 23:21:12.832812 (+ 12246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:13.806277 (+973465us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.223079 (+2416802us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b5400080a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:16.223276 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2207190272}
I20260809 23:21:16.360687 24320 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:16.360460 25365 rpcz_store.cc:269] 0809 23:21:14.644505 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.170660 (+526155us) service_pool.cc:227] Handling call
0809 23:21:16.261977 (+1091317us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:16.262133 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":101695360}
W20260809 23:21:16.268262  2052 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.000s	sys 0.015s
I20260809 23:21:16.358960 24539 rpcz_store.cc:276] 0809 23:21:15.221844 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.153055 (+931211us) service_pool.cc:227] Handling call
0809 23:21:16.153424 (+   369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.358231 (+204807us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.358431 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.365618 24435 rpcz_store.cc:276] 0809 23:21:11.333048 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.482168 (+1149120us) service_pool.cc:227] Handling call
0809 23:21:14.444235 (+1962067us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:14.445610 (+  1375us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.260933 (+1815323us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b54002f94d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:16.261113 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4511601024}
I20260809 23:21:16.294150  2469 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.429889 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:16.362170  2458 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.430564 24435 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:16.431555 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:16.357694 24534 rpcz_store.cc:276] 0809 23:21:14.743534 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.807506 (+ 63972us) service_pool.cc:227] Handling call
0809 23:21:14.888543 (+ 81037us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.074090 (+185547us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.083672 (+1009582us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:16.087723 (+  4051us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":396903808}
W20260809 23:21:16.244100  2495 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:21:16.164590  2268 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.191824  2481 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:16.359822 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1616) took 1559 ms. Trace:
I20260809 23:21:16.365289 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:16.209452  2030 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.190726  2505 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260809 23:21:16.353266  2500 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260809 23:21:16.379693 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:16.376410 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 81865.7us
I20260809 23:21:16.377226 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:16.461078 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83931.7us
I20260809 23:21:16.420084  2508 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:16.410316 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 255541us
I20260809 23:21:16.416026 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.470252 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
W20260809 23:21:16.471191 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 91754.8us
I20260809 23:21:16.421912  2194 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.421726  2221 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.374413 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:16.300336  1731 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.501322  1731 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:16.500967  2052 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260809 23:21:16.361492   877 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:16.424957 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:16.363119  2333 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:16.414964  2405 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.566s	user 0.002s	sys 0.008s
I20260809 23:21:16.295562  1677 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.126s
I20260809 23:21:16.370843  2463 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.440744 25363 rpcz_store.cc:276] 0809 23:21:14.810774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.192288 (+1381514us) service_pool.cc:227] Handling call
0809 23:21:16.192703 (+   415us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.359409 (+166706us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.359765 (+   356us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.371373  2501 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:16.450690 24543 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:16.379645 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:16.540060 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 160529us
I20260809 23:21:16.449803 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1332) took 1263 ms. Trace:
I20260809 23:21:16.543120 24424 rpcz_store.cc:276] 0809 23:21:15.186162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.448828 (+1262666us) service_pool.cc:227] Handling call
0809 23:21:16.449223 (+   395us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.449423 (+   200us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:16.449458 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:16.449512 (+    54us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:16.449556 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.449620 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:16.449736 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.544662 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3359) took 1211 ms. Trace:
I20260809 23:21:16.462661 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:16.545135 24543 rpcz_store.cc:276] 0809 23:21:15.332854 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.408240 (+1075386us) service_pool.cc:227] Handling call
0809 23:21:16.439868 (+ 31628us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.539292 (+ 99424us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.544592 (+  5300us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.166867  2493 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.472651 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:16.301302  2496 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:21:16.423022  2431 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:16.266731  2419 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.555253 24534 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:16.459038  2268 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.476121 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:16.464067  2030 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:16.476908  1776 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:16.296046  2176 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.145s
W20260809 23:21:16.439872  2497 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260809 23:21:16.418982 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:16.495405 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3361) took 2507 ms (client timeout 1786 ms). Trace:
W20260809 23:21:16.547079  2504 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260809 23:21:16.449630  2458 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.463711  2505 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:16.500653 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 126449us
I20260809 23:21:16.575048 24317 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260809 23:21:16.372208  2032 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.501829  1731 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:16.584416  1731 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.440028  2512 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:16.587476 24320 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:16.516062 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:16.399542  2315 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.227383  2200 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:16.546283 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83701.9us
I20260809 23:21:16.597645 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:16.419701  1352 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:16.547328 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.549333 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:16.513671  2508 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.518980  2486 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.622524 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:16.553478 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80956.3us
I20260809 23:21:16.371765  2322 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:16.545426  2198 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.862s	user 0.013s	sys 0.001s
I20260809 23:21:16.565953 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:16.566293 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 118986us
I20260809 23:21:16.537549  2119 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:16.408501  2222 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:16.570031 24422 rpcz_store.cc:269] 0809 23:21:13.988019 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.993334 (+  5315us) service_pool.cc:227] Handling call
0809 23:21:16.495107 (+2501773us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:16.495306 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1561771392}
I20260809 23:21:16.496357 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2927) took 4469 ms. Trace:
I20260809 23:21:16.591542  2511 raft_consensus.cc:2741] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260809 23:21:16.572670 25365 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260809 23:21:16.572695 24543 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:16.575462 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:16.513374  2221 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.551592  2006 raft_consensus.cc:2741] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260809 23:21:16.579352 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.647898 25373 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:16.581379  2461 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.539562  2405 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:16.583513  2516 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260809 23:21:16.593528 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:16.567181  2378 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:16.599635 24309 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:21:16.600193 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:16.624135 24430 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260809 23:21:16.624171 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:16.669090 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45008.9us
I20260809 23:21:16.592300  2510 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260809 23:21:16.626185 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 175577us
I20260809 23:21:16.674116 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:16.628461 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:16.578410  2505 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:16.678900  2154 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:16.679412  2154 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:16.665827  2378 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.632444 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.544414  2501 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:16.638525 24423 rpcz_store.cc:276] 0809 23:21:12.027152 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:13.249458 (+1222306us) service_pool.cc:227] Handling call
0809 23:21:13.310874 (+ 61416us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:14.207421 (+896547us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.496064 (+2288643us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b5400061654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:16.496273 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1994924288}
I20260809 23:21:16.572908  1776 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.623684  2194 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:16.656038  1352 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:16.653463 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:16.642370 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67042.8us
I20260809 23:21:16.727999 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:16.653311  2475 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.635597  2198 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:16.664544 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:16.755306 25365 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:16.636230  1677 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:16.663090 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69676.5us
I20260809 23:21:16.680274  2378 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:16.641149  2200 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:16.673071 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 74128.9us
I20260809 23:21:16.772681 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:16.676002 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3365) took 1416 ms. Trace:
W20260809 23:21:16.676836  2413 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.564s	user 0.010s	sys 0.005s
I20260809 23:21:16.677731 24539 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:16.677784  2461 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.691004  2419 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.681447 25363 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:16.692955 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2951) took 1639 ms. Trace:
I20260809 23:21:16.629977  2486 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.752699 24543 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:16.635842  2119 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.790319  2119 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:16.764271  2431 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:16.683528  2006 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:16.625628  2508 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.673712  2510 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:16.769660 25368 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:16.770462 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.680605  2176 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:16.685338  2501 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:16.679430  2405 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.662725  1429 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.102s
I20260809 23:21:16.782330  1770 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:16.782442  1776 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.783195  2032 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.783613  2221 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.721076  2194 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:16.795804  2006 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:16.796025  2519 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260809 23:21:16.784668  2421 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:16.798163  2461 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.766022  2475 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.775718  1677 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.843989  1677 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.831569  2032 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:16.813419  2510 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:16.845904  2313 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:16.848908  2472 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.058s	user 0.000s	sys 0.004s
W20260809 23:21:16.852859  2514 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.395s	user 0.000s	sys 0.001s
I20260809 23:21:16.799580  2230 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260809 23:21:16.855763  2230 consensus_queue.cc:1225] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:21:16.856266  1977 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:16.831267  2006 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:16.863349  2315 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.870146  2148 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:16.870749  2189 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.871438  2189 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.870657  2148 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:16.873521  2486 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.873221  2148 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.874339  2148 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:16.877496  2510 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:16.881256  2494 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.723s	user 0.003s	sys 0.000s
I20260809 23:21:16.882329  2475 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.882958  2469 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:16.883502  2469 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.886060  2512 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:16.886310  1746 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:16.886680  1746 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:16.827621  2194 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:16.887569  1746 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.888778  1746 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:16.839182  2421 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:16.893417  2189 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.897775  2171 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:16.898242  2171 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:16.825470  2493 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.899727  2222 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.901648  2508 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:16.901885  2483 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.154s	user 0.005s	sys 0.006s
I20260809 23:21:16.903313  2242 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.904511  2468 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.908313  2127 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:16.909287  2458 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.911216  2530 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:16.911726  2530 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:16.912125  2530 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:16.912586  2338 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:16.917200  2461 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.919548  2461 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.897660  2268 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.893458  2421 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:16.933511  2510 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.898481  2322 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.949587  2333 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:16.978459  2333 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.980152  2505 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.983321  2505 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:16.839573  2544 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.142s
I20260809 23:21:16.997334  2171 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:16.998733  2171 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:16.955672  2469 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:16.961236  2115 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.585s	user 0.004s	sys 0.005s
I20260809 23:21:17.003661  2115 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.006330  2115 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:16.987048  2119 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.012791  2119 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.019040  2378 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.021754  2378 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:17.029115  2194 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.054051  2501 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.057802  2501 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:17.055038  2421 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.063490  2421 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:17.063616  2530 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.066401  2530 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:16.670615 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:17.110476 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 439729us
I20260809 23:21:17.115769 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.118462 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3340) took 3005 ms. Trace:
W20260809 23:21:17.119540 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 445349us
I20260809 23:21:17.120440 24314 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:17.126860 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3364) took 1270 ms. Trace:
I20260809 23:21:17.131523 24309 rpcz_store.cc:276] 0809 23:21:15.856232 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.159540 (+303308us) service_pool.cc:227] Handling call
0809 23:21:17.126277 (+966737us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540032fad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.126597 (+   320us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":789166976}
I20260809 23:21:17.133481 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:17.128758 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3351) took 1470 ms. Trace:
I20260809 23:21:17.135823 24312 rpcz_store.cc:276] 0809 23:21:15.657899 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.677385 (+ 19486us) service_pool.cc:227] Handling call
0809 23:21:17.128169 (+1450784us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54001afc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.128511 (+   342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":480962816}
I20260809 23:21:17.139183 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:17.139655 24312 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:17.141366 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2813) took 1481 ms. Trace:
I20260809 23:21:17.141794 24312 rpcz_store.cc:276] 0809 23:21:15.660199 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.137217 (+1477018us) service_pool.cc:227] Handling call
0809 23:21:17.138507 (+  1290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.141071 (+  2564us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.141291 (+   220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.143082 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3348) took 1485 ms. Trace:
I20260809 23:21:17.140472 24309 raft_consensus.cc:595] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:17.143481 24311 rpcz_store.cc:276] 0809 23:21:15.657067 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.427777 (+770710us) service_pool.cc:227] Handling call
0809 23:21:16.469659 (+ 41882us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.625025 (+155366us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:17.123848 (+498823us) log.cc:842] Serialized 14 byte log entry
0809 23:21:17.139900 (+ 16052us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:17.140937 (+  1037us) log.cc:842] Serialized 26 byte log entry
0809 23:21:17.141080 (+   143us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:17.141107 (+    27us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.141211 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.142599 (+  1388us) raft_consensus.cc:1671] finished
0809 23:21:17.142620 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.142853 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24135,"prepare.queue_time_us":206968,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":62195,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9759,"thread_start_us":644116,"threads_started":3,"wal-append.queue_time_us":11694}
I20260809 23:21:17.119050 24313 rpcz_store.cc:276] 0809 23:21:14.112608 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.113009 (+   401us) service_pool.cc:227] Handling call
0809 23:21:14.113365 (+   356us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:14.129574 (+ 16209us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:14.955561 (+825987us) log.cc:842] Serialized 14 byte log entry
0809 23:21:15.437092 (+481531us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:15.443131 (+  6039us) log.cc:842] Serialized 52 byte log entry
0809 23:21:15.443308 (+   177us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:15.443350 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.443433 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:16.848288 (+1404855us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:17.117379 (+269091us) raft_consensus.cc:1671] finished
0809 23:21:17.117646 (+   267us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.118137 (+   491us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15781,"prepare.queue_time_us":869003,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":213222,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":13554,"spinlock_wait_cycles":6961408,"thread_start_us":1254264,"threads_started":2,"wal-append.queue_time_us":705637}
I20260809 23:21:17.135250 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2817) took 1426 ms. Trace:
I20260809 23:21:17.146646 24314 rpcz_store.cc:276] 0809 23:21:15.708892 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.714587 (+  5695us) service_pool.cc:227] Handling call
0809 23:21:17.134911 (+1420324us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.135142 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3229143296}
I20260809 23:21:17.148869 24314 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:17.157078 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3357) took 1344 ms. Trace:
W20260809 23:21:17.160115 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.006s	sys 0.009s
W20260809 23:21:17.169516 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.006s	sys 0.009s
I20260809 23:21:17.170758 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3366) took 1897 ms. Trace:
I20260809 23:21:16.765760 24423 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:17.171298 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3349) took 1514 ms. Trace:
I20260809 23:21:16.759820 24380 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:17.172390 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3371) took 1833 ms. Trace:
I20260809 23:21:17.173134 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1330) took 1989 ms. Trace:
I20260809 23:21:17.173960 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3364) took 1920 ms. Trace:
I20260809 23:21:17.158118 24312 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:17.163610 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.025s
I20260809 23:21:17.159399 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3353) took 1352 ms. Trace:
I20260809 23:21:17.168809 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1329) took 1985 ms. Trace:
I20260809 23:21:16.686802 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2943) took 2533 ms. Trace:
I20260809 23:21:16.785436 24429 rpcz_store.cc:276] 0809 23:21:15.053445 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.505654 (+452209us) service_pool.cc:227] Handling call
0809 23:21:15.519431 (+ 13777us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:15.519779 (+   348us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.519816 (+    37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:15.594150 (+ 74334us) log.cc:842] Serialized 52 byte log entry
0809 23:21:15.998345 (+404195us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:15.998405 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.998513 (+   108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:16.692672 (+694159us) raft_consensus.cc:1671] finished
0809 23:21:16.692702 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:16.692880 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":527,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":36344,"thread_start_us":339077,"threads_started":1,"wal-append.queue_time_us":618753}
I20260809 23:21:16.774139 24430 rpcz_store.cc:276] 0809 23:21:15.259819 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.591493 (+1331674us) service_pool.cc:227] Handling call
0809 23:21:16.613328 (+ 21835us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.675652 (+ 62324us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.675932 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:16.774410 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:16.795200 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:16.837405   877 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:17.174723 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 416002us
I20260809 23:21:17.202224 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1599) took 2081 ms. Trace:
I20260809 23:21:17.185473 24316 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:17.187657 24541 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:17.216272 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3278) took 2228 ms. Trace:
I20260809 23:21:17.160279 24314 rpcz_store.cc:276] 0809 23:21:15.812382 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.147873 (+1335491us) service_pool.cc:227] Handling call
0809 23:21:17.148497 (+   624us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.156754 (+  8257us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.156990 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.204802 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3344) took 1549 ms. Trace:
I20260809 23:21:17.248409 25373 rpcz_store.cc:276] 0809 23:21:15.120663 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.820987 (+1700324us) service_pool.cc:227] Handling call
0809 23:21:17.200970 (+379983us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.201538 (+   568us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.201576 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:17.201651 (+    75us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:17.201715 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.201820 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.201980 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.213862 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3274) took 2316 ms. Trace:
I20260809 23:21:17.154179 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.245774 24309 rpcz_store.cc:276] 0809 23:21:15.806737 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.144644 (+1337907us) service_pool.cc:227] Handling call
0809 23:21:17.145085 (+   441us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.158865 (+ 13780us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.158909 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:17.159012 (+   103us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:17.159121 (+   109us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.159202 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.159336 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.186646 24539 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:17.188273 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3347) took 2027 ms. Trace:
W20260809 23:21:17.191205 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1600) took 3013 ms (client timeout 2142 ms). Trace:
I20260809 23:21:17.217314 25369 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260809 23:21:17.218698 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3285) took 2061 ms. Trace:
I20260809 23:21:17.176496 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2949) took 2173 ms. Trace:
I20260809 23:21:17.172914 24423 rpcz_store.cc:276] 0809 23:21:15.338505 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.684282 (+1345777us) service_pool.cc:227] Handling call
0809 23:21:16.765109 (+ 80827us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.171754 (+406645us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.172166 (+   412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.171293 24422 rpcz_store.cc:276] 0809 23:21:15.273511 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.638284 (+1364773us) service_pool.cc:227] Handling call
0809 23:21:16.660100 (+ 21816us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.660342 (+   242us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:16.660377 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:16.660888 (+   511us) log.cc:842] Serialized 26 byte log entry
0809 23:21:17.169940 (+509052us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:17.170059 (+   119us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.170209 (+   150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.170290 (+    81us) raft_consensus.cc:1671] finished
0809 23:21:17.170304 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.170565 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":416971,"threads_started":1,"wal-append.queue_time_us":386492}
I20260809 23:21:17.192111 25366 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:17.212780 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1597) took 3050 ms. Trace:
I20260809 23:21:17.189550 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3280) took 2175 ms. Trace:
W20260809 23:21:17.211197 25375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1617) took 2366 ms (client timeout 1590 ms). Trace:
I20260809 23:21:17.255475 24545 rpcz_store.cc:276] 0809 23:21:14.987354 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.987595 (+   241us) service_pool.cc:227] Handling call
0809 23:21:15.135253 (+147658us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.811182 (+675929us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.215108 (+1403926us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b54001e3820. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:17.215904 (+   796us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.216117 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532173824}
W20260809 23:21:17.221166  2052 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.019s	sys 0.109s
W20260809 23:21:17.259557 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.001s	sys 0.029s
I20260809 23:21:17.259630 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2814) took 1585 ms. Trace:
W20260809 23:21:17.210103 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1614) took 2487 ms (client timeout 1588 ms). Trace:
W20260809 23:21:17.208823 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1600) took 2087 ms (client timeout 2588 ms). Trace:
W20260809 23:21:17.186738 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2803) took 1754 ms (client timeout 2209 ms). Trace:
I20260809 23:21:17.211719 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3349) took 2050 ms. Trace:
W20260809 23:21:17.227512  2030 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.658s	user 0.010s	sys 0.000s
I20260809 23:21:17.271816 24312 rpcz_store.cc:276] 0809 23:21:15.670774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.143149 (+1472375us) service_pool.cc:227] Handling call
0809 23:21:17.157738 (+ 14589us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.253782 (+ 96044us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.255967 (+  2185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.183944 24428 raft_consensus.cc:1203] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Deduplicated request from leader. Original: 13.12->[15.13-15.13]   Dedup: 15.13->[]
I20260809 23:21:17.306447 24428 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260809 23:21:17.267627 24540 rpcz_store.cc:276] 0809 23:21:15.013661 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.135175 (+121514us) service_pool.cc:227] Handling call
0809 23:21:15.135687 (+   512us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.415562 (+279875us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.415606 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.415660 (+    54us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:15.415704 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.188979 (+1773275us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b54001fc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.189048 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.189361 (+   313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":643080448}
I20260809 23:21:17.296178 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.257855 24544 rpcz_store.cc:276] 0809 23:21:14.897200 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.897498 (+   298us) service_pool.cc:227] Handling call
0809 23:21:14.915515 (+ 18017us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.099757 (+184242us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.099791 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.099833 (+    42us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:15.099871 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.213335 (+2113464us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.213419 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.213685 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":423220096}
I20260809 23:21:17.259092 24535 rpcz_store.cc:276] 0809 23:21:15.160785 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.467515 (+306730us) service_pool.cc:227] Handling call
0809 23:21:15.467889 (+   374us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.891853 (+423964us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.187679 (+1295826us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.188031 (+   352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":832522496}
I20260809 23:21:17.223793  1485 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.259684 24542 rpcz_store.cc:276] 0809 23:21:15.157419 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.466430 (+309011us) service_pool.cc:227] Handling call
0809 23:21:15.906879 (+440449us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.146694 (+239815us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.218245 (+1071551us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b5400115b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.218503 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012266368}
W20260809 23:21:17.272615 25368 rpcz_store.cc:269] 0809 23:21:15.121034 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.748612 (+627578us) service_pool.cc:227] Handling call
0809 23:21:17.208373 (+1459761us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.208620 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1438937984}
W20260809 23:21:17.260870 25371 rpcz_store.cc:269] 0809 23:21:14.177206 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.185184 (+  7978us) service_pool.cc:227] Handling call
0809 23:21:17.190631 (+3005447us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b540071dc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.190894 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91291392}
W20260809 23:21:17.272709 24323 rpcz_store.cc:269] 0809 23:21:15.431988 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.432108 (+   120us) service_pool.cc:227] Handling call
0809 23:21:17.186400 (+1754292us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54003a34d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.186574 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896813952}
W20260809 23:21:17.267297 25375 rpcz_store.cc:269] 0809 23:21:14.844741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.974704 (+129963us) service_pool.cc:227] Handling call
0809 23:21:17.210774 (+2236070us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.210976 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1804955008}
I20260809 23:21:17.232147  2119 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:17.271975 25372 rpcz_store.cc:269] 0809 23:21:14.722625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.728146 (+  5521us) service_pool.cc:227] Handling call
0809 23:21:17.209649 (+2481503us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.209858 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":724865408}
I20260809 23:21:17.223609  1527 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:17.224622  1539 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:17.188849 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.274274 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "b9b54436d0e94a5595cdded73fe33da9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:17.174613 24424 rpcz_store.cc:276] 0809 23:21:15.253928 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.544278 (+1290350us) service_pool.cc:227] Handling call
0809 23:21:16.544652 (+   374us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.544887 (+   235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:16.544911 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:16.545308 (+   397us) log.cc:842] Serialized 26 byte log entry
0809 23:21:16.685319 (+140011us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:16.685355 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.685425 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.173363 (+487938us) raft_consensus.cc:1671] finished
0809 23:21:17.173411 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.173730 (+   319us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":139692,"threads_started":1,"wal-append.queue_time_us":235138}
I20260809 23:21:17.257643 24317 rpcz_store.cc:276] 0809 23:21:15.655789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.198089 (+542300us) service_pool.cc:227] Handling call
0809 23:21:16.216658 (+ 18569us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:16.575415 (+358757us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.204236 (+628821us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.204633 (+   397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":823264000}
I20260809 23:21:17.319247 24309 raft_consensus.cc:595] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Received request to transfer leadership to TS b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.272768 24546 rpcz_store.cc:276] 0809 23:21:15.161400 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.745968 (+584568us) service_pool.cc:227] Handling call
0809 23:21:15.856822 (+110854us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.857032 (+   210us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.857067 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:15.857119 (+    52us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:15.857162 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:15.857225 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.211298 (+1354073us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b54001fca50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.211515 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":95747968}
I20260809 23:21:17.171890 24320 rpcz_store.cc:276] 0809 23:21:15.657363 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.588194 (+930831us) service_pool.cc:227] Handling call
0809 23:21:16.588528 (+   334us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.588852 (+   324us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:16.588881 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:16.589286 (+   405us) log.cc:842] Serialized 26 byte log entry
0809 23:21:17.170523 (+581237us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:17.170629 (+   106us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.170715 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.170781 (+    66us) raft_consensus.cc:1671] finished
0809 23:21:17.170790 (+     9us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.171221 (+   431us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75870,"raft.queue_time_us":225711,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":9017,"thread_start_us":580411,"threads_started":1,"wal-append.queue_time_us":301809}
W20260809 23:21:17.280683 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 506134us
I20260809 23:21:17.292120 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1622) took 2240 ms. Trace:
I20260809 23:21:17.276258 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.235399  2200 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:17.283614  2416 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.033s
I20260809 23:21:17.260241 24431 rpcz_store.cc:276] 0809 23:21:15.002553 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.455086 (+452533us) service_pool.cc:227] Handling call
0809 23:21:15.455504 (+   418us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:15.623494 (+167990us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:15.623538 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:15.640027 (+ 16489us) log.cc:842] Serialized 52 byte log entry
0809 23:21:15.908809 (+268782us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:16.081334 (+172525us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.081646 (+   312us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.867660 (+786014us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b18005536c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:17.175817 (+308157us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.175888 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.175990 (+   102us) raft_consensus.cc:1671] finished
0809 23:21:17.176003 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.176297 (+   294us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8540,"prepare.queue_time_us":48073,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":8402,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":6058,"spinlock_wait_cycles":393304320,"thread_start_us":252404,"threads_started":1,"wal-append.queue_time_us":557507}
I20260809 23:21:17.341148 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1613) took 2696 ms. Trace:
I20260809 23:21:17.267225 25370 rpcz_store.cc:276] 0809 23:21:14.162429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:14.162542 (+   113us) service_pool.cc:227] Handling call
0809 23:21:15.172280 (+1009738us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.326596 (+154316us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.212311 (+1885715us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54007062d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.212565 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2310301056}
I20260809 23:21:17.322523 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.245110 24425 rpcz_store.cc:276] 0809 23:21:15.183572 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.152148 (+968576us) service_pool.cc:227] Handling call
0809 23:21:16.152611 (+   463us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.207186 (+ 54575us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:21:16.225630 (+ 18444us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.448038 (+222408us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.448225 (+   187us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:16.457237 (+  9012us) log.cc:842] Serialized 26 byte log entry
0809 23:21:16.457390 (+   153us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:16.457772 (+   382us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.463934 (+  6162us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.464027 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:16.859573 (+395546us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b1800527080. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:16.885177 (+ 25604us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b1800527080. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:16.899652 (+ 14475us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b1800527080. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f
0809 23:21:16.900917 (+  1265us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b48001701d8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:17.168075 (+267158us) raft_consensus.cc:1671] finished
0809 23:21:17.168145 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.168474 (+   329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45852,"prepare.queue_time_us":430568,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":98,"raft.queue_time_us":62866,"raft.run_cpu_time_us":4434,"raft.run_wall_time_us":456907,"spinlock_wait_cycles":167127680,"thread_start_us":199446,"threads_started":1,"wal-append.queue_time_us":304989}
I20260809 23:21:17.341164 25371 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:17.324030 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:17.324553 25363 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:17.223131  1429 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 18.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:21:17.292760  2222 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:17.280791 24427 rpcz_store.cc:276] 0809 23:21:14.153288 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.440813 (+1287525us) service_pool.cc:227] Handling call
0809 23:21:15.456708 (+ 15895us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:15.457385 (+   677us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:16.686549 (+1229164us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b54002f9ed0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:16.686731 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35763584}
I20260809 23:21:17.346131 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3355) took 1512 ms. Trace:
I20260809 23:21:17.287649  2329 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.039s
I20260809 23:21:17.327559 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:17.329303 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3322) took 4373 ms (client timeout 3266 ms). Trace:
I20260809 23:21:17.328846  1485 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.282192  2421 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.283859  2469 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:17.276803  2315 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260809 23:21:17.280896  2115 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:17.364660  2315 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:17.365149  2315 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:17.330726 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.330775 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1546) took 1182 ms. Trace:
I20260809 23:21:17.333928  2268 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.334306 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.173664 24435 rpcz_store.cc:276] 0809 23:21:15.183890 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.432340 (+1248450us) service_pool.cc:227] Handling call
0809 23:21:16.432607 (+   267us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:16.470753 (+ 38146us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:16.470783 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:16.475891 (+  5108us) log.cc:842] Serialized 26 byte log entry
0809 23:21:16.681174 (+205283us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:16.767200 (+ 86026us) log.cc:842] Serialized 14 byte log entry
0809 23:21:16.780043 (+ 12843us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.172270 (+392227us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.172347 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.172440 (+    93us) raft_consensus.cc:1671] finished
0809 23:21:17.172594 (+   154us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.172945 (+   351us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24609,"prepare.queue_time_us":178367,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":30417,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":12825,"spinlock_wait_cycles":62301440,"thread_start_us":290408,"threads_started":2,"wal-append.queue_time_us":326913}
I20260809 23:21:17.335290  1539 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:17.349148 24316 rpcz_store.cc:269] 0809 23:21:12.955826 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:12.956940 (+  1114us) service_pool.cc:227] Handling call
0809 23:21:17.329039 (+4372099us) spinlock_profiling.cc:243] Waited 4.13 s on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.329201 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9495517568}
I20260809 23:21:17.336652 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.337226 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1328) took 2154 ms. Trace:
I20260809 23:21:17.338872 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.340137   877 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.293098  2194 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.287096  2345 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.055s
I20260809 23:21:17.343654 24423 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:17.311098  2052 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.027s	sys 0.113s
I20260809 23:21:17.343861 24429 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:17.320007 24309 raft_consensus.cc:603] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:21:17.316553 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 684272us
I20260809 23:21:17.282548  2435 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:17.320395 25369 rpcz_store.cc:276] 0809 23:21:15.051463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.193713 (+1142250us) service_pool.cc:227] Handling call
0809 23:21:17.216530 (+1022817us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.259380 (+ 42850us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.291875 (+ 32495us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540063e550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.292032 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2351268224}
I20260809 23:21:17.355396 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1605) took 1443 ms. Trace:
I20260809 23:21:17.346158 25366 rpcz_store.cc:276] 0809 23:21:14.644793 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.158624 (+1513831us) service_pool.cc:227] Handling call
0809 23:21:17.191281 (+1032657us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.271281 (+ 80000us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.340871 (+ 69590us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540071dc50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.341088 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2373820288}
I20260809 23:21:17.348688 24311 rpcz_store.cc:276] 0809 23:21:15.809921 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.146388 (+1336467us) service_pool.cc:227] Handling call
0809 23:21:17.146872 (+   484us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.149290 (+  2418us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:17.149780 (+   490us) log.cc:842] Serialized 14 byte log entry
0809 23:21:17.321319 (+171539us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:17.321402 (+    83us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:17.321460 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.321544 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.321688 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10843,"prepare.queue_time_us":114140,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":143,"raft.queue_time_us":175497,"raft.run_cpu_time_us":493,"raft.run_wall_time_us":495,"thread_start_us":153977,"threads_started":1}
I20260809 23:21:17.357271  2222 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.359017  2030 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:17.359880 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.360975 24321 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:17.361945 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.362895  2421 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.336232 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:17.345676  1731 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.006s	sys 0.003s
I20260809 23:21:17.357142 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.371635 24428 rpcz_store.cc:276] 0809 23:21:15.183261 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.892224 (+708963us) service_pool.cc:227] Handling call
0809 23:21:17.183379 (+1291155us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:17.309810 (+126431us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.336874 (+ 27064us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54003336d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.337135 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2968207232}
I20260809 23:21:17.369302 24541 rpcz_store.cc:276] 0809 23:21:16.148596 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.378657 (+230061us) service_pool.cc:227] Handling call
0809 23:21:16.379007 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.303199 (+924192us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.330518 (+ 27319us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:17.330713 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1857680640}
I20260809 23:21:17.381709 25371 rpcz_store.cc:276] 0809 23:21:15.907295 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.339898 (+1432603us) service_pool.cc:227] Handling call
0809 23:21:17.340476 (+   578us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.347187 (+  6711us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.350891 (+  3704us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.366307  2315 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.384456 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3372) took 1665 ms. Trace:
I20260809 23:21:17.386550 24429 rpcz_store.cc:276] 0809 23:21:15.716416 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.342280 (+1625864us) service_pool.cc:227] Handling call
0809 23:21:17.343127 (+   847us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.379041 (+ 35914us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.381429 (+  2388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.382910 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.388388 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.387017 24422 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:17.391625  2378 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.392599 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3378) took 1250 ms. Trace:
I20260809 23:21:17.392252  2378 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.384915 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.393024 24430 rpcz_store.cc:276] 0809 23:21:16.142054 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.391452 (+1249398us) service_pool.cc:227] Handling call
0809 23:21:17.391925 (+   473us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.392239 (+   314us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.392265 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:17.392335 (+    70us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:17.392374 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.392429 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.392522 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.388542 24542 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:17.390537 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:17.390295  1731 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.006s	sys 0.003s
I20260809 23:21:17.396091 24543 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:17.403590 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.405406 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:17.407672 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.414984  2508 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:17.415395 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.415970 24435 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:17.416823 24542 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:21:17.416957 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2959) took 1678 ms. Trace:
I20260809 23:21:17.417456 24435 rpcz_store.cc:276] 0809 23:21:15.738800 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.379161 (+1640361us) service_pool.cc:227] Handling call
0809 23:21:17.415622 (+ 36461us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.416689 (+  1067us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.416899 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.418000 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.411765 24546 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:17.422462 24431 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:17.426105 24430 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 17. Preceding OpId from leader: term: 18 index: 18. (index mismatch)
I20260809 23:21:17.425988  1539 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.426908 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3379) took 1284 ms. Trace:
I20260809 23:21:17.427284 24430 rpcz_store.cc:276] 0809 23:21:16.142506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.407466 (+1264960us) service_pool.cc:227] Handling call
0809 23:21:17.425715 (+ 18249us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.426624 (+   909us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.426832 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.430536 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:21:17.432320  2222 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:17.435632 24540 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:17.439788 25371 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:17.440084 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.440866 24427 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:17.443024 25371 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260809 23:21:17.437086 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:17.445197 24429 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:17.394165  2268 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.448590  2268 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:17.449716  2154 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 17.17, Last appended by leader: 17, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:17.452309 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 116281us
W20260809 23:21:17.452991 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2800) took 2040 ms (client timeout 2036 ms). Trace:
W20260809 23:21:17.453472 24321 rpcz_store.cc:269] 0809 23:21:15.412733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:15.413208 (+   475us) service_pool.cc:227] Handling call
0809 23:21:17.452721 (+2039513us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.452888 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3920350592}
I20260809 23:21:17.454802  2194 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:17.455854 25366 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:17.443312  2333 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:17.464246 25372 pending_rounds.cc:76] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 12
I20260809 23:21:17.464664 25372 pending_rounds.cc:97] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260809 23:21:17.464967 25372 raft_consensus.cc:2866] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:17.468622 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1608) took 1385 ms. Trace:
I20260809 23:21:17.469034 24435 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: [16.11-18.12]
W20260809 23:21:17.470432 25365 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.714s	user 0.013s	sys 0.004s
I20260809 23:21:17.472023 25371 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:17.443481 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.482951 25363 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:17.487697 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2963) took 1109 ms. Trace:
I20260809 23:21:17.454502 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:17.493194 25372 rpcz_store.cc:276] 0809 23:21:16.083399 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.405200 (+1321801us) service_pool.cc:227] Handling call
0809 23:21:17.463959 (+ 58759us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:17.465649 (+  1690us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.465677 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:17.466088 (+   411us) log.cc:842] Serialized 26 byte log entry
0809 23:21:17.468264 (+  2176us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:17.468304 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.468371 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:17.468414 (+    43us) raft_consensus.cc:1671] finished
0809 23:21:17.468424 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.468547 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.495221 24424 rpcz_store.cc:276] 0809 23:21:16.377959 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.405065 (+1027106us) service_pool.cc:227] Handling call
0809 23:21:17.486810 (+ 81745us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.487182 (+   372us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.487219 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:17.487313 (+    94us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:17.487365 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.487441 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:17.487591 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.481518 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.474789 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.442783 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.507156 24422 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:17.508667 24423 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:17.445977  2315 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:17.507977  2222 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260809 23:21:17.477038  2171 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:17.512055 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 69476.8us
I20260809 23:21:17.512684  2329 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:17.513137 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:17.477770  1429 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260809 23:21:17.518401 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:17.519058  2194 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:17.519690  1485 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.408820  2200 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.508984 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.545243 24431 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:17.523553 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.507592 25365 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:17.536653 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:17.480729  2421 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:17.541935 24435 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:17.543663 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2962) took 1175 ms. Trace:
I20260809 23:21:17.404177  1527 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.545972 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.504725  2345 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:17.553058 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 399879us
I20260809 23:21:17.525045  2508 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:17.547104 24423 rpcz_store.cc:276] 0809 23:21:16.363944 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.397445 (+1033501us) service_pool.cc:227] Handling call
0809 23:21:17.508487 (+111042us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.509043 (+   556us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.538549 (+ 29506us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54003bae50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.538728 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":237889792}
I20260809 23:21:17.575886 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 2961) took 1783 ms. Trace:
I20260809 23:21:17.576294 24429 rpcz_store.cc:276] 0809 23:21:15.792222 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.396652 (+1604430us) service_pool.cc:227] Handling call
0809 23:21:17.424673 (+ 28021us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.499495 (+ 74822us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.575677 (+ 76182us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b540023e450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:17.575826 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4269824}
W20260809 23:21:17.577924 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75754.7us
I20260809 23:21:17.580994 24542 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:17.543166  2194 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:17.607528 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:17.611729 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 157437us
I20260809 23:21:17.603070 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1580) took 1089 ms. Trace:
I20260809 23:21:17.607692 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:17.621768 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.616729 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:17.620589 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.617156 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:17.586191 24543 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:17.616771 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.614387  2378 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:17.678790  1485 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:21:17.604084 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 85982.9us
W20260809 23:21:17.683848 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76435.5us
W20260809 23:21:17.677505 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 124257us
I20260809 23:21:17.648288 25365 rpcz_store.cc:276] 0809 23:21:16.513336 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:16.659665 (+146329us) service_pool.cc:227] Handling call
0809 23:21:17.577821 (+918156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:17.679615 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:17.679513  1485 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:17.690035 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:17.599195  2154 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:17.691967  1485 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:17.596477  2200 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.697321  2463 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:17.607849  2115 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:17.612313  2322 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:17.652010  2194 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:17.696731 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.604517  2505 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:17.703819 24435 raft_consensus.cc:2362] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21. Re-sending same reply.
I20260809 23:21:17.606709  2435 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:17.613462  1527 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.553185  2171 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.613669  2544 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:17.688899  2378 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:17.703502  2463 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:17.705476 24164 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 16, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 16 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:17.708302 24544 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 15: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:17.715481 24534 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260809 23:21:17.698231  2469 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:17.704118  2322 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:17.616297  2345 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.722028 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.724530  2268 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.734791  2268 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.709604  2333 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:17.726357  2378 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:17.728569  2469 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:17.738862  2469 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:17.738999 24429 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:17.709832  2505 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:17.720989  2435 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:17.743268 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.724053  2544 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:17.745721  2333 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.740306  2378 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:17.746361  2435 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:17.749229  2115 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.004s
I20260809 23:21:17.756428 24164 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 12, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 12 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:17.607017  2508 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:17.765446 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 148943us
I20260809 23:21:17.744673  2505 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:17.768247 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:17.758482 24164 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:17.778119 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.732897  2463 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.787911 24164 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 16, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:17.789529 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.782877 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:17.797160 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.772105  2421 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.025s
I20260809 23:21:17.787369 24427 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:17.808310  2189 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260809 23:21:17.814308 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72858.2us
I20260809 23:21:17.774245  2322 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.811990  2189 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:17.796455  2333 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.823225  2322 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:17.827983  2189 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:17.828845  2463 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:17.819697  2052 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:17.827631  2345 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.827307  1677 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:17.851055  1731 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:17.867715 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.868798 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.821802 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:17.834398  2435 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.858680 24164 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:17.875871 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.884617  1485 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.887432  2501 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260809 23:21:17.888512  2501 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:17.890354  2127 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:17.890753  2127 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:17.891780  2127 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.893483  2127 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:17.877926 24435 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:17.864717  2171 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:17.902439 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 113079us
I20260809 23:21:17.866092  1677 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:17.911533 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 103830us
I20260809 23:21:17.908933  2345 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.006s
I20260809 23:21:17.900650  1731 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.917392  1731 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:17.918359 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 166673us
I20260809 23:21:17.918896  1677 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.919480 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:17.921801  2435 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:17.922446  2435 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:17.923296  2268 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:17.924533  2333 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.924705 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.925343  2333 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:17.925813  2333 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:17.926203  2333 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:17.927376 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.928287 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.864015  1527 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:17.896764  1485 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:17.931833 24424 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260809 23:21:17.933148 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:17.922789  2435 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:17.882582  2194 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:17.937722 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.939107 24164 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:17.948223 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:17.950757 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.030s	sys 0.041s
W20260809 23:21:17.972817 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.030s	sys 0.042s
I20260809 23:21:17.975255 24534 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:17.975901 24427 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:17.926425  2006 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.068s	user 0.006s	sys 0.010s
I20260809 23:21:17.979256 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:17.987581 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:17.990809 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.991756 24429 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:17.856326  2544 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.973058  2508 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.003073 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:18.049218 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.050552 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:17.968977  1527 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:17.992542  1731 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:18.059648 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:18.068190 24429 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:21:18.071511 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:18.055610 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.008s	sys 0.011s
W20260809 23:21:18.073967 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 287534us
I20260809 23:21:17.992940  2006 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:18.082314 25375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.016s	sys 0.031s
W20260809 23:21:18.086642 25375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.016s	sys 0.032s
I20260809 23:21:18.090193  2416 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:18.154956 24164 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:18.077096  2469 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.069864  2435 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.115505  1731 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:18.196703 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56326.5us
W20260809 23:21:18.179971 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 242419us
W20260809 23:21:18.222292 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 370758us
I20260809 23:21:18.161373 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.185077 24164 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.181527 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1611) took 1694 ms. Trace:
I20260809 23:21:18.252103  2544 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:18.254350 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:18.256532  1731 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.257372  2189 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.271879 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:18.263185 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 259997us
W20260809 23:21:18.263381  2566 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260809 23:21:18.140921  2268 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.264262 24427 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:18.166496  2194 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:18.273818  2222 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:18.271456  2006 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.278187 24164 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 16. New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.276681  2435 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:18.297461 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60480.1us
I20260809 23:21:18.122027  2505 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.291749 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:18.291092 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 90848.5us
I20260809 23:21:18.301898 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.299587 24539 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:18.242375  2378 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.312534 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1587) took 1733 ms. Trace:
I20260809 23:21:18.239643 25369 rpcz_store.cc:276] 0809 23:21:16.472240 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.393587 (+921347us) service_pool.cc:227] Handling call
0809 23:21:17.433623 (+ 40036us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.434230 (+   607us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:17.793609 (+359379us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.166061 (+372452us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:18.166134 (+    73us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:18.166216 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.166298 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.166636 (+   338us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":611500,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":20540,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":46307,"thread_start_us":721339,"threads_started":2}
I20260809 23:21:18.315047 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1609) took 2176 ms. Trace:
I20260809 23:21:18.321570  2577 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.322376 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.312027 24164 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.324822 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.314774 25366 rpcz_store.cc:276] 0809 23:21:16.592440 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.456842 (+864402us) service_pool.cc:227] Handling call
0809 23:21:17.457325 (+   483us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:17.457674 (+   349us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.457700 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:17.495910 (+ 38210us) log.cc:842] Serialized 26 byte log entry
0809 23:21:18.074057 (+578147us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:18.074091 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.074151 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:18.312313 (+238162us) raft_consensus.cc:1671] finished
0809 23:21:18.312333 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.312473 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":527999,"threads_started":1,"wal-append.queue_time_us":740245}
I20260809 23:21:18.330163 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:18.281111  2268 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.336457  1527 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.336308  2345 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:18.303607  2194 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.340847 24546 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:18.337095  2189 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.328600  2421 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:21:18.322824  2200 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.337291  2345 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:21:18.299145  2333 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:18.353704 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 434349us
I20260809 23:21:18.321806 25368 rpcz_store.cc:276] 0809 23:21:16.139013 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.414858 (+1275845us) service_pool.cc:227] Handling call
0809 23:21:17.415487 (+   629us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.415991 (+   504us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:17.806359 (+390368us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.298151 (+491792us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:18.298204 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:18.314783 (+ 16579us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.314857 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.314993 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3845,"prepare.queue_time_us":569769,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":47766,"thread_start_us":864413,"threads_started":2}
I20260809 23:21:18.301222  2222 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:18.360694 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73870.9us
W20260809 23:21:18.341295  2030 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.008s	sys 0.008s
W20260809 23:21:18.374889  2030 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.009s	sys 0.008s
I20260809 23:21:18.367555  2469 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.368367  2222 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:18.375667 24163 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 372) took 43357 ms (client timeout 15000 ms). Trace:
W20260809 23:21:18.372575 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 486753us
I20260809 23:21:18.372875 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.377329  2115 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:18.365657 24164 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 13 to 14, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 14 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.379551 24164 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.380966 24540 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:18.346045  2416 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:18.381784  2268 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:18.383512 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:18.384763  2315 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.859s	user 0.008s	sys 0.006s
I20260809 23:21:18.373987  2200 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.377925  2115 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:18.394522 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46965us
I20260809 23:21:18.390563 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3359) took 2553 ms. Trace:
I20260809 23:21:18.389649 24164 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 8 to 13, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 13 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:18.360769  2189 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:18.400372 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:18.406446 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:18.405041 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1610) took 1933 ms. Trace:
I20260809 23:21:18.404986 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3354) took 2597 ms. Trace:
I20260809 23:21:18.411247 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1627) took 3238 ms. Trace:
I20260809 23:21:18.407160 24314 rpcz_store.cc:276] 0809 23:21:15.813776 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.259808 (+1446032us) service_pool.cc:227] Handling call
0809 23:21:17.348359 (+ 88551us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:17.400120 (+ 51761us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.400151 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:17.465033 (+ 64882us) log.cc:842] Serialized 52 byte log entry
0809 23:21:17.862817 (+397784us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:17.864103 (+  1286us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:17.864702 (+   599us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:18.361961 (+497259us) raft_consensus.cc:1671] finished
0809 23:21:18.361991 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.366721 (+  4730us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5618,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9831,"thread_start_us":395982,"threads_started":1,"wal-append.queue_time_us":590113}
I20260809 23:21:18.388849  2421 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:18.329582  2577 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.442674  2505 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.415205 25373 rpcz_store.cc:276] 0809 23:21:15.180233 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.351895 (+2171662us) service_pool.cc:227] Handling call
0809 23:21:17.401446 (+ 49551us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:17.447346 (+ 45900us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:17.447437 (+    91us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:21:17.448464 (+  1027us) log.cc:842] Serialized 78 byte log entry
0809 23:21:17.978268 (+529804us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:18.407875 (+429607us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.408371 (+   496us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.410657 (+  2286us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.410851 (+   194us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.410955 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:18.411013 (+    58us) raft_consensus.cc:1671] finished
0809 23:21:18.411030 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.411181 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8587,"prepare.queue_time_us":91039,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":45,"raft.queue_time_us":2012,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":194,"thread_start_us":877722,"threads_started":2,"wal-append.queue_time_us":740064}
I20260809 23:21:18.444300  2115 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.445358  2577 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.446264  2115 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:18.431035  2598 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260809 23:21:18.447635  2333 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.421149  2416 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:18.451216  2463 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:18.451702  2416 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:18.452219  2416 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:18.452960  2463 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:18.453349  2463 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:18.455025  2592 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:18.455601  2592 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.456346 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:18.457356  2127 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.563s	user 0.013s	sys 0.001s
I20260809 23:21:18.458149  2592 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.459111  2189 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:18.460786  2189 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:18.462105  2189 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:18.462549  2189 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:18.454241  2463 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.463068  2315 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260809 23:21:18.463349 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45829.2us
I20260809 23:21:18.463994  2463 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:18.464604 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:18.465679 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:18.467006  2200 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.467876  2006 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:18.406628  2501 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:18.405756  2469 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.405753  2345 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:18.389330  2194 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.415055 24313 rpcz_store.cc:276] 0809 23:21:15.807506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.145889 (+1338383us) service_pool.cc:227] Handling call
0809 23:21:17.329182 (+183293us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.400538 (+ 71356us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:17.400989 (+   451us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.180071 (+779082us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:18.180143 (+    72us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:18.180204 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.386434 (+206230us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:18.386490 (+    56us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.404773 (+ 18283us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b540007e250. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:18.404926 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1838,"prepare.queue_time_us":347,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":4791,"raft.queue_time_us":6792,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":13949,"spinlock_wait_cycles":544708352,"thread_start_us":699624,"threads_started":2,"wal-append.queue_time_us":627620}
I20260809 23:21:18.410050 25370 rpcz_store.cc:276] 0809 23:21:16.471939 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.377993 (+906054us) service_pool.cc:227] Handling call
0809 23:21:17.435086 (+ 57093us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.488600 (+ 53514us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:17.969002 (+480402us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.404542 (+435540us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:18.404611 (+    69us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:18.404658 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.404748 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.404905 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":567951,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":10365,"thread_start_us":856933,"threads_started":2}
I20260809 23:21:18.458796  2592 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:18.474751 24427 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:18.478751 24164 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 14, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 14 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:18.463114  2508 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:18.469009  2378 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:18.485055 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 126815us
I20260809 23:21:18.485870 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:18.486472 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 103102us
I20260809 23:21:18.487349 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1606) took 2577 ms. Trace:
I20260809 23:21:18.482908  2588 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:18.497864 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:18.501602 24164 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 15, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 15 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.512392 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:18.487797 25375 rpcz_store.cc:276] 0809 23:21:15.910426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.392957 (+1482531us) service_pool.cc:227] Handling call
0809 23:21:17.393438 (+   481us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:17.507095 (+113657us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:18.089680 (+582585us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.483559 (+393879us) log.cc:842] Serialized 14 byte log entry
0809 23:21:18.483744 (+   185us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:18.483795 (+    51us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:18.487120 (+  3325us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.487187 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:18.487292 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":533990,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":2634,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":15830,"spinlock_wait_cycles":24580096,"thread_start_us":922534,"threads_started":2}
I20260809 23:21:18.536432 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:18.538074  2378 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:18.539073  2378 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:18.539791  2378 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:18.538961  2611 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.002s
I20260809 23:21:18.541031  2378 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.538275 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:18.543033  2189 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.544914  2189 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:18.546375  2611 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:18.526877  2508 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.552177  2601 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:18.559864  2601 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:18.560316  2601 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:18.561273  2508 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:18.551918 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57564.7us
I20260809 23:21:18.527169  2588 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:18.546788  2611 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:18.563525  2611 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:18.563239  2588 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.564973 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.546792 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:18.566378 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.562779 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.571128 24539 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:18.572049 24534 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.526s	user 0.007s	sys 0.002s
I20260809 23:21:18.567157  2505 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:18.577711 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:18.579664 24164 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:18.574280 24534 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:18.577903 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.583211  2587 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:18.583813 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.583969 24314 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:18.584451 24428 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:18.585409  2587 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:18.585798  2587 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:18.530438  2345 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:18.558916  2154 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.831s	user 0.010s	sys 0.001s
I20260809 23:21:18.591554 24429 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:18.592298  2268 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:18.593185 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 339005us
I20260809 23:21:18.592867  2268 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.597339  2268 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.600174 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:18.601528  2268 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:18.376149 24163 rpcz_store.cc:269] 0809 23:20:35.022701 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:35.032790 (+ 10089us) service_pool.cc:227] Handling call
0809 23:21:18.375422 (+43342632us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:21:12.668609 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:21:12.669404 (+   795us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:21:12.669473 (+    69us) write_op.cc:435] Acquired schema lock
0809 23:21:12.669498 (+    25us) tablet.cc:660] Decoding operations
0809 23:21:12.671065 (+  1567us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:21:12.671187 (+   122us) write_op.cc:641] Partition lock acquired for write op
0809 23:21:12.671243 (+    56us) tablet.cc:683] Acquiring locks for 13 operations
0809 23:21:12.672857 (+  1614us) tablet.cc:699] Row locks acquired
0809 23:21:12.672896 (+    39us) write_op.cc:260] PREPARE: finished
0809 23:21:12.673105 (+   209us) write_op.cc:270] Start()
0809 23:21:12.673277 (+   172us) write_op.cc:276] Timestamp: P: 1786317672673023 usec, L: 0
0809 23:21:12.673321 (+    44us) op_driver.cc:331] REPLICATION: starting
0809 23:21:12.674219 (+   898us) log.cc:842] Serialized 5104 byte log entry
0809 23:21:15.054485 (+2380266us) op_driver.cc:444] REPLICATION: finished
0809 23:21:15.700827 (+646342us) write_op.cc:301] APPLY: starting
0809 23:21:15.718749 (+ 17922us) tablet.cc:1365] starting BulkCheckPresence
0809 23:21:16.904564 (+1185815us) tablet.cc:1368] finished BulkCheckPresence
0809 23:21:16.904736 (+   172us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:21:17.024622 (+119886us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:21:17.025100 (+   478us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0809 23:21:17.025182 (+    82us) write_op.cc:312] APPLY: finished
0809 23:21:17.025704 (+   522us) log.cc:842] Serialized 121 byte log entry
0809 23:21:17.282820 (+257116us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:17.283721 (+   901us) write_op.cc:454] Released schema lock
0809 23:21:17.283875 (+   154us) write_op.cc:341] FINISH: Updating metrics
0809 23:21:17.284508 (+   633us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:17.284544 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1010680,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":584508,"apply.run_cpu_time_us":34725,"apply.run_wall_time_us":1609385,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58415,"num_ops":13,"prepare.queue_time_us":440464,"prepare.run_cpu_time_us":32706,"prepare.run_wall_time_us":539557,"raft.queue_time_us":1536918,"raft.run_cpu_time_us":72912,"raft.run_wall_time_us":488382,"replication_time_us":2332413,"thread_start_us":2127967,"threads_started":5,"wal-append.queue_time_us":783969,"wal-append.run_cpu_time_us":128009,"wal-append.run_wall_time_us":2791257}]]}
I20260809 23:21:18.605315  2333 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:18.606086  2333 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.608564  2333 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.551559  2200 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.612546  2200 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.613627 24321 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
W20260809 23:21:18.614287 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 318014us
W20260809 23:21:18.594288  2582 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.160s	user 0.000s	sys 0.004s
I20260809 23:21:18.620133 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:18.620851 24534 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:18.620931 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.621667 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:18.627701 24428 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:18.484601  2469 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.631488 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:18.632501 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63634.9us
I20260809 23:21:18.636974 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.598178  2416 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:18.641551 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 155748us
I20260809 23:21:18.639695 24164 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 13, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:18.648814 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:18.660925 24431 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:18.662098 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:18.645486 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:18.678439 24164 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 2 to 12, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 12 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:18.690825 24313 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260809 23:21:18.695576 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:18.645776  2268 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.645992  2505 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:18.696780 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:18.717638 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85301.8us
I20260809 23:21:18.698963 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:18.718480 24425 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:18.567745  2421 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.707499  1539 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:18.707934 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:18.656948  2154 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.635353  2333 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260809 23:21:18.726419 24431 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260809 23:21:18.690541  1485 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.726504 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.724021 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:18.726822 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:18.730888 24546 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:18.727017 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:18.722532 24430 raft_consensus.cc:2354] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:18.705727  2119 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.740247 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:18.744582 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:18.724855  2030 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:18.748791 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.752072 25368 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 17. Preceding OpId from leader: term: 18 index: 18. (index mismatch)
W20260809 23:21:18.734110  2322 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.874s	user 0.008s	sys 0.004s
I20260809 23:21:18.762166 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:18.762181 24545 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:18.768083 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.772255 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:18.768807 24541 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:18.744095  2194 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.789945 24534 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:18.724277  2345 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260809 23:21:18.807679 24496 leader_election.cc:398] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:18.818379 24496 leader_election.cc:401] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260809 23:21:18.723801  2268 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.761471  2200 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.814267 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:18.814821 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:18.815639 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 173216us
I20260809 23:21:18.733989  1485 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.719369  2628 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260809 23:21:18.733733  2154 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.811468 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:18.796770  2030 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:18.828761 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 72800.3us
I20260809 23:21:18.827886 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.836570 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:18.842669 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:18.836651 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:18.825449 24498 connection.cc:232] RPC call timeout handler was delayed by -1.6369820940308273s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260809 23:21:18.844892 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.853549 24430 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:18.852591 24498 connection.cc:232] RPC call timeout handler was delayed by -1.6144820938352495s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
W20260809 23:21:18.858230 24498 connection.cc:232] RPC call timeout handler was delayed by -1.4704820939805359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260809 23:21:18.827613  2463 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:18.860175 24534 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:18.769084  2119 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.816622  1539 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.032s
I20260809 23:21:18.874706 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:18.875020 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.868638  2463 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:18.897014 24317 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:18.870381  2333 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:18.888804 25368 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:18.880731  2416 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.908782 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.909176 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.912957 24316 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:18.913211  2650 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260809 23:21:18.914633  2650 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:18.915028  2650 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:18.915884  2650 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.917291  2650 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:18.919221  2600 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:18.919835  2600 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.922073  2600 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:18.922533  2623 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:18.923203  2623 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:18.923238  2333 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:18.924325  2623 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:18.925515  2333 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.885252  2469 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:18.911538 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:18.903843  2463 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:18.841864  2505 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:18.927099  2596 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.344s	user 0.000s	sys 0.003s
I20260809 23:21:18.953197  1485 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:18.954129  2579 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:18.954605  2322 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:19.020220 24311 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260809 23:21:18.959726  2633 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260809 23:21:18.960314 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:18.960871 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:19.029110 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68354.6us
W20260809 23:21:18.961541  2620 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
W20260809 23:21:18.963943 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 142824us
W20260809 23:21:18.965082  2627 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260809 23:21:18.965859  2608 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260809 23:21:18.970384  2639 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
W20260809 23:21:19.043823 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 177051us
I20260809 23:21:18.989112 25373 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:18.996148 25366 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:18.996968 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:18.998876 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.001873 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:19.027314  2469 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:19.049232 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.054111 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:19.054718 24317 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.037876  2577 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:19.000947 24425 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:19.035048 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.039485 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:19.042025 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 220134us
I20260809 23:21:19.042778 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:19.046454 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:18.906062  2593 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.285s
I20260809 23:21:19.058648 25368 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:19.058694 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.045378 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:19.020049  2345 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:19.051486  2619 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260809 23:21:19.051756  1677 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:19.028018  2579 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.063982 24316 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:21:19.041296  2030 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.073669  2268 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.043941  2639 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:19.074715 24312 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.076998 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.059754  2592 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:19.079759 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.088455 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.080715 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3374) took 1141 ms. Trace:
I20260809 23:21:19.079507 24428 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:19.054638  2469 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:19.107517 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.109382 24311 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:19.084545  2115 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.638s	user 0.009s	sys 0.006s
I20260809 23:21:19.061177  2577 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.040822  2322 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.157941 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.083913  2625 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:19.153131 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:19.161824 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 86043.7us
W20260809 23:21:19.108979  2587 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.003s
I20260809 23:21:19.168521  2587 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.169397 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.170336  2508 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:19.170876  2508 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:19.096942 24544 rpcz_store.cc:276] 0809 23:21:17.939350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.939491 (+   141us) service_pool.cc:227] Handling call
0809 23:21:17.976046 (+ 36555us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:18.335644 (+359598us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:18.335693 (+    49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:18.374332 (+ 38639us) log.cc:842] Serialized 52 byte log entry
0809 23:21:18.382835 (+  8503us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:18.656498 (+273663us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:18.656560 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:18.656653 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:19.080362 (+423709us) raft_consensus.cc:1671] finished
0809 23:21:19.080386 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:19.080653 (+   267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23051,"raft.queue_time_us":362452,"raft.run_cpu_time_us":2457,"raft.run_wall_time_us":31395,"spinlock_wait_cycles":12567424,"thread_start_us":609169,"threads_started":2,"wal-append.queue_time_us":335081}
I20260809 23:21:19.171905  2508 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.159904 24541 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:19.167526 24312 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260809 23:21:19.173266  2659 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260809 23:21:19.173363  2508 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:19.084188  2154 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.082782  2579 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:19.161957  1731 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:19.175735 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:19.177049  2378 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:19.177587  2378 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.177968  2378 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:19.170527  2587 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:19.083729  2119 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.180249 24317 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:19.181447 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.182252 24317 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.177776 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:19.188788 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 130169us
I20260809 23:21:19.189680 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.189914  2322 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.190347 24314 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 14.
I20260809 23:21:19.200800 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.191252  2598 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:19.190791  2639 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.201756  2598 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:19.202099  2611 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.005s
I20260809 23:21:19.192771  1677 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:19.175735  1731 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:19.193614 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.204320  1731 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.202756  2598 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.207394  2469 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.208932  2469 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.194324  2625 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:19.194945  2194 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:19.210991  2625 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.195780  2593 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:19.211449 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.212136  2115 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:19.195878  2587 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:19.212797  2115 consensus_queue.cc:1225] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:19.196765 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.197333  2588 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:19.214829 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.199893 24312 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:19.202170  2639 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:19.159058  2577 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.212620 24317 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:19.187160 24309 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:19.229209  2119 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:19.186160 24313 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:19.229967  2119 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.232365 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.233915 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:19.192807  2601 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.000s
I20260809 23:21:19.243633 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:19.243768 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.244378 24425 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.240628 24316 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:19.249477 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.156234  2592 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:19.252143 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.253764 24422 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:19.222112 24321 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:19.254963 24311 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260809 23:21:19.255120 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:19.243026 25323 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.603s (SENT)
I20260809 23:21:19.252750 24428 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:19.258971 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:19.258533 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.247951  2623 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:19.276386 24313 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:19.254438 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.233052  2119 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.247897  2601 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.264000  2671 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:19.264220 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.247764  2579 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:19.266078 24535 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: [15.11-20.13]
I20260809 23:21:19.267063 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:19.268110 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.270506 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.271190 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.273111 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.256498 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.228261  2600 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:19.260181 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.277411 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:19.282835 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:19.290999 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.292323 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.293530 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.278748 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.279562 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.262640 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.248020  2628 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:19.294878 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.299685 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.282088 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.265580 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.217515  2611 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.303288 24541 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:19.303288 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.305540 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.273311  2154 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:19.314296 24321 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:19.276919  2593 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:19.320586 24428 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:19.319283 24312 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:19.321496 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.322242 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.284687  2189 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:19.326426 24316 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 15.
I20260809 23:21:19.328050 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.322822 24425 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:19.286494  2671 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.331632 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.331765  2333 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:19.327647  2590 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:19.332976  2590 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.333364  2590 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.333854  2671 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.334045 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:19.335317  2006 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:19.335847  2006 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:19.336738  2006 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.337421 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.338577 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.339267 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.339349 24311 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:19.340756 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.340971 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.341922 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.287976  2600 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:19.341686 24311 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:19.342859 24320 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260809 23:21:19.357726 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.357865 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.358518 24313 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:19.358982 24311 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:19.360002 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.361341 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.324596  2154 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.329149  2189 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.367465 24311 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:19.328665  2171 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:19.370146 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.367758 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.371181 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.368752  2052 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:19.373512 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.384824 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.385521 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:19.383524 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:19.394405 24317 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.395716 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.331897  2652 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:19.375067  2333 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:19.297089  2579 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.367342  2154 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:19.375489 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.378844 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.432760 25375 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:19.379915 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.433655 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
W20260809 23:21:19.400586 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 157582us
W20260809 23:21:19.400170 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 204720us
I20260809 23:21:19.374475 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.434986 24321 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.385509 24316 raft_consensus.cc:2362] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19. Re-sending same reply.
I20260809 23:21:19.402434 24427 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:19.381629  2189 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:19.328809  2593 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.437566 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.438237 24311 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:19.441211 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.384260  2052 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:19.440882 24320 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:19.345350  2600 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.445959 24312 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:19.415946  2333 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:19.441764 25323 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.110s (SENT) [suppressed 20 similar messages]
I20260809 23:21:19.459136 24317 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:19.372195  2463 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.461340 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.461982 24544 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:19.476917 25373 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:19.478133 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.366060  2592 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.446190  2052 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:19.479367 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.484625 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:19.484414  2333 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.486426  2593 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:19.487152  2593 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:19.487653  2593 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:19.491966  2611 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.493752 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.494818 24425 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:19.486516 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:19.498325 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.462055  2623 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:19.505883 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.483516 25375 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 14.
I20260809 23:21:19.504530 24323 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 18. Ops: []
I20260809 23:21:19.476331  2600 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:19.515403 24317 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
W20260809 23:21:19.523133 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3311) took 1221 ms (client timeout 1534 ms). Trace:
W20260809 23:21:19.523669 24542 rpcz_store.cc:269] 0809 23:21:18.282196 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:18.302828 (+ 20632us) service_pool.cc:227] Handling call
0809 23:21:19.441706 (+1138878us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b54001e0620. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863eff 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86
0809 23:21:19.498723 (+ 57017us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001d5e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:19.503836 (+  5113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2555912832}
I20260809 23:21:19.529999 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.530846 25375 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:19.538363  2650 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.621s	user 0.009s	sys 0.004s
I20260809 23:21:19.543462 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:19.544085 25369 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.547084  2333 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:19.547789 25370 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:19.549189 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.549867 25365 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:19.550992 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.555799 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.556775 25371 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260809 23:21:19.559299 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.560482 24425 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:19.564110 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:19.565068 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 67607.4us
I20260809 23:21:19.573112 25372 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 13.
I20260809 23:21:19.574131 25373 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:19.576916 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.514956  2623 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.512589  2687 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:19.439145  2652 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.588552 24425 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260809 23:21:19.583554 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:19.626128 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.626559 24323 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260809 23:21:19.585451 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:19.593608 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 248999us
W20260809 23:21:19.628430 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.633s (SENT)
I20260809 23:21:19.629972 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.631132 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:19.631875 25371 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:19.559149  1677 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:19.636059 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:19.642536 25375 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:19.638402 25373 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:19.644744 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:19.645483 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:19.637218 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.593662  2623 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:21:19.696671 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59534.4us
I20260809 23:21:19.705062 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.696998 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.592598  2601 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:19.704105 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74258.8us
I20260809 23:21:19.707309 24543 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:19.711570 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:19.643843  2416 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260809 23:21:19.583324  2650 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.117s
I20260809 23:21:19.719693 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:19.731076 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:19.703859  2171 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260809 23:21:19.643851  1677 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.704254  2268 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:19.746388 25372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:19.751446 25372 raft_consensus.cc:595] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:19.750470 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 312987us
I20260809 23:21:19.751900 25372 raft_consensus.cc:603] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:19.711170  2625 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:19.755654 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:19.745862  2006 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:19.755663 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:19.758504 24425 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:19.742473  2601 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:19.703394  2189 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:19.653021  2590 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.755489  1677 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:19.746125  2171 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:21:19.794652 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 193939us
I20260809 23:21:19.766999  2416 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260809 23:21:19.783353 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76269us
I20260809 23:21:19.705295  2687 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260809 23:21:19.786727 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:19.837677  2577 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.582s	user 0.007s	sys 0.003s
I20260809 23:21:19.790922 25369 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:19.787840  2639 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:19.758956  2030 raft_consensus.cc:2741] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260809 23:21:19.763321  2006 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:19.780174 24425 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:19.766986  2579 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:19.841586 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.662s (SENT)
I20260809 23:21:19.868809 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:19.797231 24320 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260809 23:21:19.583582  2593 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.795375  2171 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:19.840221  2463 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:19.865913 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:19.903260 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:19.910075 24427 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:19.910293 24313 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:19.869614  2378 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.000s
W20260809 23:21:19.919730  2677 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.518s	user 0.000s	sys 0.002s
W20260809 23:21:19.884862  2682 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.371s	user 0.001s	sys 0.000s
I20260809 23:21:19.927634 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:19.884702  2006 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:19.934767 24545 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260809 23:21:19.938265 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 205716us
I20260809 23:21:19.909322  2171 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:19.927932  2643 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260809 23:21:19.867710  2639 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:19.959805 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:19.931586  2322 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:19.945741 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 170064us
I20260809 23:21:19.969295 24425 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260809 23:21:19.887277  2577 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:19.974697 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.977864 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:19.959126  2670 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:21:19.980756 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 185236us
W20260809 23:21:19.990026 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.762s (SENT)
I20260809 23:21:19.990561 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:19.947208  2587 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260809 23:21:19.998304 24316 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:19.934201  2378 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.001240 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.965312  2652 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:19.996922 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:19.997699 24424 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:20.002061 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:19.983078  2639 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:20.023072  2189 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:19.982698 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:20.027872 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45243us
I20260809 23:21:20.029050  1539 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:20.029748  2590 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.030215  2691 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:20.030638  2691 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:20.030484  1539 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:20.031432  1539 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 21.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:20.032495 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:20.008526  2689 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.310s	user 0.001s	sys 0.001s
I20260809 23:21:19.993376  2670 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:19.975196  1429 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.036s
I20260809 23:21:20.014667 25371 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:20.015173  2690 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260809 23:21:20.015607 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:20.016434  2577 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.067926 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:20.016897 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:20.069808 25373 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:20.016386 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:20.070613 24316 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260809 23:21:20.018612 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.072469 24543 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:20.018746  2707 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260809 23:21:20.019266  2688 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
W20260809 23:21:20.020054  2692 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260809 23:21:20.020157 25323 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.060s (SENT)
W20260809 23:21:20.020797  2695 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
W20260809 23:21:20.021595  2701 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260809 23:21:20.021819  2721 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:20.015169 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:20.008080  2700 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260809 23:21:19.946310  2697 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260809 23:21:20.036090 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:20.033843  2650 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 23:21:20.037817  2691 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.040606 24427 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260809 23:21:20.120654 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100522us
I20260809 23:21:20.049312 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.049312 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:19.989589  2667 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:20.008982  2469 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.800s	user 0.006s	sys 0.005s
I20260809 23:21:20.055822 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.051906 25375 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
W20260809 23:21:20.132174 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76454.5us
I20260809 23:21:20.062922 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:20.049839  2333 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.002s
I20260809 23:21:20.071425 25366 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:20.062508  2670 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:20.076900  2508 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.903s	user 0.012s	sys 0.002s
I20260809 23:21:20.074501  1429 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260809 23:21:20.117251 25365 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:20.056599  2322 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
I20260809 23:21:20.118058 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:20.125041 24427 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:20.059607 24317 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:20.134217 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:20.154275 24546 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260809 23:21:20.154697 24546 pending_rounds.cc:76] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:21:20.154919 24546 pending_rounds.cc:97] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 18.12
W20260809 23:21:20.136518 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 170346us
I20260809 23:21:20.144868 24309 raft_consensus.cc:2396] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260809 23:21:20.155241 24546 raft_consensus.cc:2866] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:20.150290 25366 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
W20260809 23:21:20.136595 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87355.4us
I20260809 23:21:20.156085 25369 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260809 23:21:20.157693 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:20.152366 25370 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:20.161087 25373 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260809 23:21:20.122157  2601 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:20.137979  1539 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:20.148852  2652 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:20.144124  2579 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:20.175777 24270 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.015s	sys 0.010s
W20260809 23:21:20.176309 24270 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.015s	sys 0.010s
W20260809 23:21:20.171878 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122649us
I20260809 23:21:20.149329  2667 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:20.176733 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 698051us
I20260809 23:21:20.171452 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:20.182313 24430 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:20.185776 24535 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:20.186470 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:20.187117 25373 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:20.187907 25369 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260809 23:21:20.193572 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:20.194185 24427 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:20.187184 25366 pending_rounds.cc:76] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 13
I20260809 23:21:20.194712 25366 pending_rounds.cc:97] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260809 23:21:20.172449  2691 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:20.196528 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.195094 25366 raft_consensus.cc:2866] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:20.220709 24323 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:20.159976  1677 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.201421 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.249693 24545 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:20.202108 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.203562 24422 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:20.204859 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:20.207620  2601 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.208557  2378 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:20.209168  2600 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.000s
W20260809 23:21:20.211530  2154 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.767s	user 0.010s	sys 0.001s
I20260809 23:21:20.215245  2587 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:20.154577  2650 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.032s
I20260809 23:21:20.217025 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:20.200035 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61862.1us
I20260809 23:21:20.180806  2652 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:20.191035  2598 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.229410 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.262533 24543 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:20.230238  2722 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260809 23:21:20.197348 24321 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 13
W20260809 23:21:20.222524  2052 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.697s	user 0.009s	sys 0.003s
I20260809 23:21:20.187407  2667 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.252137 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.194921  2748 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:20.177554  2508 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.055s
I20260809 23:21:20.264590 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:20.263015  2154 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.269922  2598 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.222942  2728 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260809 23:21:20.263659  2652 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:20.271898  1677 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:20.275357 24321 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260809 23:21:20.320708 24321 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:20.319067 24545 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:20.279675 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:20.321722 24431 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:20.278496  2720 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:20.283783 24540 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260809 23:21:20.281762 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.876s (SENT)
I20260809 23:21:20.306754 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.313426 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.246833  2662 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:20.274565 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:20.322826 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70774.2us
I20260809 23:21:20.329239 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.329944 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.330731 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.335127 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.335909 24320 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:20.318012  2707 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:20.327728 24316 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:20.318921 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:20.298712  2592 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:20.349604 24427 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:20.341964 25363 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:20.340735  1731 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:20.366485 25370 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:20.320092  2579 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:20.316315  2748 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:20.361883  2598 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:20.362455 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88120.5us
I20260809 23:21:20.305408  2600 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:20.366496 24320 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:20.367501 25370 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:20.367616 24313 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:20.336383  2720 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.367920 25373 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:20.341588  2333 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:20.361548 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79833.6us
I20260809 23:21:20.394996 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.284461  2171 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.369755  2322 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260809 23:21:20.401201  2623 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.000s
I20260809 23:21:20.396319  2579 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:20.410101 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:20.415868  2697 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:20.400632  2154 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.412061 25370 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.363487  2416 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:20.417418 24430 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:20.430018 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.437479 25373 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:20.419088 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.420606 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:20.414979  2333 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:20.423166 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.305121  2052 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.423542 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:20.427639 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89455.9us
I20260809 23:21:20.376096  2587 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:20.416908 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87072.1us
I20260809 23:21:20.459956 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.394377  2667 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.433269 25372 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 19. Ops: []
I20260809 23:21:20.443787 24540 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:20.365902  2662 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.458123 24430 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: [19.12-21.13]
W20260809 23:21:20.461112 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66232.4us
I20260809 23:21:20.396574  2671 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260809 23:21:20.396708  2592 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.422859  2639 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.471335 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.455701  2333 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:20.424345  2469 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.034s
I20260809 23:21:20.470996 24431 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:20.483985  2675 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
I20260809 23:21:20.469161  2748 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.443195  2416 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.490239  2416 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.491760 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1661) took 1221 ms. Trace:
I20260809 23:21:20.491192  2579 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.430754  2623 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.492182 24309 rpcz_store.cc:276] 0809 23:21:19.270098 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.270249 (+   151us) service_pool.cc:227] Handling call
0809 23:21:20.491563 (+1221314us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540043b2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:20.491700 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1957385984}
I20260809 23:21:20.493701  2736 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:20.494139  2736 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:20.494596  2736 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:20.494951  2588 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:20.495198 24320 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:20.495410  2588 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:20.496299  2588 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.497394  2588 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:20.498063 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.491988  2662 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:20.497494  2706 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.517s	user 0.001s	sys 0.004s
I20260809 23:21:20.502293 24423 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:20.502445 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:20.500130 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.498703  2764 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.027s
I20260809 23:21:20.483101  2667 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:20.507965  2746 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260809 23:21:20.508124 24535 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:20.509557  2268 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:20.527684  2119 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:20.530004  2119 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:20.509690  2171 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.479780  2587 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:20.530910  2593 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:20.518085 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.519856 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.534873 24543 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:20.524561 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:20.515654 24545 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:20.535851 24430 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:20.530447  2119 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:20.540241 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68991.1us
I20260809 23:21:20.540406 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:20.541013 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.494092  2623 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:20.581125 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63220.1us
I20260809 23:21:20.582777 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.594089 24430 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:20.610581 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:21:20.620620 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 120562us
I20260809 23:21:20.623546 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:20.528528  2667 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:20.628615 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:20.632422 24430 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:20.590878  2662 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260809 23:21:20.634526 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 215535us
W20260809 23:21:20.635999 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95080.6us
I20260809 23:21:20.620110  2268 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:20.658913  2765 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:20.662161  2760 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260809 23:21:20.640607 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:20.671370 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88678.2us
I20260809 23:21:20.672436 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.642078 24378 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:21:20.655105  2702 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:20.657198 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3326) took 1340 ms. Trace:
I20260809 23:21:20.657665  2416 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.658350  2469 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260809 23:21:20.632797  2601 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.679972 24546 rpcz_store.cc:276] 0809 23:21:19.317146 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.317264 (+   118us) service_pool.cc:227] Handling call
0809 23:21:19.328096 (+ 10832us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:20.160938 (+832842us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:20.656929 (+495991us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b5400080a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:20.657128 (+   199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1899255552}
I20260809 23:21:20.678229 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1635) took 1299 ms. Trace:
I20260809 23:21:20.681888 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:20.686794 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46302.1us
I20260809 23:21:20.680261 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.686749 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.675483  2579 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:20.676852 24378 leader_election.cc:401] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:20.690454 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.676286  2768 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:20.685218 25368 rpcz_store.cc:276] 0809 23:21:19.378896 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.379073 (+   177us) service_pool.cc:227] Handling call
0809 23:21:19.379650 (+   577us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:19.379923 (+   273us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:19.379953 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:19.380669 (+   716us) log.cc:842] Serialized 26 byte log entry
0809 23:21:19.747542 (+366873us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:20.171895 (+424353us) log.cc:842] Serialized 14 byte log entry
0809 23:21:20.665232 (+493337us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b540073f120. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f3fb3 00007fac23860f0f 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c
0809 23:21:20.665304 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:20.665425 (+   121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:20.665490 (+    65us) raft_consensus.cc:1671] finished
0809 23:21:20.677957 (+ 12467us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:20.678155 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":680,"prepare.queue_time_us":214571,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":71,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":7153,"spinlock_wait_cycles":59371264,"thread_start_us":761893,"threads_started":2,"wal-append.queue_time_us":462981}
I20260809 23:21:20.673712  2463 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.638043  2662 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:20.677114  2052 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.782456  2734 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:20.782853  2766 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
W20260809 23:21:20.782613 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92266.9us
W20260809 23:21:20.783444 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2837) took 1588 ms (client timeout 1554 ms). Trace:
I20260809 23:21:20.783799 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:20.783799 24314 rpcz_store.cc:269] 0809 23:21:19.195114 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.200566 (+  5452us) service_pool.cc:227] Handling call
0809 23:21:20.783227 (+1582661us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:20.783372 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2238477056}
W20260809 23:21:20.784601  2723 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.641s	user 0.005s	sys 0.005s
I20260809 23:21:20.785723  2734 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:20.782750 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3388) took 1514 ms. Trace:
I20260809 23:21:20.786743 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.787236 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3002) took 1030 ms. Trace:
I20260809 23:21:20.787905 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.788234  2631 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:20.790158 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.790697 24545 pending_rounds.cc:76] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 10
I20260809 23:21:20.791071 24545 pending_rounds.cc:97] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260809 23:21:20.791381 24545 raft_consensus.cc:2866] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:20.787739 24423 rpcz_store.cc:276] 0809 23:21:19.756570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.756763 (+   193us) service_pool.cc:227] Handling call
0809 23:21:19.782814 (+ 26051us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:20.524710 (+741896us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:20.780207 (+255497us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54000af354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:20.787176 (+  6969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1654209920}
I20260809 23:21:20.787132 24534 rpcz_store.cc:276] 0809 23:21:19.268086 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.268393 (+   307us) service_pool.cc:227] Handling call
0809 23:21:19.268687 (+   294us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:19.692836 (+424149us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:19.692879 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:19.958242 (+265363us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:19.958598 (+   356us) log.cc:842] Serialized 26 byte log entry
0809 23:21:20.141338 (+182740us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:20.141371 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:20.782417 (+641046us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:20.782488 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:20.782539 (+    51us) raft_consensus.cc:1671] finished
0809 23:21:20.782551 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:20.782691 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1121453312,"thread_start_us":147155,"threads_started":1,"wal-append.queue_time_us":226506}
I20260809 23:21:20.689370  2268 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:20.799494 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:20.679762  2702 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:20.683462  2469 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:20.769824  2720 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:20.791489  2670 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.634s	user 0.008s	sys 0.004s
I20260809 23:21:20.802637 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.796270 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:20.809113 24546 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
W20260809 23:21:20.802039  2691 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.575s	user 0.005s	sys 0.005s
I20260809 23:21:20.688112  2416 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.678864  2721 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:20.738698  2598 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260809 23:21:20.781937  2662 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:20.762003  2725 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260809 23:21:20.811138  2734 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:20.822891 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.758069  2601 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:20.853035 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 211123us
I20260809 23:21:20.879666 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.798827  2052 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:20.776388  2463 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.803547  2768 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:20.798600  2631 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:20.806985  2469 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 14 FOLLOWER]: Advancing to term 15
W20260809 23:21:20.812862 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 126210us
W20260809 23:21:20.875530 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 195341us
I20260809 23:21:20.822515  2623 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:20.875730 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52970.6us
I20260809 23:21:20.878518 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:20.859081  2639 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.843640  2378 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:20.823312  2579 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:20.842221  1429 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:20.917316  1429 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.919520  1429 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.844344  2765 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 15. Current term is 16.
I20260809 23:21:20.919710 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.922076 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.914515  2623 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:20.941469 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.899389 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:20.901772 24546 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:20.789101  2667 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.904397  2754 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.951491  2463 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.951323  2754 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:20.952332  2463 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:20.952068  2770 consensus_queue.cc:1159] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 18 status { last_received { term: 18 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 18 index: 11 } }
I20260809 23:21:20.953043  2463 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.904878  2779 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:20.905422  2631 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:20.906216  2776 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260809 23:21:20.864477  2030 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:20.871814  2728 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.812110  2670 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.873750  2592 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.922966  2726 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
W20260809 23:21:20.923667  2774 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260809 23:21:20.923884  2508 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:20.924226  2598 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:20.904901  2768 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.893268  2052 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:20.939536 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:20.946511 24534 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:20.994236 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:20.898967 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:20.854480  2721 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:20.904772  2763 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260809 23:21:20.962327 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:20.913122  2720 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:21:20.963658 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64355us
I20260809 23:21:21.005898  2720 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:21.006366  2720 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:20.964716 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 162131us
I20260809 23:21:21.007134  2764 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:21.007612  2764 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.008023  2764 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:20.967324 24546 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:20.954818  2754 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.023869 24380 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:20.980271 24317 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:20.951602  2667 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:20.991549 24425 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:20.963766  2588 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:20.995414 24435 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:20.978544  2770 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:21.001142 24535 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:20.997325  2052 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:21.066295  2720 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.067632  2416 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.068596  2463 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.068957  2194 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:21.069267  2463 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.069862 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:21.069998  2463 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.964139  2736 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.070921 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3417) took 1075 ms. Trace:
I20260809 23:21:21.071334 24425 rpcz_store.cc:276] 0809 23:21:19.995419 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:20.006385 (+ 10966us) service_pool.cc:227] Handling call
0809 23:21:20.006719 (+   334us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:21.027217 (+1020498us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:21.070666 (+ 43449us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:21.070856 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2254252928}
I20260809 23:21:21.071762 24316 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:21.072501  2736 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:20.976271  2728 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.000487  2721 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:21.004709  2006 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.029s	user 0.006s	sys 0.007s
W20260809 23:21:21.005407 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.070s	sys 0.126s
I20260809 23:21:21.002810 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:21.080304 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.071s	sys 0.126s
I20260809 23:21:21.080060 24535 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:21.080653  2720 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:20.875154  2702 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:21.082854 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80161.3us
I20260809 23:21:21.083060  2720 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:21.083326  2702 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:21.085140 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:21.086100  2322 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.086288  2720 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.086843  2720 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:20.994129  2741 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:21.087548 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:21.028498  2748 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:21.032739  2333 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260809 23:21:21.023940  1677 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:21.036521  2670 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.036795  2781 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:21:21.038777  2754 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:21.041198  2795 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:21.028374  1485 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.014s
I20260809 23:21:21.043823  2592 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.050665  2764 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.000538  2598 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.054749 25370 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:21.056632 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1648) took 1043 ms. Trace:
I20260809 23:21:21.058177  2119 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.069396  2194 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:21.004199 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105335us
I20260809 23:21:21.145413 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:21.146720 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:21.077992 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:21.133399 25363 rpcz_store.cc:276] 0809 23:21:20.013244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:20.015292 (+  2048us) service_pool.cc:227] Handling call
0809 23:21:21.056436 (+1041144us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:21.056573 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":604577152}
I20260809 23:21:21.147764 24543 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:21.079586 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:20.979339  2726 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.087193 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:21.017745 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:21.038345  2784 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:21:21.093487 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:21.174082 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80729us
I20260809 23:21:21.095258 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.038607  2378 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:21.074337  2774 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:21:21.181779 25370 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260809 23:21:21.112334  2725 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.101133 24431 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: [18.11-19.12]
I20260809 23:21:21.101289 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:21.185092 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83888us
I20260809 23:21:21.103480 24435 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:21.106953 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:21.108002 24425 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:21.186655 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.108991 24535 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:21.099404  2154 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:21.002214  2508 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260809 23:21:21.110908 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:21.114013 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.052107  1527 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:21.089354  2736 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.113137 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.086588  2322 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.152453 25323 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:21.146023  2592 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.164686  2119 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.204082 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 242165us
I20260809 23:21:21.181463  2625 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:21.165596  2194 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.166879  2598 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.103986  2333 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.168708  2463 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.148718 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 69217.1us
I20260809 23:21:21.170632  2726 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.218468 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:21.171988 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 154321us
I20260809 23:21:21.172550  1485 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:21.034539  2623 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.221320 24427 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:21.178315 24430 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 16: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:21.075309  2652 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.701s	user 0.012s	sys 0.000s
I20260809 23:21:21.182153 25370 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 12
I20260809 23:21:21.233918 25370 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260809 23:21:21.234184 25370 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:21.234805 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1612) took 1152 ms. Trace:
I20260809 23:21:21.192349 24423 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:21.173282  2268 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:21.128346  2754 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:21.196931  2717 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:21.197260 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:21.198509 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46143.5us
I20260809 23:21:21.092828  2006 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.102824  2748 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.182621  2593 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:21.211503  2601 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.214323  2119 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:21.203778  1429 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:21.227698 24431 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260809 23:21:21.211208  2030 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.303684  2030 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.232081  2194 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:21.235136 25370 rpcz_store.cc:276] 0809 23:21:20.082554 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:20.365693 (+283139us) service_pool.cc:227] Handling call
0809 23:21:20.366358 (+   665us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:21.054094 (+687736us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b5800c3ddb0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:21:21.234585 (+180491us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:21.234755 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105440640}
I20260809 23:21:21.306830 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:21.230188 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.244134 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:21.191030  2115 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260809 23:21:21.316859 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.213995  2728 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.191745  2378 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:21.171841  2416 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:21.296685 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:21.297817 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56092.3us
I20260809 23:21:21.332281 25369 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
W20260809 23:21:21.297536 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56631.8us
I20260809 23:21:21.307926 24430 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:21.248327 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.308735 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.307547 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:21.307528 25372 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:21.273604 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61052.2us
I20260809 23:21:21.308145 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:21.338446 25363 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:21.208225  2154 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:21.249176 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:21.232612  1485 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:21.340453 24316 raft_consensus.cc:595] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:21.192211  2725 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.204895  2736 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.146519  2795 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.205065  1527 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.320931 24542 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:21.331195 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87153.2us
I20260809 23:21:21.340837 24316 raft_consensus.cc:603] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:21.300141  2765 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260809 23:21:21.244118  2770 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:21.248664  2577 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260809 23:21:21.300136  2726 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:21.339040 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:21.235220  2741 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.349694 24429 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:21.243976  2579 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
W20260809 23:21:21.418645 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79723.2us
I20260809 23:21:21.240677  2652 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.351148  2754 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.355664  2807 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.357954 24425 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260809 23:21:21.370695 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:21.373603 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 144501us
I20260809 23:21:21.320173  2788 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.386142 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:21.393028 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:21.372061  2587 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.841s	user 0.005s	sys 0.007s
I20260809 23:21:21.347682  1485 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:21.346256  2720 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.373818  2675 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.091s
I20260809 23:21:21.356740  2171 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:21.347251  2767 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.356752  2115 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:21.425886 25369 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
W20260809 23:21:21.444264  2662 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.005s
I20260809 23:21:21.240716  2463 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260809 23:21:21.390966  2770 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:21.450588  2806 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:21.336088  2469 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:21.427199  2785 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
I20260809 23:21:21.423133  2741 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:21.331178  2593 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.479125 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3393) took 1637 ms. Trace:
I20260809 23:21:21.489599 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:21.469524  2788 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.456326 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:21.373956  2801 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.120s
I20260809 23:21:21.465881  2725 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.527653 24542 rpcz_store.cc:276] 0809 23:21:19.841378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.841698 (+   320us) service_pool.cc:227] Handling call
0809 23:21:19.864908 (+ 23210us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:21.372431 (+1507523us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:21.478868 (+106437us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003dca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:21.479068 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3347897728}
W20260809 23:21:21.519529 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 133485us
I20260809 23:21:21.463837  2809 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:21.473320  2416 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:21.473160  2587 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:21.472563  2322 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.486647  2115 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.484483  2171 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.496073  2767 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.509934  2652 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.510753  2333 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.507401  2154 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.512537  2592 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.478720  2790 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:21.506810  2691 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:21.467231  2807 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.494669  2806 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:21.494789  2770 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:21.514510  2268 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:21.542047  2675 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.082s
I20260809 23:21:21.505270 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.479064  2764 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.553925 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:21.555519 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 179840us
I20260809 23:21:21.523754  2469 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.520069  2813 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.664916  2813 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:21.666198  2798 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.460s	user 0.000s	sys 0.002s
I20260809 23:21:21.667232  2813 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.587169  2734 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.695s	user 0.006s	sys 0.006s
I20260809 23:21:21.667928  2734 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:21.630815 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 236999us
I20260809 23:21:21.646534  2779 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:21.671273 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.671337  2779 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.672824 24427 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260809 23:21:21.491845  2662 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.557336  2416 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:21.676738  2416 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.679112 24422 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:21.679175  2268 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:21.582964  2171 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:21.681830  1527 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.649430 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:21.675724  2662 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.686973 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:21.531886 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:21.688678 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 134848us
I20260809 23:21:21.556094  2809 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.559964  2725 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260809 23:21:21.692513 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:21.692914  2809 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.693636  2795 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.697062  2736 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.697625  2593 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.698455  2748 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.622483  2006 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.626566  2631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.002s	sys 0.009s
I20260809 23:21:21.549175  2728 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.078s
I20260809 23:21:21.622402  2378 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:21.626106  2691 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:21.716092 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:21.713124 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:21.710177  2631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.002s	sys 0.009s
W20260809 23:21:21.710769 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46284.6us
I20260809 23:21:21.723090  2469 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:21.724434  2378 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.727577  2691 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.729209  2691 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:21.729983  2670 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:21.730453  2670 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:21.731455  2670 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.647550  2806 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:21.737787 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:21.654074  2816 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.644675  2119 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:21.744670 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3386) took 1569 ms. Trace:
I20260809 23:21:21.745074 24321 rpcz_store.cc:276] 0809 23:21:20.175472 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:20.175671 (+   199us) service_pool.cc:227] Handling call
0809 23:21:20.197167 (+ 21496us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:20.327518 (+130351us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:20.327551 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:20.952130 (+624579us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:20.952528 (+   398us) log.cc:842] Serialized 26 byte log entry
0809 23:21:20.987183 (+ 34655us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:20.987234 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:20.987331 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:21.665491 (+678160us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:21.744459 (+ 78968us) raft_consensus.cc:1671] finished
0809 23:21:21.744484 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:21.744612 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32915968,"thread_start_us":34306,"threads_started":1,"wal-append.queue_time_us":223812}
W20260809 23:21:21.721129  2721 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.612s	user 0.009s	sys 0.000s
I20260809 23:21:21.674770  2779 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:21.751505 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 219706us
W20260809 23:21:21.750991 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.007s	sys 0.025s
I20260809 23:21:21.753324 24498 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:21.754812 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.007s	sys 0.026s
I20260809 23:21:21.745234  2725 consensus_queue.cc:1225] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 21.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:21.780158 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:21.743824  2720 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:21.783749 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.789661 24430 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:21.769549  2625 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.555s	user 0.003s	sys 0.006s
I20260809 23:21:21.754089  2717 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:21.790759 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.784621 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.751721  2469 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:21.792678 24317 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260809 23:21:21.815943 24430 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:21.822775 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:21.787577  2052 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.719s	user 0.011s	sys 0.000s
I20260809 23:21:21.823453 24428 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:21.829985 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.830797 24429 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:21.703589  2115 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 18.18, Last appended by leader: 18, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.778685  2748 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:21.828635 24317 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:21.747181  2809 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260809 23:21:21.781482  2598 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:21.824579 24320 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260809 23:21:21.824302 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.758180  2721 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.825554  2469 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.854909  2801 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:21.888823  2721 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.835005 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:21.815969  2720 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.890363 24321 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 16: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:21.836603 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:21.891453 24316 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:21.836712 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:21.844097  2815 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:21.892074  2720 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.892445  2815 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:21.893407  2815 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.894467  2815 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:21.817854  2030 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260809 23:21:21.767845  2813 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260809 23:21:21.750209  2816 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.845974  2600 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:21.846206  2767 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.920802 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:21.921707 24323 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:21.849316  2119 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.850466  1731 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:21.791383  2725 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:21.814031  2764 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:21.853888 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.020s	sys 0.033s
W20260809 23:21:21.928061 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.020s	sys 0.034s
W20260809 23:21:21.928467 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 707853us
I20260809 23:21:21.929414 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:21.834484  2052 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.943651  2052 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:21.822382  2625 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:21.833492 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:21.880348 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:21.835660  2812 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260809 23:21:21.945255 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 111852us
I20260809 23:21:21.945709  2588 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:21.946365  2588 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260809 23:21:21.900664 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 527226us
I20260809 23:21:21.947885  2588 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260809 23:21:21.902858 24429 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:21.904532 24539 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:21.786350  2652 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:21.953917 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:21.954968  2508 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:21.930156  2788 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.915262  2600 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.956010  2817 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:21.957000  2817 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Stepping down as leader of term 14
I20260809 23:21:21.957446  2817 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.957039  2600 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.957962  2816 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:21.958473  2600 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:21.959471 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:21.888840  2801 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.960147 24316 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:21.960403 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.961102 24429 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:21.961787  2801 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:21.962205 24435 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:21.964457 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.965042 24435 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:21.965972  2115 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:21.966184 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:21.967190  1677 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:21.967557  1677 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:21.967905  1677 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:21.969022  2587 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:21.970484 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:21.971084 24317 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:21.909150  2030 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:21.971508  2030 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:21.914958  2171 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:21.857641  2702 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.774s	user 0.002s	sys 0.011s
I20260809 23:21:21.955112 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:21.957892 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:21.940877  1485 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:21.937143 24540 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
W20260809 23:21:21.941202  2808 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.586s	user 0.003s	sys 0.001s
I20260809 23:21:21.976475 25372 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:22.010842 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:22.012396 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:21.985270 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55981.7us
I20260809 23:21:21.986193 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:22.000248 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.015803 25368 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:22.015851 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:21.958318  2817 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.004990 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:22.005523 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:22.006292 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:22.006856  2650 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:22.008033 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:22.001164  2820 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:22.019095 24539 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260809 23:21:22.043936 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57841.1us
I20260809 23:21:21.883579  2748 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:21.927578  2725 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.042207 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.074555 25375 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:22.074752 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:21.927285  1731 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:22.044369 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:22.046705 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 290138us
I20260809 23:21:22.084831 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:22.086848 24428 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260809 23:21:22.092527 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60604.4us
W20260809 23:21:22.047319 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 300303us
I20260809 23:21:22.055580 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:22.048498 25371 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:22.067965 24316 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:22.096830 24539 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:22.098402 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:22.099695 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.041563  2333 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:22.101679 24540 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:22.048652  2816 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:22.111742 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:22.112560 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57063.5us
I20260809 23:21:22.051930  2807 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:22.089359  2809 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:22.082386  2725 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:22.076541  2671 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:22.058452  2702 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:22.071440 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:22.073323 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.008s	sys 0.020s
I20260809 23:21:21.978894  2764 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.117980 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:22.223059 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.008s	sys 0.020s
W20260809 23:21:22.223574 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 105725us
I20260809 23:21:22.224336 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:22.094570  2820 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:22.194823 24316 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:22.125702 25372 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:22.200943 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:22.119130  2817 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:22.119971  2623 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:22.205168 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.216759  2806 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.134829 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.058264  2598 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:21:22.207250 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.211361 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.212023  2752 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:22.212636 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.213001  2670 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:22.213385  2777 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.156s
I20260809 23:21:22.215561  1485 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:22.216614  2844 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260809 23:21:22.121497  2795 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:22.122411  2813 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:22.144011  2809 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:22.220304 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55652us
W20260809 23:21:22.191524 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.012s	sys 0.017s
I20260809 23:21:22.104075  2754 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:22.117179  2816 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:22.223613 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 767339us
I20260809 23:21:22.052923  2030 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.110658  2333 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.193792  2807 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.231022 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:22.222311  2720 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.137s
I20260809 23:21:22.223321  1731 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.284480  2592 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:22.349996  2194 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:22.396139  2333 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:22.397343 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.398134 25375 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:22.399384 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.202822  1527 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.328662 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.459410 25372 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:22.329634 25365 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:22.332866 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 261520us
I20260809 23:21:22.335229 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:22.228673  2838 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260809 23:21:22.344617 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:22.449815 24317 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:22.466820 25368 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 14 FOLLOWER]: Advancing to term 15
I20260809 23:21:22.226830  2820 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.205951  2767 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.284574  2671 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:22.396180  2194 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.403309 24539 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:22.403578  2816 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:22.404043  2795 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:22.404551  2754 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:22.404986 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:22.406903 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:22.367588  2847 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260809 23:21:22.409538 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.404559 24546 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:22.411223 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.517858 24541 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:22.412811 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:22.518406 24540 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:22.413672 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.414492  2807 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.415966  2662 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:22.290642  2052 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:22.355096  2752 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:22.357676  2631 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:22.358206  2577 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.395128 25366 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:22.424758  2764 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:22.425691  2650 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.435469  2841 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:22.350438  2600 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:22.444073 25375 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:22.354112 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.512899  2754 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:22.461159 24542 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:22.110415  2667 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
W20260809 23:21:22.329864  2792 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260809 23:21:22.354554 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:22.496316 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:22.499300 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.468758  2691 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260809 23:21:22.540530 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.558154 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.512341 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:22.405026  2813 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.513518 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:22.514894 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:22.564589 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49826.3us
I20260809 23:21:22.513428 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.521376 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.458673  2809 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:22.226778  2702 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:22.529099 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:22.487223  2592 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.488111  2505 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:22.534948 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.025s	sys 0.027s
I20260809 23:21:22.498723  2820 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:22.536556 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:22.389878 24320 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:22.536260 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.520318  2639 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.494858 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:22.550988 24540 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:21:22.555153 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.527174  2171 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:22.557989  1677 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.000s
I20260809 23:21:22.534759  2650 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:22.532871  2593 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:22.532359  2052 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:22.602425 24317 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:22.596339  2598 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:22.539188  2119 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.013s	sys 0.033s
I20260809 23:21:22.536814  2748 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:22.604569 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.607831 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:22.772435 25375 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:22.538734  2600 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:22.774434  2600 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:22.776393 25375 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:22.487391  1527 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.777714 25375 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:22.779204  1527 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:22.612501 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:22.618021 25372 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:22.618431 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:22.780889 25365 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:22.620769 24540 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:22.782637 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.010s	sys 0.009s
I20260809 23:21:22.625504 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.631594 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1599) took 1211 ms. Trace:
W20260809 23:21:22.783082 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.010s	sys 0.009s
I20260809 23:21:22.587261  2813 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:22.599027 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.026s	sys 0.027s
I20260809 23:21:22.784283 24435 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:22.531651  2767 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.596606  2702 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:22.535004  2577 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.601174  2662 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:22.612360  2675 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:22.635421 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:22.800395 24428 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:22.682632  2864 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:22.603955  2505 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:22.677587  2416 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.012s	sys 0.018s
I20260809 23:21:22.686859  2623 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:22.634716  2593 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.628875  2171 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
I20260809 23:21:22.717712  2194 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:22.723565  2052 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.622565  1677 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.726286  2806 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:22.732314  2807 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:22.738075  2815 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.843s	user 0.012s	sys 0.000s
I20260809 23:21:22.745169  2819 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:22.755081  2792 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
W20260809 23:21:22.756686  2741 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.207s	user 0.008s	sys 0.003s
W20260809 23:21:22.676280  2119 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.013s	sys 0.033s
I20260809 23:21:22.601419  2592 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.588697  2795 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.759543  2828 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.860158  2828 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.861023  2592 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:22.861342 24320 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: [17.11-17.11]
W20260809 23:21:22.861629  2823 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.914s	user 0.006s	sys 0.000s
I20260809 23:21:22.862128  2828 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:22.620195  2639 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.615240  2671 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.863343  2671 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:22.624167 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.010s	sys 0.019s
I20260809 23:21:22.863889  2639 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:22.864090 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.010s	sys 0.019s
I20260809 23:21:22.537009  2631 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:22.619480  2469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.008s	sys 0.086s
I20260809 23:21:22.783263 24545 rpcz_store.cc:276] 0809 23:21:21.420631 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.420838 (+   207us) service_pool.cc:227] Handling call
0809 23:21:21.421275 (+   437us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:21.459947 (+ 38672us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:21.459993 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:21.474070 (+ 14077us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.403240 (+929170us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:22.403297 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.403394 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:22.631326 (+227932us) raft_consensus.cc:1671] finished
0809 23:21:22.631354 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:22.631522 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2718,"raft.queue_time_us":38318,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":7938,"thread_start_us":870514,"threads_started":1,"wal-append.queue_time_us":606781}
I20260809 23:21:22.788937 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:22.798319  2598 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:22.865177 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:22.793792 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.005s	sys 0.013s
W20260809 23:21:22.866109 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.005s	sys 0.013s
I20260809 23:21:22.804500 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.806881 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:22.812342  2115 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.000s
W20260809 23:21:22.643540  2378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.036s	sys 0.038s
W20260809 23:21:22.828877 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 193571us
I20260809 23:21:22.831547 24422 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:22.836814 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.672260  2752 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:22.882156 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93325.5us
I20260809 23:21:22.840198  2729 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.882797 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:22.835052  2416 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.016s	sys 0.018s
W20260809 23:21:22.857088  2670 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.509s	user 0.012s	sys 0.000s
I20260809 23:21:22.857722  2741 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.870229 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:22.871323 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.872156 24535 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17.
I20260809 23:21:22.872720 24539 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:22.816768  1527 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:21:22.841207  2625 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:22.881774  2378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.038s	sys 0.040s
I20260809 23:21:22.883208 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:22.890224 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.849743  2652 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:22.870883  2469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.013s	sys 0.087s
I20260809 23:21:22.887643 25370 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:22.947801 24428 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260809 23:21:22.890434 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:22.824651  2662 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:22.919507 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:22.887187 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.008s	sys 0.008s
I20260809 23:21:22.926805 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:22.957051 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.008s	sys 0.008s
I20260809 23:21:22.957577 24545 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260809 23:21:22.844323  2588 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.071s
I20260809 23:21:22.889331  2639 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:22.960081  2639 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:22.960557  2639 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:22.829814  2675 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.964664  2675 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:22.883668  2592 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:22.965312  2592 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.952955 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.968786  2592 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:22.971341 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2858) took 1137 ms. Trace:
I20260809 23:21:22.968672 25368 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 15.
I20260809 23:21:22.974323 25375 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 14. Current term is 15. Ops: [14.13-14.13]
W20260809 23:21:22.978307 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95166.9us
I20260809 23:21:22.979609 24311 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:22.925328  2194 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.016s
I20260809 23:21:22.997164 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:22.970573 24545 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:22.865667  2819 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:22.942502  2579 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:23.015050 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63964.7us
I20260809 23:21:23.032812 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.034216 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:22.998350  2670 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:23.052098 24540 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:23.055459 24428 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260809 23:21:23.055671 24542 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.502s	user 0.005s	sys 0.005s
I20260809 23:21:23.055995 24542 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:23.022539 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:22.891839  2729 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.059319 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:23.027587  2867 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260809 23:21:23.027434  2675 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:22.980211  2652 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.024849  2879 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.071553 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:23.072750  2817 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.729s	user 0.006s	sys 0.005s
I20260809 23:21:23.032137  2579 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.036070  2006 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.030s
I20260809 23:21:23.103453 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:23.110533 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51837.6us
I20260809 23:21:23.111225 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.100857 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:23.103943  2333 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.707s	user 0.012s	sys 0.000s
I20260809 23:21:23.011920  2416 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260809 23:21:23.113775 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.115130 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:23.197319 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129624us
W20260809 23:21:23.107723  2882 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260809 23:21:23.050606  2878 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:23.068403  2894 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260809 23:21:23.241847 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 361177us
I20260809 23:21:23.073691  2854 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:23.077229  2863 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260809 23:21:23.205169 24543 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
W20260809 23:21:23.102649  2816 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.575s	user 0.010s	sys 0.000s
I20260809 23:21:23.221938 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:23.096814  2725 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.893s	user 0.008s	sys 0.003s
I20260809 23:21:23.228657 24545 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260809 23:21:23.119606 24428 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:23.229153  2897 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:23.229701  2895 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:23.086560  1677 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:23.230183  2860 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
W20260809 23:21:23.294646 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45249.6us
W20260809 23:21:23.231498  2876 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260809 23:21:23.328320  2817 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:23.328536  2899 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260809 23:21:23.328868  2817 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.331972 24545 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:23.248504  2670 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:23.192480  2879 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.335749  2030 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:23.231859 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1640) took 1115 ms. Trace:
I20260809 23:21:23.232270 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:23.233810  2873 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.278s	user 0.000s	sys 0.000s
I20260809 23:21:23.235409 24540 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260809 23:21:23.087066  2741 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.069167  2675 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:23.200845 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.088449  2729 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.102250  2577 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.220629  2652 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.363447 24540 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:23.092546  2600 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.230729  2588 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:23.291836 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 201446us
I20260809 23:21:23.389389 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:23.270182 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 249456us
I20260809 23:21:23.326426 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.270629  2505 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.338459 25366 rpcz_store.cc:276] 0809 23:21:22.116502 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.134491 (+ 17989us) service_pool.cc:227] Handling call
0809 23:21:23.231674 (+1097183us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:23.231799 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":234793728}
I20260809 23:21:23.065026  2598 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:23.221242  2875 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260809 23:21:23.400018  2623 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:23.236501 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 125326us
I20260809 23:21:23.400578  2623 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.402108 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1628) took 1427 ms. Trace:
I20260809 23:21:23.350977 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2863) took 1402 ms. Trace:
I20260809 23:21:23.076691  2748 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.171s
I20260809 23:21:23.222800  2768 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:23.403338  2768 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:23.328567  2801 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
I20260809 23:21:23.404285  2768 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:23.358647 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 159189us
I20260809 23:21:23.405716  2768 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:23.406008  2777 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:23.406497  2777 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:23.359906  2904 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260809 23:21:23.407337  2777 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.363507  2741 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:23.408430 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.408736  2777 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:23.409360 24540 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260809 23:21:23.347465  1677 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.315523  2593 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.409883  2579 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.300791  2416 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.348140  2725 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.347823  2816 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.417011 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:23.364164 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94660.1us
I20260809 23:21:23.274296  2767 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.201730  2333 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.389736 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1635) took 1403 ms. Trace:
I20260809 23:21:23.393759 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.422178 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:21:23.227121  2770 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.545s	user 0.017s	sys 0.006s
I20260809 23:21:23.397068 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.227005  2469 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.137s
I20260809 23:21:23.389658  2631 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.396770  2667 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.364719  2662 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:23.404860  2849 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.189s	user 0.000s	sys 0.003s
I20260809 23:21:23.406338 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.463171 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:23.463577 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57347us
I20260809 23:21:23.464926 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.465279 25363 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260809 23:21:23.466019 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.359544  2670 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:23.245368  2639 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.407652 24323 rpcz_store.cc:276] 0809 23:21:21.948076 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.950672 (+  2596us) service_pool.cc:227] Handling call
0809 23:21:21.972541 (+ 21869us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:21.972863 (+   322us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:21.972891 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:21.984127 (+ 11236us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.866592 (+882465us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:22.921608 (+ 55016us) log.cc:842] Serialized 14 byte log entry
0809 23:21:22.927510 (+  5902us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.927584 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:23.350661 (+423077us) raft_consensus.cc:1671] finished
0809 23:21:23.350695 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:23.350872 (+   177us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":51061,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":23191,"raft.queue_time_us":32585,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":88970,"spinlock_wait_cycles":13728640,"thread_start_us":772003,"threads_started":1,"wal-append.queue_time_us":832288}
W20260809 23:21:23.412810 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 86452.6us
I20260809 23:21:23.402496 25369 rpcz_store.cc:276] 0809 23:21:21.974863 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.975047 (+   184us) service_pool.cc:227] Handling call
0809 23:21:21.975435 (+   388us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.007044 (+ 31609us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:22.007075 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:22.007513 (+   438us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.864509 (+856996us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:22.864549 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.864623 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:23.401869 (+537246us) raft_consensus.cc:1671] finished
0809 23:21:23.401893 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:23.402051 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5080,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":560,"thread_start_us":856609,"threads_started":1,"wal-append.queue_time_us":839095}
I20260809 23:21:23.414403 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:23.466836  2650 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.832s	user 0.012s	sys 0.000s
I20260809 23:21:23.470150  2650 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.471462  2905 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:23.471869  2905 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:23.472234  2905 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.472303  2650 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.474455  2894 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:23.476545  2194 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:23.415506 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.416260 25366 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260809 23:21:23.477082  2194 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.420151 24541 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:23.410230  1677 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.478840  2670 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.479598 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3343) took 1067 ms. Trace:
I20260809 23:21:23.356118  2675 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.480515  2907 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:23.481069  2907 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.432545 25370 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:23.483914  2639 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.484917 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.486078 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3397) took 1403 ms. Trace:
I20260809 23:21:23.486213 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.486702 24320 rpcz_store.cc:276] 0809 23:21:22.082399 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.621275 (+538876us) service_pool.cc:227] Handling call
0809 23:21:22.621643 (+   368us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.861801 (+240158us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.485831 (+624030us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:23.486020 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":550722816}
I20260809 23:21:23.487457 24544 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:23.487977 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.488735 24320 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:23.467166  2914 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:23.490224 24323 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:23.491086 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.477527 25373 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:23.491814 24323 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:23.491909  2820 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.872s	user 0.011s	sys 0.000s
I20260809 23:21:23.492331  2820 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.492655 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.436955 24312 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:23.493307  2741 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:23.494374  2820 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.493973  2741 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:23.494715 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.495200 24542 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:23.495419 24320 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:23.495589 24542 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:23.496275 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.496340 24542 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.492720 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.438284 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:23.356956 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.498865 24430 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:23.499753 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.402685  2623 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.503448  2650 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.396997  2577 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:23.359828  2879 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.361304  2819 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.505450  2650 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.506668 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2854) took 1712 ms. Trace:
I20260809 23:21:23.507112 24311 rpcz_store.cc:276] 0809 23:21:21.794088 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.794219 (+   131us) service_pool.cc:227] Handling call
0809 23:21:21.823163 (+ 28944us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:22.996949 (+1173786us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.506457 (+509508us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:23.506612 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2606166784}
I20260809 23:21:23.508337 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.441443 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.448621 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3394) took 1461 ms. Trace:
I20260809 23:21:23.509254 24540 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:23.453488  2725 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.454905  2816 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.455989  2909 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
W20260809 23:21:23.456714  2888 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260809 23:21:23.410532  2579 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260809 23:21:23.410554  2416 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.458680  1731 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.459682  2765 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:23.460836  2333 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.461254  2667 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.462440 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3032) took 1263 ms. Trace:
W20260809 23:21:23.470455 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56171.8us
I20260809 23:21:23.477111  2469 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:23.431809 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.497500 24542 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.437750 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.480031 24541 rpcz_store.cc:276] 0809 23:21:22.412370 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.522567 (+110197us) service_pool.cc:227] Handling call
0809 23:21:23.479405 (+956838us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:23.479539 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1874424576}
I20260809 23:21:23.420002  2593 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260809 23:21:23.455554  2874 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260809 23:21:23.509078 24309 rpcz_store.cc:276] 0809 23:21:21.987693 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.078004 (+ 90311us) service_pool.cc:227] Handling call
0809 23:21:22.078706 (+   702us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.079094 (+   388us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:22.079120 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:22.090217 (+ 11097us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.622448 (+532231us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:22.899349 (+276901us) log.cc:842] Serialized 14 byte log entry
0809 23:21:23.014689 (+115340us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.014794 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:23.448313 (+433519us) raft_consensus.cc:1671] finished
0809 23:21:23.448350 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:23.448535 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119024,"prepare.queue_time_us":92532,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":115380,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":10973,"thread_start_us":632092,"threads_started":2,"wal-append.queue_time_us":1122404}
I20260809 23:21:23.429911 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:23.425050 25371 rpcz_store.cc:276] 0809 23:21:21.986089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.986274 (+   185us) service_pool.cc:227] Handling call
0809 23:21:22.008225 (+ 21951us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:22.110124 (+101899us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.367117 (+1256993us) spinlock_profiling.cc:243] Waited 40 ms on lock 0x7b54003dfed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:23.389648 (+ 22531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92057344}
I20260809 23:21:23.511857  2577 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.523658 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.523783 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.524643 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.558297 25373 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:23.551242 24428 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:23.525854 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.559705 25370 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:23.540851  2765 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:23.560552 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.561663 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:23.561285  2765 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.527752 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.530426 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.563136  2765 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:23.532545 24430 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260809 23:21:23.551416 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 119717us
W20260809 23:21:23.565676 24580 heartbeater.cc:644] Failed to heartbeat to 127.21.255.62:37921 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.255.62:37921 timed out after 15.000s (SENT)
I20260809 23:21:23.526634 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.567279 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.567890 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.568389 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.568972 24541 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:23.569836 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:23.570632  2601 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:23.353332  2854 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.417495  2588 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.535080 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.538084 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.480345  1677 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.527318  2764 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.507489  2650 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.540154 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.438683  2770 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.596915 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.490260  2817 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:22.973309 24314 rpcz_store.cc:276] 0809 23:21:21.834131 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.835891 (+  1760us) service_pool.cc:227] Handling call
0809 23:21:21.863435 (+ 27544us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:21.863671 (+   236us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:22.221071 (+357400us) log.cc:842] Serialized 14 byte log entry
0809 23:21:22.970212 (+749141us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:22.970497 (+   285us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:22.970635 (+   138us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.970779 (+   144us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:22.971180 (+   401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80245,"prepare.queue_time_us":751298,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":89557,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8081,"thread_start_us":1050885,"threads_started":3,"wal-append.queue_time_us":757287}
I20260809 23:21:23.549923 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.614143 24309 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 15: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:23.615008 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.563256 25366 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:21:23.616770  2851 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.405s	user 0.002s	sys 0.001s
I20260809 23:21:23.617623 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:23.619608 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3392) took 1645 ms (client timeout 2187 ms). Trace:
I20260809 23:21:23.619959 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:23.620124 24312 rpcz_store.cc:269] 0809 23:21:21.973815 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.974348 (+   533us) service_pool.cc:227] Handling call
0809 23:21:23.619428 (+1645080us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:23.619549 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3203241600}
I20260809 23:21:23.562692 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.525590 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.570785 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:23.580353 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:23.539755  2736 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:23.569810 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.546545 24431 rpcz_store.cc:276] 0809 23:21:22.198726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.198871 (+   145us) service_pool.cc:227] Handling call
0809 23:21:22.210073 (+ 11202us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.210472 (+   399us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:22.210505 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:22.210999 (+   494us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.671938 (+460939us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:22.671980 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.672052 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:23.462173 (+790121us) raft_consensus.cc:1671] finished
0809 23:21:23.462196 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:23.462363 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10939,"raft.queue_time_us":1126,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":16895,"thread_start_us":460551,"threads_started":1,"wal-append.queue_time_us":642598}
I20260809 23:21:23.587699 24311 raft_consensus.cc:2354] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:23.623364 24541 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:23.583431  2702 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.766s	user 0.012s	sys 0.000s
I20260809 23:21:23.596508 24323 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:23.542673 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.630002 24320 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:23.630508 24542 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260809 23:21:23.542392  2725 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:23.606806  2194 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:23.608963  2868 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.897s	user 0.003s	sys 0.000s
W20260809 23:21:23.536832 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 98605.1us
I20260809 23:21:23.587137  2726 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.625049  2817 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.625255 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:23.621996 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51278.6us
I20260809 23:21:23.645884 24311 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:23.587795  2505 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.591331  2662 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.660712  2662 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.628568 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.662997 24540 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:23.589931 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62266.1us
I20260809 23:21:23.626073 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.666072 24541 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:23.667291 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.667418 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.622097  2691 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.633960 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1620) took 1097 ms. Trace:
I20260809 23:21:23.571334  2879 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:23.571097  2819 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:23.640267 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.641009 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:23.687388 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46501.3us
I20260809 23:21:23.590760 24422 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:23.643114 24423 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:23.644796 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.591238  2577 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.648600 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.633098  1429 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.696194  1429 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.627043 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.699787 24314 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:23.700632 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.701900 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.650328 24498 leader_election.cc:398] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
W20260809 23:21:23.703914 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53784.6us
I20260809 23:21:23.704773 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:23.706447 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:23.633175  2736 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.710045 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.598318  2764 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.711211  2729 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:23.711721  2729 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.712052  2729 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.712929  2764 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.657264  2588 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.657830  2600 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.659785  2770 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.662776 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.716871 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.718273 24320 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:23.675730 25375 rpcz_store.cc:276] 0809 23:21:22.536162 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.778408 (+242246us) service_pool.cc:227] Handling call
0809 23:21:22.973973 (+195565us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.974829 (+   856us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.633708 (+658879us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540055f0d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:23.633887 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":448884608}
I20260809 23:21:23.631174 24542 pending_rounds.cc:76] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 11
I20260809 23:21:23.719431 24542 pending_rounds.cc:97] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260809 23:21:23.719794 24542 raft_consensus.cc:2866] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:23.625977  2900 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:23.637974  2725 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.688835 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.635720  2171 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.635509  2777 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:23.664776  2813 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.863s	user 0.008s	sys 0.000s
I20260809 23:21:23.693409 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:23.655138  2667 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.707762  2508 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:23.698786  1429 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.710852 24314 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:23.720597 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.720142  2469 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:23.692468  1677 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.643793  2726 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.645483  2817 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.736299  2725 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:23.738577  2052 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:23.674958  2691 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.735780 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.759519 24542 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:23.678855  2879 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.676450  2820 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:23.762171 24314 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260809 23:21:23.741096 24323 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:23.638988  2702 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.743392 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.745220  2508 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.779426  2508 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:23.758908  2911 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:23.737304  2900 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.743739 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:23.735351 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.783213 25370 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.783385 24540 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:23.784585 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.785197 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:23.785797 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.786064 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.786195 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.786520 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1381) took 2115 ms. Trace:
I20260809 23:21:23.746002 24541 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260809 23:21:23.787205  2926 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:23.786698 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.787636 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.788362  2667 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.788727  2667 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.788827 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.789098  2667 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.789400 24541 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:23.664510  2194 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.790432 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.790867  2194 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.791613 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:23.688278  2593 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.792585  2662 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.792395  2593 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.792900  2194 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.755251  1677 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.787658  2926 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:23.794261  2926 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:23.794454  2593 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.644706  2779 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.088s
I20260809 23:21:23.692905  2767 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:23.748808 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:23.749279 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 155935us
I20260809 23:21:23.733599  2907 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.797351 24542 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:23.797520  1539 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:23.797995  1539 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:23.801385  2667 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.804188  2900 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.749663 24320 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:23.732384 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43621.4us
W20260809 23:21:23.804731  2887 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.362s	user 0.003s	sys 0.000s
I20260809 23:21:23.805439 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:23.731483 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62188.3us
W20260809 23:21:23.747198  2754 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.160s	user 0.011s	sys 0.004s
I20260809 23:21:23.806315  2754 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.806697 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.795611  1677 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.763816 24323 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:23.814389 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3401) took 1673 ms. Trace:
I20260809 23:21:23.814805 24323 rpcz_store.cc:276] 0809 23:21:22.140564 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.763265 (+1622701us) service_pool.cc:227] Handling call
0809 23:21:23.763615 (+   350us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:23.814067 (+ 50452us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.814307 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:23.816322 24323 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:23.816987 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3402) took 1675 ms. Trace:
I20260809 23:21:23.817418 24323 rpcz_store.cc:276] 0809 23:21:22.141502 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.815724 (+1674222us) service_pool.cc:227] Handling call
0809 23:21:23.816112 (+   388us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:23.816743 (+   631us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.816929 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:23.741735 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:23.765480 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3400) took 1651 ms. Trace:
I20260809 23:21:23.766950  2577 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.819290 24314 rpcz_store.cc:276] 0809 23:21:22.114111 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.748014 (+1633903us) service_pool.cc:227] Handling call
0809 23:21:23.759430 (+ 11416us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:23.765150 (+  5720us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.765425 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:23.771524  2854 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:23.761643  2671 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.898s	user 0.009s	sys 0.003s
I20260809 23:21:23.762058  2691 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.775403  2416 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.761623  2736 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.741786  2777 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.764950  2879 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:23.777019  2702 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.752380  2726 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.755415  2817 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.735049  2171 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:23.674332  2505 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:23.807787 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59051.6us
I20260809 23:21:23.805317 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3398) took 1711 ms. Trace:
W20260809 23:21:23.748518  2809 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.143s	user 0.005s	sys 0.009s
I20260809 23:21:23.809126 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.786881 24425 rpcz_store.cc:276] 0809 23:21:21.670901 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.689656 (+ 18755us) service_pool.cc:227] Handling call
0809 23:21:21.689965 (+   309us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:21.719692 (+ 29727us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:21.719736 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:21.739594 (+ 19858us) log.cc:842] Serialized 26 byte log entry
0809 23:21:21.968152 (+228558us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b180017a520. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:22.464363 (+496211us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:22.466327 (+  1964us) log.cc:842] Serialized 14 byte log entry
0809 23:21:22.466964 (+   637us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.467033 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:23.608238 (+1141205us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:23.613424 (+  5186us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b180017a520. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:23.786286 (+172862us) raft_consensus.cc:1671] finished
0809 23:21:23.786309 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:23.786464 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121837,"prepare.queue_time_us":121,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":67446,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":8175,"spinlock_wait_cycles":126310400,"thread_start_us":496027,"threads_started":1,"wal-append.queue_time_us":971394}
I20260809 23:21:23.864562 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.865204 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.828502 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:23.829493  2667 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:23.824973 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.889681 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.738822  2741 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.022s
I20260809 23:21:23.742443  2813 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.835685 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.764645  2820 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:23.832969  2662 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260809 23:21:23.847504 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.812924  2469 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.822732  2052 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Leader election won for term 15
I20260809 23:21:23.795611  2779 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:23.868875 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.869764 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.869426  2508 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:23.870604  2788 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:23.871174  2846 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260809 23:21:23.871564  2171 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:23.920279  2729 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.922317  2729 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.922885  1485 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.923383  1485 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.924188  2508 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.892145  2593 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.873718  2505 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:23.924928  2593 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:23.874616 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65591.2us
I20260809 23:21:23.926023 24316 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23.
I20260809 23:21:23.926126 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.875283  2671 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:23.910319  2469 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.930519 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.880192  2754 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:23.821755  2914 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:23.821584  2904 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:23.833633  2119 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260809 23:21:23.886282 24541 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:23.864173 24320 rpcz_store.cc:276] 0809 23:21:22.094288 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.723624 (+1629336us) service_pool.cc:227] Handling call
0809 23:21:23.749446 (+ 25822us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:23.804966 (+ 55520us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.805158 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:23.890868 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65996us
I20260809 23:21:23.938872 24380 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:23.939673 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.891633  2809 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.900068 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:23.906281 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.910045 25369 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 15.
I20260809 23:21:23.900544 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:23.911113 25368 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:23.911728 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.915864 24430 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: [16.11-19.12]
I20260809 23:21:23.917364  2923 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:23.919201 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.001190 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:23.922518  2675 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:23.898612  2820 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:23.940730  2937 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260809 23:21:23.933840 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:23.977001 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:24.008934 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1684) took 1014 ms. Trace:
I20260809 23:21:23.917405  1731 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:23.947795 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:24.023080 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:23.909407  2631 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.098s
I20260809 23:21:23.932018 24424 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:23.929745  2030 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.594s	user 0.007s	sys 0.006s
I20260809 23:21:24.009253 24312 rpcz_store.cc:276] 0809 23:21:22.994530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.720127 (+725597us) service_pool.cc:227] Handling call
0809 23:21:24.008881 (+288754us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.028595 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.028983 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.029357 25366 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260809 23:21:24.010135 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 110129us
I20260809 23:21:24.029732 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.935022  2770 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:24.033298 24312 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:23.941789 24270 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 1.534s (SENT)
I20260809 23:21:23.914964  2788 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:24.014356 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.038622 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:23.946388 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:24.016237 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.040534 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.013451 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:23.932653  2593 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:24.017526 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:23.832170  1539 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.002996 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:23.938786  2904 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:24.026731 25368 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:23.934851  2052 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:23.911434  2779 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:24.011443 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 110998us
I20260809 23:21:24.030486 24425 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:24.031279 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.031463 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:23.935563  2662 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:24.035069 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93272.9us
I20260809 23:21:24.036005 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:24.037230 24428 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:24.000609  2671 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:24.000980  2846 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.038138 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.039186 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:24.000824  2923 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:23.941156  2841 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.097s
I20260809 23:21:24.007817  2905 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.027557  1731 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.023198  1677 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:24.120095 25370 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:24.026273  2729 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.118374 24534 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:24.286785 25371 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:24.123001 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.027155  2900 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:24.298298 24312 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:24.219447 24423 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3429) took 2977 ms (client timeout 3292 ms). Trace:
W20260809 23:21:24.175462 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3391) took 2201 ms (client timeout 1970 ms). Trace:
W20260809 23:21:24.288296 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 342006us
W20260809 23:21:24.181561 24311 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.003s
W20260809 23:21:24.124780 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100105us
W20260809 23:21:24.287448 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 251630us
W20260809 23:21:24.290429 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 252371us
I20260809 23:21:24.287569 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.172753  2726 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.308508 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.336692 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:21:24.299057 24423 rpcz_store.cc:269] 0809 23:21:21.240404 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.240916 (+   512us) service_pool.cc:227] Handling call
0809 23:21:24.217511 (+2976595us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.217700 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5306150144}
I20260809 23:21:24.337456 24431 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:24.337824 24427 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:24.338670 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:24.299185 24313 rpcz_store.cc:269] 0809 23:21:21.973503 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.982171 (+  8668us) service_pool.cc:227] Handling call
0809 23:21:24.175239 (+2193068us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b54003859d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.175366 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1851835648}
I20260809 23:21:24.340233 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.329064 24311 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:24.341394 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1697) took 1014 ms. Trace:
I20260809 23:21:24.341892 24311 rpcz_store.cc:276] 0809 23:21:23.327373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.624992 (+297619us) service_pool.cc:227] Handling call
0809 23:21:24.341319 (+716327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.342761 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.335217 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:24.343643 24313 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:24.344367 24320 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:24.176862  2765 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.613s	user 0.005s	sys 0.005s
I20260809 23:21:24.122709  2577 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:24.123118  2600 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:24.160145  2875 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260809 23:21:24.344085 24311 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:24.344830 25370 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:24.173719  1429 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.299557  2006 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.052664  2813 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.352800 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.290582  2777 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.358093 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:24.358111 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.358336 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.385337 24313 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:24.313485  2671 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:24.350256 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.375527 24422 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:24.294705  2378 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:24.140515  2904 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:24.292840  2809 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.037333  2194 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.307s
I20260809 23:21:24.342473  2770 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.377494 24312 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:24.384829 25368 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:24.336673  2926 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.542s	user 0.005s	sys 0.006s
I20260809 23:21:24.329432  1677 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:24.299780  2469 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.412122  2905 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.042528  2702 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.302s
I20260809 23:21:24.412850 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.413640 24311 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:24.404241 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.415223 24423 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:24.416908 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 255546us
I20260809 23:21:24.342779  2764 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.342142  2807 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:24.420301 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:24.387476 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:24.393200 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.421145 24430 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:24.197928  2788 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.414413  2926 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:24.424484  2933 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.528s	user 0.002s	sys 0.000s
I20260809 23:21:24.424609  2728 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:24.425133  2728 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:24.377010  1429 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.301578  2779 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.426656 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.427395 24311 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:21:24.418459  2725 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.657s	user 0.012s	sys 0.000s
I20260809 23:21:24.428228 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.428411  2725 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.429279 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:24.429903  2925 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.536s	user 0.000s	sys 0.001s
W20260809 23:21:24.430706  2587 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260809 23:21:24.388797  2662 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
W20260809 23:21:24.432822  2943 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
W20260809 23:21:24.433133 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3347) took 1902 ms (client timeout 2201 ms). Trace:
W20260809 23:21:24.433534 24546 rpcz_store.cc:269] 0809 23:21:22.531069 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.564967 (+ 33898us) service_pool.cc:227] Handling call
0809 23:21:24.432952 (+1867985us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.433081 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2256433024}
W20260809 23:21:24.434778  2924 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.541s	user 0.000s	sys 0.001s
I20260809 23:21:24.358556  2949 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260809 23:21:24.435681  2949 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:24.407416  2505 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:24.436017  2949 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:24.436300  2505 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.407996 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.347409  2171 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.439919 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.440543  2171 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.440601 24312 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:24.441850 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.442510 24312 raft_consensus.cc:2362] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19. Re-sending same reply.
I20260809 23:21:24.388630  2006 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.443912  1429 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.434631 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:24.446177 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.446437  2006 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.409595  2577 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260809 23:21:24.447242 24323 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:24.447691  2878 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:24.392061 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.448385 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2864) took 2253 ms. Trace:
I20260809 23:21:24.448572 24320 raft_consensus.cc:2354] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
W20260809 23:21:24.440465 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3335) took 2039 ms (client timeout 1534 ms). Trace:
I20260809 23:21:24.445358 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:24.449522 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3431) took 1045 ms. Trace:
I20260809 23:21:24.382423  2600 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.450555 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.451174 24431 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:24.451885  2600 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.452479 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:24.449223 24535 rpcz_store.cc:269] 0809 23:21:22.401215 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.413365 (+ 12150us) service_pool.cc:227] Handling call
0809 23:21:24.440246 (+2026881us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54001d5e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.440392 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810206464}
I20260809 23:21:24.410383  2923 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.414511 24313 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:24.377549  2897 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.415611 25368 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:24.425184  2926 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.430835  2052 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.393373  2675 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.388018  2809 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.449445 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.487551 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3344) took 2029 ms. Trace:
I20260809 23:21:24.487989 24539 rpcz_store.cc:276] 0809 23:21:22.458422 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.533745 (+ 75323us) service_pool.cc:227] Handling call
0809 23:21:24.487374 (+1953629us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.487492 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":721621248}
I20260809 23:21:24.488615 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.489100 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:24.489588 24320 raft_consensus.cc:2354] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:24.411453  1677 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.495775 24498 leader_election.cc:398] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:24.496155 24498 leader_election.cc:401] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260809 23:21:24.448773 24323 rpcz_store.cc:276] 0809 23:21:22.195130 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.818431 (+1623301us) service_pool.cc:227] Handling call
0809 23:21:23.818769 (+   338us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.447634 (+628865us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.448161 (+   527us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:24.448332 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1444924672}
I20260809 23:21:24.498834 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.489771 24539 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:24.499625 24323 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:24.455822 24423 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:24.501255 24431 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:24.501447 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.502255 24423 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:24.409751  2817 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.516s	user 0.006s	sys 0.003s
I20260809 23:21:24.502624 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:24.504554 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:24.506111 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:24.490661 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.449548 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.045182  2907 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.304s
I20260809 23:21:24.450016 24540 rpcz_store.cc:276] 0809 23:21:23.403953 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.805459 (+401506us) service_pool.cc:227] Handling call
0809 23:21:23.805780 (+   321us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:23.808401 (+  2621us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:23.808443 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:23.823663 (+ 15220us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.026853 (+203190us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:24.026896 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.027012 (+   116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:24.449313 (+422301us) raft_consensus.cc:1671] finished
0809 23:21:24.449335 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.449466 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2230,"raft.queue_time_us":4228,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":23428,"thread_start_us":127653,"threads_started":1,"wal-append.queue_time_us":314760}
I20260809 23:21:24.430608  2725 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.462862 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:24.447911 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3342) took 2035 ms (client timeout 1887 ms). Trace:
W20260809 23:21:24.520535 24534 rpcz_store.cc:269] 0809 23:21:22.412055 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.512099 (+100044us) service_pool.cc:227] Handling call
0809 23:21:24.447740 (+1935641us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000b5c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.447839 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3389376768}
I20260809 23:21:24.468267 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:24.468470  2728 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.464159  2881 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:24.523548  2006 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:24.470182  2897 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.524128  2930 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:24.524593  2930 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:24.524935  2930 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:24.525658  2897 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.469919 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:24.527746 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:24.528072 24546 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260809 23:21:24.528548  2600 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:24.528677 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3369) took 1022 ms. Trace:
I20260809 23:21:24.529021 24546 rpcz_store.cc:276] 0809 23:21:23.506626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.527534 (+1020908us) service_pool.cc:227] Handling call
0809 23:21:24.527864 (+   330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.528442 (+   578us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.528626 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.529219  2650 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:24.530262  2650 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:24.530614  2650 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:24.438848  2505 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.475307 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:24.529038  2600 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:24.481709 24270 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.3:37765 timed out after 1.887s (SENT) [suppressed 10 similar messages]
I20260809 23:21:24.483260  2734 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:24.483281  2923 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:24.376907  2846 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
I20260809 23:21:24.484963 24430 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:24.485960  2726 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:24.557559 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:24.486586  2927 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.589s	user 0.003s	sys 0.000s
I20260809 23:21:24.415079  2770 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.511130 25370 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:24.512424 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.519135 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:24.523382 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:24.476752 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3349) took 1851 ms. Trace:
I20260809 23:21:24.520016 24313 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:24.535300 24323 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260809 23:21:24.555078 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:24.564342 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:24.530972  2587 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:24.583344 24540 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:24.580828 24423 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:24.580996 24544 rpcz_store.cc:276] 0809 23:21:22.624918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.891674 (+266756us) service_pool.cc:227] Handling call
0809 23:21:24.476569 (+1584895us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.476697 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1194514176}
I20260809 23:21:24.588263 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:24.558418 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:24.587047 24313 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260809 23:21:24.583681 24323 pending_rounds.cc:76] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 11
I20260809 23:21:24.592739 24323 pending_rounds.cc:97] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260809 23:21:24.592810 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2870) took 2025 ms. Trace:
I20260809 23:21:24.593205 24323 raft_consensus.cc:2866] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:21:24.595170 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3346) took 2083 ms (client timeout 1955 ms). Trace:
I20260809 23:21:24.530989  2650 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:24.523461  2809 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
W20260809 23:21:24.610358 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 260857us
I20260809 23:21:24.593199 24313 rpcz_store.cc:276] 0809 23:21:22.567844 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.581490 (+2013646us) service_pool.cc:227] Handling call
0809 23:21:24.586699 (+  5209us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.592560 (+  5861us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.592755 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:24.537273  2691 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.689s	user 0.009s	sys 0.000s
W20260809 23:21:24.582002 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62970.8us
I20260809 23:21:24.611877 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.612056 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:24.614957 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:24.595635 24543 rpcz_store.cc:269] 0809 23:21:22.511517 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.513283 (+  1766us) service_pool.cc:227] Handling call
0809 23:21:24.588971 (+2075688us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:24.595093 (+  6122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1460881664}
I20260809 23:21:24.616814 24543 pending_rounds.cc:76] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 13
I20260809 23:21:24.537468  2734 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:24.617354 24543 pending_rounds.cc:97] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 21.14
W20260809 23:21:24.597612 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 262431us
I20260809 23:21:24.620558 24543 raft_consensus.cc:2866] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:24.594973 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2866) took 2393 ms. Trace:
I20260809 23:21:24.611207 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.621033 24323 rpcz_store.cc:276] 0809 23:21:22.202918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.520324 (+2317406us) service_pool.cc:227] Handling call
0809 23:21:24.520976 (+   652us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.594720 (+ 73744us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.594909 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.622107 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.623518 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.602833 24320 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:24.577594  2600 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:24.636303 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.635560 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3405) took 1839 ms. Trace:
I20260809 23:21:24.645756 24323 rpcz_store.cc:276] 0809 23:21:22.795980 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.628750 (+1832770us) service_pool.cc:227] Handling call
0809 23:21:24.629037 (+   287us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.635249 (+  6212us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.635275 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:24.635316 (+    41us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:24.635350 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.635407 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.635508 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5893}
I20260809 23:21:24.518414  2817 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.589255  2587 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:24.636881 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2869) took 2066 ms. Trace:
I20260809 23:21:24.596689  2897 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:24.633982 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45812.3us
I20260809 23:21:24.650542 24323 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:24.640589 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.653858 24320 rpcz_store.cc:276] 0809 23:21:22.567513 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.538335 (+1970822us) service_pool.cc:227] Handling call
0809 23:21:24.598461 (+ 60126us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.633147 (+ 34686us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.633409 (+   262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.668329 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1689) took 1672 ms. Trace:
I20260809 23:21:24.668988 24323 rpcz_store.cc:276] 0809 23:21:22.996179 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.647097 (+1650918us) service_pool.cc:227] Handling call
0809 23:21:24.647450 (+   353us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.667843 (+ 20393us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.668225 (+   382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.630541  2881 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:24.672421 24323 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260809 23:21:24.674548 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46761.7us
I20260809 23:21:24.677974 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:24.681435 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 45235.7us
I20260809 23:21:24.683663 24544 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:24.684070 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.684653 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1693) took 1592 ms. Trace:
I20260809 23:21:24.685954 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.688936 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:24.689672 24320 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:24.690464 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1690) took 1680 ms. Trace:
I20260809 23:21:24.690770 24320 rpcz_store.cc:276] 0809 23:21:23.014708 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.654865 (+1640157us) service_pool.cc:227] Handling call
0809 23:21:24.689501 (+ 34636us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.690245 (+   744us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.690415 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.689704 24546 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:24.630802  2951 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:24.634249  2725 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.633947  2828 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:24.630920  2954 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:24.692538 24540 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:24.693092 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1632) took 1163 ms. Trace:
I20260809 23:21:24.691123 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:24.701344 25365 rpcz_store.cc:276] 0809 23:21:23.530074 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.530874 (+   800us) service_pool.cc:227] Handling call
0809 23:21:23.542963 (+ 12089us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:23.543394 (+   431us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:23.543434 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:23.555863 (+ 12429us) log.cc:842] Serialized 26 byte log entry
0809 23:21:23.739301 (+183438us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:23.739356 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.739444 (+    88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:24.485963 (+746519us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:24.675837 (+189874us) raft_consensus.cc:1671] finished
0809 23:21:24.675861 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.693023 (+ 17162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7025,"raft.queue_time_us":99,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":29300,"spinlock_wait_cycles":15172864,"thread_start_us":110033,"threads_started":1,"wal-append.queue_time_us":297390}
I20260809 23:21:24.634600  2909 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:24.685468 24323 rpcz_store.cc:276] 0809 23:21:23.080816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.670101 (+1589285us) service_pool.cc:227] Handling call
0809 23:21:24.670580 (+   479us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.672852 (+  2272us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.673047 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:24.694293 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 319937us
I20260809 23:21:24.634368  2809 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:24.698793 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:24.704406 24323 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:24.665673  2577 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.665913  1731 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:24.705085 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1696) took 1409 ms. Trace:
W20260809 23:21:24.737025 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46010.2us
I20260809 23:21:24.737463 24323 rpcz_store.cc:276] 0809 23:21:23.295963 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.703648 (+1407685us) service_pool.cc:227] Handling call
0809 23:21:24.704013 (+   365us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.704804 (+   791us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.705029 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.737843 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.739173 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.626821  2691 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.744402 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:24.696413  2869 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.897s	user 0.004s	sys 0.000s
I20260809 23:21:24.690239  2505 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:24.695673  1429 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:24.751441 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:24.757061 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2876) took 1266 ms. Trace:
I20260809 23:21:24.707751  2945 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:24.735345 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:24.754740 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.742031 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:24.757773 24323 rpcz_store.cc:276] 0809 23:21:23.490599 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.756310 (+1265711us) service_pool.cc:227] Handling call
0809 23:21:24.756599 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.756765 (+   166us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.756789 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:24.756827 (+    38us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:24.756859 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.756914 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.757009 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.763222 24320 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:24.762421 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 342189us
W20260809 23:21:24.766104 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 284408us
I20260809 23:21:24.798590 24323 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260809 23:21:24.749449  2171 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:24.741104  2846 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:24.810240 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:24.812235 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57611.5us
I20260809 23:21:24.827143 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:24.700301  2600 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:21:24.753297  2909 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:24.810832 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:24.882685 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71978us
I20260809 23:21:24.699680  2593 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
I20260809 23:21:24.818832  2171 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:24.890005  2171 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:24.890654  2944 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:24.819171  2907 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260809 23:21:24.891196  2944 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.892033  2593 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:24.892217  2948 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:24.892483  2593 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:24.752266  2828 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:24.756858  2954 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:24.814447 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3406) took 1258 ms. Trace:
I20260809 23:21:24.919564 24323 rpcz_store.cc:276] 0809 23:21:23.555907 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.763456 (+1207549us) service_pool.cc:227] Handling call
0809 23:21:24.763758 (+   302us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.813888 (+ 50130us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.814317 (+   429us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.921231 24323 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:24.920225  2828 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.922012 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3408) took 1314 ms. Trace:
I20260809 23:21:24.810190  2945 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:24.922556 24323 rpcz_store.cc:276] 0809 23:21:23.607406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.920618 (+1313212us) service_pool.cc:227] Handling call
0809 23:21:24.921031 (+   413us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.921667 (+   636us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.921881 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.924070  2945 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.692155  2052 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:24.886021 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3407) took 1290 ms. Trace:
I20260809 23:21:24.944705 24320 rpcz_store.cc:276] 0809 23:21:23.595235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.784076 (+1188841us) service_pool.cc:227] Handling call
0809 23:21:24.885424 (+101348us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.885648 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.885681 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:24.885731 (+    50us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:24.885779 (+    48us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.885841 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.885957 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:24.696755  2949 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.893273 25370 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:24.707903  2734 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.894164  2944 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.894198  2948 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:24.906528  2006 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:24.907925  2691 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:24.908897 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3388) took 2951 ms. Trace:
I20260809 23:21:24.908252  2171 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.708787  2752 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260809 23:21:24.811866  2505 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:24.697530  2897 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:24.746546  2577 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.825264  2154 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:24.753530  2587 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:24.709082  2951 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.979190 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.972332 24321 rpcz_store.cc:276] 0809 23:21:21.957167 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:21.967028 (+  9861us) service_pool.cc:227] Handling call
0809 23:21:21.967381 (+   353us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:21.967800 (+   419us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:21.967828 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:21.968244 (+   416us) log.cc:842] Serialized 26 byte log entry
0809 23:21:22.786582 (+818338us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:22.786633 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:22.786707 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:24.666856 (+1880149us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b240019ba58. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:21:24.908562 (+241706us) raft_consensus.cc:1671] finished
0809 23:21:24.908588 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:24.908819 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8558,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":30337,"spinlock_wait_cycles":73811200,"thread_start_us":816066,"threads_started":1,"wal-append.queue_time_us":796729}
I20260809 23:21:24.992364 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:25.012787 25375 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:21:24.994330 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195666us
I20260809 23:21:24.889382  2909 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:24.997299 24320 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:24.927330  2955 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:24.674090  2671 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.696941  2806 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.275s
I20260809 23:21:24.880734  2846 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:24.989773  2052 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:25.018343 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:24.989662  2907 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.014817 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:24.811908  2809 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:25.032086 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3399) took 2931 ms (client timeout 1956 ms). Trace:
I20260809 23:21:25.001750  2736 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.080s
I20260809 23:21:25.125870  2988 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260809 23:21:25.125252 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 110587us
I20260809 23:21:25.127168  3004 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:25.127602  3004 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:25.127871 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:25.128785  2988 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:25.129220  2988 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:25.128592  3004 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.129562  2988 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:25.133231  1485 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.133723  2579 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.135316  3004 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.136399 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:25.137084 25370 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:25.138286  2639 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:25.138778  2900 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:25.138844  2639 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.141083  2639 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.141721  2639 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.142201  2639 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:25.143275  2936 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.247s	user 0.000s	sys 0.003s
I20260809 23:21:25.144881 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3429) took 1741 ms. Trace:
I20260809 23:21:25.145306 24545 rpcz_store.cc:276] 0809 23:21:23.403365 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.801169 (+397804us) service_pool.cc:227] Handling call
0809 23:21:23.801481 (+   312us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:23.801719 (+   238us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:23.801743 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:23.802180 (+   437us) log.cc:842] Serialized 26 byte log entry
0809 23:21:23.895423 (+ 93243us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:23.895460 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:23.895521 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.142530 (+1247009us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:25.144534 (+  2004us) raft_consensus.cc:1671] finished
0809 23:21:25.144556 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.144795 (+   239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10537344,"thread_start_us":92888,"threads_started":1,"wal-append.queue_time_us":93351}
I20260809 23:21:25.147578 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.147974  2577 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.149313 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:25.143275  2904 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.755s	user 0.009s	sys 0.002s
I20260809 23:21:25.149909  2788 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:25.149986  2904 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.150672 24545 raft_consensus.cc:2354] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:25.008263  2878 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.151268  2878 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.152230 24541 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 19: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:25.152434 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.153314 24545 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260809 23:21:25.153873  2878 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:25.024621  2969 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260809 23:21:25.164933  2662 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:25.019287  2909 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.163904 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3438) took 1258 ms. Trace:
I20260809 23:21:25.183699 24541 rpcz_store.cc:276] 0809 23:21:23.905648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.000871 (+ 95223us) service_pool.cc:227] Handling call
0809 23:21:25.163635 (+1162764us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400270bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.163825 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2645107584}
I20260809 23:21:25.162083 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3374) took 1232 ms. Trace:
I20260809 23:21:25.019641  2752 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:25.115761 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3367) took 1609 ms. Trace:
I20260809 23:21:25.184854 24543 rpcz_store.cc:276] 0809 23:21:23.929790 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.616061 (+686271us) service_pool.cc:227] Handling call
0809 23:21:24.616536 (+   475us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.622341 (+  5805us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.622382 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.623442 (+  1060us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.974510 (+351068us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:24.974561 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.974659 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.161864 (+187205us) raft_consensus.cc:1671] finished
0809 23:21:25.161885 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.162026 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":350430,"threads_started":1,"wal-append.queue_time_us":534379}
I20260809 23:21:25.185159 24534 rpcz_store.cc:276] 0809 23:21:23.506077 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.534305 (+1028228us) service_pool.cc:227] Handling call
0809 23:21:24.534627 (+   322us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.535007 (+   380us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.535046 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.553759 (+ 18713us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.753016 (+199257us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:24.753065 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.753146 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.115493 (+362347us) raft_consensus.cc:1671] finished
0809 23:21:25.115520 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.115690 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":136332,"threads_started":1,"wal-append.queue_time_us":523921}
I20260809 23:21:25.113686  2951 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.158238  2639 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:25.166723 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148477us
I20260809 23:21:25.194429 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:25.167058 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2868) took 2678 ms. Trace:
I20260809 23:21:25.195374 24312 rpcz_store.cc:276] 0809 23:21:22.488549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.531731 (+2043182us) service_pool.cc:227] Handling call
0809 23:21:24.532048 (+   317us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.532382 (+   334us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.532409 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.532850 (+   441us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.990999 (+458149us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:24.991097 (+    98us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.991195 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.166827 (+175632us) raft_consensus.cc:1671] finished
0809 23:21:25.166865 (+    38us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.167004 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1646,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":19903,"thread_start_us":457692,"threads_started":1,"wal-append.queue_time_us":630344}
I20260809 23:21:25.032878 24323 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:25.168509  2006 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:25.027781 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3456) took 1113 ms. Trace:
I20260809 23:21:25.203161 24428 rpcz_store.cc:276] 0809 23:21:23.914395 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.915053 (+   658us) service_pool.cc:227] Handling call
0809 23:21:23.915424 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.294806 (+379382us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.027471 (+732665us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.027703 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271782784}
W20260809 23:21:25.167198 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 338067us
W20260809 23:21:25.023712 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 140088us
I20260809 23:21:25.168531 24430 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:25.139415  2900 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:25.125286 24316 rpcz_store.cc:269] 0809 23:21:22.100565 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.287955 (+187390us) service_pool.cc:227] Handling call
0809 23:21:25.031824 (+2743869us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001afc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.031998 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3138224000}
I20260809 23:21:25.169543 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:25.170194 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:25.024629 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.173098 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:25.163528  2988 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:25.123463  2879 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.235s	user 0.015s	sys 0.000s
I20260809 23:21:25.180536  2662 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:25.134318  2579 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:25.222746 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49718us
I20260809 23:21:25.154284  2904 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.181645 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1390) took 2811 ms. Trace:
I20260809 23:21:25.225342 24435 rpcz_store.cc:276] 0809 23:21:22.365972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.366126 (+   154us) service_pool.cc:227] Handling call
0809 23:21:22.366430 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:22.784650 (+418220us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.173825 (+2389175us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.177073 (+  3248us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":958603648}
I20260809 23:21:25.169165  2907 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.202170  2982 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.205824  2600 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:25.095310 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:25.023321  2817 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.028254  2171 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.176683  2670 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:25.027069  2949 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.202836 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:25.182781  2923 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.005s
I20260809 23:21:25.216809 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:25.217595 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47479us
I20260809 23:21:25.127638  2505 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.219244 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.229034 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.223960  2717 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260809 23:21:25.016855  2691 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:25.264323 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1631) took 1734 ms. Trace:
W20260809 23:21:25.024621  2964 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
I20260809 23:21:25.226668  2993 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:25.016651  2770 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.225957  2878 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:25.220578  2577 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:25.265648  2930 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.733s	user 0.013s	sys 0.002s
I20260809 23:21:25.028972  2593 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:25.245864 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.004s	sys 0.020s
I20260809 23:21:25.223474  2879 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.220083  2767 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260809 23:21:25.221473  2951 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260809 23:21:25.260095 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:25.264603 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61862.8us
W20260809 23:21:25.264827 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57734.4us
I20260809 23:21:25.264278  2691 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:25.272025  2993 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:25.264671 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.270702 24543 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260809 23:21:25.270756 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:25.263764  2171 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.028856  2897 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.285714 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:25.286983 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.004s	sys 0.020s
I20260809 23:21:25.265192 25363 rpcz_store.cc:276] 0809 23:21:23.529782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.529905 (+   123us) service_pool.cc:227] Handling call
0809 23:21:23.531977 (+  2072us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:23.532629 (+   652us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:23.532674 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:23.533245 (+   571us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.816675 (+1283430us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b180012eaa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:25.181694 (+365019us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:25.181759 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.181929 (+   170us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.252162 (+ 70233us) raft_consensus.cc:1671] finished
0809 23:21:25.252188 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.264252 (+ 12064us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2314,"raft.queue_time_us":101,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4757,"spinlock_wait_cycles":3586816,"thread_start_us":364828,"threads_started":1,"wal-append.queue_time_us":1713709}
W20260809 23:21:25.272905  2728 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.004s	sys 0.010s
I20260809 23:21:25.265843  2469 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:25.294920 24316 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:25.271950  2817 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:25.238137  2877 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260809 23:21:25.241232  2795 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:25.297300 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:25.244481  2670 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.304474 24534 raft_consensus.cc:2354] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16.
I20260809 23:21:25.305794 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.274014  2878 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:25.308171  2988 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.308332 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.346648 24312 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:25.347074 24312 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:25.347970 24312 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.348875  2907 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.349167 24312 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:25.310870 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:25.359346 25373 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:25.311115  2820 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.284s	user 0.014s	sys 0.000s
I20260809 23:21:25.304018  2993 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.363257  2820 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.324326 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:25.320921 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.325088 24535 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:25.371683 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47462.2us
I20260809 23:21:25.287251  2930 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.373283 24378 leader_election.cc:398] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:25.373595 24378 leader_election.cc:401] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260809 23:21:25.375685 25368 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:25.375312  2988 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:25.376267  2988 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.376935  2988 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.380098  2579 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.380698  2907 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:25.374584 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:25.382623 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:25.338711 24270 leader_election.cc:398] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 15 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16.
W20260809 23:21:25.383683 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47458.8us
I20260809 23:21:25.385170 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:25.333346  2877 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.387315  2954 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:25.387732  2930 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:25.324047  2728 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.005s	sys 0.011s
I20260809 23:21:25.337749  2817 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:25.388160 25370 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:25.354660  2879 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.388474  2817 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:25.236474  2982 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.388760 24542 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:25.388979  2817 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:25.357664 24309 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:25.325649  2171 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.315641  2846 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.315409  2995 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.387092  2877 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.413717  2670 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.286350  2577 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.414642  2877 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.416115  2577 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:25.416604  2650 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.788s	user 0.014s	sys 0.000s
I20260809 23:21:25.416790  2691 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.414824  2982 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.418387  2691 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:25.356776  2878 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.386445  2909 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.343389 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.421080  2909 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.421537 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.421902  2770 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260809 23:21:25.389664 24317 raft_consensus.cc:595] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:25.423740 24317 raft_consensus.cc:603] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:25.389997 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.332353 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:25.425097 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92846us
I20260809 23:21:25.390314 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:25.397423 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.425976 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:25.261256  2662 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.407459  2995 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.408603 24314 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260809 23:21:25.409011  2171 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:25.409433 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1404) took 1574 ms (client timeout 1568 ms). Trace:
I20260809 23:21:25.404300  2945 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:25.396010  2907 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:25.428519  2119 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.435726 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3370) took 1858 ms. Trace:
I20260809 23:21:25.436133 24546 rpcz_store.cc:276] 0809 23:21:23.577879 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.529940 (+952061us) service_pool.cc:227] Handling call
0809 23:21:24.530240 (+   300us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:24.692027 (+161787us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.435501 (+743474us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.435667 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366366464}
I20260809 23:21:25.435951  2119 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.411656  2900 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.412518 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1622) took 2814 ms. Trace:
I20260809 23:21:25.417054  2650 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.390493 24311 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:25.322093  2587 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:25.426293 24541 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:25.427737 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.429503 25375 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:25.429633 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:25.430064  2662 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.430375  2879 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:25.464073  2720 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.464620  2720 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.449970  2877 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260809 23:21:25.466125  2877 consensus_queue.cc:1225] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:25.468052  2988 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.470408  2988 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.431396 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.471817 24423 raft_consensus.cc:2396] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260809 23:21:25.472148  2981 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.335s	user 0.002s	sys 0.000s
I20260809 23:21:25.432770 24316 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:25.434263  2945 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:25.439700 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2865) took 3244 ms. Trace:
I20260809 23:21:25.473397  2945 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:25.432852 24424 rpcz_store.cc:269] 0809 23:21:23.834580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.835329 (+   749us) service_pool.cc:227] Handling call
0809 23:21:25.409205 (+1573876us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.409320 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83984128}
I20260809 23:21:25.473341 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.366705  2820 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.473481 24314 rpcz_store.cc:276] 0809 23:21:22.195479 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.820487 (+1625008us) service_pool.cc:227] Handling call
0809 23:21:23.849100 (+ 28613us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.408369 (+1559269us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b54001eec20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:21:25.433633 (+ 25264us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.439633 (+  6000us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31297280}
I20260809 23:21:25.475008 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1689) took 1134 ms. Trace:
W20260809 23:21:25.475404  2950 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.991s	user 0.000s	sys 0.003s
I20260809 23:21:25.475643 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.475409 25369 rpcz_store.cc:276] 0809 23:21:24.340619 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.340749 (+   130us) service_pool.cc:227] Handling call
0809 23:21:24.341079 (+   330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.468032 (+126953us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.513428 (+ 45396us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.595222 (+ 81794us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.745916 (+150694us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:24.745954 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.746017 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.471591 (+725574us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:25.474651 (+  3060us) raft_consensus.cc:1671] finished
0809 23:21:25.474673 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.474904 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8440,"raft.queue_time_us":67984,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":27302144,"thread_start_us":198097,"threads_started":2,"wal-append.queue_time_us":507560}
I20260809 23:21:25.477216 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.477471  2154 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:25.477975  2154 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:25.406134  1429 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.638s	user 0.006s	sys 0.005s
I20260809 23:21:25.478689  1429 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:25.441905  2907 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:25.446156 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3410) took 1838 ms (client timeout 1552 ms). Trace:
I20260809 23:21:25.431911  2725 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:25.451447 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.387120  3004 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:25.448779 25372 rpcz_store.cc:276] 0809 23:21:22.598604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.795838 (+197234us) service_pool.cc:227] Handling call
0809 23:21:22.826256 (+ 30418us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:22.873602 (+ 47346us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:22.873632 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:22.874119 (+   487us) log.cc:842] Serialized 26 byte log entry
0809 23:21:23.493769 (+619650us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b1800466d20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:23.687523 (+193754us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:23.687562 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.412203 (+1724641us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.412260 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.412312 (+    52us) raft_consensus.cc:1671] finished
0809 23:21:25.412324 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.412457 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32149,"raft.queue_time_us":32275,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":10526,"spinlock_wait_cycles":37549696,"thread_start_us":193550,"threads_started":1,"wal-append.queue_time_us":882635,"wal-append.run_cpu_time_us":5940,"wal-append.run_wall_time_us":1049394}
I20260809 23:21:25.460187 24544 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:25.463271  2900 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:25.465071  2987 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.335s	user 0.001s	sys 0.001s
I20260809 23:21:25.482450 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2887) took 1143 ms. Trace:
I20260809 23:21:25.483088 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1621) took 1839 ms. Trace:
I20260809 23:21:25.483038 24311 rpcz_store.cc:276] 0809 23:21:24.339381 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.429067 (+ 89686us) service_pool.cc:227] Handling call
0809 23:21:25.482214 (+1053147us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.482381 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2208753920}
I20260809 23:21:25.483564 24539 rpcz_store.cc:276] 0809 23:21:23.643731 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.594338 (+950607us) service_pool.cc:227] Handling call
0809 23:21:24.632527 (+ 38189us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.673473 (+ 40946us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.673509 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.680462 (+  6953us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.948867 (+268405us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:24.948909 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.948980 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.464484 (+515504us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:25.482643 (+ 18159us) raft_consensus.cc:1671] finished
0809 23:21:25.482737 (+    94us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.483029 (+   292us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40331,"raft.queue_time_us":42996,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":19835,"spinlock_wait_cycles":43337472,"thread_start_us":197657,"threads_started":1,"wal-append.queue_time_us":402473}
I20260809 23:21:25.485268  2725 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:25.480646  2907 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.486857 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3375) took 1198 ms. Trace:
I20260809 23:21:25.487299 24540 rpcz_store.cc:276] 0809 23:21:24.288550 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.706868 (+418318us) service_pool.cc:227] Handling call
0809 23:21:24.707248 (+   380us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.082829 (+375581us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.082876 (+    47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:25.082956 (+    80us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:25.083001 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.486570 (+403569us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.486630 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.486781 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":863065856}
I20260809 23:21:25.488785 24313 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:25.488938 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3420) took 1114 ms. Trace:
I20260809 23:21:25.489486 24320 rpcz_store.cc:276] 0809 23:21:24.374477 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.020264 (+645787us) service_pool.cc:227] Handling call
0809 23:21:25.106484 (+ 86220us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.183167 (+ 76683us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.183212 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:25.183264 (+    52us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:25.183318 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.488678 (+305360us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b540043add4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.488738 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.488872 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175785856}
I20260809 23:21:25.490242 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3404) took 2698 ms. Trace:
W20260809 23:21:25.480402 24309 rpcz_store.cc:269] 0809 23:21:23.608019 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.614714 (+  6695us) service_pool.cc:227] Handling call
0809 23:21:25.445930 (+1831216us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.446071 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4005781376}
I20260809 23:21:25.459992  2993 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.490787 24313 rpcz_store.cc:276] 0809 23:21:22.791663 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.628619 (+1836956us) service_pool.cc:227] Handling call
0809 23:21:25.488563 (+859944us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.489356 (+   793us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.490033 (+   677us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b54000db6d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.490177 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1930467712}
I20260809 23:21:25.518126 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:25.518776 25372 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.519603 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:25.521209  2993 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.521927  2993 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.522234 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3363) took 2017 ms. Trace:
I20260809 23:21:25.529219 24542 rpcz_store.cc:276] 0809 23:21:23.504992 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.033189 (+528197us) service_pool.cc:227] Handling call
0809 23:21:24.440390 (+407201us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.389242 (+948852us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.521427 (+132185us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54001d5e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.521992 (+   565us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54001d5e50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.522144 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2906030848}
I20260809 23:21:25.535683 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1289) took 2141 ms. Trace:
I20260809 23:21:25.523773  2154 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.466640  2877 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:25.560305 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 108978us
I20260809 23:21:25.536386 24313 rpcz_store.cc:276] 0809 23:21:23.394691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.500512 (+2105821us) service_pool.cc:227] Handling call
0809 23:21:25.535566 (+ 35054us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:25.596941 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 169484us
W20260809 23:21:25.600782 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 81331.2us
I20260809 23:21:25.474877 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.603320 24316 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:25.593978 25373 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:25.612450 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.625463 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.593319  2779 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:25.610669 24312 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
W20260809 23:21:25.631906 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1403) took 1817 ms (client timeout 1696 ms). Trace:
W20260809 23:21:25.634804 24422 rpcz_store.cc:269] 0809 23:21:23.814492 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:23.819780 (+  5288us) service_pool.cc:227] Handling call
0809 23:21:25.631727 (+1811947us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b54003ebcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:25.631844 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095857920}
I20260809 23:21:25.635339 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.635649 24312 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 7
I20260809 23:21:25.635900 24312 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 16.8
I20260809 23:21:25.635950 24424 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:25.636253 24312 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:25.637267 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.639950 24431 raft_consensus.cc:2396] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260809 23:21:25.640290 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 260178us
I20260809 23:21:25.594842  2877 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:25.641741 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.614537  2806 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.640697 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:25.645290 24313 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:25.644057 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.647851 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.648710 24430 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:25.650847  2827 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.651119  2957 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:25.651443  2827 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.651736  2957 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.653610  2827 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.653410  2878 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.641803  1539 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:25.622701  2741 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:25.659404 24431 raft_consensus.cc:2354] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:25.659617 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.664755 24431 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:25.666249 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1686) took 1628 ms. Trace:
I20260809 23:21:25.666661 25366 rpcz_store.cc:276] 0809 23:21:24.037602 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.039457 (+  1855us) service_pool.cc:227] Handling call
0809 23:21:24.039784 (+   327us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:24.155980 (+116196us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:24.156023 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:24.183726 (+ 27703us) log.cc:842] Serialized 26 byte log entry
0809 23:21:24.338449 (+154723us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:24.338481 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:24.338529 (+    48us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.474684 (+1136155us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:25.643397 (+168713us) raft_consensus.cc:1671] finished
0809 23:21:25.643425 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.666182 (+ 22757us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":61032,"raft.queue_time_us":83399,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":13551,"spinlock_wait_cycles":25723264,"thread_start_us":99670,"threads_started":1,"wal-append.queue_time_us":186388}
I20260809 23:21:25.670295 25373 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:25.673979 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:25.674427  2957 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.674715 25373 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:25.678586 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:25.680471 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.682858 24435 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260809 23:21:25.613257  2801 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260809 23:21:25.613571  2194 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260809 23:21:25.701086 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.696420 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:25.684890 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 380389us
I20260809 23:21:25.686896 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.711541 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.626891  2900 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260809 23:21:25.651470  2720 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.792032 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.794063 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.720264 25325 leader_election.cc:398] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:25.720355  3004 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.741254  2995 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:25.745985  2788 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.595s	user 0.005s	sys 0.005s
I20260809 23:21:25.703770  2795 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.043s
I20260809 23:21:25.836982 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.839372 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.747498 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:25.704705  2770 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.708415  2879 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.751616  2877 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.755299  2736 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.755693 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.683589  2650 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.769972 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.808907  2670 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:25.860754  2897 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.012s	sys 0.031s
I20260809 23:21:25.866901  2770 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.771068 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:25.726158 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:25.896191 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.896997 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:25.899137 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:25.795778 25325 leader_election.cc:401] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260809 23:21:25.796924 24543 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260809 23:21:25.800639 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.720568  2639 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:25.907202  2639 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:25.907317  2770 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.807367  3004 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.908154  2639 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.909727  2650 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:25.910032  2949 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:25.910564  2949 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:25.891031  2828 consensus_queue.cc:1159] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 18 status { last_received { term: 18 index: 14 } last_committed_idx: 13 last_received_current_leader { term: 18 index: 14 } }
I20260809 23:21:25.660998  2982 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.065s
I20260809 23:21:25.667127  2806 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.859266 24428 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:25.930001 24534 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
W20260809 23:21:25.901602 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 131740us
I20260809 23:21:25.930847 24496 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:25.797580  2508 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.904568 24541 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:25.888583  2670 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:25.888582  2897 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.012s	sys 0.033s
I20260809 23:21:25.874128  2736 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.946494  2736 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.886809  2993 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:25.867240  2879 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.947819  2929 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:25.948961  3026 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.208s	user 0.002s	sys 0.000s
I20260809 23:21:25.949072  2879 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.911429  2949 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:25.860272 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:25.934582 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.936266 24496 leader_election.cc:398] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:25.858780  2881 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.960858 24422 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:25.808476  2955 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.124s
I20260809 23:21:25.938443 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:25.864372  2878 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:25.940797  2508 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.941517  2962 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:25.941890  2806 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:25.912077  2416 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:25.911886  2996 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260809 23:21:25.861891  2846 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
I20260809 23:21:25.958295 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.981622 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.937077 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.950857 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1693) took 1208 ms. Trace:
I20260809 23:21:25.806660  2667 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.130s
I20260809 23:21:25.982621 25365 rpcz_store.cc:276] 0809 23:21:24.742194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.742743 (+   549us) service_pool.cc:227] Handling call
0809 23:21:24.748173 (+  5430us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.022122 (+273949us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.022151 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:25.083288 (+ 61137us) log.cc:842] Serialized 26 byte log entry
0809 23:21:25.240994 (+157706us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:25.241046 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:25.950141 (+709095us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400660154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:25.950199 (+    58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:25.950251 (+    52us) raft_consensus.cc:1671] finished
0809 23:21:25.950274 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:25.950766 (+   492us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":61484,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":19780,"spinlock_wait_cycles":629196032,"thread_start_us":94125,"threads_started":1,"wal-append.queue_time_us":161449}
I20260809 23:21:25.968844 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.960433  2736 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:25.987073 24541 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:25.988153 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.989197 24428 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:25.969408 24425 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260809 23:21:25.962764  2817 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.531s	user 0.012s	sys 0.003s
I20260809 23:21:25.998214 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:25.942219  2670 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.970320  3004 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:25.972108 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.993253 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:25.996590 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.998432 24543 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:26.003398 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:25.990070 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.004786 24496 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:26.005235 24428 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:25.998247 24545 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.033s	sys 0.038s
W20260809 23:21:26.009069 24545 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.033s	sys 0.038s
I20260809 23:21:25.939265  3025 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260809 23:21:25.987241  2806 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:26.019201 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.019021 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:25.965130  2881 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.021704 24428 raft_consensus.cc:2354] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
W20260809 23:21:26.024503 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.009s	sys 0.002s
W20260809 23:21:26.024991 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.009s	sys 0.002s
W20260809 23:21:25.937597 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 223079us
W20260809 23:21:26.027392 24311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.006s	sys 0.010s
I20260809 23:21:26.028584 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:26.001632  2587 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.528s	user 0.007s	sys 0.008s
I20260809 23:21:26.036442 24430 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:25.999532  2982 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:26.033242 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:26.026834 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:26.057269 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.046016 24424 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:26.059520 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.008581  2974 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.007715  2598 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:26.009966  2650 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:26.002086  2662 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:26.085202 24425 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:26.023226  2691 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.604s	user 0.017s	sys 0.000s
I20260809 23:21:26.020468  1677 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:26.037626  2807 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:26.067870 24313 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:26.072297 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.067309  2463 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.001278  2817 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:25.987054  2416 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.080366 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:26.073076  2788 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.087323 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.071616 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.008292  2879 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.063s
I20260809 23:21:26.089069 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:25.998211  2736 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:26.067920  2505 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.004s	sys 0.096s
I20260809 23:21:26.072638 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.037377  3028 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260809 23:21:26.128165 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:26.085856  2777 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.097236  2779 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:26.130481  2463 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.096805  2807 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:26.081480  2974 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.100955  2662 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:26.131258 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51027.8us
I20260809 23:21:26.202175 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 52) took 3740 ms. Trace:
I20260809 23:21:26.131659 24321 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:26.204267 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.144923  2777 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.148131  2779 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.138417  2006 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:26.209753  2006 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:26.210526  2779 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.148744  2508 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.158841 24539 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260809 23:21:26.211594  2006 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 18.18, Last appended by leader: 18, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.162134  2877 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:26.205528  2994 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:26.216435  2994 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:26.217729  2994 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.203552  2119 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.225155  2994 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:26.156854  2416 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.202564 24317 rpcz_store.cc:276] 0809 23:21:22.462160 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.602284 (+140124us) service_pool.cc:227] Handling call
0809 23:21:26.201908 (+3599624us) spinlock_profiling.cc:243] Waited 2.76 s on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:26.202037 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6343903104}
I20260809 23:21:26.227592  2770 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:26.228006  2770 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:26.229626  2770 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.184122  2817 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.198228 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.232074  2770 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:26.232865  2995 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.233570  2995 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.235635 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:26.236472 24317 raft_consensus.cc:2354] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23.
I20260809 23:21:26.237226 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:26.238081  2819 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:26.239760  2947 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:26.171857 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:26.137579  2788 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:26.241997 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70238.4us
I20260809 23:21:26.242522  2929 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:26.243642 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:26.244314  2788 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.159159  2807 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:26.220134  2993 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:26.245864  2993 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:26.167872  2505 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.004s	sys 0.096s
I20260809 23:21:26.247753  2993 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.249985  2993 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:26.250195  2884 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:26.169427 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:26.180852  2945 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.707s	user 0.010s	sys 0.018s
I20260809 23:21:26.194263  2962 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:26.256174  2962 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:26.256559  2947 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:26.257217  2947 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:26.259490  2884 consensus_queue.cc:1159] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 24 status { last_received { term: 23 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:26.161150  2463 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.192735  2736 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.230453  2881 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.237833 24317 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:26.264138 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:26.199680 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.278518 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.130798 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:26.264550 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:26.281270 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 150735us
I20260809 23:21:26.266034 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.283514 24430 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:26.273847 25365 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:26.278650 24309 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:26.289044 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:26.282459  2929 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:26.290647 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.283974 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.293011 24545 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260809 23:21:26.200562  1539 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.545s	user 0.002s	sys 0.011s
I20260809 23:21:26.296896 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:26.298032 24321 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:26.297077 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.300097 24309 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:26.275903  2884 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:26.291644  2947 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.240841  2819 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:26.188743  2662 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.167284  2974 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:26.294325  2691 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:26.308120 24317 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:26.301028 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:26.355577 24317 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:26.356971 24317 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:26.315313 24427 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:26.320109 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 254893us
I20260809 23:21:26.335597 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1711) took 1024 ms. Trace:
I20260809 23:21:26.360781 24314 rpcz_store.cc:276] 0809 23:21:25.311559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.478208 (+166649us) service_pool.cc:227] Handling call
0809 23:21:25.478574 (+   366us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.600878 (+122304us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.600926 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:25.611701 (+ 10775us) log.cc:842] Serialized 26 byte log entry
0809 23:21:26.097561 (+485860us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:26.097775 (+   214us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:26.097916 (+   141us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:26.335338 (+237422us) raft_consensus.cc:1671] finished
0809 23:21:26.335364 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:26.335530 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42714,"raft.queue_time_us":49617,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":7091,"thread_start_us":439227,"threads_started":1,"wal-append.queue_time_us":677834}
I20260809 23:21:26.344528 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1694) took 1343 ms. Trace:
I20260809 23:21:26.387885 25366 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:26.255589  2945 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.401216 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3412) took 2443 ms. Trace:
I20260809 23:21:26.401714 24323 rpcz_store.cc:276] 0809 23:21:23.958280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.924034 (+965754us) service_pool.cc:227] Handling call
0809 23:21:25.032639 (+108605us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.202368 (+169729us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:26.400972 (+1198604us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54001afc50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:26.401141 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":248296832}
I20260809 23:21:26.391330 24431 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:26.393759 25366 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:26.398108 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.405486 24430 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:26.368126 25371 rpcz_store.cc:276] 0809 23:21:25.001552 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.001718 (+   166us) service_pool.cc:227] Handling call
0809 23:21:25.017295 (+ 15577us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.035491 (+ 18196us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:25.237635 (+202144us) log.cc:842] Serialized 14 byte log entry
0809 23:21:25.237857 (+   222us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:25.237902 (+    45us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:25.237945 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:26.344225 (+1106280us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:26.344295 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:26.344456 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":274847,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":14187,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":18875,"spinlock_wait_cycles":41197312,"thread_start_us":121510,"threads_started":1}
I20260809 23:21:26.416035 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:26.352270 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70129.2us
I20260809 23:21:26.349445 24321 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:26.422075 24540 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.061s	sys 0.045s
W20260809 23:21:26.422456 24540 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.062s	sys 0.045s
I20260809 23:21:26.422609 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.427637 25321 leader_election.cc:398] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
I20260809 23:21:26.426468 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.416962 24309 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260809 23:21:26.428412 24314 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.10-19.10]
W20260809 23:21:26.446257 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 474490us
I20260809 23:21:26.445046 25321 leader_election.cc:401] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:26.450817 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:26.435083 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.314316  2819 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:26.367550  2923 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.452209 24431 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:26.506829 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.507671 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.458138 25371 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:26.458686 25366 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:26.470623 24435 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260809 23:21:26.473824 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.511889 25370 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:26.520349 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.521202 24431 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:26.418299  2752 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.316792  2416 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.138s
I20260809 23:21:26.313547  2907 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.142s
I20260809 23:21:26.301689  2884 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:26.471290  2639 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.033s	sys 0.031s
I20260809 23:21:26.304239  2995 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.528721  2923 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:26.549364 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.029s	sys 0.059s
W20260809 23:21:26.646229 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.029s	sys 0.059s
W20260809 23:21:26.549911 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 501379us
I20260809 23:21:26.647523  2945 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:26.652160 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.652992 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:26.555254 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.656339 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.565582  2752 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.661164  2650 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:26.627750  2884 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:26.532613 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.559320 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.704272 25368 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:26.342279  2807 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.453200  2929 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
I20260809 23:21:26.471045  2764 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260809 23:21:26.624239  2639 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.033s	sys 0.031s
I20260809 23:21:26.712359 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:26.525408  2779 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:26.743937 24311 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.039s	sys 0.062s
I20260809 23:21:26.744904 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
W20260809 23:21:26.753077 24311 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.039s	sys 0.062s
W20260809 23:21:26.744607 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.024s	sys 0.032s
W20260809 23:21:26.753676 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.025s	sys 0.033s
I20260809 23:21:26.661928  2650 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.609158  2806 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.779678 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:26.781046 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.791317 24320 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:26.807019 25368 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:26.807598 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.809535 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:26.754328  2819 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.779490  2779 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.813066 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:26.813768  2819 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:26.825106 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:26.760545  2949 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.023s	sys 0.033s
I20260809 23:21:26.839871  2779 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.841518 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:26.825305 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:26.851361 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.021s	sys 0.053s
I20260809 23:21:26.760615  2625 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:26.851675 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:26.760967 24541 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:26.760865  2764 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:26.841230  2949 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.024s	sys 0.033s
W20260809 23:21:26.852129 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.022s	sys 0.053s
I20260809 23:21:26.824117  1731 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:26.846647  2006 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.021s	sys 0.020s
I20260809 23:21:26.877967 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.880585 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:26.883945 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96749.8us
I20260809 23:21:26.887143 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.846457  2639 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:26.851931  1539 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.894553 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3387) took 1294 ms. Trace:
I20260809 23:21:26.895119 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.768432 24431 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260809 23:21:26.816653  2806 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:26.861315  2764 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:26.863428  2779 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:26.948163 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.957945 24534 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:26.967641 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:26.948927 24540 rpcz_store.cc:276] 0809 23:21:25.600459 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.600993 (+   534us) service_pool.cc:227] Handling call
0809 23:21:25.601308 (+   315us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.601504 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:26.422842 (+821338us) log.cc:842] Serialized 14 byte log entry
0809 23:21:26.894133 (+471291us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:26.894198 (+    65us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:26.894259 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:26.894341 (+    82us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:26.894483 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":278004,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":3963,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":12296,"thread_start_us":1241838,"threads_started":2}
W20260809 23:21:26.879563  2006 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.021s	sys 0.021s
I20260809 23:21:27.001559 25366 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:27.011528 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.018626 24431 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:27.019235 24431 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:26.997419 25370 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:27.028046 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.007s	sys 0.007s
I20260809 23:21:26.907374  1731 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:27.030886 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.007s	sys 0.007s
I20260809 23:21:27.036175 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:26.949338  2752 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:26.957477  2779 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:27.067622 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.015282  2949 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:26.972505  2974 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260809 23:21:27.049177  2006 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:26.957050  1539 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:26.965334  2639 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.105176 24534 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:27.073809  2779 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:27.021593 24431 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:27.122565 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.014s	sys 0.010s
W20260809 23:21:27.123016 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.015s	sys 0.010s
I20260809 23:21:27.099495  2961 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.096900  2807 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.041172  1731 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:27.122339  2959 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.007s	sys 0.018s
I20260809 23:21:27.081382  2949 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.124655  2650 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.142465  3075 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260809 23:21:27.143586  3061 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260809 23:21:27.145622 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:27.145892 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:27.103874  2006 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.118816 24431 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:27.147595 25372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.007s	sys 0.025s
W20260809 23:21:27.138826  2959 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.007s	sys 0.018s
I20260809 23:21:27.039417  2593 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:27.151033 24422 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
W20260809 23:21:27.150775  3073 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
W20260809 23:21:27.137851  2962 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.881s	user 0.008s	sys 0.008s
W20260809 23:21:27.166399 25372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.008s	sys 0.025s
I20260809 23:21:27.149895 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.177090 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:27.180836 24534 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:27.182600 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.161711 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:27.186290 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1644) took 1115 ms. Trace:
I20260809 23:21:27.187777 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3449) took 1418 ms. Trace:
I20260809 23:21:27.188138 24546 rpcz_store.cc:276] 0809 23:21:25.769296 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.770639 (+  1343us) service_pool.cc:227] Handling call
0809 23:21:25.858096 (+ 87457us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.957991 (+ 99895us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.958036 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:25.958558 (+   522us) log.cc:842] Serialized 26 byte log entry
0809 23:21:26.291607 (+333049us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:26.291646 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.187340 (+895694us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.187403 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:27.187460 (+    57us) raft_consensus.cc:1671] finished
0809 23:21:27.187474 (+    14us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.187720 (+   246us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2431,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":1465,"spinlock_wait_cycles":228763520,"thread_start_us":307347,"threads_started":1,"wal-append.queue_time_us":581116}
W20260809 23:21:27.194242  3071 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260809 23:21:27.195343  2806 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.195886  2806 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:27.196404  2806 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.186728 25365 rpcz_store.cc:276] 0809 23:21:26.070868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.071043 (+   175us) service_pool.cc:227] Handling call
0809 23:21:27.186114 (+1115071us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54004a5bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.186232 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":257405184}
I20260809 23:21:27.203240  2768 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:27.203878  2768 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.205688  2949 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.209313  1677 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:27.209860  1677 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:27.213255  2588 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.213863  2588 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.215700 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1419) took 1349 ms. Trace:
I20260809 23:21:27.216086 24435 rpcz_store.cc:276] 0809 23:21:25.894362 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.894546 (+   184us) service_pool.cc:227] Handling call
0809 23:21:25.906575 (+ 12029us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:26.510907 (+604332us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.215447 (+704540us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.215638 (+   191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1257196800}
W20260809 23:21:27.171025  3082 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.001s	sys 0.001s
I20260809 23:21:27.222157 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3447) took 2529 ms. Trace:
I20260809 23:21:27.222628 24544 rpcz_store.cc:276] 0809 23:21:24.692685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.761923 (+ 69238us) service_pool.cc:227] Handling call
0809 23:21:24.817182 (+ 55259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.481551 (+664369us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.221893 (+1740342us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.222095 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1453518080}
W20260809 23:21:27.224870  2993 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.974s	user 0.015s	sys 0.000s
I20260809 23:21:27.226135  2854 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:27.171363  2667 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:27.235589  2734 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:27.236378  2768 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.237001  2588 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.237516  2588 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:27.237798  2588 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:27.238094  2588 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:27.239099 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3368) took 3732 ms. Trace:
I20260809 23:21:27.239549 24535 rpcz_store.cc:276] 0809 23:21:23.506349 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.533921 (+1027572us) service_pool.cc:227] Handling call
0809 23:21:24.586921 (+ 53000us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:25.372002 (+785081us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.238720 (+1866718us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.239033 (+   313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1690904832}
I20260809 23:21:27.243274 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1426) took 1212 ms. Trace:
I20260809 23:21:27.243763 24428 rpcz_store.cc:276] 0809 23:21:26.031265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.072793 (+ 41528us) service_pool.cc:227] Handling call
0809 23:21:26.073082 (+   289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:26.186090 (+113008us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:26.186137 (+    47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:26.186189 (+    52us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:26.186232 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.242970 (+1056738us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540017f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.243038 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.243200 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":259287424}
W20260809 23:21:27.249147 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3429) took 1489 ms (client timeout 1689 ms). Trace:
W20260809 23:21:27.249687 24320 rpcz_store.cc:269] 0809 23:21:25.759373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.036556 (+277183us) service_pool.cc:227] Handling call
0809 23:21:27.248888 (+1212332us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.249068 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1594995456}
I20260809 23:21:27.251534 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3072) took 1278 ms. Trace:
I20260809 23:21:27.251974 24425 rpcz_store.cc:276] 0809 23:21:25.973165 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.992732 (+ 19567us) service_pool.cc:227] Handling call
0809 23:21:27.251342 (+1258610us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.251467 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189743872}
W20260809 23:21:27.255672 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3056) took 2578 ms (client timeout 2646 ms). Trace:
W20260809 23:21:27.256114 24427 rpcz_store.cc:269] 0809 23:21:24.677487 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:24.677642 (+   155us) service_pool.cc:227] Handling call
0809 23:21:27.255461 (+2577819us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.255602 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3762738432}
I20260809 23:21:27.262832 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2908) took 1698 ms. Trace:
I20260809 23:21:27.265967 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:27.267647 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3428) took 1508 ms (client timeout 1589 ms). Trace:
W20260809 23:21:27.268144 24313 rpcz_store.cc:269] 0809 23:21:25.758775 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.897735 (+138960us) service_pool.cc:227] Handling call
0809 23:21:27.267431 (+1369696us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.267572 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230295296}
I20260809 23:21:27.271386 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.235601 24435 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:27.273830 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:27.263350 24312 rpcz_store.cc:276] 0809 23:21:25.563956 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.659300 (+ 95344us) service_pool.cc:227] Handling call
0809 23:21:25.659624 (+   324us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.706545 (+ 46921us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.706575 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:25.714325 (+  7750us) log.cc:842] Serialized 26 byte log entry
0809 23:21:26.073408 (+359083us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:26.086837 (+ 13429us) log.cc:842] Serialized 14 byte log entry
0809 23:21:27.261420 (+1174583us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.262353 (+   933us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b540019d2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:27.262413 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:27.262477 (+    64us) raft_consensus.cc:1671] finished
0809 23:21:27.262603 (+   126us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.262762 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1180655,"prepare.queue_time_us":68034,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":8089,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":10756,"spinlock_wait_cycles":91777920,"thread_start_us":266211,"threads_started":1,"wal-append.queue_time_us":419407,"wal-append.run_cpu_time_us":3516,"wal-append.run_wall_time_us":1024957}
I20260809 23:21:27.272113 24544 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:27.275081 24534 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:27.131538  2961 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.325729 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.326444 24544 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:27.180330  2650 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:27.272855  3057 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:27.324133  2671 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.069s	user 0.024s	sys 0.072s
I20260809 23:21:27.320538  2691 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:27.351329 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:27.329541  2768 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
W20260809 23:21:27.348543 24543 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.031s	sys 0.079s
I20260809 23:21:27.353911 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3433) took 1592 ms. Trace:
W20260809 23:21:27.358274 24543 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.032s	sys 0.079s
I20260809 23:21:27.358351 24317 rpcz_store.cc:276] 0809 23:21:25.761186 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.357735 (+596549us) service_pool.cc:227] Handling call
0809 23:21:26.357989 (+   254us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:26.358178 (+   189us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:26.407826 (+ 49648us) log.cc:842] Serialized 14 byte log entry
0809 23:21:27.353319 (+945493us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:27.353423 (+   104us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:27.353584 (+   161us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.353682 (+    98us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.353841 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":321466,"prepare.queue_time_us":59791,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":321734,"thread_start_us":623372,"threads_started":1,"wal-append.queue_time_us":763306}
W20260809 23:21:27.373601 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 405869us
I20260809 23:21:27.330535  2993 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.308193  2962 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.330091  2923 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.341413  2671 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.086s	user 0.024s	sys 0.072s
I20260809 23:21:27.396198 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:27.400213 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:27.455375 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.462427 24544 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
W20260809 23:21:27.469888 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 70872.7us
W20260809 23:21:27.479955 24378 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.956s (SENT)
I20260809 23:21:27.485150 25365 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:27.357247  2691 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:27.378189  2994 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.152s	user 0.007s	sys 0.009s
I20260809 23:21:27.451697  2052 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:27.484934  2748 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.491873  2846 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.492026  2788 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:27.492194  2623 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:27.492406  3004 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:27.503664 24429 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260809 23:21:27.356524  2416 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:27.509377 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.512338 24535 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260809 23:21:27.508284 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.528852 24320 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:27.516696 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:27.521423 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.522323 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.515074 24539 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:27.524638 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:27.526964 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:27.485435  3024 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:27.485127  3066 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.517599  2846 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.517293  2052 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:27.549835  3076 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.409s	user 0.000s	sys 0.001s
I20260809 23:21:27.526450  2623 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.549871 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:27.518347  2748 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.527029  2841 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.345072  2650 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.505442  2691 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:27.555737 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:27.526904  2806 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.610791  2961 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.613569 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1649) took 1328 ms. Trace:
I20260809 23:21:27.613961 25372 rpcz_store.cc:276] 0809 23:21:26.352385 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.352575 (+   190us) service_pool.cc:227] Handling call
0809 23:21:26.353047 (+   472us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:26.353520 (+   473us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:26.353560 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:26.393306 (+ 39746us) log.cc:842] Serialized 26 byte log entry
0809 23:21:27.180177 (+786871us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:27.180218 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.180297 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:27.613363 (+433066us) raft_consensus.cc:1671] finished
0809 23:21:27.613392 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.613510 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":127,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":8578,"thread_start_us":725640,"threads_started":1,"wal-append.queue_time_us":1154204}
I20260809 23:21:27.617625  1677 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.353571  3057 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.527318  2881 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:27.473883  2993 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.676362 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:27.545509  2827 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.527180  2119 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:27.526844  2631 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:27.599133  3084 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260809 23:21:27.683851  2119 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.684765 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3066) took 2022 ms. Trace:
I20260809 23:21:27.685128 24423 rpcz_store.cc:276] 0809 23:21:25.662028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.662221 (+   193us) service_pool.cc:227] Handling call
0809 23:21:27.684577 (+2022356us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.684704 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2452953344}
I20260809 23:21:27.599638  2623 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.599918  3085 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260809 23:21:27.600798  2052 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.601743  2748 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.602010  2650 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.602371  2846 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.572669  2841 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.526597  2788 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:27.599175  3086 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260809 23:21:27.605263 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1699) took 1829 ms. Trace:
W20260809 23:21:27.606009  3038 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260809 23:21:27.553087  2416 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.608438  2154 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:27.613978  2006 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.518639  2667 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.624046  2962 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.627614  2779 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.630575  2961 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:27.683610  2631 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.741267  2961 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.742215  2841 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.743050  2631 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.743777  2961 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.561729  3024 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.745631  3024 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.565606  3066 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.746002  2416 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.544821  3004 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.748579  3004 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:27.610844  2819 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.796s	user 0.006s	sys 0.006s
I20260809 23:21:27.674345  2801 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:27.749404  2801 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:27.750262  2801 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.751271  2801 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:27.674467  2881 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.739215  2667 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.756814  2881 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.757345  2667 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.741276  2119 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.733163  2741 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.759588  3066 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:27.723621  2884 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.049s	user 0.013s	sys 0.000s
I20260809 23:21:27.637472  1539 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.718480 25375 rpcz_store.cc:276] 0809 23:21:25.776207 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.776370 (+   163us) service_pool.cc:227] Handling call
0809 23:21:25.776831 (+   461us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:25.777356 (+   525us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.777386 (+    30us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:21:25.828083 (+ 50697us) log.cc:842] Serialized 78 byte log entry
0809 23:21:26.754314 (+926231us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:26.754382 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:26.754474 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:27.604988 (+850514us) raft_consensus.cc:1671] finished
0809 23:21:27.605017 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:27.605197 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":754,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8725,"thread_start_us":817116,"threads_started":1,"wal-append.queue_time_us":1183100}
I20260809 23:21:27.554760  2639 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.763037  2974 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.021s	sys 0.041s
I20260809 23:21:27.742453  2788 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.768141  2741 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.821162  2777 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.822723  2588 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.584s	user 0.013s	sys 0.006s
I20260809 23:21:27.823421  2588 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.826128  2588 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.827867 25369 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:27.858983 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1669) took 1050 ms. Trace:
I20260809 23:21:27.859539 25369 rpcz_store.cc:276] 0809 23:21:26.808789 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.809279 (+   490us) service_pool.cc:227] Handling call
0809 23:21:27.858737 (+1049458us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:27.858889 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2339086208}
W20260809 23:21:27.803561  2974 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.022s	sys 0.042s
I20260809 23:21:27.828038  2947 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:27.855230  2995 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:27.884107  2993 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.894021  2720 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:27.894562  2720 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:27.895653  2720 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.903381  2720 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:27.911161  2995 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.864315  2741 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.827298  2779 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.832330  2625 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:27.695199  2827 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.801522  2884 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:27.947981  2884 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:27.949079  3057 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:27.950971  2827 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:27.952080  2764 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.940s	user 0.012s	sys 0.000s
I20260809 23:21:27.935078  2995 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:27.968640  2764 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:27.973744  2764 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:28.017683 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 490594us
I20260809 23:21:28.123634 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.123986 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:28.125335  2662 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.003s	sys 0.006s
W20260809 23:21:28.125852  2662 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.004s	sys 0.006s
I20260809 23:21:28.128769 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3031) took 5909 ms. Trace:
I20260809 23:21:28.129181 24429 rpcz_store.cc:276] 0809 23:21:22.198365 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:22.198491 (+   126us) service_pool.cc:227] Handling call
0809 23:21:24.218033 (+2019542us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.519273 (+3301240us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.041091 (+521818us) spinlock_profiling.cc:243] Waited 3.28 s on lock 0x7b540023e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:28.092369 (+ 51278us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b540023e450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:28.107194 (+ 14825us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12156270592}
I20260809 23:21:28.132014 24541 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:28.135242  2650 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.135974  2650 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.141441 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:28.143102  2726 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.007s	sys 0.002s
W20260809 23:21:28.143618  2726 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.007s	sys 0.002s
I20260809 23:21:28.146272 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.147167 24539 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:28.026367 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1712) took 2685 ms. Trace:
I20260809 23:21:28.147819 24311 rpcz_store.cc:276] 0809 23:21:25.340688 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.484158 (+143470us) service_pool.cc:227] Handling call
0809 23:21:25.484565 (+   407us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:25.484964 (+   399us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:25.485002 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:25.485510 (+   508us) log.cc:842] Serialized 26 byte log entry
0809 23:21:26.028120 (+542610us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:26.760921 (+732801us) log.cc:842] Serialized 14 byte log entry
0809 23:21:27.340307 (+579386us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.340405 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:27.501154 (+160749us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b1800318940. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:28.025360 (+524206us) raft_consensus.cc:1671] finished
0809 23:21:28.025675 (+   315us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.026082 (+   407us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":692074,"prepare.queue_time_us":112928,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":601285,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":32764,"spinlock_wait_cycles":28106496,"thread_start_us":1221902,"threads_started":2,"wal-append.queue_time_us":577710}
I20260809 23:21:28.025867 24317 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:28.028258 24320 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:28.150683 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.151844  3112 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:28.152261 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:27.566644 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:28.152787 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:28.153138 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:28.153663 24312 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:28.154366 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:28.161547 24320 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:28.042799 24546 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.008s	sys 0.004s
W20260809 23:21:28.162366 24546 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.009s	sys 0.004s
I20260809 23:21:28.142762 24535 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:28.163654 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3471) took 1016 ms. Trace:
W20260809 23:21:28.043026 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.022s	sys 0.025s
I20260809 23:21:28.164798 24535 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:28.165077 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.022s	sys 0.026s
W20260809 23:21:28.165628 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 972382us
W20260809 23:21:28.044039 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.007s	sys 0.008s
W20260809 23:21:28.167339 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.007s	sys 0.008s
W20260809 23:21:27.734107 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.022s	sys 0.039s
W20260809 23:21:28.168390 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.022s	sys 0.039s
I20260809 23:21:28.168684 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3469) took 1454 ms. Trace:
W20260809 23:21:28.168929 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.15693e+06us
W20260809 23:21:28.047278 24424 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.004s	sys 0.008s
W20260809 23:21:28.171254 24424 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.004s	sys 0.009s
W20260809 23:21:28.050568 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.003s	sys 0.002s
W20260809 23:21:28.173169 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.004s	sys 0.003s
I20260809 23:21:28.051245 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1639) took 1006 ms. Trace:
I20260809 23:21:28.170817 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:28.019315 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 544138us
I20260809 23:21:28.174386 24540 rpcz_store.cc:276] 0809 23:21:27.045080 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:27.050219 (+  5139us) service_pool.cc:227] Handling call
0809 23:21:27.050624 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:27.168515 (+117891us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.168561 (+    46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:27.168629 (+    68us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:27.168679 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.050577 (+881898us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540029cf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:28.050658 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.050979 (+   321us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":270544128}
I20260809 23:21:28.176179 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:28.059238 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.003s	sys 0.005s
W20260809 23:21:28.177577 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.003s	sys 0.006s
W20260809 23:21:28.178081 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 756791us
I20260809 23:21:28.178174 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.179127 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.180177 24435 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:28.181335 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:28.170790 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:28.060027 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.006s	sys 0.003s
W20260809 23:21:28.182614 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.006s	sys 0.003s
I20260809 23:21:28.064674  2795 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.066860  2816 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:28.068318  1677 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:28.096850 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.007s	sys 0.015s
I20260809 23:21:28.182781 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:28.097343 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.000s	sys 0.008s
W20260809 23:21:28.098656 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.009s	sys 0.008s
W20260809 23:21:28.110311 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.005s	sys 0.005s
I20260809 23:21:28.113263  1731 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.114050  2671 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.147102 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:28.149657 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.152379  3112 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.025904 24316 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:28.029495 24430 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:28.154829 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:27.927070  2625 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:28.155746  3093 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.157989 24311 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:28.160602 24317 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:28.030624 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3435) took 2268 ms. Trace:
I20260809 23:21:28.179255 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
W20260809 23:21:28.056236 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.004s	sys 0.008s
W20260809 23:21:28.063931  2469 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.009s	sys 0.014s
I20260809 23:21:28.028445 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:28.195067 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.007s	sys 0.016s
W20260809 23:21:28.186165 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.002s	sys 0.014s
W20260809 23:21:28.197018 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.000s	sys 0.009s
W20260809 23:21:28.169839  2593 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.990s	user 0.013s	sys 0.002s
W20260809 23:21:28.199903 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.010s	sys 0.008s
W20260809 23:21:28.200366 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.005s	sys 0.005s
I20260809 23:21:28.183435  2795 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.164126 24546 rpcz_store.cc:276] 0809 23:21:27.147608 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:27.189712 (+ 42104us) service_pool.cc:227] Handling call
0809 23:21:27.190021 (+   309us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:28.163111 (+973090us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:28.163168 (+    57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:28.163210 (+    42us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:28.163252 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.163362 (+   110us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.163570 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":149544,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":17964,"thread_start_us":972392,"threads_started":1}
I20260809 23:21:28.211695  2671 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.214041 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3077) took 1794 ms. Trace:
W20260809 23:21:28.214697  2469 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.009s	sys 0.014s
I20260809 23:21:28.185592  2816 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.215261 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.169184 24543 rpcz_store.cc:276] 0809 23:21:26.713929 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.714079 (+   150us) service_pool.cc:227] Handling call
0809 23:21:26.714461 (+   382us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.358742 (+644281us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.358774 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:27.368778 (+ 10004us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.168074 (+799296us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:28.168173 (+    99us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.168294 (+   121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.168355 (+    61us) raft_consensus.cc:1671] finished
0809 23:21:28.168370 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.168598 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13337,"raft.queue_time_us":126849,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":11125,"thread_start_us":1429083,"threads_started":2,"wal-append.queue_time_us":593040}
I20260809 23:21:28.222200 24539 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:28.220458 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.002s	sys 0.015s
W20260809 23:21:28.218283 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.005s	sys 0.009s
I20260809 23:21:28.194058  3112 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:28.220000 24541 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1633) took 1339 ms (client timeout 1583 ms). Trace:
I20260809 23:21:28.227834 24435 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:28.229131 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:28.224397  2816 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:28.233475 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:28.211725 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1717) took 2509 ms (client timeout 1744 ms). Trace:
I20260809 23:21:28.235008  2923 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:28.236526 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:28.237728 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 49307.5us
W20260809 23:21:28.235230 24316 rpcz_store.cc:269] 0809 23:21:25.702042 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:25.715509 (+ 13467us) service_pool.cc:227] Handling call
0809 23:21:28.211344 (+2495835us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:28.211599 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4964933376}
I20260809 23:21:28.233896  1731 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.236829  2854 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:28.226075 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3467) took 1513 ms. Trace:
W20260809 23:21:28.224586 24541 rpcz_store.cc:269] 0809 23:21:26.880194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.880338 (+   144us) service_pool.cc:227] Handling call
0809 23:21:28.206497 (+1326159us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:28.219922 (+ 13425us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2839228160}
I20260809 23:21:28.250669 24544 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:28.219424 24430 rpcz_store.cc:276] 0809 23:21:26.419364 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.422375 (+  3011us) service_pool.cc:227] Handling call
0809 23:21:28.213650 (+1791275us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:28.213913 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3692330752}
I20260809 23:21:28.238896 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:28.240437 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91010.9us
I20260809 23:21:28.200829 24323 rpcz_store.cc:276] 0809 23:21:25.761806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.402765 (+640959us) service_pool.cc:227] Handling call
0809 23:21:26.403174 (+   409us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:26.403443 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:26.403478 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:26.403972 (+   494us) log.cc:842] Serialized 26 byte log entry
0809 23:21:27.123586 (+719614us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:27.123645 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:27.123740 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.029760 (+906020us) raft_consensus.cc:1671] finished
0809 23:21:28.029835 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.030330 (+   495us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":608566,"threads_started":1,"wal-append.queue_time_us":908870}
I20260809 23:21:28.244591 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:28.248212  2416 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.270507  2416 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.246860 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.230736 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:28.243120 24542 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:28.244897  1731 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.279601 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:28.247787  2923 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:28.240991  2827 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:28.239135  2795 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.289237  2995 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.289445  2881 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.289669  2992 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:28.290138 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.010s	sys 0.008s
W20260809 23:21:28.290448 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s	user 0.007s	sys 0.015s
I20260809 23:21:28.295310  2741 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.274063  2416 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:28.300275  2691 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.008s
I20260809 23:21:28.302063 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:28.303234 24435 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23.
W20260809 23:21:28.303494 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.893s	user 0.012s	sys 0.010s
I20260809 23:21:28.344441  2691 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:28.345196 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.018s	user 0.012s	sys 0.010s
I20260809 23:21:28.311297  2764 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.346520  2764 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.347455 24429 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:28.348243  2691 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:28.312273 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73557.2us
I20260809 23:21:28.312542  1731 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:28.352124 24309 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:28.316545  3104 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:28.318210  2809 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:28.244968  3127 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:28.250203  2947 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:28.338667  2729 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:28.339280  2881 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.340523  2508 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.340557  2030 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:28.342017  2995 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:28.343482 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 172998us
W20260809 23:21:28.333783 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87268.9us
I20260809 23:21:28.254726 24545 rpcz_store.cc:276] 0809 23:21:26.712216 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.712442 (+   226us) service_pool.cc:227] Handling call
0809 23:21:26.713007 (+   565us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.485610 (+772603us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.485636 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:27.486049 (+   413us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.173886 (+687837us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:28.173988 (+   102us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.200123 (+ 26135us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002710d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:28.200179 (+    56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.200232 (+    53us) raft_consensus.cc:1671] finished
0809 23:21:28.200242 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.225994 (+ 25752us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":124818,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":14945,"spinlock_wait_cycles":721916160,"thread_start_us":1145490,"threads_started":2,"wal-append.queue_time_us":473272}
W20260809 23:21:28.356009 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s	user 0.008s	sys 0.016s
W20260809 23:21:28.356319 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.011s	sys 0.009s
I20260809 23:21:28.359530  2691 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:28.359683  2736 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:28.360520 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.332418  2827 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.330835  2923 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:28.358287 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.355228  2795 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:28.360767 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3397) took 1095 ms. Trace:
I20260809 23:21:28.351706  1731 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.348115  2625 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.241279  3093 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.357942  3127 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.360733  2729 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:28.378702 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.369536  2827 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:28.386624  2881 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.387149  2030 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.397378  2736 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:28.397807  2992 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:28.388245 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 212307us
I20260809 23:21:28.391669 24541 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:28.392454 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.393311 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.393527  2741 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.394069 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2913) took 2429 ms. Trace:
I20260809 23:21:28.394503 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.383637  3127 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:28.372221  2795 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.375732  1731 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:28.408460 24546 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:28.383845  2729 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:28.409909 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 124252us
I20260809 23:21:28.402608 24539 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:28.370786 24534 rpcz_store.cc:276] 0809 23:21:27.265419 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:27.517446 (+252027us) service_pool.cc:227] Handling call
0809 23:21:27.517795 (+   349us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.528324 (+ 10529us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.528360 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:27.528779 (+   419us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.288046 (+759267us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:28.331205 (+ 43159us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.331306 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.331367 (+    61us) raft_consensus.cc:1671] finished
0809 23:21:28.331382 (+    15us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.360714 (+ 29332us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10840,"raft.queue_time_us":10878,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":5767,"thread_start_us":673690,"threads_started":1,"wal-append.queue_time_us":560048}
I20260809 23:21:28.413977 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.417006 24545 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260809 23:21:28.417807 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 173511us
I20260809 23:21:28.418991 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:28.420212  2650 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:28.422307 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3464) took 2224 ms (client timeout 1738 ms). Trace:
I20260809 23:21:28.384189  2764 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.411743 24540 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:28.400230  2809 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260809 23:21:28.422734 24542 rpcz_store.cc:269] 0809 23:21:26.197596 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.197783 (+   187us) service_pool.cc:227] Handling call
0809 23:21:28.422059 (+2224276us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b54000b6654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:28.422194 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4578549120}
I20260809 23:21:28.427573 24496 leader_election.cc:398] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23.
I20260809 23:21:28.407470  2795 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:28.430430 24496 leader_election.cc:401] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260809 23:21:28.428815 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.401340 24314 rpcz_store.cc:276] 0809 23:21:25.964568 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.456540 (+491972us) service_pool.cc:227] Handling call
0809 23:21:26.547696 (+ 91156us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:27.326070 (+778374us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.326100 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:27.326593 (+   493us) log.cc:842] Serialized 52 byte log entry
0809 23:21:28.345877 (+1019284us) raft_consensus.cc:1631] Marking committed up to 7
0809 23:21:28.345940 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.346089 (+   149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.393763 (+ 47674us) raft_consensus.cc:1671] finished
0809 23:21:28.393787 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.393988 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":889494,"raft.queue_time_us":776529,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8398,"thread_start_us":1018789,"threads_started":1,"wal-append.queue_time_us":1007483}
I20260809 23:21:28.439155 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:28.439728  2720 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.010s
I20260809 23:21:28.440217  2881 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:28.440361  3093 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:28.440697  2881 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.441115  2881 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:28.444859  2995 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:28.447347 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.447561  2030 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.448060 24544 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:28.448614  3127 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.450212 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:28.368342  2854 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.429675 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:28.455205 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:28.457938 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.29617e+06us
I20260809 23:21:28.454075 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.451236  2779 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:28.531953 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:28.533504 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.450778  3127 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:28.550743 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91865.8us
W20260809 23:21:28.551805 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 113576us
I20260809 23:21:28.584766 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.585559 24541 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:28.595260 24535 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:28.399597  2508 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.555480  2779 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:28.609063 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3085) took 1132 ms. Trace:
I20260809 23:21:28.556387  2625 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:28.608774  3146 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260809 23:21:28.555763  2741 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:28.622709 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 222598us
I20260809 23:21:28.623611  2726 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:28.451203  3057 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.182s
I20260809 23:21:28.627128  2030 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:28.626737  2593 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23.
I20260809 23:21:28.427963 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c831643e28e34208b6edf775607155e4"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:28.620770  2508 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:28.640594 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 227453us
W20260809 23:21:28.624249 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 201369us
I20260809 23:21:28.644407  2736 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260809 23:21:28.631923 24427 rpcz_store.cc:276] 0809 23:21:27.460640 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:27.460832 (+   192us) service_pool.cc:227] Handling call
0809 23:21:27.461323 (+   491us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.461599 (+   276us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.461635 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:27.462159 (+   524us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.412785 (+950626us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:28.430954 (+ 18169us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.456937 (+ 25983us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.533273 (+ 76336us) raft_consensus.cc:1671] finished
0809 23:21:28.533292 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.593090 (+ 59798us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6912,"thread_start_us":903202,"threads_started":1,"wal-append.queue_time_us":664012}
W20260809 23:21:28.638695  3142 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260809 23:21:28.656157  2736 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:28.656672  2736 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:28.657733 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:28.658291 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3078) took 2238 ms. Trace:
I20260809 23:21:28.659209  2741 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:28.436236  3104 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.659727 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:28.660036 24309 raft_consensus.cc:595] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:28.660554 24309 raft_consensus.cc:603] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:28.661029 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:28.661579  3104 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.662318  2820 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:28.662791  2820 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.663169  2820 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:28.667213  2469 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:28.667726  2469 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:28.641922  2726 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:28.668169  2469 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:28.668251 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 136559us
I20260809 23:21:28.669494 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:28.671128  2995 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:28.657954 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1429) took 1871 ms. Trace:
I20260809 23:21:28.671775  2995 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:28.668537  2726 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:28.671842 24422 rpcz_store.cc:276] 0809 23:21:26.786648 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:27.192007 (+405359us) service_pool.cc:227] Handling call
0809 23:21:27.192329 (+   322us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.192526 (+   197us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.192548 (+    22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:28.036786 (+844238us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:28.037754 (+   968us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.186281 (+148527us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:28.186312 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.204026 (+ 17714us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.657437 (+453411us) raft_consensus.cc:1671] finished
0809 23:21:28.657464 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.657779 (+   315us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":232598400,"thread_start_us":148013,"threads_started":1,"wal-append.queue_time_us":321767}
I20260809 23:21:28.673632  2795 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.668169  2816 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.675712 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:28.658705 24424 rpcz_store.cc:276] 0809 23:21:26.419691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.419812 (+   121us) service_pool.cc:227] Handling call
0809 23:21:26.420106 (+   294us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:26.444764 (+ 24658us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:27.248086 (+803322us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b1800170c20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:21:28.172073 (+923987us) log.cc:842] Serialized 14 byte log entry
0809 23:21:28.382164 (+210091us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:28.441092 (+ 58928us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.441261 (+   169us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:28.441294 (+    33us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.441407 (+   113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.658011 (+216604us) raft_consensus.cc:1671] finished
0809 23:21:28.658104 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:28.658219 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3170,"prepare.queue_time_us":1501156,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":41431,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":63777,"spinlock_wait_cycles":37437696,"thread_start_us":1113539,"threads_started":2,"wal-append.queue_time_us":258919}
W20260809 23:21:28.677193 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 200262us
I20260809 23:21:28.680217 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:28.681362 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:28.682188  2720 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:28.682587  2720 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.682976  2720 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:28.685709 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.686524 24429 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:28.689152 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:28.676075  2854 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:28.696935 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.657956  2736 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.705148 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:28.710966  2508 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260809 23:21:28.699586  3146 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260809 23:21:28.712167 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.659725  2741 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.711947 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.717132 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.666852  2881 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:28.722506  3145 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260809 23:21:28.723011  2795 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.723651  2881 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.723920  2795 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:28.694739  2671 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.718820 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:28.721091 24427 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:28.728783 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.724622  2795 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:28.740437  2030 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.711468  2508 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.723131 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.747139 24534 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:28.750284 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 79452.9us
I20260809 23:21:28.753695  1731 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.747057 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.757259  3109 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:28.757795  3109 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:28.757611  3112 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.758281  3109 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:28.735787  2995 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:28.760236  2508 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:28.719718  2854 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:28.751318  2030 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.763652  2854 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.765162  2820 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.765967  2854 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:28.767306  2820 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.771780 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:28.772481 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 16.
I20260809 23:21:28.773692  3104 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:28.773950 24534 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:28.774106  3104 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:28.774505  3104 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:28.779897  2795 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.726614  2741 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:28.743475 24535 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:28.800077  3112 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.801355  2923 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.803560  2764 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:28.804522  2923 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.805663  2736 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:28.823092 24435 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:28.834002  2992 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.804430  2764 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.838616 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:28.840685 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:28.897464 24546 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:28.862711  2378 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:28.857200  3004 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Stepping down as leader of term 15
I20260809 23:21:28.840406  2816 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:28.871086 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:28.861279  2726 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:28.873163  1731 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:28.843840  2729 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:28.853792  2720 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:28.853700  3120 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.626s	user 0.002s	sys 0.015s
W20260809 23:21:28.900074 24498 connection.cc:232] RPC call timeout handler was delayed by -2.1754807389806956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
W20260809 23:21:28.919486 24498 connection.cc:232] RPC call timeout handler was delayed by -2.0250512999482453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260809 23:21:28.920056 24498 connection.cc:232] RPC call timeout handler was delayed by -1.9414807390421629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
I20260809 23:21:28.900594  2378 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:28.918471 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120342us
I20260809 23:21:28.904335  3004 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:28.940263  3108 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:28.940793  3108 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:28.941182  3108 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:28.939034 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:29.007673 24498 connection.cc:232] RPC call timeout handler was delayed by -1.5256691628601402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
W20260809 23:21:29.008165 24498 connection.cc:232] RPC call timeout handler was delayed by -1.5129650509916246s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1967s for a response
W20260809 23:21:29.014050 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 563956us
W20260809 23:21:29.007031 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70979.6us
I20260809 23:21:29.028064  2378 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.030018  2378 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:29.016758  2593 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260809 23:21:29.040869  2884 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.019s	sys 0.025s
W20260809 23:21:29.051260  2884 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.020s	sys 0.026s
I20260809 23:21:29.052922 24546 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:29.037971  2593 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:29.067464 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.068287 24431 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:29.062804  2593 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.073266  3004 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.082304 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 241) took 36534 ms (client timeout 15000 ms). Trace:
I20260809 23:21:29.080369  2726 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.094448 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.095642  3004 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 15 FOLLOWER]: Advancing to term 16
I20260809 23:21:29.089897  2729 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:29.103490  3093 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:29.104794  3093 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:29.105161  3093 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.093185 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2910) took 3294 ms. Trace:
I20260809 23:21:29.106204  3093 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.091662  2827 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.000s
I20260809 23:21:29.090433  2720 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.109097 24321 rpcz_store.cc:276] 0809 23:21:25.798336 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:26.422432 (+624096us) service_pool.cc:227] Handling call
0809 23:21:26.422842 (+   410us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:27.324139 (+901297us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:27.324171 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:27.330119 (+  5948us) log.cc:842] Serialized 26 byte log entry
0809 23:21:28.391104 (+1060985us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:28.391161 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:28.391278 (+   117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:28.852817 (+461539us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:29.070889 (+218072us) raft_consensus.cc:1671] finished
0809 23:21:29.070918 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:29.093104 (+ 22186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1108812,"raft.queue_time_us":903853,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":2394,"spinlock_wait_cycles":137936768,"thread_start_us":996890,"threads_started":1,"wal-append.queue_time_us":888241}
I20260809 23:21:29.121498  3104 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.134414  3104 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.111371  2764 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.149557 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.114447  2469 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.120060  2508 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.146481  2720 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.165089  2993 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.077s
I20260809 23:21:29.123442  2827 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.145394  3104 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:29.171442 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 493372us
I20260809 23:21:29.172120 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:29.157583  3087 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:29.161367  2764 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.161186  3093 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:29.176698 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.007s	sys 0.023s
W20260809 23:21:29.178323 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.007s	sys 0.023s
I20260809 23:21:29.153333  2741 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.155892  2729 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260809 23:21:29.191615 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:29.198959 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 447681us
W20260809 23:21:29.189306 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.053s	sys 0.047s
W20260809 23:21:29.206422 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.053s	sys 0.047s
I20260809 23:21:29.179380  2720 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.212615 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.026s	sys 0.029s
W20260809 23:21:29.227759 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 79072.9us
I20260809 23:21:29.197549  2995 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:29.229585 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.026s	sys 0.033s
W20260809 23:21:29.231045 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 549373us
I20260809 23:21:29.187857  2764 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.236447 25325 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:29.184645  3104 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.243736 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 568135us
I20260809 23:21:29.233461  2736 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:29.243268 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.247385  2508 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:29.232923  2809 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.010s	sys 0.017s
W20260809 23:21:29.083446 24164 rpcz_store.cc:269] 0809 23:20:52.501812 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:52.502013 (+   201us) service_pool.cc:227] Handling call
0809 23:21:29.032338 (+36530325us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:21:18.928007 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:21:19.240605 (+312598us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:21:19.253264 (+ 12659us) write_op.cc:435] Acquired schema lock
0809 23:21:19.254504 (+  1240us) tablet.cc:660] Decoding operations
0809 23:21:19.856893 (+602389us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:21:19.912686 (+ 55793us) write_op.cc:641] Partition lock acquired for write op
0809 23:21:19.933661 (+ 20975us) tablet.cc:683] Acquiring locks for 18 operations
0809 23:21:20.205439 (+271778us) tablet.cc:699] Row locks acquired
0809 23:21:20.205481 (+    42us) write_op.cc:260] PREPARE: finished
0809 23:21:20.205644 (+   163us) write_op.cc:270] Start()
0809 23:21:20.205789 (+   145us) write_op.cc:276] Timestamp: P: 1786317680205587 usec, L: 0
0809 23:21:20.205826 (+    37us) op_driver.cc:331] REPLICATION: starting
0809 23:21:20.206546 (+   720us) log.cc:842] Serialized 7052 byte log entry
0809 23:21:20.888565 (+682019us) op_driver.cc:444] REPLICATION: finished
0809 23:21:21.005520 (+116955us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac2091a5fe 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac27270410
0809 23:21:21.460652 (+455132us) write_op.cc:301] APPLY: starting
0809 23:21:21.472818 (+ 12166us) tablet.cc:1365] starting BulkCheckPresence
0809 23:21:22.114982 (+642164us) tablet.cc:1368] finished BulkCheckPresence
0809 23:21:22.140121 (+ 25139us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:21:23.486279 (+1346158us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:21:23.486645 (+   366us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=0,mrs_lookups=0
0809 23:21:23.486703 (+    58us) write_op.cc:312] APPLY: finished
0809 23:21:23.624851 (+138148us) log.cc:842] Serialized 164 byte log entry
0809 23:21:23.752094 (+127243us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:23.809199 (+ 57105us) write_op.cc:454] Released schema lock
0809 23:21:23.821651 (+ 12452us) write_op.cc:341] FINISH: Updating metrics
0809 23:21:23.850903 (+ 29252us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:23.850959 (+    56us) write_op.cc:454] Released schema lock
0809 23:21:24.429330 (+578371us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
Metrics: {"mutex_wait_us":1156724,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":470252,"apply.run_cpu_time_us":64133,"apply.run_wall_time_us":2433368,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79295,"num_ops":18,"prepare.queue_time_us":186430,"prepare.run_cpu_time_us":47530,"prepare.run_wall_time_us":1470463,"raft.queue_time_us":79637,"raft.run_cpu_time_us":35660,"raft.run_wall_time_us":297904,"replication_time_us":682612,"spinlock_wait_cycles":22214656,"thread_start_us":706846,"threads_started":5,"wal-append.queue_time_us":481268,"wal-append.run_cpu_time_us":17919,"wal-append.run_wall_time_us":3013835}]]}
W20260809 23:21:29.251587 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60111us
I20260809 23:21:29.253556 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.254271  3154 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.254727 24430 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260809 23:21:29.255704 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.254333 24435 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:29.256918 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.008s	sys 0.049s
W20260809 23:21:29.261018 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.008s	sys 0.049s
W20260809 23:21:29.261304 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 563552us
W20260809 23:21:29.251047  2809 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.011s	sys 0.017s
I20260809 23:21:29.268437 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.269043 24546 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:29.269459 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.270010 24534 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.255311  3154 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.275676 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.280238 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.283289 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:29.276171 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.285804 24535 raft_consensus.cc:2354] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:29.253916  3156 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:29.281446 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.287892 24542 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:29.265748  2469 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.290123 24540 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:29.291112 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:29.292238 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57710.2us
I20260809 23:21:29.296059 24539 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:29.274643  3087 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:29.273849  2508 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260809 23:21:29.263221  3109 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.301904 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.309986 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.315515 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:29.310511 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.310987 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:42134
I20260809 23:21:29.311443  2764 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.311798 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.313372  2720 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:29.313819  3155 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.075s	user 0.001s	sys 0.001s
I20260809 23:21:29.300634  2884 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:29.270479  2593 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.320749 24539 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:29.293424  3171 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.328895 24546 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:29.275619  2827 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:29.331923 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 82255.7us
I20260809 23:21:29.297226  3169 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:29.345767 24168 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 352) took 128582 ms (client timeout 15000 ms). Trace:
W20260809 23:21:29.346267 24168 rpcz_store.cc:269] 0809 23:19:20.757761 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:20.758044 (+   283us) service_pool.cc:227] Handling call
0809 23:21:29.337398 (+128579354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":639205,"reported_tablets":45}
W20260809 23:21:29.302313  2813 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.001s	sys 0.027s
I20260809 23:21:29.300325  2741 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:29.341325 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.341953 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.343334 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.329195  2854 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:29.349431 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93906.7us
I20260809 23:21:29.330559  2720 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.352487 24270 leader_election.cc:398] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:29.354568 24270 leader_election.cc:401] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260809 23:21:29.355418 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.357004 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.357668 24429 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 19
I20260809 23:21:29.357601 24428 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:29.359618 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.356802 24423 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:29.343798  2820 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.365034 24534 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:29.367272 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 106709us
I20260809 23:21:29.296787  2995 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.374862 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.376103  2469 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:29.345062  3087 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.369506 24425 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:29.377115 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:29.381336 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79657.9us
I20260809 23:21:29.372838  2508 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:29.384402 24423 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:29.385102 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:29.388037 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.376840  2741 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.348893  3171 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.391058 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:29.384088  3177 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.054s	user 0.001s	sys 0.001s
W20260809 23:21:29.392478 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71666.8us
I20260809 23:21:29.359230  2720 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.395252 24321 raft_consensus.cc:595] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:29.365916  2820 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.397004 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:29.386523  3104 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.392567  2469 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:29.402475  2884 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:29.408289  2884 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:29.409670 24431 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:29.409332  2884 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.409962 25369 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 15. Current term is 16. Ops: []
I20260809 23:21:29.411855 25372 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260809 23:21:29.412072  3087 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.413172 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.413877 25365 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:29.415961 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:29.417469 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.419412 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.420109 24430 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.422349  3154 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:29.424253  3171 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.425060  3109 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:29.425658  3109 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:29.426142  3109 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.427170  3109 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.428623 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.429646  2729 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:29.430716 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.431442 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.431674  2720 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.432402 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.459542 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.455960 24430 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:29.461014 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45211.8us
I20260809 23:21:29.460315 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.461817 24435 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:29.458369 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.461326  3154 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.005s
I20260809 23:21:29.467433  2827 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:29.468073  2827 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.399657  2813 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:29.469372  2813 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:29.470165  2827 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:29.470309  2813 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.471083  3156 raft_consensus.cc:2741] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:29.471392  2508 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:29.465816  2720 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:29.458580 24542 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:29.472617 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:29.457890 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 57400.4us
I20260809 23:21:29.476742 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.480388 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.456039  3161 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:29.481775 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:29.486913 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:29.489924 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.491150 25365 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:29.473948 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.492661 25372 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260809 23:21:29.455963  2469 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:29.489880 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.493945 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.475848  2720 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:29.500933 24429 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:29.503466 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.508580 24314 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:29.505546 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:29.513548 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55264.2us
I20260809 23:21:29.507046 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.515698  3109 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:29.516386  3109 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.502641  2816 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:29.517634  3109 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.518824  2508 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader election won for term 16
I20260809 23:21:29.519466  2508 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.520380  2508 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.523406 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.524036 24313 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:29.524713  2884 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:29.525259  2884 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:29.496543  3154 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:29.527348  3154 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:29.527546 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47468us
I20260809 23:21:29.528313  3154 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:29.530426 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.912s (SENT) [suppressed 17 similar messages]
I20260809 23:21:29.490023  2593 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:29.509279  2720 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:29.533558 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.533800 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.534390 24316 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:29.535483  3172 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260809 23:21:29.536518 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.536772 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.509553  3156 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:29.537405  3156 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:29.537712  3156 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:29.538021 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:29.538734 24313 raft_consensus.cc:2354] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19.
I20260809 23:21:29.540417 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.540803 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.541185 24316 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:29.541608 24314 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:29.542457 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.526158  2884 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.543507 24323 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:29.547325 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:29.547686 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.548198 24309 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:29.548621 25366 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:29.549160 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.495457  2378 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:29.553853 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.495747  3161 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:29.571961 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:29.573122 24313 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:29.573362 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.583850 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.584929 24539 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:29.601909 24429 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40954
W20260809 23:21:29.594611 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87676.8us
I20260809 23:21:29.615757 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.616230 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.616632 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.602886 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.571820  2820 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:29.560448  2816 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:29.621254 25366 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:29.602553 24429 raft_consensus.cc:595] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:29.622128 24429 raft_consensus.cc:603] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:29.622716 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:29.623227 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 164177us
I20260809 23:21:29.623683 24427 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.617187  3093 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:29.602406  2378 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:29.633213 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.633421 24540 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:29.635291 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:29.636034 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130051us
I20260809 23:21:29.637135 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.643450 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.630293  2995 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:29.648283 24535 raft_consensus.cc:2354] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20.
I20260809 23:21:29.615257  2720 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.652160 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.644531 25372 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:29.655128 24539 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:29.655193 24380 leader_election.cc:398] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20.
I20260809 23:21:29.656883 24380 leader_election.cc:401] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:29.656769 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:29.647954  2741 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:29.663039 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:21:29.668655 24309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:29.669267 24309 raft_consensus.cc:595] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.669588 24309 raft_consensus.cc:603] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:29.655745 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.648799  3154 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260809 23:21:29.640558  3093 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.679338 24321 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:29.682420 25366 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:29.686201 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 155720us
I20260809 23:21:29.690285 24312 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:29.649979  2995 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:29.696148 24312 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:29.656415  2993 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:29.621724  3161 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.691670  3108 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.750s	user 0.013s	sys 0.002s
I20260809 23:21:29.721043 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:29.710621 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.703426  2995 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.732918 24427 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:29.707883  2993 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:29.738565 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:29.741484 25372 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.695678  3203 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.036s
I20260809 23:21:29.647282  2469 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.745091 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.741456 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:29.745247  2625 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:29.752909 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63065.8us
I20260809 23:21:29.744894  2993 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:29.756580 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.763639 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
W20260809 23:21:29.764062 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59777.7us
I20260809 23:21:29.764819 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:29.766202  3203 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20.
I20260809 23:21:29.766467 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:29.725960  2820 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260809 23:21:29.762951 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:29.767422 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48152us
I20260809 23:21:29.767333  3104 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:29.721774  3108 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.766871 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:29.775810 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:29.778419 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57506.8us
I20260809 23:21:29.779673  3171 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:29.780109  3171 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:29.780710  3171 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:29.782069  3195 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260809 23:21:29.782653  3193 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260809 23:21:29.783409  2625 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:29.791251  2625 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:29.791703  2625 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:29.780052 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:29.801237 25323 leader_election.cc:398] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19.
I20260809 23:21:29.802361 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:29.803126  3169 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:29.725786  2378 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.786229 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:29.786826 24546 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:29.787408 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.761318  3164 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:29.814983 24539 raft_consensus.cc:2354] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:29.824564 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:29.824685 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.825515 24540 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:29.826321 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.826968 24545 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:29.827836 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.828444 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.829180 24427 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:29.828996 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.829628  2884 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:29.829944 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.830423 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:29.831032 25366 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.831290 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.831993 24430 raft_consensus.cc:2354] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:29.832787 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.833987 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.820163 25372 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:29.822979 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:29.830029  2884 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:29.835179 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.763146  3087 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:29.836596  3087 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:29.836194  2884 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.837544  2884 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:29.837512  3087 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.838106  3169 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:29.840312  3169 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.850732 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:29.839840 24161 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 16, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 16 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:29.843384  3170 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:29.845228 24542 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:29.844962 24544 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:29.855871 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.861832 24428 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:29.866154 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.869594 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.870319 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:29.872094 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.865325 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:29.874069 25373 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.879740 24541 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:29.879840 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.880602 24427 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.880729 24431 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:29.882023 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.844622  2854 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:29.892184 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.894114 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:29.893146 25366 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:29.823460  3104 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:29.823412  3164 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:29.899996 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.859810  2764 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:29.909544 24435 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 21
I20260809 23:21:29.900389 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:29.810168  3171 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:29.926232 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:29.938670 24425 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:29.939414 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.940673 24422 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:29.883503  2720 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:29.941421 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47396.3us
I20260809 23:21:29.922765 24423 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:29.893662  2854 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:29.777290  3093 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:29.960479 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:29.915427  3164 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:29.959517 24435 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:29.872822  3170 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:29.938378  3186 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.291s	user 0.001s	sys 0.001s
I20260809 23:21:29.967705 24427 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:29.967394 24428 raft_consensus.cc:2354] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:29.965498 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.001832 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.002449 24424 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:29.925447  2992 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:30.005120 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.008684 24312 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:21:30.005802 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 354904us
W20260809 23:21:30.006424 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44343.3us
I20260809 23:21:29.961119  2854 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:30.007256 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 150671us
I20260809 23:21:29.964773  2813 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:30.018110 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:29.964910  2720 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:29.951028  2593 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260809 23:21:30.030264 24435 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:30.030380 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.031312 24429 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:30.028014 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.031857 24428 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:29.944993  2764 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:30.045630 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56731.8us
I20260809 23:21:30.046702 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:30.055799 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.076864 25369 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:30.083763 24313 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:21:30.090581 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 72578.9us
I20260809 23:21:30.090660  3234 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:30.092109 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:30.093473 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:30.104039 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 281117us
I20260809 23:21:30.105352 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:30.110345 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.026127  2992 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.113391 24428 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:30.117576 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:30.118772 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.124053 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.123248 24313 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:30.088802  3108 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.124203 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.128677 24321 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
W20260809 23:21:30.126976 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80307.8us
I20260809 23:21:30.130650 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.131315 24429 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:30.098713  2593 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:30.084786  2625 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.143237 24161 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 18, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 18 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:30.098327  2816 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:30.153111 25373 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:30.111083  3167 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:30.154785 24312 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260809 23:21:30.156018 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.149385 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:30.163486 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.157089 24312 pending_rounds.cc:76] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 12
I20260809 23:21:30.167268 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.167970 25365 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:30.127739  2720 raft_consensus.cc:2741] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:30.172353  2625 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:30.172941  2625 raft_consensus.cc:2741] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:30.172995 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.173591  2625 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:30.174269  2625 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:30.175994  3156 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.638s	user 0.009s	sys 0.006s
I20260809 23:21:30.127163  3169 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:30.177726 25366 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:30.165438 24312 pending_rounds.cc:97] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260809 23:21:30.178989 24312 raft_consensus.cc:2866] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:30.176558 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.076658  3104 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.174719  2625 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:30.192406 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.165046  2593 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.192976 24161 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 16, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 16 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:30.193850  2992 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.195003 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.195235  2764 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:21:30.197273  3214 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260809 23:21:30.203936 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.091284  3234 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.207417 24309 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23.
I20260809 23:21:30.215651 24428 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:30.216177 24429 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
W20260809 23:21:30.297253  3224 log.cc:924] Time spent T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260809 23:21:30.300244  2593 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.307076 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:30.227698 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.193922  2809 raft_consensus.cc:979] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:30.177919  3171 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:30.231427 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97565.7us
I20260809 23:21:30.176541  3156 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.287277 24311 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:30.177075  3169 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:30.300263 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:30.239558 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109851us
I20260809 23:21:30.309744 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.331358 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:30.331650 24313 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:30.307958 24317 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:30.198678  3167 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:30.325526 24323 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:30.207902  2720 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:30.326807 24430 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260809 23:21:30.321130 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.317682 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.208295  3170 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:30.336169 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 241529us
I20260809 23:21:30.329164 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.177884  2816 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:30.350867 24321 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:30.347929 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:30.351625 24323 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:30.353305 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.353058 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.356840 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:30.357900 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.358021 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.358640 24311 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:30.330919  2795 raft_consensus.cc:1067] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:30.361644 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.330608  3169 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:30.338660  3167 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:30.365454 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.298448 24161 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:30.375790 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:30.377523 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46256.7us
I20260809 23:21:30.367995  3239 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:30.368799  3156 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.353025  2809 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:30.370054 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:30.365634 25369 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:30.366647 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.347366  2720 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:30.403458 24431 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:30.375711 24312 raft_consensus.cc:2362] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18. Re-sending same reply.
I20260809 23:21:30.384382 24313 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:21:30.387297 24320 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260809 23:21:30.457110 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.457762 24317 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:30.453689 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.302822  2993 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.398571 24321 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:30.401607 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.401803 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:30.402774 24498 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.523s (SENT)
I20260809 23:21:30.411899 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.412144 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.381450  3167 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:30.312964  3236 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.090s
I20260809 23:21:30.452334 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.377205  2764 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:30.456027 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.487372 24312 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:30.375351  3169 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:30.491297 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.492513 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.493180 24431 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:30.493933  2995 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.750s	user 0.005s	sys 0.004s
I20260809 23:21:30.495527 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.496119  2993 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.496469 25375 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
W20260809 23:21:30.496748 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 118597us
I20260809 23:21:30.497200  3197 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:30.493960 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.497668  3197 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:30.471657 24309 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:30.495623 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.395617  2827 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:30.473078 24429 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:30.502075 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:30.505453  2992 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:30.504900 24161 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 17, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 17 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:30.505957  3229 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:30.506392  3229 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:30.506606  3206 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.373s	user 0.002s	sys 0.000s
I20260809 23:21:30.506781  3229 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:30.508344 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:30.510761  3221 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
W20260809 23:21:30.515873  3245 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260809 23:21:30.402305  3239 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:30.522675  2884 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:30.523237  2884 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.526129  2884 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.411432  2720 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:30.505589  3219 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260809 23:21:30.545305 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.500075 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:30.537891 24425 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.586s	user 0.006s	sys 0.005s
W20260809 23:21:30.538237 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63413.5us
I20260809 23:21:30.547196 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.547300 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.545226 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.548077 24425 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:30.548892  3236 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:30.549191  3239 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.550154  3236 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:30.550508  3236 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:30.551797  3236 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:30.553910 24424 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.000s
I20260809 23:21:30.554270 24424 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:30.557447 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:30.558426 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58432.2us
I20260809 23:21:30.540083 24311 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:30.559149 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:30.558816 24317 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:30.567880 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.568725 24323 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:30.498968  3228 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:30.587150  3228 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:30.587580  3228 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:30.591069  2378 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:30.591202 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.592521 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.593940 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.594635 25363 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:30.570887 24424 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:21:30.603402 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.604048 24311 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:30.558284 24320 raft_consensus.cc:2362] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20. Re-sending same reply.
I20260809 23:21:30.570592 24428 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:30.575198 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.575371  3236 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:30.575536 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.576269 24431 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:30.494395  2995 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.576828  3243 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260809 23:21:30.577574 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 185806us
I20260809 23:21:30.573741 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:30.580430  3251 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260809 23:21:30.581375  3171 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:30.524107  3153 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:30.583698  3248 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
W20260809 23:21:30.584141  3250 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260809 23:21:30.571759  2469 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:30.597551 24313 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:30.567296 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:30.600754 24378 connection.cc:232] RPC call timeout handler was delayed by -1.5673514478839934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260809 23:21:30.606415 24422 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.631s	user 0.006s	sys 0.002s
I20260809 23:21:30.578233 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:30.603825 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.505970  2992 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:30.618860 24425 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260809 23:21:30.621614 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48182.3us
I20260809 23:21:30.631515 24309 raft_consensus.cc:2362] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18. Re-sending same reply.
I20260809 23:21:30.623054 24423 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:30.631865 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:30.649955 24323 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260809 23:21:30.623699 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:30.624974 25325 leader_election.cc:398] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:30.554352  2993 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:30.627278 24378 connection.cc:232] RPC call timeout handler was delayed by -1.0995344577822834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260809 23:21:30.629918 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:30.655270 24541 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:30.627421 24422 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:30.657321 24544 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:30.664216 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:30.665099 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:30.661693 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:30.617038  2378 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:30.685557 25365 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260809 23:21:30.539806  2827 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.688623 24429 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:30.652197  2992 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:30.701893 24539 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:30.632105  3161 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:30.698050 24535 raft_consensus.cc:2354] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
W20260809 23:21:30.697789 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76443.2us
I20260809 23:21:30.705111 25370 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:30.506698 24161 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 17, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 17 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:30.705067 24311 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:30.697204 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:30.696449 24534 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:21:30.751010 24534 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:21:30.751612 24534 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:30.701462  3248 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260809 23:21:30.759713 24321 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260809 23:21:30.685045  3259 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260809 23:21:30.764006 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:30.767050 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:30.768561 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:30.769833 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:30.772055 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.626314  3153 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:30.769734 24534 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.705739  2736 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:30.774992 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88323.3us
I20260809 23:21:30.705425  3161 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:30.739966  2816 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:30.663498  2993 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.753934  2469 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:30.789307 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:30.772055  3171 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:30.811506 24534 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:30.809953 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 186367us
I20260809 23:21:30.748257  2729 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260809 23:21:30.782984  3232 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:30.787142  3161 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:30.857883 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63589.8us
I20260809 23:21:30.856112 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:30.865504 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.011s	sys 0.000s
W20260809 23:21:30.875125 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.012s	sys 0.000s
I20260809 23:21:30.875319 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.876358 24427 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
I20260809 23:21:30.874308 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:30.877372 24429 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:30.877346  3254 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:30.878377  3239 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:30.878185  3254 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.879690 24535 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:30.878832  3239 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:30.885313  3254 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.886921  3239 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:30.888221 24535 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:30.890475 24429 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:30.890560 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:30.892251 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:30.896136 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:30.899309 24542 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:30.892642 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:30.895943 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:30.740005  3249 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.205s
W20260809 23:21:30.905037 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 296941us
I20260809 23:21:30.900893 25370 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 20
W20260809 23:21:30.932279 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76448.6us
I20260809 23:21:30.933694 24266 leader_election.cc:398] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:30.934978 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:30.939622 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:30.870963  3171 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:30.869247  2764 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:30.783571  3167 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:30.950619 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54724.1us
W20260809 23:21:30.926522  3164 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.921s	user 0.007s	sys 0.004s
W20260809 23:21:30.959208  3258 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260809 23:21:30.942487 24161 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 15, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 15 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:30.962144  3218 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:30.926728  3154 raft_consensus.cc:2741] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260809 23:21:31.034899 24544 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:30.972836  3272 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:30.974902  2720 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.963012  3244 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:30.979856  2993 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.950558  2795 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:30.900870  3232 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:30.946210  2995 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:30.977872  3229 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.069522 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.069994 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.950899  2741 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:31.039496  2813 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.017s	user 0.014s	sys 0.005s
I20260809 23:21:31.036131  2827 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:30.992735 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:30.895596  2378 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.020035 24540 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:31.023015 24545 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:30.957412  3203 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
W20260809 23:21:31.033703 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 263941us
I20260809 23:21:30.907509  2469 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.043783 24314 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
W20260809 23:21:31.055709  3270 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.187s	user 0.001s	sys 0.001s
I20260809 23:21:31.099933 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:30.990913  3228 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.040053  3272 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:31.059719  3244 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260809 23:21:30.993428 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120684us
W20260809 23:21:30.989629 24498 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.643s (SENT)
W20260809 23:21:31.074103 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 134553us
I20260809 23:21:31.132653 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.085568 24535 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:31.094478  2625 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.803s	user 0.012s	sys 0.000s
I20260809 23:21:31.082872  2813 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:31.064731  2729 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:31.088047 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:31.140028 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.135426 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:31.130738 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.039512  2881 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:31.107736 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.141242 24540 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:31.150889 25366 raft_consensus.cc:2354] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:31.141237 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:31.145807 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53581.2us
I20260809 23:21:31.151712 24323 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:31.158883  3171 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.041857  3164 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.099740  3153 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:31.171182 25321 leader_election.cc:398] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
I20260809 23:21:31.172297  3279 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:31.172726  2813 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:31.176301 25321 leader_election.cc:401] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260809 23:21:31.178037 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.174639  3283 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:31.180826 24535 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:31.129679  2741 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260809 23:21:31.187131 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.089468  2795 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.069s
I20260809 23:21:31.179399 25325 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
W20260809 23:21:31.172798  3262 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260809 23:21:31.155573  2816 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:31.206723  3153 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:31.188508  3244 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:31.203359 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:31.108258  3272 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:31.234017 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.001s	sys 0.006s
I20260809 23:21:31.211146 25325 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:31.208701 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.210487 24313 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:31.135529  3212 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.244s
I20260809 23:21:31.153250  2625 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.240386 24311 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:31.201570 24309 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:31.176294  2881 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:31.248629  2625 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:31.249346  2625 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:31.250443  3164 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.251137 25372 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:31.188298  2813 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:31.251933  2813 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:31.235826 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56639.4us
I20260809 23:21:31.255781 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:31.234378 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.001s	sys 0.006s
I20260809 23:21:31.156191  2729 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.209609  3279 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.242278 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:31.139207  2827 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
W20260809 23:21:31.259196 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50626.9us
I20260809 23:21:31.138908  2764 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.225577  2720 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:31.215150  3263 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:31.264482  2741 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:31.295643  3301 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:21:31.251183 24161 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 19, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:31.296622  3263 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:31.297348  3263 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.297550 24161 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:31.268388  2881 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:31.268925  2854 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.243s	user 0.012s	sys 0.002s
W20260809 23:21:31.299726  3290 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260809 23:21:31.301753 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:31.302379 24429 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:31.303793  3279 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.299727  2854 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.243228  3291 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260809 23:21:31.309489 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:31.311204 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.311964 24422 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:31.312536 25365 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:31.312548  3170 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:31.313227 24423 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:31.313380  3170 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.313560  3197 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:31.314421  3170 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.315630 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2924) took 1143 ms. Trace:
I20260809 23:21:31.315992  3170 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:31.317986  3234 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:31.315990 24314 rpcz_store.cc:276] 0809 23:21:30.172637 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.187732 (+ 15095us) service_pool.cc:227] Handling call
0809 23:21:30.194533 (+  6801us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:31.099746 (+905213us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:31.315405 (+215659us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54000dc5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:31.315568 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1947796736}
I20260809 23:21:31.319406 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.320079 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:31.320833 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 217901us
I20260809 23:21:31.321422 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.321879 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:31.322774  3208 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:31.322824 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.323364  3208 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:31.323524  3228 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:31.323783  3208 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:31.324430 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.281289 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.275763  3232 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.281883 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:31.328570 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:31.328715 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46957.5us
I20260809 23:21:31.284461 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.285535  3296 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:31.286383  3300 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:31.287647 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:31.288455  3169 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.800s	user 0.009s	sys 0.002s
I20260809 23:21:31.289104  3229 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:31.253918  3304 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260809 23:21:31.289815 24266 leader_election.cc:398] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
W20260809 23:21:31.291322  3236 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.675s	user 0.006s	sys 0.016s
W20260809 23:21:31.292311  3287 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260809 23:21:31.234313  3299 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:31.324452  3212 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:31.294574  3286 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:31.233453  3244 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:31.277336 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 287734us
I20260809 23:21:31.281981 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.325958 25369 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 16.
I20260809 23:21:31.327446 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:31.269949  2992 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.511s	user 0.002s	sys 0.007s
I20260809 23:21:31.330083 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.384334 25371 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:31.330837 24430 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:31.322646 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:31.342816 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55283.5us
I20260809 23:21:31.386515 25325 leader_election.cc:398] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:31.386843 25325 leader_election.cc:401] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260809 23:21:31.343632 25370 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:21:31.347339 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 57712.5us
I20260809 23:21:31.354218 24429 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:31.306438  3161 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.356660 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:31.264391  3167 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.293677  2625 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21.
I20260809 23:21:31.374364 25375 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260809 23:21:31.387856  3305 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:31.408854 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:21:31.435256  3256 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.838s	user 0.003s	sys 0.001s
I20260809 23:21:31.435734  3288 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:31.437317  3273 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:31.437811  3273 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:31.438532 24161 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 17, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 17 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:31.375931  3229 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:31.441884 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55526.9us
W20260809 23:21:31.374428 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50063us
I20260809 23:21:31.377089 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.378146 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:31.443804 24422 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:31.313577  2854 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:31.327740  3218 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:31.223402  3171 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.337342  3300 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:31.342777  3249 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:31.388213 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:31.355803 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:31.453378 24425 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:31.387079 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57087.3us
I20260809 23:21:31.355166  3291 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260809 23:21:31.401398 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.439284  3273 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:31.394510  3299 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:31.378109  3286 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:31.346413  3254 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260809 23:21:31.455333 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:31.356853  3234 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:31.346416  3169 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.461230 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.390993  2992 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:31.472049 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70774.1us
I20260809 23:21:31.457132  3300 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:31.461807  3307 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260809 23:21:31.451591  3236 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260809 23:21:31.451342  3304 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20.
I20260809 23:21:31.487870 24540 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:31.476871  3167 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:31.490310 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:31.489661 24422 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:31.458068 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.496913 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.510192 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:31.537807 24539 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 20
W20260809 23:21:31.537987 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 80018.6us
I20260809 23:21:31.484306 24425 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:31.493288  3315 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:31.527376  2854 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:31.528189 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:31.446019  2947 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260809 23:21:31.529168 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:31.510828 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:31.555245 24543 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:31.544546 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:21:31.555761 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58963.5us
I20260809 23:21:31.472455  2795 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260809 23:21:31.557461 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:31.493610  2809 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.091s	user 0.010s	sys 0.003s
I20260809 23:21:31.475517  3249 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.546684 24312 raft_consensus.cc:2354] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18.
I20260809 23:21:31.546542 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.491103  3273 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:31.485579  3218 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:31.471691  3212 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260809 23:21:31.452495  3229 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.530036  2378 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:31.550613  3156 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:31.528780  2729 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.573179 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.573927 24309 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:31.574754 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.550787  3316 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:31.575738 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.603634 24323 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:31.583531 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.587054 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.590827 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:31.594621 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:31.610666 24539 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:31.567266  2809 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:31.452270 24161 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:31.620851 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.622336 24541 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260809 23:21:31.626972 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:31.626497 24541 pending_rounds.cc:76] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 13
I20260809 23:21:31.596908  2378 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:31.591591  2854 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.628185 24541 pending_rounds.cc:97] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260809 23:21:31.674165 24541 raft_consensus.cc:2866] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:31.630501 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.676064 25369 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:31.671069 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:31.566824  3288 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
W20260809 23:21:31.612494  3222 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.749s	user 0.006s	sys 0.007s
I20260809 23:21:31.567015  2827 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260809 23:21:31.602521  3167 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:31.704195 24541 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:31.673300 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.705592 24323 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:31.676560 25369 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.684376 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:31.554092  3315 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.710136 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.713912 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3468) took 1105 ms. Trace:
I20260809 23:21:31.685671 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.606455  2881 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.078s
I20260809 23:21:31.722885 25370 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:31.691361  3259 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:31.624888 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 236743us
I20260809 23:21:31.688761 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.688896  2854 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.629551  3316 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:31.632146  3164 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:31.795591 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:31.707190 25372 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:31.671205  2378 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:31.594448  3299 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.118s
I20260809 23:21:31.692165  3161 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.738315  2992 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:31.769276 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.591580  3218 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:31.722597  3283 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:31.787004 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.794365 25370 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:31.591455  3286 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.799921 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:31.802263 25372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:31.807636 25372 raft_consensus.cc:595] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:31.808125 25372 raft_consensus.cc:603] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:31.714362 24320 rpcz_store.cc:276] 0809 23:21:30.608193 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.608353 (+   160us) service_pool.cc:227] Handling call
0809 23:21:30.646406 (+ 38053us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:30.923139 (+276733us) raft_consensus.cc:1520] Early marking committed up to index 10
0809 23:21:31.044998 (+121859us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.360583 (+315585us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:31.373634 (+ 13051us) log.cc:842] Serialized 52 byte log entry
0809 23:21:31.374001 (+   367us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:31.374525 (+   524us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.382690 (+  8165us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.383051 (+   361us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:31.383146 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:31.713643 (+330497us) raft_consensus.cc:1671] finished
0809 23:21:31.713671 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:31.713832 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40850,"prepare.queue_time_us":113492,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":5176,"raft.queue_time_us":138965,"raft.run_cpu_time_us":525,"raft.run_wall_time_us":28078,"spinlock_wait_cycles":97533824,"thread_start_us":545312,"threads_started":4,"wal-append.queue_time_us":188481}
I20260809 23:21:31.804952 24323 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:31.813584 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.816110  3331 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:31.816874 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:31.819990 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:31.820407  3153 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:31.820758  2881 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:31.821159  3153 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:31.823113  2881 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:21:31.823763  3229 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:31.825150  3313 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260809 23:21:31.825203  2816 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:31.825779  2816 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:31.744366  3259 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.826686  2816 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:31.828349  3314 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260809 23:21:31.828248  2816 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:31.734221  3308 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260809 23:21:31.831451  2993 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:31.803896  3169 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:31.834219  3311 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.349s	user 0.001s	sys 0.000s
I20260809 23:21:31.828187 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:31.837774 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:31.797355  2813 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.003s
W20260809 23:21:31.801658  3310 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260809 23:21:31.842653 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:21:31.851080 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 145462us
I20260809 23:21:31.709154  3249 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.857090 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.858237 24320 raft_consensus.cc:2396] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260809 23:21:31.854575 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:31.860270 24323 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:31.814071  3316 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:31.846004  3104 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:31.876250 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:31.849148  3291 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:31.876888 25372 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:31.847918  2813 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.888833 24313 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:31.890203 24321 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:31.893635 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:31.880997 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.001s	sys 0.010s
I20260809 23:21:31.894550  3339 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:31.894645 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.001s	sys 0.010s
I20260809 23:21:31.896669 24321 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:31.906322 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:31.907217 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:31.907447 25372 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:31.926129 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:31.928172  3309 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.309s	user 0.000s	sys 0.001s
I20260809 23:21:31.928607  3167 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:31.929222  3167 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:31.930092  3167 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:31.812924  3283 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:31.931669 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1685) took 1235 ms. Trace:
I20260809 23:21:31.932034 25365 rpcz_store.cc:276] 0809 23:21:30.696246 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.748476 (+ 52230us) service_pool.cc:227] Handling call
0809 23:21:31.931321 (+1182845us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:31.931584 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1240917248}
I20260809 23:21:31.933246  3283 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:31.937475 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1681) took 1411 ms. Trace:
I20260809 23:21:31.937862 25375 rpcz_store.cc:276] 0809 23:21:30.526303 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.619465 (+ 93162us) service_pool.cc:227] Handling call
0809 23:21:30.650206 (+ 30741us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:31.407238 (+757032us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:31.937193 (+529955us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b5400442ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:31.937397 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1633624320}
I20260809 23:21:31.939229 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:31.940006  3328 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.202s	user 0.000s	sys 0.000s
I20260809 23:21:31.940840 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.941319 24161 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 11 to 19, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:31.947345  3316 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:31.949296 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1669) took 1580 ms (client timeout 1912 ms). Trace:
I20260809 23:21:31.950369  3104 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.852042  3156 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260809 23:21:31.872202  2741 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:31.963195 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:31.955719 25366 rpcz_store.cc:269] 0809 23:21:30.368912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.380449 (+ 11537us) service_pool.cc:227] Handling call
0809 23:21:31.949053 (+1568604us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b540055f854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:31.949217 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1560014592}
I20260809 23:21:31.958863 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:31.979892 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:31.983659 24313 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:31.987179 25371 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:31.998195 25371 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:31.999789 24320 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:31.999919 24314 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:32.000769 24314 raft_consensus.cc:595] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.001444 24314 raft_consensus.cc:603] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:32.003516 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:31.970290  3291 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:31.969719  2809 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:32.026096 24320 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:32.027709 24323 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:31.980198  2741 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.037563 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3509) took 1147 ms. Trace:
I20260809 23:21:32.037956 24535 rpcz_store.cc:276] 0809 23:21:30.890091 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.890304 (+   213us) service_pool.cc:227] Handling call
0809 23:21:32.037440 (+1147136us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400147154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:32.037520 (+    80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424394112}
I20260809 23:21:32.028327  3339 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:32.041519 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 148007us
I20260809 23:21:32.048981 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:32.041492 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:32.030328  3170 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:32.034363  3315 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.024837  2992 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260809 23:21:32.055902 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:32.062810 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:32.066154 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:32.068038 25366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:32.068714 25366 raft_consensus.cc:595] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:32.068969 25366 raft_consensus.cc:603] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:32.070154 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:32.071316 24431 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:32.072546 24266 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:32.072966 24266 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260809 23:21:32.054843  2813 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.081144  3242 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:32.081756  3242 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:32.082122  3242 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:32.084010  3247 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.503s	user 0.002s	sys 0.002s
I20260809 23:21:32.040822  3104 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.049093 24161 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 16, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:32.057878  3153 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:32.059561  3315 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260809 23:21:32.093580  3353 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260809 23:21:32.094578  3352 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260809 23:21:32.095110  3333 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:32.125882 24428 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:32.135386 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54830.7us
W20260809 23:21:32.134783 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 729068us
I20260809 23:21:32.087356  3156 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:32.139392  2741 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.136148  3333 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.141758 25372 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260809 23:21:32.138780 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:32.147223 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:32.147910  3164 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:32.080601  3167 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22.
I20260809 23:21:32.148665  3164 consensus_queue.cc:1225] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:32.055143  3170 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.147933 24423 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:32.099498  3208 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.775s	user 0.008s	sys 0.005s
I20260809 23:21:32.161053 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:32.162789 24498 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.816s (SENT)
W20260809 23:21:32.124233  3272 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.866s	user 0.012s	sys 0.000s
I20260809 23:21:32.165482  3208 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.165728  2741 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:32.176163 24422 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:32.178078  3170 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.179632  3218 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.182166 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1675) took 1659 ms. Trace:
I20260809 23:21:32.182655 25368 rpcz_store.cc:276] 0809 23:21:30.372073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.520222 (+148149us) service_pool.cc:227] Handling call
0809 23:21:30.520659 (+   437us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:30.520922 (+   263us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:30.663635 (+142713us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.942159 (+1278524us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b180033cee0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:21:32.030591 (+ 88432us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:32.030644 (+    53us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:32.030693 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:32.030763 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:32.030875 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":412,"prepare.queue_time_us":236061,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":6703,"raft.queue_time_us":1754,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":6939,"spinlock_wait_cycles":4810112,"thread_start_us":226372,"threads_started":2,"wal-append.queue_time_us":1395282}
W20260809 23:21:32.191497  3366 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260809 23:21:32.191638  3244 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.124431  3229 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:32.086560 24161 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:21:32.203956  3358 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.102s	user 0.000s	sys 0.002s
W20260809 23:21:32.204296 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 250541us
W20260809 23:21:32.206878  3354 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260809 23:21:32.159533  3315 raft_consensus.cc:2741] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260809 23:21:32.206700  3355 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260809 23:21:32.175379  2741 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:32.215241  2741 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:32.218487 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.174332  3208 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.143923  3242 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.219658 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.170800  3272 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.286266  3306 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:32.227936  2995 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:32.229203 24320 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:32.214466  2378 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.231763 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.210594  3229 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.233932  3263 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.235538 24161 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:32.239912  2820 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:32.242304 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1743) took 1173 ms. Trace:
I20260809 23:21:32.061537  3291 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.244138 24314 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:32.244688 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46198.3us
I20260809 23:21:32.287968  2995 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:32.251870 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1729) took 1555 ms. Trace:
I20260809 23:21:32.260079 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:32.266628 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.269447 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:32.271924  2720 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:32.276762 24540 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:32.278573 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.368929 24535 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
W20260809 23:21:32.281105 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66548.4us
I20260809 23:21:32.285199 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.285490 24546 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:32.305408 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 151935us
I20260809 23:21:32.288241 25369 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.376536 25369 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:32.321520 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:32.378118 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56719.1us
I20260809 23:21:32.212643  3170 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:32.318833 24312 rpcz_store.cc:276] 0809 23:21:31.069127 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.069261 (+   134us) service_pool.cc:227] Handling call
0809 23:21:32.242059 (+1172798us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b5400386dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:32.242223 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1058140416}
W20260809 23:21:32.369279 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 109295us
W20260809 23:21:32.370420 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47553.3us
I20260809 23:21:32.374534 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:32.384732 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 115368us
I20260809 23:21:32.386307 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.391263 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:32.339089 25363 rpcz_store.cc:276] 0809 23:21:30.696512 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.761492 (+ 64980us) service_pool.cc:227] Handling call
0809 23:21:30.808478 (+ 46986us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:31.053936 (+245458us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:31.057799 (+  3863us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.452779 (+394980us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.609633 (+156854us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.614844 (+  5211us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:31.615188 (+   344us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:31.615322 (+   134us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:32.251556 (+636234us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:32.251605 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:32.251818 (+   213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":131109,"prepare.queue_time_us":1931,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":6605,"raft.queue_time_us":58234,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":59960,"spinlock_wait_cycles":655945728,"thread_start_us":495115,"threads_started":2,"wal-append.queue_time_us":538335}
I20260809 23:21:32.396885 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.397787 24546 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:32.318343  2820 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:32.338415  3153 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:32.371703  3276 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:32.373560  3228 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.407648 24431 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260809 23:21:32.410425  3330 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:32.384406  3156 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:21:32.373387  2720 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:32.446378  2720 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.449975  3197 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:32.436290 24427 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:32.455667  3259 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.456909 24539 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:21:32.457912 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.396636  3306 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:32.392970  3170 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.458523 24546 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:32.435528 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3467) took 1874 ms. Trace:
I20260809 23:21:32.432824  3242 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.432981  2741 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.442624  2995 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:32.445232  3329 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.862s	user 0.001s	sys 0.002s
I20260809 23:21:32.411520  3263 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:32.469024 24317 rpcz_store.cc:276] 0809 23:21:30.561408 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:30.574392 (+ 12984us) service_pool.cc:227] Handling call
0809 23:21:30.608792 (+ 34400us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:30.624865 (+ 16073us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:30.624910 (+    45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:30.625572 (+   662us) log.cc:842] Serialized 52 byte log entry
0809 23:21:31.222388 (+596816us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:31.258256 (+ 35868us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:31.279220 (+ 20964us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.279753 (+   533us) log.cc:842] Serialized 14 byte log entry
0809 23:21:31.288232 (+  8479us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:31.288331 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:31.848664 (+560333us) raft_consensus.cc:1671] finished
0809 23:21:31.848693 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:32.435367 (+586674us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b54002535d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:32.435472 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14644,"prepare.queue_time_us":27223,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":1861,"raft.run_cpu_time_us":3362,"raft.run_wall_time_us":595703,"spinlock_wait_cycles":85439232,"thread_start_us":552224,"threads_started":1,"wal-append.queue_time_us":741672}
I20260809 23:21:32.473018 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:32.411844  3300 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.864s	user 0.013s	sys 0.000s
I20260809 23:21:32.412205 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:32.437429  3171 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
W20260809 23:21:32.479240 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67151.1us
I20260809 23:21:32.411616  3212 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:32.500067 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93646.9us
I20260809 23:21:32.406991  3288 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.077s
I20260809 23:21:32.501745 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:32.480615  3273 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.883s	user 0.014s	sys 0.000s
I20260809 23:21:32.461658  3276 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:32.478021  3306 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:32.460043  3234 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260809 23:21:32.441546  3228 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.491382  2884 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:32.520601  2884 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:32.521034  2884 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:32.523699 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:32.524341  2947 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.524855  2947 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.525805  3153 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.507771  3242 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.019s
W20260809 23:21:32.527071  3373 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260809 23:21:32.527643  3276 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.528791  3272 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.515465  3243 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.015s
I20260809 23:21:32.529883  3272 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:32.508178 24161 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 16, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 16 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:32.532068 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3548) took 1138 ms. Trace:
W20260809 23:21:32.476739  3334 consensus_queue.cc:1159] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "808fed3ed3f04882ad757a3db9abfdd9" responder_term: 21 status { last_received { term: 20 index: 15 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:32.518333  2378 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.533090 24430 rpcz_store.cc:276] 0809 23:21:31.393721 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.394114 (+   393us) service_pool.cc:227] Handling call
0809 23:21:31.409839 (+ 15725us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:31.411479 (+  1640us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:31.411524 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:31.412020 (+   496us) log.cc:842] Serialized 26 byte log entry
0809 23:21:31.545912 (+133892us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:31.545989 (+    77us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:31.546132 (+   143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:32.444659 (+898527us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:32.531564 (+ 86905us) raft_consensus.cc:1671] finished
0809 23:21:32.531597 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:32.531895 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1160,"raft.queue_time_us":2441,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":111,"spinlock_wait_cycles":7595392,"thread_start_us":133208,"threads_started":1,"wal-append.queue_time_us":154602}
I20260809 23:21:32.516319 24546 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:32.546806 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:32.509303  3339 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:32.501062  3171 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:32.553171 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:32.467252  3330 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.541548  3154 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.008s
W20260809 23:21:32.559316 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 135352us
I20260809 23:21:32.560935 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:32.561506 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:32.562269 24431 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:32.541606  3272 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:32.563596 24498 leader_election.cc:398] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:32.564779 24498 leader_election.cc:401] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:32.563452 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.565999 24314 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:32.483018  3300 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.570238 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:32.570750 24320 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260809 23:21:32.572257 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:32.572312 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 164008us
I20260809 23:21:32.555665  3369 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:32.578194 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:32.552162  3170 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.581400 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:32.501238  3212 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.599828 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.607466 24535 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:32.492337  2820 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.610142 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.605991 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.557315  2378 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.560315  2741 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.606686 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:32.608168 24314 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:32.613847 25370 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:32.614176 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:32.615015 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:32.615621 25368 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:32.608578 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:32.565644  3208 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:32.623821 25363 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:32.613402 25372 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:32.613384 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.628230 25366 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:32.620119 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 161796us
I20260809 23:21:32.611814 24320 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:32.567273  3272 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:32.575874  3207 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:21:32.639135 24535 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:32.601406  2795 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:32.645371 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:32.601596  3273 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.604727  3218 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:32.564023  3253 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.068s
I20260809 23:21:32.706116 24430 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:32.731312 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.531675 24161 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 13 to 19, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:21:32.723448 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113376us
I20260809 23:21:32.723017 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.734197 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:32.734771 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89521.5us
I20260809 23:21:32.736773 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.623039  3229 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:32.742071 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 138037us
I20260809 23:21:32.640407  3263 raft_consensus.cc:2741] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260809 23:21:32.743414 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:32.731385  3316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.024s	sys 0.040s
I20260809 23:21:32.703048  3334 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:32.645583 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:32.705003  3400 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.800576 25369 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260809 23:21:32.624297  2378 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.816021 25369 pending_rounds.cc:76] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 15
W20260809 23:21:32.777321  3316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.024s	sys 0.040s
I20260809 23:21:32.626332  3276 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.820382 25369 pending_rounds.cc:97] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260809 23:21:32.824451 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:32.828529 24321 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260809 23:21:32.787685  2741 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.733938  3273 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.615031  3170 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.202s
I20260809 23:21:32.821650  3339 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.839493 25366 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260809 23:21:32.844702 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:32.845233 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.845706 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.839320 25369 raft_consensus.cc:2866] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:32.826618  2795 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:32.856911 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:32.865235 24320 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:32.794548  3218 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260809 23:21:32.861078 24317 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:32.862005 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.721208  3272 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.865780 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:32.868590 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:32.787209  3315 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.053s
I20260809 23:21:32.840147  3393 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:32.847198  2736 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.885150 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:32.887454 24542 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:21:32.849169  3291 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.848147  2741 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.887243  2795 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:32.893594  2741 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.893878  3407 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
W20260809 23:21:32.895469  3341 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260809 23:21:32.895903  2736 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.896298  2884 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.893719  2795 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:32.895362  3405 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260809 23:21:32.926261 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 108475us
I20260809 23:21:32.866725  3228 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:32.946262 24431 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:32.898339  2736 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:32.813906  3171 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.970192 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:32.970404 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:32.880271  3393 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:32.973996 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:32.974668 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:32.979099 24545 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:32.904824  3413 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:32.643688  3306 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.902057  3414 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.821962  3400 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.990427 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:32.999079 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:32.999783 24428 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:32.976521  3300 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:32.983033 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:32.911402  3156 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260809 23:21:32.976708  3330 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:32.893616  3266 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.346s
I20260809 23:21:32.908707  3333 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.955893  3407 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:32.864324  3339 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.003542 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:32.898344  2884 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.010402 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:32.982692  3272 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:32.982409  3393 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:32.980247  3212 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:32.982399  3399 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:33.047441  3399 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.942023  3291 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:32.902559  3421 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.105s
I20260809 23:21:32.988536  3170 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.019646 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:33.008507 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 142812us
I20260809 23:21:33.022660  3417 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:33.023500  3390 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
W20260809 23:21:33.024152  3380 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
W20260809 23:21:33.026512  3409 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260809 23:21:33.031031  3333 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:33.031198  3377 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:33.032557  2378 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.032883  3400 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.018455  3405 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:33.039477  3339 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.015236  3296 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260809 23:21:33.091583 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.040190  3273 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.041522 24431 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:33.050490 25368 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:33.053241 24545 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
W20260809 23:21:33.017736 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 179271us
I20260809 23:21:33.057137 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.097163 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.098470 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.059034 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.061615 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.063829 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.063678  3170 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.064620 25365 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:33.065142  3291 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:33.065196  3432 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:33.067546 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 330864us
I20260809 23:21:33.068671 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.099926 24534 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:33.100121 24541 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260809 23:21:33.100414 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:33.070677  3334 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:33.017269  3414 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.090291  3377 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:33.073909  3417 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.094296  2881 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.110725 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.015162  3407 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.092768  3405 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.094719  3333 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:33.094821  3413 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:32.879316 24161 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 10 to 17, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 17 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:33.102481  3228 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:33.129235 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.117611  3291 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:33.126575 24498 leader_election.cc:398] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23.
I20260809 23:21:33.138833 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.121487  3377 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:33.123059  2469 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader election won for term 17
I20260809 23:21:33.124122  3339 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:33.187177 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48464.4us
I20260809 23:21:33.126260  3431 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:33.192601 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:33.194543 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:33.199922 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.119659  2992 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260809 23:21:33.180886  3421 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:33.128948  3405 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:33.206768  2469 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:33.215546  2469 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.218852  3339 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:33.219400  3339 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:33.189517 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:33.210079 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 83640.3us
I20260809 23:21:33.209549  3421 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:33.128782  3423 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.069s
I20260809 23:21:33.213486  3393 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.189427  3291 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:33.212730  3430 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260809 23:21:33.127704  2881 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.230873 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:33.230595 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.226967 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:33.228489 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.206851  2992 consensus_queue.cc:1225] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 21.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:33.197124  3288 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.231683 24320 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:33.224112  3421 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:33.271602  3393 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:33.223992  3428 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:33.273550  3428 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:33.273955  3428 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:33.278527 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50077.8us
I20260809 23:21:33.280350 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.281978 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.231881  3296 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.299328  3296 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.300800  3286 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:33.302394  3296 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.304642  3399 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:33.306232  3441 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260809 23:21:33.258975  3396 raft_consensus.cc:2741] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260809 23:21:33.140630  3413 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:33.285629  3369 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.000s
I20260809 23:21:33.269300  2741 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.285928  3407 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.287132  3263 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:33.286257  3393 raft_consensus.cc:2741] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election decision vote started in defunct term 24: won
I20260809 23:21:33.290030  3283 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:33.336062 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 237701us
I20260809 23:21:33.340559 24320 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:33.321532  3369 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:33.342713 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.343601 24320 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:33.290202  3239 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260809 23:21:33.259181  3288 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.367014 24427 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:33.325255  3170 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.333721  3263 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:33.382697 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:33.398535 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113788us
I20260809 23:21:33.399415 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.258312  3333 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.334909  3424 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:33.356273  3350 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.336200  3283 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.372516  3445 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.324369  2741 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.339327  3414 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.312958  3339 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.419941 24320 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:33.367838  3393 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:33.403472  3396 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:33.388103  3218 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.437896  3283 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.387378  3263 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:33.411134  3423 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:33.405258 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:33.443439 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 250927us
W20260809 23:21:33.446820 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 169568us
W20260809 23:21:33.452900 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48499.6us
W20260809 23:21:33.447211 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 178325us
I20260809 23:21:33.453881 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:33.456058  3169 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:33.442996 24427 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:33.447561 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.416290  3417 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:33.411244  3369 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.458570 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.459179  3369 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.464500 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.465112 24534 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:33.415102  2726 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:21:33.475251 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.441704  3170 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.481914  2741 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:33.483528  3452 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260809 23:21:33.441036  3445 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.489445  3445 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.498382 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.500236  3413 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.417243  3431 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.058s
I20260809 23:21:33.507274  3431 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:33.451340  2881 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.448189  3349 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.512450  3349 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.515173 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.430214  3350 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.516076 24435 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:33.517783  3350 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.518306  3423 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:33.520788  2947 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:33.521775  3272 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.001s
W20260809 23:21:33.526181  3451 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260809 23:21:33.528189 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1752) took 1821 ms. Trace:
I20260809 23:21:33.448881  3393 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.529559  3405 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.531951  3369 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.532793 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1685) took 1979 ms. Trace:
I20260809 23:21:33.533166 25369 rpcz_store.cc:276] 0809 23:21:31.554824 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.555049 (+   225us) service_pool.cc:227] Handling call
0809 23:21:31.580980 (+ 25931us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:32.287877 (+706897us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5400294120. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f6951 00007fac23857321 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612
0809 23:21:32.861753 (+573876us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:33.532574 (+670821us) spinlock_profiling.cc:243] Waited 94.8 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:33.532735 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":223138432}
I20260809 23:21:33.534428  3349 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.448153  3218 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.528573 24321 rpcz_store.cc:276] 0809 23:21:31.706830 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.900595 (+193765us) service_pool.cc:227] Handling call
0809 23:21:31.900894 (+   299us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:32.829924 (+929030us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:33.527941 (+698017us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:33.528129 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2132708736}
I20260809 23:21:33.467393  3417 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:33.537742 24534 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:33.538411 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63870.8us
I20260809 23:21:33.452445  3396 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.518695  3423 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.552234 25369 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:33.552901 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.572312 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.578728 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.579571 25368 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:33.545146  3393 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.587494 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.591518 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.549980  2947 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:33.585566 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.594532 24428 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 20 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 18.
I20260809 23:21:33.549615  3431 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:33.549625  3253 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.578792  3411 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.612501 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.562690  3423 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:33.555899  3417 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.604718  3431 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:33.622843 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 169060us
I20260809 23:21:33.624401 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.625167 24428 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:33.544883  3170 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.627722 24428 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:33.558665  3396 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.639268 25365 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:33.625047  3431 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:33.610056  3253 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.645995  3253 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.629844  3405 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.654980 24541 pending_rounds.cc:76] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 13
I20260809 23:21:33.655309 24541 pending_rounds.cc:97] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260809 23:21:33.655586 24541 raft_consensus.cc:2866] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:33.641891  3478 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260809 23:21:33.662456  3398 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.663648  3218 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.653980  3304 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.665390  3304 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:33.666594  3408 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.777s	user 0.000s	sys 0.004s
I20260809 23:21:33.667673  3304 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.668251  3369 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:33.669255  3446 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.401s	user 0.000s	sys 0.006s
I20260809 23:21:33.671042  3377 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.671597  2720 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:33.671351  3398 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.672241  3396 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:33.637028  3288 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.664532  3207 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:33.663777  3239 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:33.668834  3369 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.678704 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:33.689929 25369 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:33.691138 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.662237 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:33.680629 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.673207  3377 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.696509 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.697268 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.698199 25369 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:33.698320 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.616882  3411 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.700784 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.701449 25373 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:33.703320 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:33.703384 24316 raft_consensus.cc:595] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.704254 24316 raft_consensus.cc:603] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:33.708745  2995 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:33.710641  3369 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.689015  3239 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:33.712181  3239 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.713639 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40854 (request call id 1291) took 2215 ms. Trace:
I20260809 23:21:33.713888  3239 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:33.714043 24311 rpcz_store.cc:276] 0809 23:21:31.497879 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.498941 (+  1062us) service_pool.cc:227] Handling call
0809 23:21:33.713338 (+2214397us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238667c5 00007fac2385ec1a 00007fac287a26d9 00007fac21e8c184 00007fac21e8c112 00007fac21e8c041 00007fac21e8bfbc 00007fac21e8b272 00007fac21c8a94c 00007fac21c89d86
0809 23:21:33.713556 (+   218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4870451584}
I20260809 23:21:33.723297 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.722990  3443 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.123s
I20260809 23:21:33.724099  3304 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:33.725087  3170 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.725963  3170 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:33.730736  3473 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:33.731309  3473 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.733395  3473 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:33.733974  3427 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.710s	user 0.004s	sys 0.002s
I20260809 23:21:33.736193  3417 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:33.736826 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1700) took 1142 ms. Trace:
I20260809 23:21:33.737198 25371 rpcz_store.cc:276] 0809 23:21:32.594378 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.595424 (+  1046us) service_pool.cc:227] Handling call
0809 23:21:32.614233 (+ 18809us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:32.783687 (+169454us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:32.783758 (+    71us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:32.824323 (+ 40565us) log.cc:842] Serialized 26 byte log entry
0809 23:21:32.930895 (+106572us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:32.969494 (+ 38599us) log.cc:842] Serialized 14 byte log entry
0809 23:21:32.975093 (+  5599us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:32.975169 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:33.733440 (+758271us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:33.736559 (+  3119us) raft_consensus.cc:1671] finished
0809 23:21:33.736651 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:33.736771 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12088,"prepare.queue_time_us":36440,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":119983,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":49766,"spinlock_wait_cycles":58661248,"thread_start_us":209347,"threads_started":2,"wal-append.queue_time_us":158296}
I20260809 23:21:33.690078  3218 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.742688 25366 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:33.743011  3218 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.745745  3218 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.747879  3411 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.749730 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.750342 24312 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 16.
I20260809 23:21:33.751185  3462 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.751271 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.752023  3462 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.752612 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.753077 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.754211 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:33.754455 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:33.755309 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.755792 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.755836 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.756453 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.757042 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.757226 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.757531 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.758284 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.758450 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.759099  3405 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:33.762079  3432 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.000s
I20260809 23:21:33.766079 24317 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:33.765597  3428 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.767740  2947 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:33.718720 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1696) took 2125 ms. Trace:
I20260809 23:21:33.768270  2947 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:33.768607 25370 rpcz_store.cc:276] 0809 23:21:31.592987 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.837771 (+244784us) service_pool.cc:227] Handling call
0809 23:21:31.853961 (+ 16190us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:32.627906 (+773945us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:33.718487 (+1090581us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:33.718660 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1746997120}
I20260809 23:21:33.769794 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.769402  2947 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.759117 24540 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:33.758783 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.770892 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.772895 24542 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:33.773659 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3134) took 1234 ms. Trace:
I20260809 23:21:33.792590  3315 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:33.774122 24311 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:33.793184  3315 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:33.794956 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:33.795897 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.796551 24544 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:33.777810 24312 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:33.797509 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.798600 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.798844 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.703670  3377 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.800266 24545 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:33.800437  3377 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.801510 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 57) took 1681 ms. Trace:
I20260809 23:21:33.801882 24316 rpcz_store.cc:276] 0809 23:21:32.120262 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.121358 (+  1096us) service_pool.cc:227] Handling call
0809 23:21:33.801329 (+1679971us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:33.801429 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3586332544}
I20260809 23:21:33.772644 24320 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:33.705469  3333 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:33.803606  3333 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:33.803929 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.804136  3333 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:33.805069 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:33.806413 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.806908 24320 raft_consensus.cc:2362] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20. Re-sending same reply.
I20260809 23:21:33.775343 24543 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:33.780422 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.808606 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.791801 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.771610 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.791831 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.812109 24542 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:33.792374 24427 rpcz_store.cc:276] 0809 23:21:32.539656 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.540174 (+   518us) service_pool.cc:227] Handling call
0809 23:21:33.773445 (+1233271us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54003b9a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:33.773582 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1867547392}
I20260809 23:21:33.774341 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.814215 24545 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:33.779426 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:33.814998 24498 connection.cc:232] RPC call timeout handler was delayed by -1.4158814190886915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260809 23:21:33.818029 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.818274 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.810221 25370 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:33.786727 24321 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
W20260809 23:21:33.821354 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 119049us
I20260809 23:21:33.782298 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:33.825762 24311 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:33.826859 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.827723 25373 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:33.818892 25369 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:33.762683  3432 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:33.832702 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.836129 24321 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:33.833026 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.834672 25369 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [14.13-16.14]
I20260809 23:21:33.838376 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.850368 24321 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:33.851318 25371 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:33.838388  2947 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:33.856726 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.860046 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.861047 24320 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:21:33.863131  3368 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.451s	user 0.003s	sys 0.000s
I20260809 23:21:33.787242  3304 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:33.864281 25371 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:33.779240  3417 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260809 23:21:33.831727  2741 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:33.875445 24312 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [14.13-16.14]
W20260809 23:21:33.784864 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 60d5c69728134c8bb918a900eb6a8c21 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 21 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:33.837723  3207 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
W20260809 23:21:33.876958 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 92259.5us
I20260809 23:21:33.880288 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3123) took 2037 ms. Trace:
I20260809 23:21:33.880699 24424 rpcz_store.cc:276] 0809 23:21:31.843293 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.843412 (+   119us) service_pool.cc:227] Handling call
0809 23:21:31.854110 (+ 10698us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:31.871526 (+ 17416us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:31.871558 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:31.912828 (+ 41270us) log.cc:842] Serialized 26 byte log entry
0809 23:21:32.053971 (+141143us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:32.054006 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:32.054074 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:33.791063 (+1736989us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:33.871018 (+ 79955us) raft_consensus.cc:1671] finished
0809 23:21:33.871048 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:33.880218 (+  9170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":811,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":22125,"spinlock_wait_cycles":8449792,"thread_start_us":53886,"threads_started":1,"wal-append.queue_time_us":299926}
I20260809 23:21:33.866451  2881 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:33.882788  2881 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.883646  3417 consensus_queue.cc:1225] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260809 23:21:33.884824  2881 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.865630  2741 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.887302  2741 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.852003  3398 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.889931 25368 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:33.891927 24312 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:33.893982 24424 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260809 23:21:33.852842  3454 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.409s	user 0.000s	sys 0.004s
I20260809 23:21:33.903718 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.904803 24424 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
W20260809 23:21:33.921312 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89759us
I20260809 23:21:33.922586 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:33.924806  3266 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:33.925189  3266 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:33.925550  3266 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:33.926088  3350 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:21:33.928485  3429 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.743s	user 0.000s	sys 0.004s
I20260809 23:21:33.925405 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:33.928964  3338 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:33.929515  3338 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:33.900580  3308 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.930963  3308 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.932292  3347 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:33.933143  3308 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.933246  3347 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:33.935789  3421 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.664s	user 0.011s	sys 0.002s
I20260809 23:21:33.936223  3421 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:33.937347 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:33.937414 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.938066 24428 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:33.938310  3421 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:33.939338  3411 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:33.938812  2881 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:33.939924  3411 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.940728 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.940965 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.941272 24316 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:33.941846 24422 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:21:33.941881  3411 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.943286 24321 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 21. Ops: []
I20260809 23:21:33.944011  3315 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.945863  3315 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:33.946445 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.947234 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:33.897773  3379 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:33.950105 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.949872  3379 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:33.952840  3379 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.937533 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:33.955189 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.955555 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.956538 24428 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:33.956363  3455 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:33.956915 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:33.957909 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:33.959357  3304 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:33.958081 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:33.962199 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.962548 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:33.963572 24321 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:33.965224 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.918076 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.968911 25368 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:33.974431 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.975260 25370 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:33.979866 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.984959 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.985766 24542 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:33.987629 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.987931 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:33.898874  3377 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:33.911545 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.991606 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.992420 24424 raft_consensus.cc:2396] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260809 23:21:33.992332 24428 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:33.971261  2995 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:34.014348 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 185588us
I20260809 23:21:33.960155  3304 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.013s
I20260809 23:21:33.972688  3350 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:33.981436  3153 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:34.056381 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.930562  3338 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.072206 24543 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:33.990368 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.010185 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:34.075721 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76865.4us
I20260809 23:21:34.076020 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:34.008291 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:34.007016 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 257871us
I20260809 23:21:34.011946 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.971778  3164 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:34.081081 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:33.981204  3412 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260809 23:21:33.973302  2884 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:34.045449 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:33.979305  3432 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:33.978458  3242 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260809 23:21:34.081892 24429 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:33.966507  3411 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:34.082835 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.088979 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:33.988644 25368 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:34.079810 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:34.047816  3350 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:34.096256 24429 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:34.093113 24539 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260809 23:21:34.095149 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.099903 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:34.058818  3417 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:34.074275  3218 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.068869  3104 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.125517 24535 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:21:34.105007 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.091787  3338 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:34.088056  2884 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:34.117887 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.085721  3164 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:34.103546 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 113290us
I20260809 23:21:34.129634 25363 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:34.138295 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:34.147118 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.154794 24535 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.101042  3207 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:34.102396  3350 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:34.091260  3397 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:34.096155  3472 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:34.106300  3401 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:34.189262  3401 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.189654  3401 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:34.190308  3423 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.000s
I20260809 23:21:34.190582 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40854 (request call id 1292) took 2189 ms. Trace:
I20260809 23:21:34.190992 24323 rpcz_store.cc:276] 0809 23:21:32.001570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.028992 (+ 27422us) service_pool.cc:227] Handling call
0809 23:21:34.190132 (+2161140us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:34.190491 (+   359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3780406016}
I20260809 23:21:34.064534  3243 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260809 23:21:34.152143 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.153827 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:34.158757 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.105803  3330 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:34.109153  3448 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:34.168613  3242 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:34.105568  2881 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:34.101840  3428 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.169832 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.110783  3379 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:34.228309 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.190308  3396 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.170050  3208 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.180483 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:34.181885  3482 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260809 23:21:34.169775  3397 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:34.139231  3218 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.213902  3448 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.124619  3418 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
W20260809 23:21:34.186400  3263 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.725s	user 0.010s	sys 0.001s
I20260809 23:21:34.276568  3418 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:34.174966  3472 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.276983  3418 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:34.277340  3472 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.267035  3208 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.197924 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44201.5us
I20260809 23:21:34.187578  3347 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.279194 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.279333  3208 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.102708  3411 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.280951 24427 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260809 23:21:34.281693  3411 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.159461  3164 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:34.209128 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62083.5us
I20260809 23:21:34.285589  2820 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.285928 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.166980 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.287000 24543 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:34.288348 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:34.290416 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.099123  3431 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.124953  3253 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260809 23:21:34.292930  3253 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 17.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:34.293021  3232 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:34.293292 25365 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260809 23:21:34.293682  3253 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:34.294034  3253 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:34.294728 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1754) took 1381 ms. Trace:
I20260809 23:21:34.295172 24546 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260809 23:21:34.295598  3153 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:34.296227  3153 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.203150 24539 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:34.169019  3234 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:34.191022  3473 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:34.251924 24423 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:34.169149  3207 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.299872  3207 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:34.300420 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.258076 24534 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.301215 24423 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.256944 25368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:34.302170 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:34.302460 25368 raft_consensus.cc:595] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:34.304695  2993 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:34.304955 24427 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:34.209657  3377 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:34.306430  3377 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:34.306983  3377 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.183120  3453 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.009s
W20260809 23:21:34.308533 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 2938) took 2623 ms (client timeout 1503 ms). Trace:
I20260809 23:21:34.172668  2813 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
W20260809 23:21:34.309027 24309 rpcz_store.cc:269] 0809 23:21:31.685183 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:31.685296 (+   113us) service_pool.cc:227] Handling call
0809 23:21:34.308319 (+2623023us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b540019c3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:34.308450 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4408732416}
I20260809 23:21:34.143692  3104 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.215690  3242 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.313874  3242 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.315594 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.316955 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.317813 25368 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 18: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:34.274333 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.264480  3396 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.275511  3448 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.276232  3263 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.259915  2881 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.212515  3259 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:34.323375  3259 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.325651  3263 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:34.295135 25366 rpcz_store.cc:276] 0809 23:21:32.913954 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.946883 (+ 32929us) service_pool.cc:227] Handling call
0809 23:21:34.294579 (+1347696us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:34.294674 (+    95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1756643712}
I20260809 23:21:34.190730  3423 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.254045 24535 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:34.327702 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:34.335120 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44800.5us
I20260809 23:21:34.337383 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.339365 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:34.339610 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 148515us
I20260809 23:21:34.264698  3379 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:34.354907 24430 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:21:34.352347 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:34.360857 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.268051  3397 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:34.374135 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:34.213018  3330 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.380498 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:34.382019 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:21:34.384428 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:34.385252 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:34.381230 24321 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:34.396960 24321 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:34.331239  2993 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:34.350304  3244 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:34.406724 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:34.354516  3109 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:34.374430  3465 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
W20260809 23:21:34.416355 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55463.2us
I20260809 23:21:34.369493  3263 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:34.444689 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70658.4us
I20260809 23:21:34.262418  2884 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.422987 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:34.419641  2993 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:34.461012 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77792.2us
I20260809 23:21:34.413866  3411 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:34.460029 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.435401 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.414522  3104 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:34.435074  2625 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:34.483930 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.394107  3232 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260809 23:21:34.294847  3253 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.461074  3432 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:34.424355  3154 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260809 23:21:34.431069  3442 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:34.447599  3263 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:34.456349  2809 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:34.510306 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:34.517907 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.011s	sys 0.008s
W20260809 23:21:34.495093  2947 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.622s	user 0.006s	sys 0.006s
I20260809 23:21:34.430708  3445 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260809 23:21:34.479516  3455 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.468854  3431 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:34.536715 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.011s	sys 0.008s
I20260809 23:21:34.550182 24540 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:34.527475 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 424748us
I20260809 23:21:34.542385 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.486992  2625 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.480159  3462 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.485001  3396 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:34.555004  3505 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260809 23:21:34.535892  3263 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:34.541848  3436 consensus_queue.cc:1159] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 17 status { last_received { term: 13 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: [14.13-16.14]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:34.567719 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.568109  3350 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.548666  3442 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:34.570379  3333 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.766s	user 0.011s	sys 0.000s
I20260809 23:21:34.574707 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.575579  3330 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:34.575809  3432 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.576831  3493 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:21:34.522527  2884 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:34.526255  3239 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.812s	user 0.012s	sys 0.001s
I20260809 23:21:34.517819  2881 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.631171 24425 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 17: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:34.576514  3164 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.525879  3153 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.568933  3424 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:34.594129 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.588699  3154 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:34.600244 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.553036  3253 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:34.553741  2809 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:34.594205  3347 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260809 23:21:34.615566 24423 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:34.619376 24313 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260809 23:21:34.661710 24313 pending_rounds.cc:76] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 12
I20260809 23:21:34.661973 24313 pending_rounds.cc:97] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260809 23:21:34.662294 24313 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:34.626098 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.607849 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.577828  3259 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:34.588272  2625 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:34.457722  3418 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.635633 24544 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:34.644025 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50023.9us
I20260809 23:21:34.602684  3218 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.517609  3379 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.677757 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69985.4us
I20260809 23:21:34.652505  3417 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:34.652949  2881 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:34.653614 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.032s	sys 0.027s
I20260809 23:21:34.619380  2726 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:34.695235 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.032s	sys 0.027s
I20260809 23:21:34.653609  3154 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:34.665901 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:34.645109  3315 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.695s	user 0.013s	sys 0.001s
I20260809 23:21:34.578258  3455 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.708801  3455 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.663632 25373 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:34.664705  3369 raft_consensus.cc:2741] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260809 23:21:34.727794 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.692618 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.728488 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1757) took 1786 ms. Trace:
I20260809 23:21:34.728637 25371 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:34.728847 25375 rpcz_store.cc:276] 0809 23:21:32.942087 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.942261 (+   174us) service_pool.cc:227] Handling call
0809 23:21:32.969693 (+ 27432us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:32.975480 (+  5787us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:32.975509 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:33.804806 (+829297us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:33.805159 (+   353us) log.cc:842] Serialized 26 byte log entry
0809 23:21:34.055708 (+250549us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:34.055760 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.056675 (+   915us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:34.713466 (+656791us) raft_consensus.cc:1671] finished
0809 23:21:34.713494 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:34.728424 (+ 14930us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13301,"raft.queue_time_us":7993,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4165,"spinlock_wait_cycles":3019008,"thread_start_us":250148,"threads_started":1,"wal-append.queue_time_us":638147}
I20260809 23:21:34.731074 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.692796 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.732708 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.734633  2947 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:34.735275  2947 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:34.737278  3164 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.695652  3417 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:34.743191 24427 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:34.620198  3333 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.744236  3377 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.745036 25370 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:34.745877  3333 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:34.746158  3377 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.746536 24430 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:34.748329 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:34.695901 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 613165us
I20260809 23:21:34.466146 24534 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:34.644706  3423 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.662921  2992 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.710839  3350 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:34.679929 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.752283 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.712946 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.717769  3300 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:34.721263  2995 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.000s
I20260809 23:21:34.738592 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.727574  2809 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.679112  3259 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:34.703262  2816 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.753901 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:34.751490  2992 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.721771  3154 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.755035 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.756543 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:34.699632  3338 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.547s	user 0.007s	sys 0.004s
I20260809 23:21:34.649351 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:34.756139 24425 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:34.758778 24422 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:34.774303 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.775694 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.778481 24431 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:34.778997 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.755613  2995 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:34.800974 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 365654us
W20260809 23:21:34.777302  3515 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
W20260809 23:21:34.714712 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 116122us
I20260809 23:21:34.803678 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.784264 24425 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:34.805927 25369 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.809741 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:34.810547 25373 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:34.755859  3305 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260809 23:21:34.773676  3164 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:34.769968 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:34.756997 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44150.7us
I20260809 23:21:34.807215  3473 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:34.825982 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1763) took 1980 ms. Trace:
I20260809 23:21:34.782214  3393 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:34.764727  3333 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:34.755318  3300 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.755679  3244 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260809 23:21:34.782660  3259 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:34.817085 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.770869  2816 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.830794 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.773767  3407 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:34.770243 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.765969 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:34.829681 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59813us
W20260809 23:21:34.772373  3266 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.846s	user 0.014s	sys 0.000s
I20260809 23:21:34.801976  3207 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.834107 24313 rpcz_store.cc:276] 0809 23:21:32.845875 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.864607 (+ 18732us) service_pool.cc:227] Handling call
0809 23:21:32.882807 (+ 18200us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:34.662733 (+1779926us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.825716 (+162983us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400224f54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:34.825919 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3992925440}
I20260809 23:21:34.837313  3396 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:34.807384  3243 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:34.838671  3526 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260809 23:21:34.840639  2993 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.901525  3472 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.001s
I20260809 23:21:34.901969  3472 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.844013  3349 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:34.903168  3349 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:34.903539  3349 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.842587  3263 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.907426  3339 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:34.907871  3339 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:34.908625  3263 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:34.908917  3503 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.390s	user 0.004s	sys 0.000s
I20260809 23:21:34.778311  3397 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.829201  3521 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260809 23:21:34.908717  3339 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.910140  3339 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:34.812081  3154 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.856238 24534 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:34.923194 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:34.923995 25375 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:34.925088 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:34.927031 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 156899us
W20260809 23:21:34.927951 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 97279.5us
I20260809 23:21:34.928769 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:34.859258 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:34.851203  2947 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:34.899914 24540 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:34.902279  2993 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.910703 24435 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:34.911463 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:34.911789 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3490) took 1166 ms. Trace:
I20260809 23:21:34.913440  3417 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.915108  3207 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.915642 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:34.983342 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54668.3us
I20260809 23:21:34.983582 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:34.915858  3448 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.593s	user 0.005s	sys 0.005s
W20260809 23:21:34.758991 24429 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.001s	sys 0.014s
I20260809 23:21:34.910985 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:34.984561 24429 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.001s	sys 0.015s
W20260809 23:21:34.828111  3401 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.000s
I20260809 23:21:34.849443 24425 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
W20260809 23:21:34.920928 24311 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.010s	sys 0.011s
I20260809 23:21:34.984920 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:34.985163 24311 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.010s	sys 0.011s
I20260809 23:21:34.985643 24546 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:34.910871  3397 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:34.942054 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 133321us
I20260809 23:21:34.995653 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2952) took 1212 ms. Trace:
I20260809 23:21:34.977703 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:34.996433 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:34.998039 24546 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:34.986104 24314 rpcz_store.cc:276] 0809 23:21:33.745236 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:33.767164 (+ 21928us) service_pool.cc:227] Handling call
0809 23:21:33.767528 (+   364us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:33.891444 (+123916us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:33.891734 (+   290us) log.cc:842] Serialized 14 byte log entry
0809 23:21:34.577598 (+685864us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:34.596758 (+ 19160us) log.cc:842] Serialized 52 byte log entry
0809 23:21:34.596988 (+   230us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:34.623974 (+ 26986us) log.cc:842] Serialized 14 byte log entry
0809 23:21:34.694805 (+ 70831us) log.cc:842] Serialized 14 byte log entry
0809 23:21:34.717940 (+ 23135us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.718045 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:34.823506 (+105461us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b18003671e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:34.911552 (+ 88046us) raft_consensus.cc:1671] finished
0809 23:21:34.911591 (+    39us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:34.911732 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":249594,"prepare.queue_time_us":356957,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":225969,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":22986,"spinlock_wait_cycles":5988096,"thread_start_us":658623,"threads_started":1,"wal-append.queue_time_us":770698}
I20260809 23:21:35.002887 24425 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:35.004232 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.005486 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:35.005642  3472 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.006108  3469 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:35.007786  2816 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:35.009020  3242 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.695s	user 0.006s	sys 0.006s
I20260809 23:21:35.009438  3242 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.010336  3442 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:34.977916  3338 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:34.993759  3232 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260809 23:21:34.847815  3164 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:35.011762 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.012217  3417 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:35.012398 24427 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:35.012703 25366 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:34.978493  3333 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:35.013793 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.014401  3423 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:35.014596 24539 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260809 23:21:35.015095 24539 pending_rounds.cc:76] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 13
I20260809 23:21:35.015329 24539 pending_rounds.cc:97] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260809 23:21:34.889384  3473 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.015998 25368 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:35.013815  3350 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260809 23:21:34.994171  3401 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.898162  3266 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.019439  3401 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:34.980353 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.019848  3333 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:35.020524  3266 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.020992 24534 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260809 23:21:35.021281 24424 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:34.920549  3243 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.023177 24316 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260809 23:21:34.980856 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:34.847285  3393 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:35.023597  3243 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:35.023990  3393 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:34.982079  3330 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260809 23:21:34.983443  2947 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.025913  2625 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:34.990764 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64253.3us
I20260809 23:21:34.989812 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.991006 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.027356 24312 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:35.028306 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.029469 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.030249  3315 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:35.030544 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.015556 24539 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:35.017726  3473 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.026427  2947 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.002308  3377 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260809 23:21:34.943656 25372 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260809 23:21:35.030745  3411 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:35.074715  3411 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:35.075160  3411 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:34.940276 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.014879 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.012457  3164 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.031924 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:35.001029  3448 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.002038  2881 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:34.979360 25369 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:35.033623 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:34.995505  3462 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:35.023512  3242 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.035717  3207 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:21:35.087464  3207 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:35.087625  3377 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:35.087841  3207 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:35.088063  3377 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:35.088721  3207 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.037933  3349 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.042011 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1489) took 1106 ms. Trace:
I20260809 23:21:35.043934 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.043959  3397 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:35.044461 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2956) took 1260 ms. Trace:
I20260809 23:21:35.045135  3396 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Stepping down as leader of term 16
W20260809 23:21:35.045504 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64769.9us
I20260809 23:21:35.107293 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:35.045976  3259 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:35.046543  3330 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:34.981690  3244 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:35.046737  3393 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:34.982836  3305 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.066s
I20260809 23:21:35.048759 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.183531 24425 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:35.048939  3424 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:35.014891  3423 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:35.006222 24540 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:35.049459 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47141.6us
I20260809 23:21:35.051393  2625 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:35.052541 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.194898 24534 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:34.996229 24320 rpcz_store.cc:276] 0809 23:21:33.783120 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:33.861823 (+ 78703us) service_pool.cc:227] Handling call
0809 23:21:33.862242 (+   419us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:33.977617 (+115375us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:33.977663 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:34.011164 (+ 33501us) log.cc:842] Serialized 52 byte log entry
0809 23:21:34.399556 (+388392us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:34.889920 (+490364us) log.cc:842] Serialized 14 byte log entry
0809 23:21:34.946901 (+ 56981us) log.cc:842] Serialized 14 byte log entry
0809 23:21:34.995360 (+ 48459us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.995433 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:34.995475 (+    42us) raft_consensus.cc:1671] finished
0809 23:21:34.995485 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:34.995603 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":165640,"prepare.queue_time_us":203958,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":151697,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":13351,"thread_start_us":739461,"threads_started":2,"wal-append.queue_time_us":587586}
I20260809 23:21:35.058863 24546 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:35.059950 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:21:35.191496 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84300.7us
W20260809 23:21:35.060097 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75263.3us
I20260809 23:21:35.070531 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.070895 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:35.076653 25372 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 11
I20260809 23:21:35.080988 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:35.197474 25372 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260809 23:21:35.081465 24317 raft_consensus.cc:2354] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:35.197754 25372 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:35.076840 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.085862  3462 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.088816  3377 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:34.952252 25373 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:35.097688  3207 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:35.101155 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:35.099300 24431 rpcz_store.cc:276] 0809 23:21:33.935711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:33.935870 (+   159us) service_pool.cc:227] Handling call
0809 23:21:33.936206 (+   336us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:34.779080 (+842874us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.041773 (+262693us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.041949 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1936073472}
I20260809 23:21:35.068416  2993 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:35.074163  3469 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:35.072790 24427 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:35.103292 24311 rpcz_store.cc:276] 0809 23:21:33.788610 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:33.867274 (+ 78664us) service_pool.cc:227] Handling call
0809 23:21:33.885657 (+ 18383us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:33.885975 (+   318us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:33.886004 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:33.886393 (+   389us) log.cc:842] Serialized 26 byte log entry
0809 23:21:34.136628 (+250235us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:34.985573 (+848945us) log.cc:842] Serialized 14 byte log entry
0809 23:21:35.044149 (+ 58576us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.044227 (+    78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:35.044271 (+    44us) raft_consensus.cc:1671] finished
0809 23:21:35.044283 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:35.044406 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58312,"prepare.queue_time_us":434381,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":31,"raft.queue_time_us":60255,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":150,"thread_start_us":1078062,"threads_started":2,"wal-append.queue_time_us":543361}
I20260809 23:21:35.142866 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.105093  3347 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:35.199083 24535 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:21:35.106657  3396 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:35.197775 24313 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:35.143599  3330 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:35.227223 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.216061 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:35.222874 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.200196 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.187722  3244 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.233734 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:35.193998  3423 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:35.184496  3563 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260809 23:21:35.203979  3393 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.187920  3305 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:35.141613  3405 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:35.208149 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 137367us
I20260809 23:21:35.207973 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.260712 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:35.197716 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.141466  3442 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.144164  3473 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:35.222864  3349 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.214694 24425 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:35.272738 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.210742 24498 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.218127 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.217237 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.231209 24546 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:35.274402 24535 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:35.265681 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:35.275939 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.276535 24314 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:35.277822 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:35.255151  3379 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.003s	sys 0.016s
I20260809 23:21:35.242242 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:35.244277 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.115447  3397 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.250416 24534 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:35.252024 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.294661 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.255283 24311 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260809 23:21:35.270433 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:35.279580 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.297147 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.297501 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.218422  3557 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:35.281535 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.298357 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:35.299283 25369 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:35.281814 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71161.5us
I20260809 23:21:35.300612 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.266264  3466 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:35.281234 25373 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:35.253093  3448 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:35.309404 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43829.3us
I20260809 23:21:35.285686  3308 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:35.286206  3396 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.287262 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:35.287727  2881 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:35.245572  3401 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:35.288288 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:35.187024  3432 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.012s	sys 0.019s
I20260809 23:21:35.290999 24320 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:35.292523 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:35.385671 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65767.8us
I20260809 23:21:35.279853 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.235939  2625 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260809 23:21:35.300099 24427 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:35.305155  3556 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:35.288816  3232 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:35.307500 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:35.394564 24534 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:35.395339 24540 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
W20260809 23:21:35.397961  3539 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260809 23:21:35.270682 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:35.399191 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3554) took 1324 ms. Trace:
W20260809 23:21:35.399662 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129052us
I20260809 23:21:35.400457 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:35.393492 24546 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:35.272183  3405 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.311901 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:35.403579 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91746.8us
I20260809 23:21:35.232261  2736 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260809 23:21:35.321079 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:35.291496  3379 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.003s	sys 0.017s
I20260809 23:21:35.314970 25369 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:35.410552  2625 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:35.411173  2625 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:35.412700  3462 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.413743  3472 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:35.412349  2625 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.414165  3472 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:35.414975  3472 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.416316  3472 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:35.420574  3405 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.393435  3349 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:35.421759  3349 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:35.422624  3349 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:35.391011  3432 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.012s	sys 0.019s
I20260809 23:21:35.425887 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:35.426398  2827 consensus_queue.cc:1159] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 22 status { last_received { term: 20 index: 13 } last_committed_idx: 13 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:35.260478  3431 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
I20260809 23:21:35.350327 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.347007  3520 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:35.427604  3431 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:35.427809  3520 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:35.428002  3431 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:35.303953  3557 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:35.428937  3520 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.429690  3557 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:35.430109  3300 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:35.353041  3305 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:35.431196  3305 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.433202  3305 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.354430  3244 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:35.307864  2378 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:35.434075  2378 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:35.434810 25363 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:35.409375  3563 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.435026  2378 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.435768  3563 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.437132 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1709) took 1378 ms. Trace:
I20260809 23:21:35.437844  3563 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.320209  3243 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:35.275164  3469 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260809 23:21:35.362946  3529 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:35.279513  3417 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260809 23:21:35.302129  3473 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:35.372632 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3470) took 1609 ms (client timeout 1996 ms). Trace:
I20260809 23:21:35.376245  2881 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.297165  3330 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.310631  3208 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260809 23:21:35.505414 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64866.9us
I20260809 23:21:35.489427 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:35.442981 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92735.5us
I20260809 23:21:35.444487 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:35.243769  3333 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
W20260809 23:21:35.292316  3421 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.004s	sys 0.018s
I20260809 23:21:35.305325  3455 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260809 23:21:35.384894 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96683.5us
I20260809 23:21:35.399569 24545 rpcz_store.cc:276] 0809 23:21:34.074712 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.074870 (+   158us) service_pool.cc:227] Handling call
0809 23:21:34.101739 (+ 26869us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:34.102060 (+   321us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:34.102089 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:34.104928 (+  2839us) log.cc:842] Serialized 26 byte log entry
0809 23:21:34.606664 (+501736us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:34.606727 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.612042 (+  5315us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:35.398973 (+786931us) raft_consensus.cc:1671] finished
0809 23:21:35.399004 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:35.399139 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":479,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":15425,"thread_start_us":453611,"threads_started":1,"wal-append.queue_time_us":959980}
I20260809 23:21:35.313340 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:35.411123 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.420688  3418 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:35.425380  2992 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.353304  3397 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:35.280593  3442 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:35.308045  3203 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.010s	sys 0.026s
I20260809 23:21:35.439653 24534 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:35.344460 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.441007 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:35.437536 25363 rpcz_store.cc:276] 0809 23:21:34.059085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.279683 (+220598us) service_pool.cc:227] Handling call
0809 23:21:34.279994 (+   311us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.435227 (+1155233us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.436916 (+  1689us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.437077 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2655351552}
W20260809 23:21:35.441608 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54548.7us
I20260809 23:21:35.344309  3259 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:35.508091 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.389401 24314 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:35.624018 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:35.391832  3396 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:35.513696 24544 rpcz_store.cc:269] 0809 23:21:33.763598 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:33.797145 (+ 33547us) service_pool.cc:227] Handling call
0809 23:21:35.372446 (+1575301us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54000131d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:35.372571 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1925681536}
I20260809 23:21:35.631450 24313 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:35.517000 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:35.601820 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.603613 25369 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:35.603571 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:35.564764 24266 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.620s (SENT) [suppressed 10 similar messages]
I20260809 23:21:35.534821 24435 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:35.519161 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1778) took 1374 ms. Trace:
W20260809 23:21:35.523922 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3149) took 1468 ms (client timeout 1620 ms). Trace:
I20260809 23:21:35.403097  2809 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:35.635129 24424 rpcz_store.cc:269] 0809 23:21:34.055909 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.056074 (+   165us) service_pool.cc:227] Handling call
0809 23:21:35.523686 (+1467612us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:35.523854 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2216824064}
I20260809 23:21:35.637316 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:35.637883 25370 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:35.638335 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.525206 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1710) took 1219 ms. Trace:
I20260809 23:21:35.639299 25368 rpcz_store.cc:276] 0809 23:21:34.305364 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.318430 (+ 13066us) service_pool.cc:227] Handling call
0809 23:21:34.330051 (+ 11621us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.016370 (+686319us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.524919 (+508549us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.525135 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1577064192}
I20260809 23:21:35.521322  3242 raft_consensus.cc:979] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
W20260809 23:21:35.611490  3523 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.939s	user 0.000s	sys 0.007s
I20260809 23:21:35.612032  3442 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:35.614208  2881 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:35.612547  3203 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.011s	sys 0.035s
I20260809 23:21:35.517419  2993 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:35.489494  3232 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:35.617156 24534 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:35.517948 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3478) took 1392 ms. Trace:
W20260809 23:21:35.621389 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.018s	sys 0.015s
I20260809 23:21:35.378782  3423 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:35.619585 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 306272us
I20260809 23:21:35.438889  3448 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:35.409950  3426 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:35.644289 25369 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:35.634840 24316 rpcz_store.cc:276] 0809 23:21:34.144818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.144952 (+   134us) service_pool.cc:227] Handling call
0809 23:21:34.171221 (+ 26269us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.023586 (+852365us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.518845 (+495259us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.519095 (+   250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1958647168}
W20260809 23:21:35.644927 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 204000us
I20260809 23:21:35.608110 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.645304 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:35.517413  3347 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:35.645635 24545 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [17.13-18.14]
W20260809 23:21:35.603309  3421 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.006s	sys 0.027s
I20260809 23:21:35.438410  3556 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:35.650609 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.014s	sys 0.021s
I20260809 23:21:35.495971  3350 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260809 23:21:35.629480  3428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.013s	sys 0.021s
I20260809 23:21:35.390908  3401 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.428946  3431 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:35.728137 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.031s	sys 0.029s
W20260809 23:21:35.857204 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.034s	sys 0.031s
W20260809 23:21:35.857903 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.018s	sys 0.026s
W20260809 23:21:35.858178 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 212936us
I20260809 23:21:35.858048 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.858891 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3151) took 1707 ms. Trace:
I20260809 23:21:35.859135 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:35.859344 24429 rpcz_store.cc:276] 0809 23:21:34.151726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.153266 (+  1540us) service_pool.cc:227] Handling call
0809 23:21:34.153562 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:34.153764 (+   202us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:34.985003 (+831239us) log.cc:842] Serialized 14 byte log entry
0809 23:21:35.858433 (+873430us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:35.858484 (+    51us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:35.858526 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.858588 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:35.858701 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":369448,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":3284,"thread_start_us":1680922,"threads_started":2}
I20260809 23:21:35.669051 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3481) took 1481 ms. Trace:
I20260809 23:21:35.857147 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:35.862977 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.019s	sys 0.023s
I20260809 23:21:35.856822 24543 rpcz_store.cc:276] 0809 23:21:34.126007 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.287563 (+161556us) service_pool.cc:227] Handling call
0809 23:21:34.287871 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:34.436130 (+148259us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:34.436187 (+    57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:34.436241 (+    54us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:34.436282 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.517720 (+1081438us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.517770 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:35.517893 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340211584}
I20260809 23:21:35.669375 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1783) took 1247 ms. Trace:
I20260809 23:21:35.867167 24321 rpcz_store.cc:276] 0809 23:21:34.421537 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.422677 (+  1140us) service_pool.cc:227] Handling call
0809 23:21:35.669130 (+1246453us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:35.669303 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2032588672}
I20260809 23:21:35.666658 25370 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:35.873790 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:35.862169 24539 rpcz_store.cc:276] 0809 23:21:34.187943 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.298132 (+110189us) service_pool.cc:227] Handling call
0809 23:21:34.298402 (+   270us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.066822 (+768420us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.668746 (+601924us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:35.668975 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1646354816}
W20260809 23:21:35.880334 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 272247us
I20260809 23:21:35.882146 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:35.635177  2992 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:35.918674 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:35.465540  3339 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.920377 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:35.863610  3428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.018s	sys 0.025s
I20260809 23:21:35.935850 24311 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260809 23:21:35.939985 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.019s	sys 0.037s
W20260809 23:21:35.940780 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.020s	sys 0.037s
I20260809 23:21:35.649533  3156 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.711440  3263 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:35.713881  3243 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:35.668015  3234 raft_consensus.cc:1067] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:35.874977  3442 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.925318  2736 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:35.649931  3338 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:35.829504  3154 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:35.711715  3455 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:35.668028  3379 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.307s
I20260809 23:21:35.906803  3347 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:35.918431  3563 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:35.874881  3436 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:35.917762  2993 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.632493  3259 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.959982 24424 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:35.930888  3426 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:35.971933  2992 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:35.929694  3350 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:35.999539 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1495) took 2005 ms. Trace:
I20260809 23:21:35.736424  3467 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:36.039309 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.010s	user 0.019s	sys 0.061s
W20260809 23:21:36.106556 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s	user 0.019s	sys 0.061s
I20260809 23:21:35.652621  2809 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:35.630698  3529 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:35.984025  3203 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:35.917980  3232 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:36.069403 25365 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.010s	sys 0.023s
I20260809 23:21:36.009882  3243 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:35.959393  3339 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.107954 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:21:36.117877 25365 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.010s	sys 0.023s
I20260809 23:21:36.099462 24428 rpcz_store.cc:276] 0809 23:21:33.994064 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.038980 (+ 44916us) service_pool.cc:227] Handling call
0809 23:21:34.058759 (+ 19779us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:34.074492 (+ 15733us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:34.074524 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:34.074972 (+   448us) log.cc:842] Serialized 26 byte log entry
0809 23:21:34.410222 (+335250us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:34.410266 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.410341 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:35.438063 (+1027722us) spinlock_profiling.cc:243] Waited 8.92 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:35.966438 (+528375us) raft_consensus.cc:1671] finished
0809 23:21:35.966467 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:35.999449 (+ 32982us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":305,"raft.queue_time_us":1036,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":11527,"spinlock_wait_cycles":20524928,"thread_start_us":277245,"threads_started":1,"wal-append.queue_time_us":517760}
I20260809 23:21:36.049018  2736 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:36.085026  3411 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.009s	user 0.009s	sys 0.002s
W20260809 23:21:36.118656 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.010s	sys 0.022s
I20260809 23:21:35.936178  3473 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.000365  3442 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:36.129570 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.010s	sys 0.022s
I20260809 23:21:36.095512  3563 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:36.068351  2947 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260809 23:21:36.057019  3455 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.050453  3154 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.147537  3569 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
I20260809 23:21:36.205569 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:36.177043  3350 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:36.285792 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 983795us
W20260809 23:21:36.286089 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.006s	sys 0.035s
W20260809 23:21:36.132478  3568 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
W20260809 23:21:36.133096  3574 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260809 23:21:36.018909  3556 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.281653  3243 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:36.307555 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.007s	sys 0.035s
W20260809 23:21:36.260744  3472 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.844s	user 0.006s	sys 0.005s
W20260809 23:21:36.190724  3466 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.871s	user 0.006s	sys 0.001s
W20260809 23:21:36.131800  3572 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
W20260809 23:21:36.193547  3207 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.967s	user 0.010s	sys 0.001s
W20260809 23:21:36.196388  3557 consensus_meta.cc:287] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.766s	user 0.007s	sys 0.003s
I20260809 23:21:36.117280  3203 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:35.980367  3156 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.287616 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:36.317787 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:36.367087 24428 raft_consensus.cc:2396] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260809 23:21:36.367352 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49673.6us
I20260809 23:21:36.167869  3239 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:36.113423  3426 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:36.136905  2993 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.176358  3401 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.174290  3339 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.325681  3243 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.327805  3555 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.143s	user 0.000s	sys 0.004s
I20260809 23:21:36.203943  2736 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.164245  2992 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.354077 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:36.323290  3412 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.101959  3338 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.383394 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.036s	sys 0.132s
I20260809 23:21:36.281841  3411 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.113583  3467 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.351850  3242 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:36.283655  3563 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.351775  3369 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:36.311791  3234 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:36.384083  3239 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:36.383885  3349 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.014s	sys 0.023s
I20260809 23:21:36.357878  3466 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:36.384755  2995 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.952s	user 0.011s	sys 0.024s
I20260809 23:21:36.414161 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:36.308225  3350 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:36.445890 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.070s	user 0.037s	sys 0.134s
I20260809 23:21:36.388650  3272 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:36.424054 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1709) took 2113 ms. Trace:
W20260809 23:21:36.432328  3407 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.061s	user 0.023s	sys 0.059s
I20260809 23:21:36.483407 24431 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:36.484844 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1783) took 1661 ms. Trace:
I20260809 23:21:36.502504 25371 rpcz_store.cc:276] 0809 23:21:34.823242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.830949 (+  7707us) service_pool.cc:227] Handling call
0809 23:21:34.894672 (+ 63723us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:35.002139 (+107467us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.002175 (+    36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:35.068931 (+ 66756us) log.cc:842] Serialized 52 byte log entry
0809 23:21:35.941237 (+872306us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:35.941293 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:35.941482 (+   189us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:36.484589 (+543107us) raft_consensus.cc:1671] finished
0809 23:21:36.484616 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:36.484781 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":97701,"raft.queue_time_us":107472,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":835417,"threads_started":1,"wal-append.queue_time_us":1312700}
I20260809 23:21:36.494319  3239 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.499385 25365 rpcz_store.cc:276] 0809 23:21:34.305030 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.320125 (+ 15095us) service_pool.cc:227] Handling call
0809 23:21:34.763645 (+443520us) spinlock_profiling.cc:243] Waited 9.96 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:21:34.763835 (+   190us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:34.912164 (+148329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:34.912197 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:34.912850 (+   653us) log.cc:842] Serialized 26 byte log entry
0809 23:21:35.266850 (+354000us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:36.118251 (+851401us) log.cc:842] Serialized 14 byte log entry
0809 23:21:36.118497 (+   246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:36.418470 (+299973us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:36.418547 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:36.418617 (+    70us) raft_consensus.cc:1671] finished
0809 23:21:36.418636 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:36.418846 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7481,"raft.queue_time_us":26594,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":36175,"spinlock_wait_cycles":313392256,"thread_start_us":1179568,"threads_started":2,"wal-append.queue_time_us":820334}
W20260809 23:21:36.495118  3349 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.070s	user 0.015s	sys 0.023s
W20260809 23:21:36.510691 24423 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.025s	sys 0.078s
I20260809 23:21:36.487797  3242 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.365046  3557 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.484493  3411 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.413460  3423 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.491425  3369 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.494109  3234 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.485636  3412 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.415918  3333 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:36.501056 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:36.498835  2995 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.066s	user 0.011s	sys 0.024s
I20260809 23:21:36.492120  2809 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:36.506991  3407 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.131s	user 0.024s	sys 0.062s
I20260809 23:21:36.429786  3203 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.487810 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.009s	sys 0.024s
W20260809 23:21:36.487953  3164 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.407s	user 0.012s	sys 0.029s
I20260809 23:21:36.498392  3426 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.555003 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.009s	sys 0.029s
W20260809 23:21:36.535797 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.047s	sys 0.056s
I20260809 23:21:36.687999 24428 raft_consensus.cc:2354] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
W20260809 23:21:36.761611 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.282s	user 0.009s	sys 0.033s
W20260809 23:21:36.764501 24266 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.053s	user 0.000s	sys 0.033s
W20260809 23:21:36.764874 24266 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.053s	user 0.000s	sys 0.033s
W20260809 23:21:36.765163 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.20036e+06us
I20260809 23:21:36.544399 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:36.771193 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1510) took 1151 ms. Trace:
W20260809 23:21:36.513643 24423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.025s	sys 0.078s
I20260809 23:21:36.710011 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3484) took 2388 ms. Trace:
W20260809 23:21:36.773468 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.161s	user 0.047s	sys 0.060s
I20260809 23:21:36.719833 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1756) took 3805 ms. Trace:
I20260809 23:21:36.772580 24541 rpcz_store.cc:276] 0809 23:21:34.311386 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.316821 (+  5435us) service_pool.cc:227] Handling call
0809 23:21:34.317086 (+   265us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:34.350214 (+ 33128us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:34.350244 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:34.415286 (+ 65042us) log.cc:842] Serialized 26 byte log entry
0809 23:21:34.849077 (+433791us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:34.849117 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:34.849184 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:36.327220 (+1478036us) spinlock_profiling.cc:243] Waited 6.77 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:36.688840 (+361620us) raft_consensus.cc:1671] finished
0809 23:21:36.688885 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:36.699344 (+ 10459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10584,"raft.queue_time_us":5209,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":19420,"spinlock_wait_cycles":15578880,"thread_start_us":381841,"threads_started":1,"wal-append.queue_time_us":650510}
W20260809 23:21:36.775810 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.392s	user 0.034s	sys 0.029s
W20260809 23:21:36.779739 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.396s	user 0.034s	sys 0.029s
W20260809 23:21:36.761109 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.011s	sys 0.036s
W20260809 23:21:36.788986 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 861744us
I20260809 23:21:36.789860 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:36.555330 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.014s	sys 0.014s
I20260809 23:21:36.775214 25372 rpcz_store.cc:276] 0809 23:21:32.914699 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:32.914835 (+   136us) service_pool.cc:227] Handling call
0809 23:21:32.915199 (+   364us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.198658 (+2283459us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:36.719511 (+1520853us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b54006004d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:36.719752 (+   241us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4664166144}
W20260809 23:21:36.508617  2378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.034s	sys 0.043s
W20260809 23:21:36.794091 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.149s	user 0.019s	sys 0.018s
I20260809 23:21:36.771703 24435 rpcz_store.cc:276] 0809 23:21:35.619624 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:36.485844 (+866220us) service_pool.cc:227] Handling call
0809 23:21:36.771112 (+285268us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:36.752398  3396 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.075s	user 0.013s	sys 0.000s
W20260809 23:21:36.760375  3164 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.679s	user 0.014s	sys 0.035s
I20260809 23:21:36.512769  3239 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:36.795297 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.19171e+06us
I20260809 23:21:36.800413 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:36.511691  3272 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:36.513926  3418 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.901s	user 0.013s	sys 0.000s
W20260809 23:21:36.539537  3520 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.049s	user 0.009s	sys 0.065s
I20260809 23:21:36.752636  2993 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.532068  3338 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.518056  3529 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.875881  2993 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:36.876816  2827 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:36.877341  2827 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.517257  3154 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.543828  3339 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.672269  3411 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.801596 24498 leader_election.cc:398] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:36.904744 24498 leader_election.cc:401] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
W20260809 23:21:36.804585 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.885s	user 0.022s	sys 0.032s
I20260809 23:21:36.701797  3156 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.814019 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:36.813266 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:36.862291  3467 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.813620 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:36.811591 25368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.005s	sys 0.027s
W20260809 23:21:36.910395 25368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.147s	user 0.005s	sys 0.034s
I20260809 23:21:36.751729  3349 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:36.515113  3259 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:36.798521  2378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.036s	sys 0.056s
I20260809 23:21:36.866020 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:36.796382  3207 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:36.774369  3466 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:36.798481  3423 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.798121  3203 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:36.870088 24535 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:36.878674  2993 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:36.865975  3520 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.375s	user 0.014s	sys 0.086s
W20260809 23:21:36.902997 25375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.969s	user 0.003s	sys 0.056s
W20260809 23:21:36.967434 25375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.003s	sys 0.056s
I20260809 23:21:36.854084  3273 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:36.853924 24544 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:36.926915  3259 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.909832  3339 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.913535  3411 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.902686 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.983s	user 0.008s	sys 0.029s
W20260809 23:21:36.996763 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.077s	user 0.008s	sys 0.029s
W20260809 23:21:36.997136 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.11502e+06us
I20260809 23:21:36.671887  3333 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:36.876850  3579 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
I20260809 23:21:36.891041  3431 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:36.880847  3570 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.489s	user 0.001s	sys 0.001s
I20260809 23:21:36.869452  3164 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:36.917500  3412 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.921247 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3515) took 1649 ms. Trace:
I20260809 23:21:36.970839  3242 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:36.925423  3349 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.923700 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3516) took 1651 ms. Trace:
I20260809 23:21:36.881783  3377 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:37.017920  3607 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:36.893520  3396 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:37.018483  3607 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.889495  2827 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.973272  3557 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:36.954403 25366 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.970s	user 0.015s	sys 0.029s
I20260809 23:21:36.915556  3467 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:37.025852  3467 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.027999  3467 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.033095 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3158) took 1626 ms. Trace:
I20260809 23:21:37.033607 24422 rpcz_store.cc:276] 0809 23:21:35.406230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.409308 (+  3078us) service_pool.cc:227] Handling call
0809 23:21:35.492284 (+ 82976us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.492553 (+   269us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.492588 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.513454 (+ 20866us) log.cc:842] Serialized 26 byte log entry
0809 23:21:36.094115 (+580661us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:36.094172 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:36.094265 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:37.032813 (+938548us) raft_consensus.cc:1671] finished
0809 23:21:37.032847 (+    34us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.033023 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":474778,"threads_started":1,"wal-append.queue_time_us":1363076}
I20260809 23:21:37.035163  3333 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.983772  3369 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.035938  3234 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:37.036485  3393 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.979887  3259 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.036767 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.037019  3393 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.038645  3259 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:36.908898  3253 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.039183  3259 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:36.966053  3423 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:36.976912  3207 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:36.905694  3583 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.038311 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:36.905020  3154 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.959050 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:36.959966 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:36.810452  3243 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:36.811916  2809 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:36.907357  3418 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:36.914398  3350 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.969514 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:36.968489  3529 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:37.072561  3529 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.072783  2827 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:37.073421  2827 consensus_queue.cc:1225] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 14.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:37.024415  3164 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.077455  3607 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:37.071108  3583 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.078549  3164 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:36.970263  3273 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.080039  3583 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:37.080415  3273 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:36.906468 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 104985us
I20260809 23:21:36.987596  3232 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:36.971765  3593 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.081694  3593 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.084653  3593 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:36.991938  3455 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:36.989549 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3162) took 1571 ms (client timeout 1510 ms). Trace:
W20260809 23:21:37.085492 24424 rpcz_store.cc:269] 0809 23:21:35.417965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.637904 (+219939us) service_pool.cc:227] Handling call
0809 23:21:36.989282 (+1351378us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b5400062554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:36.989461 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":718202112}
I20260809 23:21:37.086728 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.036723  3442 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:37.087378 24424 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:37.087893 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1507) took 1680 ms. Trace:
I20260809 23:21:37.088217 24424 rpcz_store.cc:276] 0809 23:21:35.407658 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.086435 (+1678777us) service_pool.cc:227] Handling call
0809 23:21:37.087827 (+  1392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.089041 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.089315  3418 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.090175 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.091341  3418 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.091763  2809 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.094017  2809 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.095502 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:37.066540  3393 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.096642 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:37.098069 24544 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:37.098753  3411 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:37.099812  3448 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.193s	user 0.014s	sys 0.000s
I20260809 23:21:37.100157  3423 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:37.100288  3448 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:37.101099 24317 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:37.012208 24323 rpcz_store.cc:276] 0809 23:21:35.272188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.292100 (+ 19912us) service_pool.cc:227] Handling call
0809 23:21:35.292378 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.621536 (+329158us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.621581 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:35.621636 (+    55us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:35.621680 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:36.923250 (+1301570us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:36.923485 (+   235us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:36.923638 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756398976}
I20260809 23:21:37.101866  3349 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:37.009474 24312 rpcz_store.cc:276] 0809 23:21:35.271688 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.284437 (+ 12749us) service_pool.cc:227] Handling call
0809 23:21:35.284738 (+   301us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:35.295256 (+ 10518us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.295284 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:35.295786 (+   502us) log.cc:842] Serialized 52 byte log entry
0809 23:21:36.323485 (+1027699us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:36.327656 (+  4171us) log.cc:842] Serialized 14 byte log entry
0809 23:21:36.328150 (+   494us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:36.328248 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:36.920953 (+592705us) raft_consensus.cc:1671] finished
0809 23:21:36.920988 (+    35us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:36.921167 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10095,"prepare.queue_time_us":3616,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":58,"raft.queue_time_us":11461,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":5150,"thread_start_us":974068,"threads_started":1,"wal-append.queue_time_us":1505831}
I20260809 23:21:36.924924 24435 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:37.024221 25366 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.051s	user 0.016s	sys 0.030s
I20260809 23:21:37.104352 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.104780 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1712) took 2116 ms. Trace:
I20260809 23:21:37.105274 25366 rpcz_store.cc:276] 0809 23:21:34.988468 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.077617 (+ 89149us) service_pool.cc:227] Handling call
0809 23:21:35.077971 (+   354us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.078345 (+   374us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.078378 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.078833 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:21:35.871323 (+792490us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:37.104131 (+1232808us) log.cc:842] Serialized 14 byte log entry
0809 23:21:37.104398 (+   267us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.104489 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:37.104546 (+    57us) raft_consensus.cc:1671] finished
0809 23:21:37.104562 (+    16us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.104714 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57492,"prepare.queue_time_us":890559,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":3232,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":6827,"thread_start_us":1914758,"threads_started":2,"wal-append.queue_time_us":1029239}
W20260809 23:21:37.107100  3579 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260809 23:21:37.105480 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.108026 24435 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:37.108940 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:37.065789 24430 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.024s	sys 0.017s
W20260809 23:21:37.124734 24430 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.024s	sys 0.017s
I20260809 23:21:36.967595  3466 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.110348 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.112181 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:37.039738  3259 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.068964  3253 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:36.907199 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.023s	sys 0.033s
W20260809 23:21:37.145403 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.28623e+06us
I20260809 23:21:37.115432  3154 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:37.116045  3347 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.023s	user 0.012s	sys 0.000s
I20260809 23:21:36.973977  3272 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.019253  3396 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.118292 24544 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:46304
I20260809 23:21:37.157645 24544 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:21:37.157967 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.068454 24539 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:37.160995 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.157991 24544 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:37.090737 24424 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:37.114861  3339 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:37.151446  3573 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.016s	user 0.027s	sys 0.019s
I20260809 23:21:37.153419 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1502) took 2131 ms. Trace:
I20260809 23:21:37.191336 24430 rpcz_store.cc:276] 0809 23:21:35.022313 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.027525 (+  5212us) service_pool.cc:227] Handling call
0809 23:21:35.027853 (+   328us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.028216 (+   363us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.028244 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.028655 (+   411us) log.cc:842] Serialized 26 byte log entry
0809 23:21:36.107068 (+1078413us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:37.143242 (+1036174us) log.cc:842] Serialized 14 byte log entry
0809 23:21:37.152957 (+  9715us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.153107 (+   150us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:37.153165 (+    58us) raft_consensus.cc:1671] finished
0809 23:21:37.153182 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.153341 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":909906,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":12456,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":3911,"thread_start_us":2080522,"threads_started":2,"wal-append.queue_time_us":1940165}
I20260809 23:21:37.167382 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.168627 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.169493 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.147404 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:37.195894 24312 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:37.200170 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:37.194865 24422 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
W20260809 23:21:37.197407 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.276s	user 0.012s	sys 0.041s
W20260809 23:21:37.196666 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49369us
W20260809 23:21:37.201841 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.281s	user 0.012s	sys 0.041s
I20260809 23:21:37.202246 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.201380 25372 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:37.196820 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.034s	user 0.027s	sys 0.017s
W20260809 23:21:37.207576 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.044s	user 0.027s	sys 0.017s
I20260809 23:21:37.152011  3232 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:37.213534 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3167) took 1793 ms. Trace:
W20260809 23:21:37.214318 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.023s	sys 0.033s
I20260809 23:21:37.155512  3218 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:37.191267  3573 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.027s	sys 0.019s
W20260809 23:21:37.336123 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 133025us
I20260809 23:21:37.192731  3600 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.352191 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:37.339522 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.354995 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:37.355716 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.704s	user 0.015s	sys 0.035s
I20260809 23:21:37.243752  3207 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:37.162611  3243 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260809 23:21:37.236245  3577 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260809 23:21:37.345177 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s	user 0.023s	sys 0.034s
I20260809 23:21:37.345715 24422 rpcz_store.cc:276] 0809 23:21:35.419622 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.144900 (+1725278us) service_pool.cc:227] Handling call
0809 23:21:37.213452 (+ 68552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.346678 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1508) took 1938 ms. Trace:
I20260809 23:21:37.369273 24424 rpcz_store.cc:276] 0809 23:21:35.407963 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.165149 (+1757186us) service_pool.cc:227] Handling call
0809 23:21:37.165494 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.344231 (+178737us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.344276 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.344324 (+    48us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:37.346358 (+  2034us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.346449 (+    91us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.346607 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34033,"raft.queue_time_us":59085,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":77076}
I20260809 23:21:37.347811 24435 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260809 23:21:37.348453 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3164) took 1929 ms. Trace:
I20260809 23:21:37.349694 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.372807 25371 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:37.162364  3593 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.352778 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.343472 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:37.358369  3349 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:37.236918  3232 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.159569  3347 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.173544  3423 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.180s
I20260809 23:21:37.350608  3208 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:37.373879 24430 rpcz_store.cc:276] 0809 23:21:35.418547 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.193693 (+1775146us) service_pool.cc:227] Handling call
0809 23:21:37.337724 (+144031us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.343413 (+  5689us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.343453 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.343512 (+    59us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:37.343570 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.348229 (+  4659us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.348380 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.375393 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3163) took 1957 ms. Trace:
I20260809 23:21:37.418649 24435 rpcz_store.cc:276] 0809 23:21:35.418265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.171266 (+1753001us) service_pool.cc:227] Handling call
0809 23:21:37.340335 (+169069us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:37.375135 (+ 34800us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.375338 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.376389 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.370901  3472 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.010s
I20260809 23:21:37.361960  3330 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:37.428459  3347 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.385142 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.382189 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.383526 24323 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
W20260809 23:21:37.356294 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.704s	user 0.015s	sys 0.035s
I20260809 23:21:37.362345  3207 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.380445 24544 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.434077 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3165) took 2015 ms. Trace:
I20260809 23:21:37.387385 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3168) took 1871 ms. Trace:
I20260809 23:21:37.434561 24422 rpcz_store.cc:276] 0809 23:21:35.419030 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.382265 (+1963235us) service_pool.cc:227] Handling call
0809 23:21:37.433460 (+ 51195us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.433692 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.433730 (+    38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.433783 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:37.433826 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.433890 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.434012 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.435304 24424 rpcz_store.cc:276] 0809 23:21:35.516181 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.386296 (+1870115us) service_pool.cc:227] Handling call
0809 23:21:37.386644 (+   348us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.386857 (+   213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.387003 (+   146us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.387067 (+    64us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:37.387118 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.387193 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.387323 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.422878 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.426234 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:37.428051 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3588) took 2173 ms. Trace:
I20260809 23:21:37.420202 25365 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:37.432869 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.468885 24544 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.372067  3607 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:37.472491 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.474095 25371 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:37.438967 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.499078 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.499508 24323 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:37.473170 24431 rpcz_store.cc:276] 0809 23:21:35.254655 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.295267 (+ 40612us) service_pool.cc:227] Handling call
0809 23:21:35.295606 (+   339us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:36.506748 (+1211142us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.427736 (+920988us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003eb7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:37.427976 (+   240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2731124992}
I20260809 23:21:37.500174 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.500835 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:37.473448 24312 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:37.501879  2884 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:37.502290  2884 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:37.502163 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.502785 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.503445 24320 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:37.505148 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3591) took 1894 ms. Trace:
I20260809 23:21:37.505610 24435 rpcz_store.cc:276] 0809 23:21:35.610419 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.419808 (+1809389us) service_pool.cc:227] Handling call
0809 23:21:37.504571 (+ 84763us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.504769 (+   198us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.504799 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.504865 (+    66us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:21:37.504907 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.504973 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.505087 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:37.376853  3239 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:37.496214 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3590) took 1886 ms. Trace:
I20260809 23:21:37.493644 25372 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
W20260809 23:21:37.511500 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43555.6us
I20260809 23:21:37.511828 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:37.416678  3232 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:37.523025 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.521286 24430 rpcz_store.cc:276] 0809 23:21:35.609997 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.417608 (+1807611us) service_pool.cc:227] Handling call
0809 23:21:37.495600 (+ 77992us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.495809 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.495844 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.495910 (+    66us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:37.495953 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.496015 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:37.496144 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:37.527647  3564 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.596s	user 0.011s	sys 0.000s
I20260809 23:21:37.502578  3472 consensus_queue.cc:1225] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 23.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:21:37.522745 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.537053 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.431562  3208 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:37.468731  3207 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:37.431860  3418 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:37.540553 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.548502 24313 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:37.364677  3600 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.469067  3156 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:37.553617 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.022s	user 0.010s	sys 0.026s
I20260809 23:21:37.557103 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:37.561614 24425 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.016s	sys 0.020s
W20260809 23:21:37.566066 24425 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.051s	user 0.017s	sys 0.020s
W20260809 23:21:37.565820 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.010s	sys 0.026s
I20260809 23:21:37.509166  3607 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:37.416684  3593 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.567699 24309 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:37.544356  3369 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:37.603330 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.537002  3243 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:37.526731  2884 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.625319 24312 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260809 23:21:37.565438  3442 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:37.631995  3587 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:37.643997 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.565775  3466 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.539354  3239 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:37.582146  3607 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:37.647964 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:37.603196  3557 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:37.608134  3259 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:37.661033 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.898s	user 0.018s	sys 0.044s
W20260809 23:21:37.661633 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.018s	sys 0.044s
W20260809 23:21:37.617398  3377 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.601s	user 0.011s	sys 0.001s
I20260809 23:21:37.651727 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.655094 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.606812  3455 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:37.607597  3369 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:37.639389  3411 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:37.641902  3396 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.562743  3418 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.645720  3442 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.643846  2884 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:37.644539  3154 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:37.669193 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1789) took 2396 ms. Trace:
W20260809 23:21:37.718253 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.002s	sys 0.031s
W20260809 23:21:37.718852 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.002s	sys 0.031s
I20260809 23:21:37.718321 24311 rpcz_store.cc:276] 0809 23:21:35.272673 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.296141 (+ 23468us) service_pool.cc:227] Handling call
0809 23:21:35.344211 (+ 48070us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.936733 (+592522us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.668910 (+1732177us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:37.669126 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1359103360}
I20260809 23:21:37.668068  3369 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:37.675760 25371 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:21:37.662081 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 899638us
I20260809 23:21:37.576787  3156 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.691433  3455 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:37.735843  3431 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.669s	user 0.009s	sys 0.001s
I20260809 23:21:37.736382  3156 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.736945 24431 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:37.615669  2820 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260809 23:21:37.665912  3423 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:37.740031  3423 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:37.740448  3423 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:37.743906 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3161) took 2336 ms (client timeout 2171 ms). Trace:
W20260809 23:21:37.744330 24428 rpcz_store.cc:269] 0809 23:21:35.407247 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:36.389789 (+982542us) service_pool.cc:227] Handling call
0809 23:21:37.743687 (+1353898us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400332cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:37.743833 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433013632}
I20260809 23:21:37.745498 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.746136 24428 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:37.746623  3455 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.670593 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:37.747633 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:37.748248 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 77762.6us
I20260809 23:21:37.749511 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:37.693837 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:37.696018 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.016s	sys 0.051s
I20260809 23:21:37.661147  3557 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.752244  3557 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:37.640707  3243 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:37.755568 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.016s	sys 0.051s
I20260809 23:21:37.668185  3604 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.668673  3411 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.724345 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.768136 25370 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:37.659670  3208 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.698242  3442 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:37.729477  3203 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.762s	user 0.010s	sys 0.000s
I20260809 23:21:37.659402  3587 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:37.730896  3600 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:37.732177  3239 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:37.734346 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:37.829133 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:37.751855  3567 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.710s	user 0.000s	sys 0.011s
W20260809 23:21:37.691751 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.208s	user 0.010s	sys 0.030s
W20260809 23:21:37.831301 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.347s	user 0.010s	sys 0.030s
W20260809 23:21:37.831626 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.34838e+06us
W20260809 23:21:37.812186  3556 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.918s	user 0.011s	sys 0.017s
I20260809 23:21:37.759783 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:37.764945 24427 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.018s	sys 0.066s
I20260809 23:21:37.678722  3259 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260809 23:21:37.818046 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:37.759785 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:37.768255  3407 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.018s	sys 0.033s
W20260809 23:21:37.822887 24541 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.910s	user 0.015s	sys 0.036s
I20260809 23:21:37.755223  3109 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.826300  3208 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:37.767319  3218 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:21:37.827654 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.007s	sys 0.044s
I20260809 23:21:37.836022 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.758374 24431 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:37.827068  3426 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.002s	sys 0.049s
I20260809 23:21:37.776266  3411 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:37.786293  3455 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:37.859525 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:37.840377 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:21:37.840305 24427 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.019s	sys 0.066s
I20260809 23:21:37.842901 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:37.847787 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88070.2us
I20260809 23:21:37.776500  3170 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:37.768319  3431 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:37.851673 24541 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.942s	user 0.015s	sys 0.036s
W20260809 23:21:37.723903  3473 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.000s	sys 0.010s
I20260809 23:21:37.822211  3154 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:37.854467 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:37.839835  3556 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.011s	sys 0.018s
W20260809 23:21:37.896339  3426 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.003s	sys 0.049s
I20260809 23:21:37.902105 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:37.776726  3604 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:37.854800  3407 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.045s	user 0.020s	sys 0.035s
W20260809 23:21:37.863291  3563 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.011s	sys 0.010s
I20260809 23:21:37.867708 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:37.957417 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:37.963083 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95463.5us
I20260809 23:21:37.963603 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:37.865145 24312 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:37.867153 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:37.863463 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.011s	sys 0.031s
W20260809 23:21:37.965198 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.106s	user 0.013s	sys 0.031s
W20260809 23:21:37.870368 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s	user 0.007s	sys 0.044s
I20260809 23:21:37.924340 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:37.863091  2995 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.998s	user 0.008s	sys 0.057s
I20260809 23:21:37.869700  3218 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:37.950294 24422 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
W20260809 23:21:37.954020 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.018s	sys 0.046s
W20260809 23:21:37.969411 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.177s	user 0.019s	sys 0.047s
I20260809 23:21:37.970109 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:37.967159 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:37.934715  3431 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:37.979517 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.916s	user 0.019s	sys 0.027s
W20260809 23:21:37.980690 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.019s	sys 0.027s
W20260809 23:21:37.987957 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 924418us
W20260809 23:21:37.971135  3263 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.923s	user 0.013s	sys 0.027s
W20260809 23:21:37.965467  3473 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s	user 0.006s	sys 0.010s
W20260809 23:21:37.958081  3618 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260809 23:21:37.862509  3109 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:37.966462  3563 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.011s	sys 0.011s
I20260809 23:21:37.944904  3593 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.945824  3623 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.054118 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:37.969449  2995 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.127s	user 0.009s	sys 0.057s
I20260809 23:21:37.981555  3600 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:38.055994 24544 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:21:37.984697  3633 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:37.968871  3259 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:37.972934  3218 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:37.982999 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:38.057250 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:37.982121  3203 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:37.955528  3625 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:37.972018  3556 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260809 23:21:37.988757 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.006s	sys 0.015s
W20260809 23:21:38.126374 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.013s	sys 0.017s
W20260809 23:21:38.052563  3263 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.042s	user 0.014s	sys 0.028s
I20260809 23:21:37.953827  3253 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.103768 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:38.073625 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.007s	sys 0.042s
I20260809 23:21:37.939023  3170 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:38.132794 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.010s	sys 0.042s
I20260809 23:21:38.091655  3243 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:37.946055  3154 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.117131 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:38.108593  3623 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.120945  3259 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.123535  3587 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.091960  3164 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:37.989056  3316 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.908s	user 0.004s	sys 0.016s
W20260809 23:21:38.132496 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 149576us
I20260809 23:21:38.204609 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:38.124246  3626 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260809 23:21:38.137845 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:38.140388 24544 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:38.140201 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:38.208272 24539 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:37.983433  3431 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:38.141305 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.007s	sys 0.015s
W20260809 23:21:38.224341 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.007s	sys 0.015s
I20260809 23:21:38.225530 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1512) took 2590 ms. Trace:
I20260809 23:21:38.226038 24424 rpcz_store.cc:276] 0809 23:21:35.635312 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.436547 (+1801235us) service_pool.cc:227] Handling call
0809 23:21:37.436927 (+   380us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.437159 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:37.448453 (+ 11294us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.224948 (+776495us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.225157 (+   209us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:38.225217 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.225304 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.225465 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31874,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":17269,"thread_start_us":776067,"threads_started":1}
I20260809 23:21:37.984591  3442 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.063364  3339 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:38.152906 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.946s	user 0.027s	sys 0.010s
I20260809 23:21:38.129158  3556 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:38.141925 24314 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.001s	sys 0.021s
I20260809 23:21:38.132195  3263 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:38.210960  3316 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.130s	user 0.004s	sys 0.025s
I20260809 23:21:38.146682  3625 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:38.256058 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:38.245008 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.027s	sys 0.010s
I20260809 23:21:38.245240 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:38.155843  3350 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.006s	sys 0.015s
I20260809 23:21:38.230836 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3574) took 3182 ms. Trace:
I20260809 23:21:38.265897 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:38.265337 24546 rpcz_store.cc:276] 0809 23:21:35.047984 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.440728 (+392744us) service_pool.cc:227] Handling call
0809 23:21:35.501333 (+ 60605us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.501832 (+   499us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.501875 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.502358 (+   483us) log.cc:842] Serialized 26 byte log entry
0809 23:21:36.329891 (+827533us) spinlock_profiling.cc:243] Waited 45.7 ms on lock 0x7b18004ace60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:37.367563 (+1037672us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:37.367618 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.367709 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.147022 (+779313us) raft_consensus.cc:1671] finished
0809 23:21:38.147050 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.230752 (+ 83702us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":20930,"spinlock_wait_cycles":105135872,"thread_start_us":1037450,"threads_started":1,"wal-append.queue_time_us":2325467}
W20260809 23:21:38.247686 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.011s	sys 0.038s
W20260809 23:21:38.249406 24314 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.001s	sys 0.021s
I20260809 23:21:38.141492  3426 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:38.122052  3600 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.242537 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:38.155927  3253 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:38.147378  3259 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:38.262946 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.015s	sys 0.037s
I20260809 23:21:38.210162  3593 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:38.268373 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.011s	sys 0.038s
I20260809 23:21:38.129036  3203 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.324898 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:38.335803 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1710) took 3348 ms. Trace:
I20260809 23:21:38.336529 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:38.327693 24424 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:38.707465 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 2969) took 3281 ms. Trace:
W20260809 23:21:38.708611 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.008s	sys 0.007s
W20260809 23:21:38.709170 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.008s	sys 0.007s
I20260809 23:21:38.709962 24424 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:38.714920 25370 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.008s	sys 0.017s
W20260809 23:21:38.715396 25370 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.009s	sys 0.017s
I20260809 23:21:38.715507 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1515) took 2665 ms. Trace:
I20260809 23:21:38.715997 24424 rpcz_store.cc:276] 0809 23:21:36.049725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.708389 (+2658664us) service_pool.cc:227] Handling call
0809 23:21:38.709361 (+   972us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:38.715113 (+  5752us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.715402 (+   289us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:38.717199 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.010s	sys 0.022s
W20260809 23:21:38.717671 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.010s	sys 0.022s
W20260809 23:21:38.717852 25371 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.016s	sys 0.012s
W20260809 23:21:38.718550 25371 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.016s	sys 0.012s
W20260809 23:21:38.703284 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.827s	user 0.006s	sys 0.013s
W20260809 23:21:38.719277 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.007s	sys 0.014s
I20260809 23:21:38.718961 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:38.721071 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:38.704372 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.001s	sys 0.021s
W20260809 23:21:38.723206 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.002s	sys 0.022s
W20260809 23:21:38.704679 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.011s	sys 0.006s
W20260809 23:21:38.705070 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.008s	sys 0.020s
W20260809 23:21:38.724196 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.011s	sys 0.006s
W20260809 23:21:38.724390 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.008s	sys 0.020s
W20260809 23:21:38.724974 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 787629us
I20260809 23:21:38.704442 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1791) took 1235 ms. Trace:
I20260809 23:21:38.726683 24321 rpcz_store.cc:276] 0809 23:21:37.468621 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.469078 (+   457us) service_pool.cc:227] Handling call
0809 23:21:37.469522 (+   444us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:37.529366 (+ 59844us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.529402 (+    36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:37.543526 (+ 14124us) log.cc:842] Serialized 52 byte log entry
0809 23:21:38.335342 (+791816us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.335394 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.335457 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.703492 (+368035us) raft_consensus.cc:1671] finished
0809 23:21:38.703559 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.704140 (+   581us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59390,"raft.queue_time_us":66493,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":3292,"thread_start_us":727386,"threads_started":1,"wal-append.queue_time_us":914080}
I20260809 23:21:38.730815 24312 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260809 23:21:38.706239 24427 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.007s	sys 0.045s
W20260809 23:21:38.732282 24427 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.007s	sys 0.045s
I20260809 23:21:38.336174 25375 rpcz_store.cc:276] 0809 23:21:34.987801 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:34.987949 (+   148us) service_pool.cc:227] Handling call
0809 23:21:34.988279 (+   330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.029504 (+ 41225us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.029532 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.029884 (+   352us) log.cc:842] Serialized 26 byte log entry
0809 23:21:35.857952 (+828068us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:36.967817 (+1109865us) log.cc:842] Serialized 14 byte log entry
0809 23:21:36.968043 (+   226us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.265810 (+1297767us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.265867 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.265925 (+    58us) raft_consensus.cc:1671] finished
0809 23:21:38.335603 (+ 69678us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.335750 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":892840,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":3751,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":170,"spinlock_wait_cycles":94004480,"thread_start_us":1866425,"threads_started":2,"wal-append.queue_time_us":1134443,"wal-append.run_cpu_time_us":2832,"wal-append.run_wall_time_us":1889218}
I20260809 23:21:38.335167 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:38.711333 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.005s	sys 0.016s
I20260809 23:21:38.733683 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3159) took 3327 ms. Trace:
W20260809 23:21:38.734196 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.005s	sys 0.016s
I20260809 23:21:38.736171 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1514) took 2920 ms. Trace:
I20260809 23:21:38.736687 24430 rpcz_store.cc:276] 0809 23:21:35.815426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.525001 (+1709575us) service_pool.cc:227] Handling call
0809 23:21:37.525391 (+   390us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:37.530353 (+  4962us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.530423 (+    70us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:37.548133 (+ 17710us) log.cc:842] Serialized 26 byte log entry
0809 23:21:37.962239 (+414106us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b1800096fa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:38.735243 (+773004us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.735406 (+   163us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.735571 (+   165us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.735649 (+    78us) raft_consensus.cc:1671] finished
0809 23:21:38.735681 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.736072 (+   391us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4315,"raft.queue_time_us":12071,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":8839,"spinlock_wait_cycles":37041536,"thread_start_us":772391,"threads_started":1,"wal-append.queue_time_us":1062176}
I20260809 23:21:38.335976 24431 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:38.434770 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.010s	sys 0.012s
W20260809 23:21:38.739676 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.011s	sys 0.012s
W20260809 23:21:38.332070 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.016s	sys 0.038s
I20260809 23:21:38.722599 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:38.708341 24317 rpcz_store.cc:276] 0809 23:21:35.441995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.442115 (+   120us) service_pool.cc:227] Handling call
0809 23:21:35.488517 (+ 46402us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.101466 (+1612949us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.702491 (+1601025us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b540019d2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.707138 (+  4647us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3708137728}
W20260809 23:21:38.712827 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.012s	sys 0.004s
W20260809 23:21:38.747243 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 904361us
W20260809 23:21:38.747850 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.012s	sys 0.005s
I20260809 23:21:38.748835  3154 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.232302  3164 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.750921 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1795) took 1239 ms. Trace:
I20260809 23:21:38.752683 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:38.754318  3164 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.739626 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3602) took 1206 ms. Trace:
W20260809 23:21:38.629742  3634 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.514s	user 0.005s	sys 0.000s
I20260809 23:21:38.439031 24545 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:21:38.394613  2992 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.019s	sys 0.033s
I20260809 23:21:38.327312  2995 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.811139  3676 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:38.811674 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1796) took 1282 ms. Trace:
I20260809 23:21:38.812361  3676 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:38.714016 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.994s	user 0.030s	sys 0.026s
W20260809 23:21:38.813776 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.031s	sys 0.027s
I20260809 23:21:38.815295 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:38.817186  3681 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:38.817740  3681 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:38.818240  3681 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:38.819090  3671 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.820243  3671 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.752946 24316 rpcz_store.cc:276] 0809 23:21:37.511717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.524082 (+ 12365us) service_pool.cc:227] Handling call
0809 23:21:37.535013 (+ 10931us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:37.550870 (+ 15857us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:37.550900 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:37.551382 (+   482us) log.cc:842] Serialized 26 byte log entry
0809 23:21:37.961619 (+410237us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b180037fa80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:38.748836 (+787217us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.749592 (+   756us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.750094 (+   502us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.750256 (+   162us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.750345 (+    89us) raft_consensus.cc:1671] finished
0809 23:21:38.750513 (+   168us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.750832 (+   319us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16102,"raft.queue_time_us":18441,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":14139,"spinlock_wait_cycles":3427712,"thread_start_us":786599,"threads_started":1,"wal-append.queue_time_us":1059694}
W20260809 23:21:38.809373 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 87024.1us
W20260809 23:21:38.492183  3207 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.923s	user 0.010s	sys 0.002s
W20260809 23:21:38.333635  3272 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.005s	sys 0.038s
W20260809 23:21:38.265089  3350 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.006s	sys 0.016s
W20260809 23:21:38.826912  3272 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.095s	user 0.006s	sys 0.039s
I20260809 23:21:38.529597  3442 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.759747 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:38.331489  3330 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.903s	user 0.010s	sys 0.001s
W20260809 23:21:38.825747 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.004s	sys 0.016s
I20260809 23:21:38.255291  3455 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:38.829270 25368 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.012s	sys 0.004s
W20260809 23:21:38.761663  3407 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.006s	sys 0.028s
I20260809 23:21:38.762215 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:38.830894 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.004s	sys 0.016s
W20260809 23:21:38.477708  3232 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.931s	user 0.006s	sys 0.007s
W20260809 23:21:38.832322 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.012s	sys 0.004s
I20260809 23:21:38.836154  3232 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:38.832693  3407 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.007s	sys 0.029s
I20260809 23:21:38.388536  3253 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.839298  3677 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:38.839550 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:38.839970  3677 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:38.840523  3677 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:38.840785 24317 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:38.841960  3253 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.764940 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:38.481652  3263 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:38.832666 25368 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.013s	sys 0.005s
I20260809 23:21:38.734297 24427 rpcz_store.cc:276] 0809 23:21:35.406522 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.406631 (+   109us) service_pool.cc:227] Handling call
0809 23:21:35.407083 (+   452us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.439565 (+ 32482us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.439596 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.439963 (+   367us) log.cc:842] Serialized 26 byte log entry
0809 23:21:36.774093 (+1334130us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:37.869141 (+1095048us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.732987 (+863846us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.733135 (+   148us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.733212 (+    77us) raft_consensus.cc:1671] finished
0809 23:21:38.733231 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.733576 (+   345us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45928,"prepare.queue_time_us":243179,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":130867,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":56025,"thread_start_us":3083622,"threads_started":3,"wal-append.queue_time_us":1521651}
I20260809 23:21:38.844985 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:38.845988  3685 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:38.846017 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:38.252326  3339 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.848038 24157 maintenance_manager.cc:419] P e3357d0fe4ef46db9a4739044fa527d5: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033369
I20260809 23:21:38.849116  3675 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:38.405630  3426 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:38.439667  3473 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.854984  3675 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.855716  3473 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.858373  3675 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:38.624768  3529 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:38.552704  3557 consensus_meta.cc:287] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.800s	user 0.010s	sys 0.002s
I20260809 23:21:38.869510  3557 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.870290  3676 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:38.550532  3586 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.593s	user 0.005s	sys 0.001s
I20260809 23:21:38.328850  3587 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:38.886948 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3166) took 3467 ms (client timeout 1686 ms). Trace:
I20260809 23:21:38.887933 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1504) took 3844 ms. Trace:
I20260809 23:21:38.887755  3587 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.888425 24425 rpcz_store.cc:276] 0809 23:21:35.043657 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.272656 (+228999us) service_pool.cc:227] Handling call
0809 23:21:35.272962 (+   306us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:35.273307 (+   345us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.273336 (+    29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:35.306620 (+ 33284us) log.cc:842] Serialized 52 byte log entry
0809 23:21:36.499352 (+1192732us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:37.604931 (+1105579us) log.cc:842] Serialized 14 byte log entry
0809 23:21:37.606471 (+  1540us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:37.606576 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.538762 (+932186us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:38.887220 (+348458us) raft_consensus.cc:1671] finished
0809 23:21:38.887439 (+   219us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.887693 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11541,"prepare.queue_time_us":517819,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":13440,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":24959,"spinlock_wait_cycles":10329984,"thread_start_us":2175349,"threads_started":2,"wal-append.queue_time_us":1524841}
I20260809 23:21:38.409240  3593 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.501974  3600 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.628388  3604 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:38.892100  3557 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:38.531005  3623 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.479214  3625 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:38.915952  3423 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.175s	user 0.011s	sys 0.008s
W20260809 23:21:38.501924  3630 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260809 23:21:38.498834  3635 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.378s	user 0.001s	sys 0.000s
W20260809 23:21:38.601361  3637 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.339s	user 0.003s	sys 0.000s
I20260809 23:21:38.916622  3423 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:38.921175 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.003s	sys 0.007s
I20260809 23:21:38.921698  3109 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:38.502849  3641 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260809 23:21:38.922623 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.005s	sys 0.006s
W20260809 23:21:38.923331 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.005s	sys 0.006s
W20260809 23:21:38.516862  3638 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20260809 23:21:38.127384  3633 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.444456  3642 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:39.017210  3642 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:38.496413  3643 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.383s	user 0.001s	sys 0.000s
I20260809 23:21:39.020443 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:38.590601  3649 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:21:38.512712  3651 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.022995  3338 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260809 23:21:38.550776  3654 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260809 23:21:38.587924  3665 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260809 23:21:38.786774 24424 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:21:39.027173  3154 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:39.027321 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1516) took 2850 ms. Trace:
I20260809 23:21:39.027725  3154 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.028265  3154 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:38.758776  3203 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.028991  3604 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:38.770026 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:39.029736  3604 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:39.030474  3369 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.263s	user 0.010s	sys 0.014s
W20260809 23:21:38.333561 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.009s	sys 0.017s
I20260809 23:21:39.048990  3203 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:39.054342 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.221s	user 0.009s	sys 0.017s
I20260809 23:21:39.054561  3203 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.056981 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:38.390245  2736 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.019s	sys 0.024s
W20260809 23:21:39.058037  2736 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.306s	user 0.020s	sys 0.024s
I20260809 23:21:39.057964  3529 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:38.756012  3170 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.059396  3170 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:39.059873  3170 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:39.061306  3170 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.833150 24320 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 19
I20260809 23:21:38.763474 24312 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260809 23:21:38.843602  2992 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.090s	user 0.020s	sys 0.033s
I20260809 23:21:38.768592  3349 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:38.446686  3396 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.828696 25366 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:38.843379 24321 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:38.856329 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3592) took 3245 ms. Trace:
I20260809 23:21:38.251653  3556 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:38.812323 24323 rpcz_store.cc:276] 0809 23:21:37.528825 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.528995 (+   170us) service_pool.cc:227] Handling call
0809 23:21:37.529268 (+   273us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.529562 (+   294us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:37.960249 (+430687us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b18003393a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:21:38.725065 (+764816us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.810895 (+ 85830us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.811034 (+   139us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.811201 (+   167us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.811333 (+   132us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.811589 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19917,"prepare.queue_time_us":974412,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":1461,"raft.queue_time_us":29271,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":9392,"spinlock_wait_cycles":29266176,"thread_start_us":823743,"threads_started":2}
I20260809 23:21:38.858418 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1696) took 3865 ms. Trace:
I20260809 23:21:38.807189  3563 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:38.843210  3683 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:38.849452  3339 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:38.833312  3670 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:38.634027  3607 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.951s	user 0.010s	sys 0.003s
I20260809 23:21:38.896883 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1697) took 3801 ms. Trace:
I20260809 23:21:38.859436  3207 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:38.898717  3693 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:38.900094 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3490) took 3845 ms (client timeout 2401 ms). Trace:
I20260809 23:21:38.903360  3600 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:38.908411 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1809) took 1049 ms. Trace:
I20260809 23:21:38.870283  3473 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.921855  3423 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.337164 24539 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:21:38.887583 24429 rpcz_store.cc:269] 0809 23:21:35.419341 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.918420 (+499079us) service_pool.cc:227] Handling call
0809 23:21:38.886317 (+2967897us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:38.886629 (+   312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5038620032}
I20260809 23:21:39.089337 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.090337 24429 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:38.921690 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.003s	sys 0.007s
I20260809 23:21:39.091090  3683 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.091718  3683 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:39.091858 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.092828 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:39.093811 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:38.938875  3253 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.093225 24542 rpcz_store.cc:276] 0809 23:21:35.058155 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.528460 (+470305us) service_pool.cc:227] Handling call
0809 23:21:35.528784 (+   324us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:36.120382 (+591598us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:21:37.208106 (+1087724us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.129594 (+921488us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.129784 (+   190us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.129831 (+    47us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:38.129874 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.632631 (+502757us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b18001636c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:38.859434 (+226803us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.859506 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.859787 (+   281us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1704937,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":5513,"raft.queue_time_us":318,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":8933,"spinlock_wait_cycles":1463428352,"thread_start_us":1963130,"threads_started":2,"wal-append.queue_time_us":1292195}
I20260809 23:21:39.095578  3253 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.096568 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.097455 24542 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:39.097981  3253 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.100663 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.099620 24540 rpcz_store.cc:276] 0809 23:21:34.992622 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.396080 (+403458us) service_pool.cc:227] Handling call
0809 23:21:35.396390 (+   310us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:35.635756 (+239366us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:35.635786 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:35.651700 (+ 15914us) log.cc:842] Serialized 26 byte log entry
0809 23:21:37.431653 (+1779953us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:37.431712 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.857433 (+1425721us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.857527 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:38.857634 (+   107us) raft_consensus.cc:1671] finished
0809 23:21:38.857657 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.858075 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13245,"raft.queue_time_us":14398,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":13534,"spinlock_wait_cycles":519137536,"thread_start_us":1779555,"threads_started":1,"wal-append.queue_time_us":2188366,"wal-append.run_cpu_time_us":2459,"wal-append.run_wall_time_us":1110284}
I20260809 23:21:39.102980  3600 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.103381  3670 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.105474  3600 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.947093  3239 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:38.948338  3347 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:39.106303  3670 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.106492  3347 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:39.107178  3563 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.107892  3347 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.108610  3633 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:38.951740 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1794) took 1457 ms. Trace:
I20260809 23:21:38.952755  3671 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.110510 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:38.953598  3418 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.955083 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:38.832366  3455 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:39.112819 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 158027us
I20260809 23:21:39.113699  3207 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.964964  3593 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:38.966873  3557 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:39.116155  3170 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:38.969684 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.116572 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:38.971391 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:38.911648  3272 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:38.974071 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:38.975219  3587 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.118199 24317 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:39.118508 25373 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:39.119711 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:38.976361 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.115057 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:38.978513 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.120931 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.121649 24311 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:38.981128 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:38.983222 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 168286us
W20260809 23:21:38.987643 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 783049us
I20260809 23:21:39.085245 24422 rpcz_store.cc:276] 0809 23:21:35.610759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.436175 (+1825416us) service_pool.cc:227] Handling call
0809 23:21:37.511444 (+ 75269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.969234 (+457790us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.855673 (+886439us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.856109 (+   436us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1005759360}
I20260809 23:21:39.123806 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1721) took 1967 ms. Trace:
I20260809 23:21:39.124280 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.124214 25366 rpcz_store.cc:276] 0809 23:21:37.156078 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.156209 (+   131us) service_pool.cc:227] Handling call
0809 23:21:37.156495 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.083471 (+1926976us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.084321 (+   850us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:39.123726 (+ 39405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3843940096}
I20260809 23:21:39.125610 24266 leader_election.cc:398] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20.
I20260809 23:21:39.110322 24314 rpcz_store.cc:276] 0809 23:21:37.494294 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.494476 (+   182us) service_pool.cc:227] Handling call
0809 23:21:37.494859 (+   383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.495117 (+   258us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:38.320424 (+825307us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.951021 (+630597us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.951169 (+   148us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.951308 (+   139us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.951442 (+   134us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.951663 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37772,"prepare.queue_time_us":1001397,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":47,"raft.queue_time_us":79147,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":136,"thread_start_us":1254224,"threads_started":2,"wal-append.queue_time_us":630687}
I20260809 23:21:39.126312  3593 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:38.994062 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.126788  3593 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.106338  3239 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:39.127868 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 165277us
I20260809 23:21:39.101783 24539 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:39.128876 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.128441  3239 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.129841  3239 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:38.971330  3675 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:38.996290 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3593) took 2141 ms. Trace:
I20260809 23:21:38.494619  3648 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:39.027864 24424 rpcz_store.cc:276] 0809 23:21:36.177362 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.717039 (+2539677us) service_pool.cc:227] Handling call
0809 23:21:38.717615 (+   576us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.026912 (+309297us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.027223 (+   311us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:38.771260 24431 rpcz_store.cc:276] 0809 23:21:37.533250 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.324555 (+791305us) service_pool.cc:227] Handling call
0809 23:21:38.739400 (+414845us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:39.168717 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 111901us
W20260809 23:21:39.031315 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 280654us
I20260809 23:21:38.812434  2995 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.031388  3672 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:39.023684  3651 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.033939  3642 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:39.034379  3411 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.168s	user 0.012s	sys 0.006s
I20260809 23:21:38.958556 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.057183  3203 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:38.846726  3685 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:39.008253  3676 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.069152 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1718) took 3743 ms. Trace:
I20260809 23:21:39.079078 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3534) took 1391 ms. Trace:
W20260809 23:21:39.082401 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1703) took 2298 ms (client timeout 2790 ms). Trace:
I20260809 23:21:39.083628  3396 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.089933  3693 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.115725  3557 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:38.949828 25370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:39.109081 25372 rpcz_store.cc:276] 0809 23:21:37.867788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.868262 (+   474us) service_pool.cc:227] Handling call
0809 23:21:37.868613 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:38.209342 (+340729us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:38.209400 (+    58us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.209455 (+    55us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:38.209497 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.907871 (+698374us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b5400601654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:38.907937 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.908206 (+   269us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":783009152}
I20260809 23:21:39.185801 25370 raft_consensus.cc:595] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:39.186244 25370 raft_consensus.cc:603] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:21:39.115720 24535 rpcz_store.cc:269] 0809 23:21:35.054725 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.287007 (+232282us) service_pool.cc:227] Handling call
0809 23:21:38.899446 (+3612439us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:38.899658 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3505665792}
I20260809 23:21:38.967701  3232 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.117475 24323 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:38.977443 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.122778 25363 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:39.127241  3593 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:39.130147 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:38.995059  3316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.003s	sys 0.014s
I20260809 23:21:39.132885 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.135860 24425 rpcz_store.cc:276] 0809 23:21:36.855447 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.890897 (+2035450us) service_pool.cc:227] Handling call
0809 23:21:38.995534 (+104637us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:38.995848 (+   314us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:38.995889 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.995952 (+    63us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:38.995987 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.996058 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:38.996204 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.195760 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.196697 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3510) took 1053 ms. Trace:
I20260809 23:21:39.197039 24535 rpcz_store.cc:276] 0809 23:21:38.143300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.195405 (+1052105us) service_pool.cc:227] Handling call
0809 23:21:39.196632 (+  1227us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:39.199822 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48017.5us
I20260809 23:21:39.200665  3396 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.203112 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.203220 24309 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:39.205077 24539 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:39.205596 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3508) took 1078 ms. Trace:
I20260809 23:21:39.205468  3685 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.206077 24539 rpcz_store.cc:276] 0809 23:21:38.126817 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.129737 (+1002920us) service_pool.cc:227] Handling call
0809 23:21:39.205512 (+ 75775us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.208149  3232 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:39.208592  3232 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:39.208999  3232 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:39.183617  3411 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.203234 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.212510 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.213213 24431 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:39.192937  3587 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.021s
I20260809 23:21:39.214183 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.214620  3693 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.215271  3651 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.215451 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3588) took 1070 ms. Trace:
I20260809 23:21:39.215852 24540 rpcz_store.cc:276] 0809 23:21:38.145194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.102478 (+957284us) service_pool.cc:227] Handling call
0809 23:21:39.215369 (+112891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.216372 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:39.217074 24535 pending_rounds.cc:76] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
I20260809 23:21:39.154110 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1517) took 2869 ms. Trace:
W20260809 23:21:39.147712  3218 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.958s	user 0.007s	sys 0.019s
I20260809 23:21:39.115192  3207 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:39.087604  3604 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.136987  3563 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.135500  3253 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.171542 25366 raft_consensus.cc:1203] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Deduplicated request from leader. Original: 20.14->[22.15-22.15]   Dedup: 22.15->[]
I20260809 23:21:39.205114  3557 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.176926 24429 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:39.252377 24431 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:39.109309  3347 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:39.178514 24543 rpcz_store.cc:269] 0809 23:21:36.783520 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:36.783694 (+   174us) service_pool.cc:227] Handling call
0809 23:21:39.081977 (+2298283us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.082213 (+   236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4970682752}
I20260809 23:21:39.070693  3369 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.185034 24545 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:39.189720 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61027.6us
W20260809 23:21:39.190904 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 176976us
I20260809 23:21:39.217428 24535 pending_rounds.cc:97] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260809 23:21:39.219918 24430 rpcz_store.cc:276] 0809 23:21:36.284711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.739107 (+2454396us) service_pool.cc:227] Handling call
0809 23:21:38.739671 (+   564us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:38.740098 (+   427us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:38.794157 (+ 54059us) log.cc:842] Serialized 14 byte log entry
0809 23:21:39.153657 (+359500us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:39.153727 (+    70us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:39.153788 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.153873 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.154038 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":322288,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":39,"raft.queue_time_us":18235,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":18777,"thread_start_us":362158,"threads_started":2}
I20260809 23:21:39.162029  3675 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.224570 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:39.277195 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3509) took 1147 ms. Trace:
I20260809 23:21:39.242224 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3160) took 3636 ms. Trace:
I20260809 23:21:39.246217 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1811) took 1311 ms. Trace:
I20260809 23:21:39.280508 24545 rpcz_store.cc:276] 0809 23:21:38.129608 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.957680 (+828072us) service_pool.cc:227] Handling call
0809 23:21:39.277098 (+319418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.213239  3676 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.178087 24313 rpcz_store.cc:276] 0809 23:21:37.622670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.623027 (+   357us) service_pool.cc:227] Handling call
0809 23:21:37.623390 (+   363us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:37.623621 (+   231us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:38.740785 (+1117164us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.879297 (+138512us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.896275 (+ 16978us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:38.914303 (+ 18028us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.997933 (+ 83630us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.013706 (+ 15773us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93186,"prepare.queue_time_us":903381,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":72880,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":8589,"thread_start_us":1178229,"threads_started":2}
I20260809 23:21:39.214120  3587 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:39.241556  3207 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.265352 25366 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260809 23:21:39.212958  3411 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.268916  3648 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.269822  3625 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:39.177937 25368 rpcz_store.cc:276] 0809 23:21:35.306781 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.644167 (+337386us) service_pool.cc:227] Handling call
0809 23:21:35.711839 (+ 67672us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.712048 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:36.910848 (+1198800us) log.cc:842] Serialized 14 byte log entry
0809 23:21:38.881219 (+1970371us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:38.911778 (+ 30559us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:39.001070 (+ 89292us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.013470 (+ 12400us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.049644 (+ 36174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9155,"prepare.queue_time_us":968813,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":539214,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3319,"thread_start_us":3076209,"threads_started":3,"wal-append.queue_time_us":1149293}
I20260809 23:21:39.267971 24496 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.223996  3633 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:39.253191 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:39.283079 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3594) took 1034 ms. Trace:
I20260809 23:21:39.278878 24542 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:39.252140 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.286301 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.287168 24546 raft_consensus.cc:2396] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260809 23:21:39.281558 24535 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:21:39.277047  3208 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.350s	user 0.006s	sys 0.018s
I20260809 23:21:39.294580 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:39.295147 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3591) took 1131 ms. Trace:
I20260809 23:21:39.295713 24542 rpcz_store.cc:276] 0809 23:21:38.160499 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.157104 (+996605us) service_pool.cc:227] Handling call
0809 23:21:39.292000 (+134896us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.298566 24431 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 18: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.299536 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.300285 24431 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:39.302212 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.302912 24429 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25.
I20260809 23:21:39.291859 24539 rpcz_store.cc:276] 0809 23:21:38.244697 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.206843 (+962146us) service_pool.cc:227] Handling call
0809 23:21:39.267221 (+ 60378us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.278181 (+ 10960us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.278215 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:39.278257 (+    42us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:39.278289 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.278446 (+   157us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.278588 (+   142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3405,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":8398}
I20260809 23:21:39.303716 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.304535 24539 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.304845 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.305563 24429 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.306311 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.307668 24435 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:39.310209 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:39.311070 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 94825.9us
I20260809 23:21:39.312076 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1513) took 3668 ms. Trace:
I20260809 23:21:39.281530 25371 rpcz_store.cc:276] 0809 23:21:37.935165 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.939620 (+  4455us) service_pool.cc:227] Handling call
0809 23:21:37.970009 (+ 30389us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:38.719363 (+749354us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:38.719407 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:38.720509 (+  1102us) log.cc:842] Serialized 26 byte log entry
0809 23:21:38.805174 (+ 84665us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:38.805218 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.809982 (+  4764us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:39.217443 (+407461us) raft_consensus.cc:1671] finished
0809 23:21:39.217463 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.246134 (+ 28671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5174,"raft.queue_time_us":539937,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":15034,"thread_start_us":827368,"threads_started":2,"wal-append.queue_time_us":348294}
I20260809 23:21:39.312784 24429 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.318069 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:39.312520 24435 rpcz_store.cc:276] 0809 23:21:35.643660 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.506804 (+1863144us) service_pool.cc:227] Handling call
0809 23:21:37.550063 (+ 43259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.308136 (+1758073us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.311817 (+  3681us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:39.312023 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4041231616}
I20260809 23:21:39.319592 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.212864  3607 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.292958  3207 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:39.336575 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88622us
I20260809 23:21:39.265971  3218 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.275884  3253 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.320684  3418 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:39.287889  3587 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:39.284657  3600 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:39.347229 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.350956 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.293538  3633 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:39.352890 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61138.8us
I20260809 23:21:39.356626 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.359242 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.336506  3272 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.346830  3557 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:39.369464  3431 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.083s	user 0.016s	sys 0.005s
I20260809 23:21:39.285576 24423 rpcz_store.cc:276] 0809 23:21:35.406923 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:35.408107 (+  1184us) service_pool.cc:227] Handling call
0809 23:21:35.413298 (+  5191us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:35.610057 (+196759us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:36.772246 (+1162189us) log.cc:842] Serialized 14 byte log entry
0809 23:21:37.764026 (+991780us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:37.764085 (+    59us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:37.764138 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.961961 (+1197823us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b18000e6500. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9
0809 23:21:39.004422 (+ 42461us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54003322d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:39.004548 (+   126us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.043261 (+ 38713us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7548,"prepare.queue_time_us":1406690,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":7659,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":20894,"spinlock_wait_cycles":463519232,"thread_start_us":2145506,"threads_started":2,"wal-append.queue_time_us":1146935}
I20260809 23:21:39.369716  3557 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:39.377267  3711 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.172s	user 0.000s	sys 0.003s
W20260809 23:21:39.377554 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 65282.1us
I20260809 23:21:39.380141 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3595) took 2476 ms. Trace:
I20260809 23:21:39.380671 24435 rpcz_store.cc:276] 0809 23:21:36.903960 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.336401 (+2432441us) service_pool.cc:227] Handling call
0809 23:21:39.373483 (+ 37082us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.379728 (+  6245us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.379762 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:39.379804 (+    42us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:39.379857 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.379920 (+    63us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.380059 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:39.375653 24425 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:39.358274  3563 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:39.385401 24429 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:39.363819  3633 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:39.366403  3407 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:39.389818  3717 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260809 23:21:39.391390  3676 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:39.391902  3716 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.188s	user 0.000s	sys 0.002s
I20260809 23:21:39.392330  3253 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.392817  3396 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:39.393110 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:39.395314 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.395599  3369 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.395885 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3592) took 1188 ms. Trace:
I20260809 23:21:39.380210  3648 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:39.398404  3681 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.579s	user 0.011s	sys 0.003s
I20260809 23:21:39.398461  3232 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.439447 24539 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:39.398823  3431 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.387291  3563 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.382438 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3595) took 1131 ms. Trace:
I20260809 23:21:39.403174 25368 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:39.404155 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:39.404371  3713 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260809 23:21:39.392270 25373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:39.443653 24543 rpcz_store.cc:276] 0809 23:21:38.251356 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.270998 (+1019642us) service_pool.cc:227] Handling call
0809 23:21:39.343590 (+ 72592us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.382112 (+ 38522us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.382143 (+    31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:39.382182 (+    39us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:39.382207 (+    25us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.382274 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.382380 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":525,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3366}
I20260809 23:21:39.412503 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:39.421567 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111524us
I20260809 23:21:39.415987 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.387025  3625 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:39.429878 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:39.435695 24320 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:39.451054 24428 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.457366 24431 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:39.442677  3681 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.451833 24323 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:39.434556 24546 rpcz_store.cc:276] 0809 23:21:38.207141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.267828 (+ 60687us) service_pool.cc:227] Handling call
0809 23:21:39.374547 (+1106719us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.395799 (+ 21252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1041205120}
I20260809 23:21:39.450946  3660 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:39.464833  3232 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:39.467187 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 173683us
W20260809 23:21:39.469223  3259 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.045s	user 0.014s	sys 0.004s
I20260809 23:21:39.452693  3563 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:39.469720  3259 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.470674  3681 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.471895  3259 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.386515  3272 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.474241  3272 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.474867 24544 raft_consensus.cc:2379] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:39.475488 24540 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:39.476442 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46319.7us
I20260809 23:21:39.477279 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.480099  3607 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.482059 25373 raft_consensus.cc:595] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:39.482474 25373 raft_consensus.cc:603] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:39.472404  3431 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.484251 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3594) took 2600 ms. Trace:
I20260809 23:21:39.490123 24429 rpcz_store.cc:276] 0809 23:21:36.856237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.343844 (+2487607us) service_pool.cc:227] Handling call
0809 23:21:39.382980 (+ 39136us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.390786 (+  7806us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.456631 (+ 65845us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x7b54002f8ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.456832 (+   201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17855232}
I20260809 23:21:39.478631 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.424124  3396 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.489159 24496 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:39.497761 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 309300us
I20260809 23:21:39.499779  3675 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.500059 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.500649 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3171) took 2733 ms. Trace:
I20260809 23:21:39.474171  3154 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.501111 24427 rpcz_store.cc:276] 0809 23:21:36.767642 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.847451 (+2079809us) service_pool.cc:227] Handling call
0809 23:21:38.847989 (+   538us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:38.848615 (+   626us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:38.848642 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:38.849209 (+   567us) log.cc:842] Serialized 26 byte log entry
0809 23:21:38.997674 (+148465us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:38.997713 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:38.997777 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:39.500403 (+502626us) raft_consensus.cc:1671] finished
0809 23:21:39.500426 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.500563 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":160,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":172,"thread_start_us":148083,"threads_started":1,"wal-append.queue_time_us":332179}
I20260809 23:21:39.512117 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.503819 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:39.521070 25372 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:39.522959 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.526146 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.524950 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.528939 24425 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:39.535740 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.529536 24323 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260809 23:21:39.489502  3625 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:39.540992 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:39.543146  3625 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:39.543370 24323 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:21:39.547528 24435 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:39.548579 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3511) took 1288 ms. Trace:
I20260809 23:21:39.548945 24540 rpcz_store.cc:276] 0809 23:21:38.250035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.216572 (+966537us) service_pool.cc:227] Handling call
0809 23:21:39.216977 (+   405us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.475917 (+258940us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.538483 (+ 62566us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:39.538686 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":594055936}
W20260809 23:21:39.549990 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1519) took 1729 ms (client timeout 1826 ms). Trace:
I20260809 23:21:39.550432 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.550637  3316 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:39.551252  3455 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:39.551681  3455 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:39.551939 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.531601 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.531781 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.500687  3396 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:39.540234 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.520793  3557 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.541945 24542 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:39.552166  3455 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.551682 24430 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
W20260809 23:21:39.574818 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 170745us
I20260809 23:21:39.553897 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.554189 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:39.527449  3677 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.686s	user 0.010s	sys 0.007s
I20260809 23:21:39.554644 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:39.578078 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 212701us
I20260809 23:21:39.530616 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.516793  3232 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:39.540972 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3590) took 1395 ms (client timeout 1744 ms). Trace:
I20260809 23:21:39.564108 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.564718  3660 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:39.580855 24321 raft_consensus.cc:2396] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260809 23:21:39.550428 24428 rpcz_store.cc:269] 0809 23:21:37.820588 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:37.969857 (+149269us) service_pool.cc:227] Handling call
0809 23:21:39.549708 (+1579851us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.549900 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3218697472}
I20260809 23:21:39.526244  3607 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:39.570355 24422 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.572608 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.554914 24424 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:39.577314 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111742us
I20260809 23:21:39.517601  3369 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:39.584734 24544 rpcz_store.cc:269] 0809 23:21:38.145777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.244991 (+ 99214us) service_pool.cc:227] Handling call
0809 23:21:39.540718 (+1295727us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.540918 (+   200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2795722624}
I20260809 23:21:39.588714 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.565966 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:39.592562 24323 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:39.598037 24311 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:39.599432 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.599640 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:39.603029  3377 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.603456  3660 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:39.603559  2995 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.603916  3660 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:39.605057  3660 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.605510  3697 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260809 23:21:39.606482  3660 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.493642  3587 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.594903 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.611315 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.607864 24534 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:39.576995  3316 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:39.609397 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69267.3us
I20260809 23:21:39.617228 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.584743  3232 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:39.610375  3218 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.586869  3607 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:39.623608  3607 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:39.626134  3689 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:39.617473 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.629951 24424 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:39.607312 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.610846 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1700) took 2829 ms. Trace:
I20260809 23:21:39.611611 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.630864 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.627602 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.628158 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.628518 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.632623 24429 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260809 23:21:39.590297  3369 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:39.626886  3253 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:39.627269 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:39.635246  3253 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:39.635694  3253 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:39.635926  3316 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.630829 24541 rpcz_store.cc:276] 0809 23:21:36.781060 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:36.812490 (+ 31430us) service_pool.cc:227] Handling call
0809 23:21:36.812794 (+   304us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:36.863693 (+ 50899us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:38.871329 (+2007636us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b1800431860. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:21:38.871633 (+   304us) log.cc:842] Serialized 14 byte log entry
0809 23:21:39.046989 (+175356us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:39.272395 (+225406us) log.cc:842] Serialized 26 byte log entry
0809 23:21:39.272606 (+   211us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:39.272653 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.279843 (+  7190us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:39.579917 (+300074us) raft_consensus.cc:1671] finished
0809 23:21:39.595281 (+ 15364us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.610673 (+ 15392us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":922,"prepare.queue_time_us":990963,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":35070,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":18500,"spinlock_wait_cycles":56338688,"thread_start_us":1167575,"threads_started":2,"wal-append.queue_time_us":306678}
I20260809 23:21:39.638070  3316 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.638463 24266 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
W20260809 23:21:39.634315 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 241385us
I20260809 23:21:39.641125  3239 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:39.641525 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:39.641784  3239 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.643945  2820 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:39.644336  3239 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:39.645927  2820 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:39.588351  3677 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:39.647529  3556 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.558s	user 0.005s	sys 0.007s
I20260809 23:21:39.649020  3677 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.638847 24266 leader_election.cc:401] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20260809 23:21:39.633056 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67192.3us
I20260809 23:21:39.614868 24313 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:21:39.652073 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60303.3us
I20260809 23:21:39.648543  3587 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.561249  3411 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260809 23:21:39.654587 24321 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.654547 24431 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:39.660135  3604 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:39.661509 24427 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:39.618837  3232 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:39.619760  3676 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:39.646296  2820 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:39.635118  3369 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:39.650338  3154 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.671073 24431 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:39.686910 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3513) took 1267 ms. Trace:
W20260809 23:21:39.687569 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69243.4us
I20260809 23:21:39.687508 24542 rpcz_store.cc:276] 0809 23:21:38.419734 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.336194 (+916460us) service_pool.cc:227] Handling call
0809 23:21:39.541744 (+205550us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.574320 (+ 32576us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.686648 (+112328us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b5400012550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.686827 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384370048}
I20260809 23:21:39.688503 24266 leader_election.cc:398] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25.
I20260809 23:21:39.688836 24266 leader_election.cc:401] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:39.696633 24428 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:39.667249  3676 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.667551  2995 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:39.649803  3556 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.666774  3604 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:39.670077  2820 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
W20260809 23:21:39.708877 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 90310.2us
I20260809 23:21:39.680421  3349 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:39.717453 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.657265  3625 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.730458 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.694953  3689 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:39.748242  3730 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:39.748867  3730 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.709345  3676 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:39.753352 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 88044.1us
I20260809 23:21:39.754189 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.755518 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.756695  3411 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:39.757251  3411 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:39.757542  3411 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:39.757915  3411 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:39.759523 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68914.6us
I20260809 23:21:39.760620 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:39.715145  3604 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:39.769253 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3600) took 2383 ms. Trace:
I20260809 23:21:39.769646 24430 rpcz_store.cc:276] 0809 23:21:37.386118 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.352554 (+1966436us) service_pool.cc:227] Handling call
0809 23:21:39.551502 (+198948us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.574280 (+ 22778us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.769047 (+194767us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b540017edd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.769191 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311803136}
I20260809 23:21:39.771015  3208 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:39.771484  3208 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:39.771891  3208 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:39.773303  3730 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:39.774005  3726 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260809 23:21:39.779379 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3507) took 1657 ms. Trace:
I20260809 23:21:39.780280  3455 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.779713 24535 rpcz_store.cc:276] 0809 23:21:38.122501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.198128 (+1075627us) service_pool.cc:227] Handling call
0809 23:21:39.215348 (+ 17220us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:39.306647 (+ 91299us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.306678 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:39.307282 (+   604us) log.cc:842] Serialized 52 byte log entry
0809 23:21:39.352194 (+ 44912us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:39.352235 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.352301 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:39.773237 (+420936us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:39.779013 (+  5776us) raft_consensus.cc:1671] finished
0809 23:21:39.779035 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:39.779296 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1452,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":133,"spinlock_wait_cycles":5522176,"thread_start_us":44511,"threads_started":1,"wal-append.queue_time_us":145153}
W20260809 23:21:39.785557  3683 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.693s	user 0.005s	sys 0.006s
I20260809 23:21:39.785985  3683 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.787388 25375 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:39.790328  3689 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:39.790176 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.766249  3633 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.792883  3633 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.796821  3563 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.695737  3687 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25.
I20260809 23:21:39.726715  3349 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:39.803103 25365 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:39.782481  3455 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.811669 24496 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.809778 24427 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:39.856205 24425 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
W20260809 23:21:39.874577 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69039.8us
I20260809 23:21:39.876287  3431 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:39.832206  3675 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:39.876849  3431 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:39.877099  3675 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.877504  3675 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:39.877950  3431 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.878567 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1734) took 1120 ms. Trace:
I20260809 23:21:39.884928 25375 rpcz_store.cc:276] 0809 23:21:38.758759 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.759045 (+   286us) service_pool.cc:227] Handling call
0809 23:21:39.878321 (+1119276us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54002459d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.878471 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2202479488}
I20260809 23:21:39.887501  3607 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.880834  3563 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:39.871363  3207 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.889058  3253 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.880131  3556 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:39.811776  3377 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:39.870038  3593 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.004s
I20260809 23:21:39.720124  2995 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:39.919024 24545 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:39.801565  3349 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.926505 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.946635 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:39.948890 24435 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:39.920006 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3503) took 1838 ms (client timeout 1718 ms). Trace:
I20260809 23:21:39.891589  3607 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:39.951750 24534 rpcz_store.cc:269] 0809 23:21:38.081235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:38.103440 (+ 22205us) service_pool.cc:227] Handling call
0809 23:21:39.902119 (+1798679us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:39.919910 (+ 17791us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2852201216}
I20260809 23:21:39.964002 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:39.968840 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3514) took 1127 ms. Trace:
I20260809 23:21:39.944116  2995 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:39.974731 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:39.969340 24545 rpcz_store.cc:276] 0809 23:21:38.841898 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.302207 (+460309us) service_pool.cc:227] Handling call
0809 23:21:39.302485 (+   278us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:39.934600 (+632115us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.968635 (+ 34035us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54001fd954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:39.968784 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1417308416}
I20260809 23:21:39.895646  3604 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.951558  3556 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:21:39.983139 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 341679us
I20260809 23:21:39.871277  3676 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.986557 24425 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:39.964231  3648 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:39.987736 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:39.890864  2820 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:39.915968  3396 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:39.987414  3349 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:40.000658 24429 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:39.921123  3593 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.004019  3676 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.007694 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:40.008648 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92424.4us
I20260809 23:21:40.011380 24430 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:39.945822  3259 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260809 23:21:39.985558  3607 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:40.001086  3207 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:40.027952  3253 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:40.031059 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.031201 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.031078 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.035818  3232 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.084271  3232 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:40.086052  3316 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:40.053557  2820 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.055413  3396 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:40.060411  3411 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.067301 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:40.082157 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61844.6us
I20260809 23:21:40.092901 24535 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:21:40.093097 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:40.094269 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 72406.8us
I20260809 23:21:40.090371  3607 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.096819 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.095598 24545 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:40.095283 24534 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 16.
I20260809 23:21:40.102707 24380 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:40.080482  3625 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.106055  2992 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:40.091598  3207 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
W20260809 23:21:40.112238 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 350036us
W20260809 23:21:40.118336 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 477844us
W20260809 23:21:40.119616 24423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.000s	sys 0.031s
W20260809 23:21:40.119868 24155 tablet_replica.cc:1397] Time spent applying in-flights took a long time: real 0.368s	user 0.000s	sys 0.000s
W20260809 23:21:40.121284 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 496897us
I20260809 23:21:40.125687  2995 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.126336 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.130108 25372 raft_consensus.cc:2379] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:40.130596 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1741) took 1026 ms. Trace:
I20260809 23:21:40.131031 25366 rpcz_store.cc:276] 0809 23:21:39.103716 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.374559 (+270843us) service_pool.cc:227] Handling call
0809 23:21:39.410855 (+ 36296us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:39.442845 (+ 31990us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.442882 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:39.517235 (+ 74353us) log.cc:842] Serialized 26 byte log entry
0809 23:21:39.950433 (+433198us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:39.950495 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:39.950593 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:40.130380 (+179787us) raft_consensus.cc:1671] finished
0809 23:21:40.130403 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:40.130534 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":382552,"threads_started":1,"wal-append.queue_time_us":578153}
I20260809 23:21:40.158972 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:40.115705 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 310826us
I20260809 23:21:40.163492  2995 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:21:40.184217 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 87551.7us
I20260809 23:21:40.185514 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:40.187080 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 162393us
I20260809 23:21:40.187469 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.188506 24545 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:21:40.189630 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.190460 24535 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:40.191578 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.193109 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.194298 24539 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:40.195541 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.197576 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.198392 24540 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:40.199381 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.201864 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.202539 24534 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:40.202852  2995 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:40.205395  3407 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:40.205838  3407 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:40.208436 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.210456 24435 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:40.211303 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95641.6us
I20260809 23:21:40.216967 24542 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.217689 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:40.218562 24425 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:40.216249 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.109932  3563 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:40.224790 24435 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:21:40.112367  3683 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:40.207412  3751 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260809 23:21:40.226699 24424 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.561s	user 0.008s	sys 0.003s
I20260809 23:21:40.237792 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:40.115427  3207 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.242587 24541 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260809 23:21:40.249701 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.244421 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.241415 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:40.254833 24545 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:40.258544 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.259876 24422 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:40.261066 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.262465 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.265138 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.267627 24542 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.268461 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.269550 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.267030 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.267057 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:40.278030 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92535.2us
I20260809 23:21:40.280404 24539 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:40.282310 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.283108 24545 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:40.271708 24543 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
W20260809 23:21:40.285396 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48024.8us
I20260809 23:21:40.278018  3411 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:40.287252 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56720.7us
I20260809 23:21:40.206910  3407 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.287946  3689 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:40.288097 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.288512  3689 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:40.288597  3407 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:40.289004  3689 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:40.290762 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.291174 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.292153 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.294183 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.294632 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:40.111588  3607 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.301882 24422 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:40.280714 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:40.302484 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.018s	sys 0.080s
W20260809 23:21:40.302923 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.018s	sys 0.080s
W20260809 23:21:40.263439  3660 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.653s	user 0.012s	sys 0.003s
I20260809 23:21:40.296850 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.299543 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.300557 24546 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:40.300659 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.303530 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.301473 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.246480  3754 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:40.309317  3675 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.307627 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:40.311246 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48965.9us
I20260809 23:21:40.227623  3347 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260809 23:21:40.332314 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.273669  2820 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:40.333231 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:40.342815 24435 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:40.309989  3687 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:40.331400 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.352101 24309 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:40.336611 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:40.336627 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:40.336076 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 49647us
I20260809 23:21:40.241385  3396 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260809 23:21:40.341425 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:40.341696 24542 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:40.338497 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.355852 24424 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:40.305039  3563 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:40.305133  3557 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:21:40.345492  3677 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.039s	sys 0.116s
I20260809 23:21:40.310727  3625 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:40.363708 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.313634  2736 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:21:40.337993  3208 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.566s	user 0.011s	sys 0.002s
I20260809 23:21:40.364666 24541 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260809 23:21:40.366341 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.366920 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.381433 24546 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:40.367748 24422 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:40.278852  3232 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260809 23:21:40.368314 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.370227 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.352921  3556 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:40.332564  3604 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.344518  3431 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:40.367293 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.310393  3316 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.059s
I20260809 23:21:40.377213 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.371129 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.406869 24435 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:40.385074 25365 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:40.386592 25375 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:40.387748 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.389621 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.410311 25371 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 24
I20260809 23:21:40.381093 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.411312 25372 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:40.411434  3740 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.106s
I20260809 23:21:40.412403 25372 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:40.405143 24313 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:40.364500  3593 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:40.393527 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.394567 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:40.366602  3677 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.039s	sys 0.116s
I20260809 23:21:40.361720  3676 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:40.423152 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.407399 24323 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:40.358470  3754 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.382171 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.418062 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.427507 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.428037 24534 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:40.428258 24424 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:40.418660 24541 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.427389 24312 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:40.413442 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.426301 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.429754 25366 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:40.431182 24540 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.353262  3239 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260809 23:21:40.364341  2820 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260809 23:21:40.429934 24435 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:40.432981 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:40.364086  3170 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.050s
I20260809 23:21:40.439390 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.386826  3563 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:40.414727  3369 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.716s	user 0.012s	sys 0.000s
I20260809 23:21:40.444036 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.417466  3431 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.452131 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.422262  3648 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.452744 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.454538 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.453744 24435 raft_consensus.cc:2354] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:21:40.459578 24424 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260809 23:21:40.440990  3676 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.387095  3208 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.462236 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.447589  3593 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:40.463220 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.470523 24422 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260809 23:21:40.471011 24422 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 16
I20260809 23:21:40.471297 24422 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260809 23:21:40.471592 24422 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:40.450124  3316 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:40.446370  3556 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:40.464852 24321 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:40.478438 24323 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 13
I20260809 23:21:40.472345 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.478787 24323 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260809 23:21:40.456516  3557 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:40.491914 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:40.492654 24435 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:40.469969 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.479228 24323 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:40.458662  3154 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:40.495272 24427 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:40.500692 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.452700  3633 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260809 23:21:40.448802  3369 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.476233  3316 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:40.515959 25373 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260809 23:21:40.518941 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 209259us
I20260809 23:21:40.518379 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.480127  3593 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.491554  3556 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:40.525439 24435 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:40.529079 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.528396 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.500829  3557 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.552083 24428 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:40.463169  3734 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.218s
I20260809 23:21:40.478513  3604 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260809 23:21:40.582312 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.583906 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.526482  3675 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.528254  3556 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:40.581579  3349 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.001s
I20260809 23:21:40.596596 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.602648 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.619513 24435 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:40.638186 24313 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:40.592549 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.555s (SENT) [suppressed 17 similar messages]
I20260809 23:21:40.664160 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.665066 24313 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:40.599429  3734 consensus_queue.cc:1225] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:21:40.676029 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.551177  3411 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
W20260809 23:21:40.683022 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47697.9us
I20260809 23:21:40.679451 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:40.688423 24435 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:21:40.551092  3563 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.712252 25375 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:40.713541 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:40.635429  3347 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260809 23:21:40.715626 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.717689 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:40.708317  3745 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.397s	user 0.001s	sys 0.000s
W20260809 23:21:40.724733 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132225us
I20260809 23:21:40.583356  3431 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.584884  3648 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.732352 24545 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:40.593901  3676 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.530036  3754 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.714701  3604 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:40.760321 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.761498 25375 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260809 23:21:40.745229 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.641731  3316 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.766016 25373 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:40.769409 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 70705.5us
I20260809 23:21:40.772738 24435 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:40.735826  3208 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:40.697635  3557 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:40.775789  3773 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
W20260809 23:21:40.780077  2992 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.000s
I20260809 23:21:40.781123  3232 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:40.783140 24546 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:40.786064 25371 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:21:40.772225  3742 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.297s	user 0.001s	sys 0.001s
W20260809 23:21:40.793804  3587 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.129s	user 0.014s	sys 0.000s
I20260809 23:21:40.767939  3604 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.807449 24378 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:40.801110 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.808521 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.735097  2736 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260809 23:21:40.809362 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.809098 24435 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:40.811617 24314 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260809 23:21:40.727936  3593 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.791278  3369 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:40.730456  3763 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
I20260809 23:21:40.796573  2992 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:40.796420  3232 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:40.829100 24431 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:40.830555  3232 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.831688  3232 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:40.832983 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1823) took 1165 ms. Trace:
I20260809 23:21:40.833387 25368 rpcz_store.cc:276] 0809 23:21:39.667545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.673756 (+  6211us) service_pool.cc:227] Handling call
0809 23:21:39.691284 (+ 17528us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:39.691520 (+   236us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:39.691553 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:39.692007 (+   454us) log.cc:842] Serialized 26 byte log entry
0809 23:21:40.303578 (+611571us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:40.303635 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:40.303724 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:40.832787 (+529063us) raft_consensus.cc:1671] finished
0809 23:21:40.832808 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:40.832929 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":611175,"threads_started":1,"wal-append.queue_time_us":754572}
I20260809 23:21:40.813562  3208 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:40.836608  3208 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.837518  3208 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.838943 24312 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:40.841255 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.839879 24431 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:40.832547  3604 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.842085 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.844233 24534 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:40.846684 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.845314 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.853024 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.855073 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.856556 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:40.831957  3760 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260809 23:21:40.862485 25321 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:21:40.862896 25321 leader_election.cc:401] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:40.863940 24425 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:40.859153 25373 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:40.865501 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.865862 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.866650 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.866986 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.872051 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.882579 24546 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:40.870844 24534 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.871690 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:40.885962 24321 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:40.889885 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.890647 24314 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:40.891613 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.892369 24543 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:40.893028 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.894029 25373 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:40.861986  3369 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:40.896394 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:40.898518 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:40.892264 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:40.901191 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.851378  3411 raft_consensus.cc:2741] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260809 23:21:40.880434  3689 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.591s	user 0.012s	sys 0.000s
I20260809 23:21:40.869153  3675 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:40.909070 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.907747 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.924474 24425 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260809 23:21:40.925438 25369 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:40.929889 24546 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:40.917227  3633 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260809 23:21:40.936555  3633 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:40.935371  3411 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260809 23:21:40.937836  3411 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:40.938221  3411 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:40.936987  3633 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:40.942965  3633 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.944911  3633 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:40.946702 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.917851  3683 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260809 23:21:40.947777  3783 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:40.906718  3557 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:40.949054 24321 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:40.948312  3783 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.955487 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:40.956938 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:40.958318 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 239606us
I20260809 23:21:40.959342 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:40.908715  3587 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:21:40.917974  3675 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:40.965474 24266 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:40.965708 24266 leader_election.cc:401] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260809 23:21:40.948664  3557 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:40.964443 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 65999.6us
I20260809 23:21:40.919332  3529 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:40.969367 25321 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:21:40.969693 25321 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:40.949626  3272 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:40.974052 24423 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:40.939258  3411 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:40.976215 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.973439 24321 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
W20260809 23:21:40.968981  3777 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260809 23:21:40.978529 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.978879 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:40.967541  3675 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:40.913715  3689 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.979423 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:40.964411  3347 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260809 23:21:40.990909 24423 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260809 23:21:40.992980 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:40.974108  3754 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:40.989480 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.985371 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:40.984089  3793 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.012s
I20260809 23:21:40.991573  3625 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:40.975719  3529 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:41.127092 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:41.035882  3789 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:40.966008  3759 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.121s
I20260809 23:21:41.140028 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.041010 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:41.151943 25363 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:41.121351  3218 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:41.095908 24309 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:41.123847 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:40.996194  3557 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:40.968947  3763 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260809 23:21:40.993849  3411 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:41.044116  3689 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:40.988817  3687 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.633s	user 0.007s	sys 0.004s
I20260809 23:21:41.047991  3431 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:40.995524  3587 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.043s
W20260809 23:21:41.047921  3407 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.759s	user 0.012s	sys 0.001s
I20260809 23:21:40.985172  3677 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:21:41.088681 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.040203 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.977303  3788 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:41.043398 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:41.039670  3783 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:41.037598 24155 maintenance_manager.cc:643] P e3357d0fe4ef46db9a4739044fa527d5: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 2.188s	user 0.036s	sys 0.008s Metrics: {"bytes_written":4457,"delete_count":0,"lbm_write_time_us":4166,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":160}
I20260809 23:21:41.043246 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:41.136409 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:41.053594  3779 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260809 23:21:41.049933 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:41.048492 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:40.996857  3754 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:41.150821 24321 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:41.185184 24422 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:21:41.005653  3556 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.243810 25373 raft_consensus.cc:2379] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:41.186261 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:41.244355 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 248006us
I20260809 23:21:41.171950  3272 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260809 23:21:41.244967 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.246384 24157 maintenance_manager.cc:419] P e3357d0fe4ef46db9a4739044fa527d5: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.195303
W20260809 23:21:41.182789 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203979us
I20260809 23:21:41.192724  3625 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:41.188222 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:41.201093 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 390542us
I20260809 23:21:41.255450 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:41.143525  3683 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260809 23:21:41.208117 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:41.211462 24313 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:41.236821 25369 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260809 23:21:41.243080  3754 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:41.205147  3687 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:41.193607 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 210917us
I20260809 23:21:41.263943 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:41.259884 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:41.192883  3218 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.235020  3232 raft_consensus.cc:465] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Not starting pre-election -- already a leader
I20260809 23:21:41.155956  3789 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.253252 24321 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:41.265712 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:41.265175 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:41.241662  3648 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:41.269271 24320 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:41.269313 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 294809us
I20260809 23:21:41.197731  3740 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260809 23:21:41.271245  3625 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.314276 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:41.281055  3556 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:41.296636 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:41.306530 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67841.5us
W20260809 23:21:41.306563 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 118450us
I20260809 23:21:41.332799 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:41.307168 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.332079 24313 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
W20260809 23:21:41.275682  3797 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260809 23:21:41.338815 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:41.315517 24546 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:41.333108 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:41.338815 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:41.330032 24423 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:21:41.341185 24545 raft_consensus.cc:2354] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:41.270495  3604 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:41.342514 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.343873 24545 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:41.346297  3587 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:41.299068  3563 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:41.350994 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.355592 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.356125 24321 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:41.356815 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:41.358646 25368 raft_consensus.cc:2396] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260809 23:21:41.349875 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:41.353685 24313 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:41.360599 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.357589 24321 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:41.363452 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:41.364032 24313 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:41.364521 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 99403.1us
I20260809 23:21:41.369763 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:41.374751 24320 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:41.423780 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:41.424077 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:41.368081  3587 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.430116 24543 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:41.342814  3675 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.362066  3604 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:41.444934 24314 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.554s	user 0.009s	sys 0.001s
I20260809 23:21:41.330894  3648 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.457856 24542 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:41.445323 24313 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:41.454244  3377 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:21:41.447695 24314 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:41.462702 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:41.464131 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.475072 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:41.476668 24313 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:41.472154  3793 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:41.479135 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:41.481631  3604 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.441905  3170 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:41.475617 24321 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:41.263595  3529 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.427563  3823 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:41.488956  3823 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.491541  3823 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:41.492843 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 160518us
I20260809 23:21:41.477568 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.427085  3759 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260809 23:21:41.424546  3563 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260809 23:21:41.570247 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:41.522693 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90080.9us
I20260809 23:21:41.502462  3812 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260809 23:21:41.603374 24544 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:21:41.554636 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 130941us
I20260809 23:21:41.552217  3806 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:41.497496 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:41.502717 24321 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:41.572595 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3571) took 1059 ms. Trace:
I20260809 23:21:41.576141  3804 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:41.501060 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:41.460577  3683 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:41.553138 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47912.9us
I20260809 23:21:41.592697 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:41.602155 24313 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:41.559320  3816 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:41.577848  3828 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:41.614890 24323 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:41.608709 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:41.605024 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:41.625597 24313 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:41.604557 24314 rpcz_store.cc:276] 0809 23:21:40.513377 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.480221 (+966844us) service_pool.cc:227] Handling call
0809 23:21:41.480525 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:41.480744 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:41.482031 (+  1287us) log.cc:842] Serialized 14 byte log entry
0809 23:21:41.572145 (+ 90114us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:41.572215 (+    70us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:41.572284 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:41.572378 (+    94us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:41.572527 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":342,"prepare.queue_time_us":140,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":7400,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":11946,"thread_start_us":86888,"threads_started":1}
I20260809 23:21:41.625748 24542 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:41.626195 24313 raft_consensus.cc:595] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.626539 24313 raft_consensus.cc:603] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:41.612155 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1818) took 1029 ms. Trace:
I20260809 23:21:41.627254 24321 rpcz_store.cc:276] 0809 23:21:40.583347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.498232 (+914885us) service_pool.cc:227] Handling call
0809 23:21:41.498604 (+   372us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:41.611875 (+113271us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:41.612098 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:41.628729 24321 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:41.630039 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.632630 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.642592 24380 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:41.647972 24423 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:41.650581 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 342580us
W20260809 23:21:41.624233  3798 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260809 23:21:41.607595  3812 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:41.613034  3218 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:41.656936 24321 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260809 23:21:41.618306  3763 raft_consensus.cc:2741] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
W20260809 23:21:41.649376  3799 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.325s	user 0.001s	sys 0.000s
W20260809 23:21:41.671778 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101637us
I20260809 23:21:41.638677  3232 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:41.660997 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:41.680742 24314 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:41.652245  3804 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:41.683135 24270 leader_election.cc:398] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19.
I20260809 23:21:41.683934 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:41.583050  3207 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.691835 24541 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260809 23:21:41.613790  3683 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.660579  3812 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 16 FOLLOWER]: Advancing to term 17
W20260809 23:21:41.724584 25370 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.508s	user 0.009s	sys 0.007s
I20260809 23:21:41.725561 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:41.725870 25375 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:41.704798 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:41.732055 25366 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:41.735395 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:41.747311 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.690794  3563 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:41.667959  3825 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260809 23:21:41.750681 24546 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:41.625052  3828 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.767295 25369 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:41.768306 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.695331  3675 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:41.766889 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3634) took 1608 ms (client timeout 2061 ms). Trace:
W20260809 23:21:41.784317 24428 rpcz_store.cc:269] 0809 23:21:40.158078 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.218913 (+ 60835us) service_pool.cc:227] Handling call
0809 23:21:41.766658 (+1547745us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:41.766815 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":714801024}
I20260809 23:21:41.771247 24311 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
W20260809 23:21:41.739681  3778 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:21:41.728243  3804 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:41.786005 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:41.711982  3556 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:41.785466 24311 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:41.739374  3836 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:41.730472  3789 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:41.740121  3207 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:41.789127 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.792431 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:41.802034 24546 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:41.788883 25365 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:41.792631 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:41.782605  3800 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260809 23:21:41.806594 24431 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:41.756429  3587 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:41.800173 24311 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.782155  3816 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:41.741425  3844 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:41.753520  3763 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260809 23:21:41.822158 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:41.850206 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 217664us
I20260809 23:21:41.851718 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:41.788319  3648 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:41.872689 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:41.858418 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.016s	sys 0.023s
I20260809 23:21:41.860514 24311 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:41.855355  3834 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260809 23:21:41.772926  3557 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260809 23:21:41.841910  3789 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:41.876251 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.016s	sys 0.024s
I20260809 23:21:41.820997  3836 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:41.900615 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:41.806572  3810 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260809 23:21:41.915323 24428 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 19: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:41.867094  3844 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:41.787377  3820 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.125s
I20260809 23:21:41.813632  3675 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.088s
W20260809 23:21:41.996064 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 207027us
I20260809 23:21:42.055253  2736 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:42.055688  2736 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:42.055487 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.056646  2736 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:41.861071  3768 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:42.058864  3272 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:41.919803  3836 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.059340  3272 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:42.060297  3844 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:42.060868  3844 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:42.061213  3836 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.060771  3272 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:41.906256  3259 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260809 23:21:42.062552  3839 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:42.062770 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:42.064003 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.064713 24423 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:42.064579  3763 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:42.065220  3763 consensus_queue.cc:1225] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:42.065762 24423 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
W20260809 23:21:42.067274  3818 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
W20260809 23:21:42.067922  3857 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:21:41.963066 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:41.901371  3755 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
W20260809 23:21:41.984050  3633 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.039s	user 0.016s	sys 0.007s
I20260809 23:21:41.985325  3825 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.009954  3687 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:42.075892 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 112984us
I20260809 23:21:41.878628  3347 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:41.971573  3789 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:41.969553  3876 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:42.058545  3814 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:42.061487  3844 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:42.005635 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.013s	sys 0.038s
W20260809 23:21:42.103024 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.018s	sys 0.038s
I20260809 23:21:42.080776 24311 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260809 23:21:42.103549 24311 pending_rounds.cc:76] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 14
I20260809 23:21:42.103811 24311 pending_rounds.cc:97] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260809 23:21:42.085628 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1737) took 1107 ms. Trace:
I20260809 23:21:42.080412 25373 pending_rounds.cc:76] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 13
W20260809 23:21:42.072345  3831 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.418s	user 0.001s	sys 0.000s
I20260809 23:21:42.090775 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1814) took 1695 ms. Trace:
I20260809 23:21:42.090956 25372 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260809 23:21:42.096705 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1539) took 1278 ms. Trace:
I20260809 23:21:42.109202 25373 pending_rounds.cc:97] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260809 23:21:42.109637 25373 raft_consensus.cc:2866] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:42.104067 24311 raft_consensus.cc:2866] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:42.111397 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3564) took 1759 ms. Trace:
I20260809 23:21:42.108989 25368 rpcz_store.cc:276] 0809 23:21:40.978426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.978621 (+   195us) service_pool.cc:227] Handling call
0809 23:21:42.079995 (+1101374us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.085558 (+  5563us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":826000640}
I20260809 23:21:42.105558 24428 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:42.108846 24435 rpcz_store.cc:276] 0809 23:21:40.818642 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.818875 (+   233us) service_pool.cc:227] Handling call
0809 23:21:40.819562 (+   687us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:40.840821 (+ 21259us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:40.840858 (+    37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:40.841386 (+   528us) log.cc:842] Serialized 52 byte log entry
0809 23:21:41.198624 (+357238us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:41.198653 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:41.198699 (+    46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:42.096290 (+897591us) raft_consensus.cc:1671] finished
0809 23:21:42.096316 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:42.096505 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":224,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":23278,"thread_start_us":313420,"threads_started":1,"wal-append.queue_time_us":476461}
I20260809 23:21:42.109211 24309 rpcz_store.cc:276] 0809 23:21:40.395189 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.395385 (+   196us) service_pool.cc:227] Handling call
0809 23:21:40.429440 (+ 34055us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:41.192107 (+762667us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.090514 (+898407us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:42.090708 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1306891008}
I20260809 23:21:42.111784 24311 rpcz_store.cc:276] 0809 23:21:40.352226 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.352536 (+   310us) service_pool.cc:227] Handling call
0809 23:21:40.384372 (+ 31836us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:41.771149 (+1386777us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400451620. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:21:42.111120 (+339971us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.111337 (+   217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":675963136}
I20260809 23:21:42.093716  3755 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:42.095355  3825 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.097815  3683 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:42.177202 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.090086  3874 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
W20260809 23:21:42.183430 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 850655us
I20260809 23:21:42.003176  3804 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.196547 25325 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:21:42.168726  3272 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:42.198264 25325 leader_election.cc:398] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:21:42.199096 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:42.205662 24534 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.006s	sys 0.010s
I20260809 23:21:42.205696 24542 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260809 23:21:42.206182 24534 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.006s	sys 0.010s
W20260809 23:21:42.203614 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 331005us
I20260809 23:21:42.208897 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.103256  3633 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
W20260809 23:21:42.192696  3677 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.977s	user 0.012s	sys 0.001s
I20260809 23:21:42.180294  3208 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.252103 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:42.114727  3347 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.095906  3687 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:42.304697 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.051s	sys 0.063s
W20260809 23:21:42.328083 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.978s	user 0.051s	sys 0.063s
W20260809 23:21:42.341625 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164660us
I20260809 23:21:42.205890  3828 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.258850  3814 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:42.331458  3754 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.024s	user 0.008s	sys 0.004s
I20260809 23:21:42.317710  3677 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
W20260809 23:21:42.372316 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 173279us
I20260809 23:21:42.373229 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:42.376291 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.378981 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.378589  2995 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:42.382558  3735 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:42.382740  2995 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:42.383275  3735 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:42.384002  2995 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.379689 24424 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:42.384438  3735 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.386077 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.386432  3735 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:42.387126  3836 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:42.391430 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:42.327729  3842 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.517s	user 0.000s	sys 0.001s
I20260809 23:21:42.306432  3874 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:42.392664 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.316908 25369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:42.401158 25369 raft_consensus.cc:595] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:42.402685 25369 raft_consensus.cc:603] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:21:42.356837 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 241490us
W20260809 23:21:42.388460  3845 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.355s	user 0.001s	sys 0.001s
I20260809 23:21:42.427018 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:42.325784  3208 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.428541 24435 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:42.387194  3837 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:42.424295  3821 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.419s	user 0.000s	sys 0.001s
I20260809 23:21:42.442529 25363 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:42.429435  3411 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.230s	user 0.013s	sys 0.001s
I20260809 23:21:42.396400  3814 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:42.395784  3677 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:42.456053  3879 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
W20260809 23:21:42.456406  3846 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:21:42.456840  3815 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:42.457791  3863 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
I20260809 23:21:42.462031 24435 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:42.463007 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 70447.2us
I20260809 23:21:42.369129  3754 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:42.491809  3854 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
I20260809 23:21:42.486783  3660 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:42.468701  3272 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:42.495399  3852 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.493s	user 0.000s	sys 0.001s
I20260809 23:21:42.591507  3347 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:42.595558  3349 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260809 23:21:42.603868  3844 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.606166  3844 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.449414  3154 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.127s
W20260809 23:21:42.609400  3905 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260809 23:21:42.611336  3812 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.855s	user 0.007s	sys 0.007s
I20260809 23:21:42.611471  3825 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.611917  3812 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.495748  3890 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.496879  3689 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:42.489293 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:42.378904  3755 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:42.497730  3872 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260809 23:21:42.499595  3900 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260809 23:21:42.503571  3873 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260809 23:21:42.503285  3874 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:42.504444  3883 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260809 23:21:42.505615  3279 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:42.439357  3587 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:42.506979  3687 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.507012  3783 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:42.511662 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3627) took 2549 ms. Trace:
I20260809 23:21:42.630797 24427 rpcz_store.cc:276] 0809 23:21:39.962239 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:39.969777 (+  7538us) service_pool.cc:227] Handling call
0809 23:21:39.970153 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:40.503023 (+532870us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.511370 (+2008347us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:42.511574 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1201793408}
W20260809 23:21:42.512697  3865 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260809 23:21:42.513115 25370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1725) took 1877 ms (client timeout 1944 ms). Trace:
I20260809 23:21:42.520284 25371 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260809 23:21:42.522423 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3002) took 1822 ms (client timeout 1667 ms). Trace:
W20260809 23:21:42.641299 24317 rpcz_store.cc:269] 0809 23:21:40.699625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.522154 (+1822529us) service_pool.cc:214] Skipping call since client already timed out
0809 23:21:42.522308 (+   154us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260809 23:21:42.524472  3764 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.833s	user 0.003s	sys 0.000s
I20260809 23:21:42.619514  3689 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:42.526777 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3647) took 1259 ms. Trace:
I20260809 23:21:42.645983 24425 rpcz_store.cc:276] 0809 23:21:41.267322 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.291046 (+ 23724us) service_pool.cc:227] Handling call
0809 23:21:41.291442 (+   396us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:41.309972 (+ 18530us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:41.310004 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:41.367400 (+ 57396us) log.cc:842] Serialized 26 byte log entry
0809 23:21:41.586836 (+219436us) raft_consensus.cc:1631] Marking committed up to 10
0809 23:21:41.586887 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:41.589584 (+  2697us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:42.526533 (+936949us) raft_consensus.cc:1671] finished
0809 23:21:42.526561 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:42.526711 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18100,"raft.queue_time_us":38051,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4496,"thread_start_us":140782,"threads_started":1,"wal-append.queue_time_us":271807}
I20260809 23:21:42.452715  3837 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:42.533671 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1735) took 1618 ms (client timeout 1807 ms). Trace:
W20260809 23:21:42.647966 25372 rpcz_store.cc:269] 0809 23:21:40.915451 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.916033 (+   582us) service_pool.cc:227] Handling call
0809 23:21:42.533432 (+1617399us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540076c5d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.533602 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2605142784}
W20260809 23:21:42.498690 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 289847us
W20260809 23:21:42.538561 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73807.5us
I20260809 23:21:42.650123 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.448771  3633 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:42.626250  3900 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:42.555164 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 177812us
I20260809 23:21:42.425266  3683 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
W20260809 23:21:42.499614  3855 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.330s	user 0.000s	sys 0.000s
I20260809 23:21:42.518148  3593 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
I20260809 23:21:42.578848  3660 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:42.585279  3272 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.573500  3815 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:42.525614  3208 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:42.639921 25370 rpcz_store.cc:269] 0809 23:21:40.635989 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.650035 (+ 14046us) service_pool.cc:227] Handling call
0809 23:21:42.512894 (+1862859us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54000c4754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.513043 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1228077440}
W20260809 23:21:42.681528 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 437087us
W20260809 23:21:42.681622 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3565) took 2288 ms (client timeout 2156 ms). Trace:
I20260809 23:21:42.686075 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:42.645252 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.727s	user 0.003s	sys 0.006s
W20260809 23:21:42.687218 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.003s	sys 0.007s
I20260809 23:21:42.352581  3789 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.627111  3279 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:42.395511  3836 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.656529 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.650426  3812 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.640089  3783 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:42.688797 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.689985 24313 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:42.655534  3900 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:42.645095  3806 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:42.682228 24320 rpcz_store.cc:269] 0809 23:21:40.368624 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.368887 (+   263us) service_pool.cc:227] Handling call
0809 23:21:42.656575 (+2287688us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.656806 (+   231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1954037120}
W20260809 23:21:42.642796  3676 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.447s	user 0.008s	sys 0.002s
I20260809 23:21:42.706020 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.706975 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.707188 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:42.720338 24309 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:42.708424 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:42.651885  3687 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.710623 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:42.711433 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.695420 24323 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:21:42.680367  3660 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:42.702549 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:42.721694 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 307568us
I20260809 23:21:42.714365 24423 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:21:42.693104  3815 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:42.698968  3207 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.887s	user 0.011s	sys 0.001s
I20260809 23:21:42.717458 24428 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:42.706054 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.676728  3587 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:42.721325 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:42.622735  3890 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.737758 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:42.703928  3816 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:42.740808  3885 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.741381  3885 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.743258  3411 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:42.743392  3232 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.743889  3232 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.745460 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:42.746637  3807 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.562s	user 0.000s	sys 0.009s
I20260809 23:21:42.746735 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:42.747403 24542 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:42.749533 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3623) took 1883 ms (client timeout 2188 ms). Trace:
W20260809 23:21:42.749907 24544 rpcz_store.cc:269] 0809 23:21:40.865701 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.866754 (+  1053us) service_pool.cc:227] Handling call
0809 23:21:42.749388 (+1882634us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.749481 (+    93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":553945984}
I20260809 23:21:42.751394 24544 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260809 23:21:42.752211 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:42.752523 24539 raft_consensus.cc:2396] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260809 23:21:42.752774 24544 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
W20260809 23:21:42.753720 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3612) took 2455 ms (client timeout 2846 ms). Trace:
I20260809 23:21:42.753983 24541 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260809 23:21:42.754367 24541 pending_rounds.cc:76] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 14
W20260809 23:21:42.754091 24539 rpcz_store.cc:269] 0809 23:21:40.298493 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.299333 (+   840us) service_pool.cc:227] Handling call
0809 23:21:42.753536 (+2454203us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b54003dc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.753672 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5624994560}
I20260809 23:21:42.754719 24541 pending_rounds.cc:97] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260809 23:21:42.755025 24541 raft_consensus.cc:2866] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:42.756405  3279 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.757884 25366 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:42.758123 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1811) took 2469 ms. Trace:
I20260809 23:21:42.748225 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:42.722715  3676 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.531s	user 0.008s	sys 0.002s
I20260809 23:21:42.730640  3689 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.759642 24542 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:42.761791 24321 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:42.766656 24321 raft_consensus.cc:595] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:42.758479 24316 rpcz_store.cc:276] 0809 23:21:40.289196 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.296029 (+  6833us) service_pool.cc:227] Handling call
0809 23:21:40.308613 (+ 12584us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:40.309044 (+   431us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:40.309075 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:40.309517 (+   442us) log.cc:842] Serialized 26 byte log entry
0809 23:21:40.513808 (+204291us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:40.513848 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:40.513913 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:42.523550 (+2009637us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:42.757835 (+234285us) raft_consensus.cc:1671] finished
0809 23:21:42.757857 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:42.758066 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1331,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":4994,"spinlock_wait_cycles":25533696,"thread_start_us":203794,"threads_started":1,"wal-append.queue_time_us":289751}
I20260809 23:21:42.767277 24321 raft_consensus.cc:603] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:42.768081 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:42.768805 24539 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:42.756456 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3628) took 1007 ms. Trace:
I20260809 23:21:42.769697 24541 rpcz_store.cc:276] 0809 23:21:41.748969 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.749849 (+   880us) service_pool.cc:227] Handling call
0809 23:21:42.753661 (+1003812us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.755424 (+  1763us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.756301 (+   877us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54003dc550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:42.756409 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2231695104}
I20260809 23:21:42.758354  3279 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:42.773355 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56107.9us
I20260809 23:21:42.731798  3683 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:42.772830 24534 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:42.733129  3876 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:42.781791 24425 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260809 23:21:42.759490  2992 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.759644  3816 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:42.738430  3820 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:42.706256  3625 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
I20260809 23:21:42.739341  3587 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:42.760749  3812 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:42.802350 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 152112us
I20260809 23:21:42.766405  3207 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:42.697847  3892 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.099s
I20260809 23:21:42.810325 24541 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
W20260809 23:21:42.815840 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52950.9us
I20260809 23:21:42.728333  3687 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:42.836401 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74676.6us
I20260809 23:21:42.792413  3683 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:42.888473 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:42.800526  3689 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:42.889083 25372 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:42.728446  3806 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:42.836189  3818 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260809 23:21:42.890589  3750 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:42.801169  3876 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:42.887315 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 91465.3us
I20260809 23:21:42.893612 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:42.886286 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 111294us
I20260809 23:21:42.817040  3587 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:42.894719  3919 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:42.790822  3677 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.095s
I20260809 23:21:42.843918  3207 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:42.923379 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 84847.9us
I20260809 23:21:42.888257  3844 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:42.892014  3683 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:42.836138  3675 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260809 23:21:42.936225 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:42.937165 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.937570 24313 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260809 23:21:42.938742 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:42.939520 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123290us
I20260809 23:21:42.934676 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.818372  2992 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.794894  3783 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.915582  3885 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.957639 24311 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:42.958009 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.958587 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.959864 24313 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260809 23:21:42.962049  3877 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:42.962203  3870 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.168s
I20260809 23:21:42.962687  3877 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.963099  3770 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
W20260809 23:21:42.964121  3934 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:21:42.964994  3877 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.965924  3890 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:42.965758 24151 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260809 23:21:42.966554 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 227044us
I20260809 23:21:42.837153  3820 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:42.968264  2992 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.968942  3820 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.972726  3687 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.939742  3207 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:42.956514  3816 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.976336  3816 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:42.978518  3933 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260809 23:21:42.979560 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:42.973322  3754 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.980189 24320 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:42.980594 24309 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260809 23:21:42.925957  3919 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.983631  3836 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:42.985532 24316 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:42.985896 24323 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:42.842635  3812 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:42.967228  3783 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:42.994673 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60154us
I20260809 23:21:42.995611 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:42.985152  3806 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:43.092248 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 141550us
I20260809 23:21:42.990878 24542 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:43.131146 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:42.977530  3660 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.133682 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.140458  3837 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.140749  3914 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.013s
W20260809 23:21:43.142223  3937 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.209s	user 0.000s	sys 0.000s
I20260809 23:21:42.932057  3369 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.065s
I20260809 23:21:42.987617  3272 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:43.150862 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 413153us
I20260809 23:21:43.149736 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:43.153576 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 158068us
I20260809 23:21:43.154307 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:43.127022  3930 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.127565  3349 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.157338 24425 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:43.160934 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:43.161689 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.162389 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.163739 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.148290  2992 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:42.997455  3411 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.163s
I20260809 23:21:42.936506  3750 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.172688 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.168004 24430 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:43.165885 24323 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:42.985481  3919 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.169304 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.147679  3789 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.171128 24435 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:43.175946  3932 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:43.177002  3942 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
I20260809 23:21:43.169939  3349 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:43.178889  3633 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.000s
I20260809 23:21:43.179111  3604 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260809 23:21:43.182552 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:21:43.179167  3948 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:43.182999  3812 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:42.996937  3876 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.218209  3633 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:43.220968  3876 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:43.223117  3928 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.223769  3928 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.226310  3928 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.227087  3900 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.229138  3900 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:43.229296  3587 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.203526  3932 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.230520 25366 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:43.204136  3814 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:43.231230  3814 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:43.231623  3814 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:43.231722  3932 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.185068  3750 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.185197  3660 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.160858  3788 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.179837  3946 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:43.179915  3958 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.156682  3556 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:43.170465  3930 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.180253 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:43.187374  3964 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:43.200511  3922 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.304s	user 0.001s	sys 0.000s
I20260809 23:21:43.208434  3349 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.209219 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.209277  3789 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:43.174525  3735 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.784s	user 0.012s	sys 0.001s
I20260809 23:21:43.172910  3836 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.215196  3948 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:43.235360 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65333.2us
I20260809 23:21:43.248260 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.272087 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.279361 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:43.276648 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 96474us
I20260809 23:21:43.486562 24427 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:43.487111  3814 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.489420  3814 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.280725 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.490825 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:43.494269  3844 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:43.494686  3844 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:43.495077  3844 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:21:43.495230 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 764232us
I20260809 23:21:43.495458  3789 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.491744 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3532) took 1422 ms. Trace:
I20260809 23:21:43.288012  3932 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:43.275918  3964 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:43.356879  3930 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.341822  3735 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.437901 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:43.501322 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 229322us
I20260809 23:21:43.278333  3877 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.274451  3958 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.435351 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.323354  3633 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260809 23:21:43.355600 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:43.333611  3755 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.435346 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.331106  3950 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.275571  3928 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.269320  3946 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.305878  3948 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:43.278262  3556 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.259930  3788 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.326622 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.308300 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:43.320878 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.349536  3828 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.226745  3815 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.426462 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.310348  3836 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.328660 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.229328  3207 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:43.336024  3279 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260809 23:21:43.335381  3587 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:43.456243  3803 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.271s	user 0.006s	sys 0.014s
I20260809 23:21:43.345543 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:43.353708 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:43.498189 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:43.496837 24545 rpcz_store.cc:276] 0809 23:21:42.069383 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.070336 (+   953us) service_pool.cc:227] Handling call
0809 23:21:42.089112 (+ 18776us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:42.169041 (+ 79929us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:42.169078 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:42.174204 (+  5126us) log.cc:842] Serialized 26 byte log entry
0809 23:21:42.640155 (+465951us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:42.713002 (+ 72847us) log.cc:842] Serialized 14 byte log entry
0809 23:21:43.491143 (+778141us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b18002b3840. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:43.491323 (+   180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.491404 (+    81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:43.491448 (+    44us) raft_consensus.cc:1671] finished
0809 23:21:43.491544 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.491666 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":78238,"prepare.queue_time_us":70370,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":854061,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11129,"spinlock_wait_cycles":5949184,"thread_start_us":385560,"threads_started":1,"wal-append.queue_time_us":534198}
I20260809 23:21:43.509799 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3635) took 1332 ms. Trace:
I20260809 23:21:43.630468  3676 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.500603  3964 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:43.637542 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.030s	sys 0.029s
I20260809 23:21:43.509547  3735 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.534852  2995 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:43.532438 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.698766 24543 rpcz_store.cc:276] 0809 23:21:42.177394 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.178004 (+   610us) service_pool.cc:227] Handling call
0809 23:21:42.204738 (+ 26734us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.204958 (+   220us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:42.657228 (+452270us) log.cc:842] Serialized 14 byte log entry
0809 23:21:43.509433 (+852205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:43.509491 (+    58us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:43.509552 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.509627 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.509741 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":929560,"prepare.queue_time_us":250900,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":731669,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":51,"thread_start_us":342987,"threads_started":2,"wal-append.queue_time_us":265707}
I20260809 23:21:43.534469 24535 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:43.542361 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1835) took 2186 ms. Trace:
I20260809 23:21:43.707073 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 17.
I20260809 23:21:43.709205 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.707266 25373 rpcz_store.cc:276] 0809 23:21:41.356120 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.356435 (+   315us) service_pool.cc:227] Handling call
0809 23:21:42.080212 (+723777us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:42.169204 (+ 88992us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:42.169237 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:42.169672 (+   435us) log.cc:842] Serialized 26 byte log entry
0809 23:21:42.630105 (+460433us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:42.630155 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.630259 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:43.199331 (+569072us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:43.342387 (+143056us) raft_consensus.cc:1671] finished
0809 23:21:43.342409 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.542155 (+199746us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b54006d5950. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:43.542285 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1669709952,"thread_start_us":376604,"threads_started":1,"wal-append.queue_time_us":581586}
I20260809 23:21:43.713502 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.710034 24542 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:43.719560 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.721127 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.722399 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.724227 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.725238  3556 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.725513 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:43.686720  3928 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.696705  3968 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:43.728121  3735 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.728075  3968 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.728813  3815 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.730266  3968 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.682895  3950 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.731043 25370 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260809 23:21:43.732060  3683 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.744s	user 0.008s	sys 0.003s
I20260809 23:21:43.732563 25373 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:43.706565  3676 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.733275  3683 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.734668 25365 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:43.690604  3828 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.735586 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.515170  3759 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:43.736917 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:43.500552  3945 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260809 23:21:43.516433 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.545847 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:43.535188 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.543344  3253 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:43.508718  3789 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:43.536779 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1542) took 2686 ms. Trace:
I20260809 23:21:43.593353 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.666541 24380 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:43.673743 25369 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:43.677039 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3640) took 2956 ms. Trace:
I20260809 23:21:43.626608  3377 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:43.686861 25368 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:43.746134 24427 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:43.693555 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 364994us
I20260809 23:21:43.674710  3648 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:43.700459 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.031s	sys 0.030s
W20260809 23:21:43.697508 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 352012us
I20260809 23:21:43.703333 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.752904 24545 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:43.692750  3783 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.000s
I20260809 23:21:43.500821  3932 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:43.709244 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3530) took 2135 ms. Trace:
I20260809 23:21:43.755625 24535 rpcz_store.cc:276] 0809 23:21:41.574328 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.574698 (+   370us) service_pool.cc:227] Handling call
0809 23:21:41.592792 (+ 18094us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.534321 (+1941529us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b540021e6a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:21:43.708902 (+174581us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.709167 (+   265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31167872}
I20260809 23:21:43.732137  3207 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:43.732380  3950 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.696786  2992 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:43.737138  3828 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.739130 24546 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:43.741145 25375 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:43.538015  2736 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:43.749159  3816 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.767s	user 0.013s	sys 0.000s
I20260809 23:21:43.743813 24431 rpcz_store.cc:276] 0809 23:21:40.850741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.850964 (+   223us) service_pool.cc:227] Handling call
0809 23:21:41.767991 (+917027us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:41.812243 (+ 44252us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.468826 (+656583us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:43.536504 (+1067678us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b5400130450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:43.536710 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2159097600}
I20260809 23:21:43.762733 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3637) took 1581 ms. Trace:
I20260809 23:21:43.745229 24422 rpcz_store.cc:276] 0809 23:21:40.720194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.720383 (+   189us) service_pool.cc:227] Handling call
0809 23:21:40.733346 (+ 12963us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:41.243206 (+509860us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.670903 (+2427697us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b540017edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:43.676955 (+  6052us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1038385408}
W20260809 23:21:43.747880 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81434.1us
I20260809 23:21:43.781013 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:43.779016 24540 rpcz_store.cc:276] 0809 23:21:42.181604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.181819 (+   215us) service_pool.cc:227] Handling call
0809 23:21:42.182205 (+   386us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.182438 (+   233us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:42.576982 (+394544us) log.cc:842] Serialized 14 byte log entry
0809 23:21:43.762366 (+1185384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:43.762424 (+    58us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:43.762469 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.762559 (+    90us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.762679 (+   120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":871635,"prepare.queue_time_us":273153,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":752001,"raft.run_cpu_time_us":47,"raft.run_wall_time_us":47,"thread_start_us":573484,"threads_started":3,"wal-append.queue_time_us":733382}
I20260809 23:21:43.781574 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:43.783007  3369 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:43.783407  3369 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:43.783361 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:43.783916  3369 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:43.784255 24309 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:43.785295 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:43.749979 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.786747 24542 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:43.791602  3369 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.743741  3946 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.727447  3928 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.684201  3755 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.807849 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:43.771787 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:43.752904  3556 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:43.685024  3877 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.796343 25369 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:43.796720 24542 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:43.736996  3683 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:43.756538 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 256986us
I20260809 23:21:43.800400 25375 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:43.801479 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.753151  3676 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.802577 25372 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260809 23:21:43.782094 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:43.514004 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 556026us
I20260809 23:21:43.754988  3633 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:43.737317  3759 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.778090 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.762298  3932 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:43.818481 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.819474 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:43.759920  3218 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.769049  3878 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:43.780591  3876 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.559s	user 0.009s	sys 0.003s
I20260809 23:21:43.808089  3316 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:43.815241  3816 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.851225 24545 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:43.851100 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.852882 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:43.794842  3920 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.860240 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:43.860916 24540 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:43.751655  3377 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.804678  3207 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.848129 24546 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
W20260809 23:21:43.868214 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59884.2us
I20260809 23:21:43.914633 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.915457 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
W20260809 23:21:43.869511 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50158.1us
I20260809 23:21:43.848862 24321 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:43.864389  3946 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260809 23:21:43.873936 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.864318  3759 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:43.877826  3810 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:43.864346  3844 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.886905 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1770) took 1187 ms. Trace:
I20260809 23:21:43.978464 25370 rpcz_store.cc:276] 0809 23:21:42.699930 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.700061 (+   131us) service_pool.cc:227] Handling call
0809 23:21:42.700353 (+   292us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.731432 (+1031079us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.886567 (+155135us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:43.886828 (+   261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2369096064}
I20260809 23:21:43.905174 24541 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260809 23:21:43.979799 24541 pending_rounds.cc:76] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 15
I20260809 23:21:43.980104 24541 pending_rounds.cc:97] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260809 23:21:43.980400 24541 raft_consensus.cc:2866] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:43.884243  3660 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:43.906170 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.908636  3218 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.907707 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.986562 24545 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:43.987442 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:43.798915  3789 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.065s
I20260809 23:21:43.987972  3218 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:43.912887 24316 raft_consensus.cc:2396] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260809 23:21:43.988916  3958 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.977789  3793 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.989809 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:43.990104  3793 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.991083 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:43.991695 24316 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:43.992995 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:43.759552  3648 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.872602 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.994225 24543 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:43.994470 24542 raft_consensus.cc:2379] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:43.995585  3648 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:43.944445 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.944252  3870 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:43.996603 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3649) took 1011 ms. Trace:
I20260809 23:21:43.947062  3683 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.996940 24540 rpcz_store.cc:276] 0809 23:21:42.985634 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.944147 (+958513us) service_pool.cc:227] Handling call
0809 23:21:43.996544 (+ 52397us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:43.997625  3683 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.998152 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.949116 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1740) took 2912 ms. Trace:
I20260809 23:21:43.949116  3369 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:43.945406 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.960724 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1726) took 1232 ms. Trace:
I20260809 23:21:44.000002  3683 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:44.000080 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54781.3us
I20260809 23:21:44.001085 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.001322  3963 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:44.001703  3963 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.002020  3963 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:44.002297 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:43.962461 25363 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1747) took 2923 ms (client timeout 1561 ms). Trace:
W20260809 23:21:44.002710 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 128884us
W20260809 23:21:44.003080 25363 rpcz_store.cc:269] 0809 23:21:41.038896 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.152795 (+113899us) service_pool.cc:227] Handling call
0809 23:21:43.962187 (+2809392us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:43.962354 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2589775616}
I20260809 23:21:44.003494 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.004839 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.005541  3889 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:44.005688  3823 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.006083  3823 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:44.006453  3823 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:43.954841 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1823) took 1265 ms. Trace:
I20260809 23:21:44.007045 24314 rpcz_store.cc:276] 0809 23:21:42.689119 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.689249 (+   130us) service_pool.cc:227] Handling call
0809 23:21:42.712377 (+ 23128us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.712580 (+   203us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:42.771923 (+ 59343us) log.cc:842] Serialized 14 byte log entry
0809 23:21:43.953820 (+1181897us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b1800563080. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b
0809 23:21:43.954392 (+   572us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:43.954461 (+    69us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:43.954525 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.954621 (+    96us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.954765 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":172414,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":744,"raft.queue_time_us":5043,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"spinlock_wait_cycles":6121856,"thread_start_us":199823,"threads_started":2,"wal-append.queue_time_us":490006}
I20260809 23:21:44.009001 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.010078 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:43.999095 25371 rpcz_store.cc:276] 0809 23:21:41.036498 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:41.049120 (+ 12622us) service_pool.cc:227] Handling call
0809 23:21:42.519840 (+1470720us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.640896 (+121056us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.948923 (+1308027us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54000c4750. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:43.949055 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3381335936}
I20260809 23:21:44.012151 24320 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:43.999744 24544 rpcz_store.cc:276] 0809 23:21:42.728563 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.782543 (+ 53980us) service_pool.cc:227] Handling call
0809 23:21:42.790493 (+  7950us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.984947 (+194454us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:42.985290 (+   343us) log.cc:842] Serialized 14 byte log entry
0809 23:21:43.598117 (+612827us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:43.598185 (+    68us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:43.598252 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.960438 (+362186us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:43.960507 (+    69us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:43.960652 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137110,"prepare.queue_time_us":6950,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":137302,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":446659968,"thread_start_us":473761,"threads_started":1,"wal-append.queue_time_us":502033}
I20260809 23:21:44.012437  3871 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:44.012879  3871 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:44.013020 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:43.968930  3633 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:43.982728 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:43.985828 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 79775.1us
I20260809 23:21:43.912137  3556 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:43.995448 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.006398  3253 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:44.025400 24542 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:43.970573 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:43.940868  3978 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.346s	user 0.000s	sys 0.001s
I20260809 23:21:44.014104 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.031914 24544 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:44.032495 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3644) took 1073 ms. Trace:
I20260809 23:21:44.032773 24544 rpcz_store.cc:276] 0809 23:21:42.958781 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.031437 (+1072656us) service_pool.cc:227] Handling call
0809 23:21:44.031738 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:44.032270 (+   532us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.032446 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:44.034021 24544 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:44.034528 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3645) took 1075 ms. Trace:
I20260809 23:21:44.035362  3232 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.036232 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.036538 24316 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:44.036922 24535 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:44.038291 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3648) took 1054 ms. Trace:
I20260809 23:21:43.712183 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:44.042613 24317 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:44.015118 24422 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:44.038618 24535 rpcz_store.cc:276] 0809 23:21:42.983501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.037541 (+1054040us) service_pool.cc:227] Handling call
0809 23:21:44.037831 (+   290us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:44.038005 (+   174us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.038032 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:44.038070 (+    38us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:44.038104 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.038152 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:44.038241 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:44.035645 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.016925 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.045065 24314 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:21:43.999317 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.045778 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.046482 24540 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:44.012331  3844 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.020426 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.056552 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.057600 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:43.940924 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.023064 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:44.027392 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.026501 25365 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:44.038748 24316 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:44.065296  3377 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.043149  3885 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:44.068142 24427 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:44.023693  3633 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:44.051759 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.013043  3810 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.988919  3789 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:44.034803 24544 rpcz_store.cc:276] 0809 23:21:42.959096 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.033463 (+1074367us) service_pool.cc:227] Handling call
0809 23:21:44.033722 (+   259us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:44.034319 (+   597us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.034485 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:44.054836 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.081377 24314 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:44.048179 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:43.913334  3816 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:43.987973 24545 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:43.941790  3207 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:44.073238 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45966us
I20260809 23:21:43.975786  3563 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260809 23:21:44.071990  3660 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:44.090238  3154 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:44.077593  3844 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:44.090843  3895 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:44.091358  3895 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:44.092168 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44103.1us
I20260809 23:21:44.093554 24309 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:44.094590  3911 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.095125  3911 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.095613  3907 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:44.096416  3816 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.096318  3907 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:44.097492  3907 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260809 23:21:44.098361  3983 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:44.098758  3983 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:44.099614  3983 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.100121 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.100742 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:44.100958  3983 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:44.077128  2992 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:44.075193 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:44.103354 24312 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260809 23:21:44.077805 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.103482  3948 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.104282 24542 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:44.104245 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45338us
W20260809 23:21:44.105015 24312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3007) took 3232 ms (client timeout 1621 ms). Trace:
I20260809 23:21:44.078229 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.105603 24542 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:44.105824 24316 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:44.106617 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.108247  3877 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.108785  3877 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:44.109118  3877 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.022301  3964 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.109414  3968 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:44.110477  3968 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:44.111114  3968 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:44.111604  3968 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.111917  3968 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:43.889302  3878 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.113546 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.114470  3878 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.115250 24311 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:44.083077  3377 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:21:44.115978  3814 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:44.049824  3804 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260809 23:21:44.049877  3763 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.044766  3556 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.090862  3660 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:44.085943 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.116312  3377 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.117353  3763 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:43.917050  3920 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.089552 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.091432  3154 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:44.097698  3911 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.121383 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.125711 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.124137 24545 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:21:44.074044  3683 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:44.073585  3958 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:44.125088 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:44.105409 24312 rpcz_store.cc:269] 0809 23:21:40.872196 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.915889 (+ 43693us) service_pool.cc:227] Handling call
0809 23:21:44.104824 (+3188935us) spinlock_profiling.cc:243] Waited 3.15 s on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:44.104950 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7236928256}
I20260809 23:21:44.109444  3877 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:44.144742 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55266.5us
I20260809 23:21:44.147202 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:44.146840 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.111064  3964 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.092298  3895 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.163691 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.144184 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.105623  3948 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.086256  3870 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.090s
W20260809 23:21:44.177346 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1551) took 1800 ms (client timeout 1812 ms). Trace:
I20260809 23:21:44.177241 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.075515  3369 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.082s
I20260809 23:21:44.127133  3791 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:21:44.127609  3377 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:44.127485  3563 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:44.202982  3815 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
I20260809 23:21:44.172459  3889 raft_consensus.cc:2741] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260809 23:21:44.204100  3815 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:44.204520  3815 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:44.205294  3815 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.206480  3815 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:44.177716 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52088.8us
I20260809 23:21:44.207902  3810 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.208626  3455 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:21:44.209017 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.181710 24545 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:44.211405  3455 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:44.212229 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.212829  3455 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:44.124804  3789 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:44.213258  3455 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:44.213696  3455 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:44.214078  3789 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.172900 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51709.2us
I20260809 23:21:44.218129  3920 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.160782  3272 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.219240  3272 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.221340  3272 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:44.221827  3272 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.221771  3455 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.157882  2992 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.223240  2992 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.225164  2992 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.225657  3885 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:44.187309  3793 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.055730  3871 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.229800  3689 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:44.233836 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:44.233966 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3584) took 1298 ms. Trace:
I20260809 23:21:44.142184  2820 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.056s
I20260809 23:21:44.234782  2820 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:44.235415  2820 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.235769  2820 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.236621 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.237784 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.239054  3556 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.238968 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1555) took 1079 ms. Trace:
I20260809 23:21:44.239895 24429 rpcz_store.cc:276] 0809 23:21:43.159529 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.159670 (+   141us) service_pool.cc:227] Handling call
0809 23:21:43.160022 (+   352us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:43.233268 (+ 73246us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:43.233313 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:43.233849 (+   536us) log.cc:842] Serialized 26 byte log entry
0809 23:21:43.308017 (+ 74168us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:43.308063 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.238442 (+930379us) spinlock_profiling.cc:243] Waited 72.8 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:44.238513 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:44.238578 (+    65us) raft_consensus.cc:1671] finished
0809 23:21:44.238596 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:44.238871 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1816,"raft.queue_time_us":8333,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":16146,"spinlock_wait_cycles":167397248,"thread_start_us":73789,"threads_started":1,"wal-append.queue_time_us":208241}
I20260809 23:21:44.243534  3930 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:44.243914  3930 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:44.244725  3930 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.246022  3930 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:44.189100  3823 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.247650 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1728) took 1495 ms. Trace:
I20260809 23:21:44.248010 24541 rpcz_store.cc:276] 0809 23:21:42.751818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.940447 (+188629us) service_pool.cc:227] Handling call
0809 23:21:42.941071 (+   624us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.980929 (+1039858us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.247440 (+266511us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:44.247594 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2176345344}
I20260809 23:21:44.248682  3823 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.146731  3154 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:44.249090 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.249155 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3543) took 1070 ms. Trace:
I20260809 23:21:44.249694  3154 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.250284 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.173790  3587 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260809 23:21:44.250978  3154 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:44.192368 24424 rpcz_store.cc:269] 0809 23:21:42.376501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.385799 (+  9298us) service_pool.cc:227] Handling call
0809 23:21:44.177160 (+1791361us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540017f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:44.177293 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235739008}
I20260809 23:21:44.192595 24544 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:44.253147 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.253885 24544 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:44.158124  3804 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.255128  3804 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.255511  3804 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.176736  3911 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:44.256189  3911 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.256565  3911 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:44.260825  3207 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:44.249797 24543 rpcz_store.cc:276] 0809 23:21:43.179233 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.736643 (+557410us) service_pool.cc:227] Handling call
0809 23:21:44.248825 (+512182us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:44.249057 (+   232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":589891968}
I20260809 23:21:44.262385 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.234503 24317 rpcz_store.cc:276] 0809 23:21:42.935423 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.935559 (+   136us) service_pool.cc:227] Handling call
0809 23:21:44.233731 (+1298172us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:44.233870 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2491387648}
I20260809 23:21:44.261488  3763 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.264171 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1725) took 1539 ms. Trace:
I20260809 23:21:44.264550 24539 rpcz_store.cc:276] 0809 23:21:42.724586 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.781784 (+ 57198us) service_pool.cc:227] Handling call
0809 23:21:42.804848 (+ 23064us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:42.813449 (+  8601us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:42.813490 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:42.813550 (+    60us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:42.813595 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:42.813671 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:44.264009 (+1450338us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b540029de50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:44.264113 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52072320}
I20260809 23:21:44.265766 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.266311 24539 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:44.269729  3871 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:44.162168  3683 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:44.271160  3683 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:44.210265 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:44.274233 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64041.1us
I20260809 23:21:44.173964  3946 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260809 23:21:44.275617 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1773) took 1497 ms. Trace:
I20260809 23:21:44.276297  3755 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.181001  3958 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:44.277612  3958 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.279430  3836 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.279652 24167 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 9 to 16. New cstate: current_term: 16 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:44.280929  3788 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.275991 25366 rpcz_store.cc:276] 0809 23:21:42.777909 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.778217 (+   308us) service_pool.cc:227] Handling call
0809 23:21:42.799986 (+ 21769us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.230897 (+430911us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.275373 (+1044476us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:44.275555 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989674880}
I20260809 23:21:44.283201 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:44.284327 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.184131  3963 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.287626  3963 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.182374  3676 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:44.290202 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.281540 24167 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 13, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 13 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:44.300096 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.299957 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.301270 24429 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:44.301477 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.254820 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.304009 24544 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:44.317698 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.320374 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.320442 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.322470 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.321147 24544 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:44.324630 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.325651 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.239967  3947 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:44.332012 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.332752 24539 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.336351 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.336908 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.337836 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:44.222465  3272 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.331184  3843 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:44.351588  3843 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.351936  3843 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:44.352732  3272 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.236076  2820 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.331030  3253 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.201807  3979 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.355370  3253 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.355934  3979 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.341245 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.358853 24317 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:44.332407  3947 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:44.360854  3947 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.341471  3932 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.342410  3763 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:44.343290  3885 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:44.341740 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:44.358210  3253 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:44.316974  3788 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.344269 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3643) took 1596 ms. Trace:
I20260809 23:21:44.331316  3810 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.411129  3810 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.411634  3810 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.346462  3563 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:44.346799  3759 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.347720 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.348315  3870 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:44.349308 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.349860 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:44.125746  3660 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.323450  3814 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260809 23:21:44.323401  3689 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:44.323513  3556 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:44.336972 24422 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:44.364912 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.337944 24541 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:44.375590 24539 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:44.425769 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 484475us
I20260809 23:21:44.403098 24534 rpcz_store.cc:276] 0809 23:21:42.747818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:42.810801 (+ 62983us) service_pool.cc:227] Handling call
0809 23:21:42.811133 (+   332us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:43.513793 (+702660us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:43.513825 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:43.514337 (+   512us) log.cc:842] Serialized 26 byte log entry
0809 23:21:43.904675 (+390338us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:43.904713 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:43.904774 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:44.344022 (+439248us) raft_consensus.cc:1671] finished
0809 23:21:44.344051 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:44.344200 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":698206,"raft.queue_time_us":703499,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":373831,"threads_started":1,"wal-append.queue_time_us":775667}
I20260809 23:21:44.363987  3885 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.323562  3946 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:44.445664 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.402827  3316 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.323971  3963 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:44.455300 24543 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:44.449465  3272 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:44.408766  2820 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.410231  3455 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.464880  2820 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.346837  3895 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:44.415731 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:44.471695 24424 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:44.372257 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:44.428797 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 81172.5us
I20260809 23:21:44.440186 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.476449 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.477978 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.480363 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:44.480505  3556 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
I20260809 23:21:44.485332  3556 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:44.485762  3556 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:44.489521 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:44.490533 24323 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:44.492092 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.492861 24316 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:44.494890 24312 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:44.442209 24541 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260809 23:21:44.498592 24541 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
W20260809 23:21:44.437841 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73038.4us
I20260809 23:21:44.362841  3947 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:44.447117 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.499225  3529 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:44.506986 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.512413 24312 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:44.512837 24541 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:44.508409 24323 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260809 23:21:44.513922 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.514712 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:44.515340 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:44.515960 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:44.515617 24323 raft_consensus.cc:2362] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24. Re-sending same reply.
I20260809 23:21:44.517971 24323 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260809 23:21:44.519546  3823 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:44.520313  3823 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:44.521513 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 74502.1us
I20260809 23:21:44.520614 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.486709  3556 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.525314  3556 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:44.540920  3455 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:44.550907 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.555220  2992 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:44.555775  2992 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.556166  2992 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:44.565753  3920 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.566326  3920 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.568708  3920 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:44.576418 25368 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.017s	sys 0.034s
W20260809 23:21:44.584839 25368 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.018s	sys 0.034s
I20260809 23:21:44.587073 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.584307 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.438073  3563 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:44.453019  3885 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:44.455080  3689 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:44.455665  3316 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.420063  3788 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.338706  3968 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.573047  3272 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.640718 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.426510  3804 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.643507 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54961.5us
I20260809 23:21:44.652518 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:44.619258  3563 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:44.659340 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.664022 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:44.615700  3920 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:44.639837  3316 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.642407  3272 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:44.457793  3946 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.573930  3963 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.677718  3920 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:44.688336  3932 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.659024  3834 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:44.660668  3557 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:44.643289  3689 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:44.698246  3920 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:44.698457  3834 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.714210 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:44.673460  3431 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.642304  3755 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260809 23:21:44.718641 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:44.676275  3563 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:44.684402  3870 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:44.716486 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:44.693076  3676 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:44.725308 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:44.727515 25363 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:44.660871  3369 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260809 23:21:44.730767 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:44.732115 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:44.733527 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.018s	sys 0.046s
I20260809 23:21:44.661661  3793 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260809 23:21:44.738124 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:44.738471 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.019s	sys 0.047s
I20260809 23:21:44.738689 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:44.660753  3948 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260809 23:21:44.743147 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:44.743775 24541 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.745007 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:44.745402 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51185.3us
I20260809 23:21:44.745785 25371 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
W20260809 23:21:44.742013 25369 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.003s	sys 0.029s
W20260809 23:21:44.747391 25369 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.004s	sys 0.029s
I20260809 23:21:44.751262 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:44.724093  3834 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:44.724740  3529 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.727351  3870 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.724160  3979 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.760859 25363 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 18.
I20260809 23:21:44.734131  3587 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:44.692217  3877 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.691901  2992 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.784498 24314 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.018s	sys 0.009s
I20260809 23:21:44.764370 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:44.786136 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.006s	sys 0.020s
W20260809 23:21:44.808519 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.007s	sys 0.021s
W20260809 23:21:44.758147  3239 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.011s	sys 0.022s
I20260809 23:21:44.748296  3253 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:44.794759 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:44.800267 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:44.786193 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:44.808064 24314 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.018s	sys 0.009s
I20260809 23:21:44.764104  3557 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:44.791119  3870 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:44.730010  3431 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.811269  3239 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.012s	sys 0.022s
I20260809 23:21:44.759714  3932 consensus_queue.cc:1046] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260809 23:21:44.763886  3759 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:44.845840 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.016s	sys 0.033s
W20260809 23:21:44.850212 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.016s	sys 0.033s
W20260809 23:21:44.852612 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.005s	sys 0.023s
W20260809 23:21:44.853006 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.005s	sys 0.023s
I20260809 23:21:44.815834  3253 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:44.812093  3979 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260809 23:21:44.849536 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.010s	sys 0.008s
W20260809 23:21:44.863399 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.010s	sys 0.008s
I20260809 23:21:44.790144  3529 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.862287 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.010s	sys 0.026s
W20260809 23:21:44.869784 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.010s	sys 0.027s
I20260809 23:21:44.848453  3836 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:44.845888  3154 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:21:44.865402  3633 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.777s	user 0.008s	sys 0.002s
W20260809 23:21:44.859252 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59069.9us
I20260809 23:21:44.815645  3587 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.862509  3253 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:44.875644  3836 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:44.883699  3431 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.884325 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:44.886431  3377 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.646s	user 0.009s	sys 0.001s
I20260809 23:21:44.888392  3529 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.889148  3946 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:44.923540 24427 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
W20260809 23:21:44.890101  3871 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.620s	user 0.010s	sys 0.003s
I20260809 23:21:44.891718  3407 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:44.893707  2992 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.832952  3689 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.865036  3968 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:44.865170  3788 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:44.914536 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.016s	sys 0.032s
W20260809 23:21:44.981602 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.017s	sys 0.033s
I20260809 23:21:44.925640 24541 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:44.874646  3804 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:44.927070 25365 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:44.926699  3377 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.947518  3836 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:44.951493 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:44.956467  3587 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:44.971983 24155 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.013s	sys 0.009s
W20260809 23:21:44.987326 24155 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.013s	sys 0.009s
W20260809 23:21:44.902498  3892 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.952s	user 0.033s	sys 0.052s
I20260809 23:21:44.908936  3963 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:44.981995 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 888020us
I20260809 23:21:44.946512  3932 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Leader election won for term 18
W20260809 23:21:44.944905  3997 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.271s	user 0.001s	sys 0.000s
I20260809 23:21:44.987381 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:44.987843 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.006s	sys 0.018s
W20260809 23:21:45.015789 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.008s	sys 0.018s
W20260809 23:21:44.993464  3843 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.000s
I20260809 23:21:44.835811  3557 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:44.938376  3923 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:44.985955 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:45.015268 24541 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:21:45.040494 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54630.5us
W20260809 23:21:44.981674  3789 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.007s	sys 0.026s
I20260809 23:21:44.904917  3633 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:44.943120  3683 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.671s	user 0.008s	sys 0.002s
I20260809 23:21:44.968788  3407 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.027575  3377 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:45.020975 24498 connection.cc:232] RPC call timeout handler was delayed by -4.0818495929706842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260809 23:21:45.031535  3556 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.000s
I20260809 23:21:45.033983  3870 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.040978 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3657) took 1600 ms. Trace:
I20260809 23:21:45.007715 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.050025 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1782) took 1557 ms. Trace:
I20260809 23:21:45.052207 24543 rpcz_store.cc:276] 0809 23:21:43.440816 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.285339 (+844523us) service_pool.cc:227] Handling call
0809 23:21:44.285696 (+   357us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:44.573414 (+287718us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.040639 (+467225us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54001fc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:45.040876 (+   237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":379870976}
I20260809 23:21:45.042541 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:45.047660 24498 connection.cc:232] RPC call timeout handler was delayed by -3.9531495929695666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1765s for a response
I20260809 23:21:45.052914 25373 rpcz_store.cc:276] 0809 23:21:43.492961 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.733752 (+240791us) service_pool.cc:227] Handling call
0809 23:21:43.734113 (+   361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:43.734469 (+   356us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:43.734499 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:43.737871 (+  3372us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.051780 (+313909us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:44.051837 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.051942 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.049806 (+997864us) raft_consensus.cc:1671] finished
0809 23:21:45.049834 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.049967 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10618,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":26301,"thread_start_us":313530,"threads_started":1,"wal-append.queue_time_us":874278}
W20260809 23:21:45.054401 24498 connection.cc:232] RPC call timeout handler was delayed by -3.9153495931532234s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1807s for a response
W20260809 23:21:45.021694  3995 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
W20260809 23:21:45.055351 24498 connection.cc:232] RPC call timeout handler was delayed by -3.8118495929520577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
I20260809 23:21:45.055591 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:45.055841 24498 connection.cc:232] RPC call timeout handler was delayed by -3.7920495930593461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
I20260809 23:21:45.052429 24534 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:45.056756 24498 connection.cc:232] RPC call timeout handler was delayed by -3.6570495930500329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260809 23:21:45.057560 24498 connection.cc:232] RPC call timeout handler was delayed by -3.6570495930500329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260809 23:21:45.058012 24498 connection.cc:232] RPC call timeout handler was delayed by -3.6417495929636061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21110000000000004s for a response
W20260809 23:21:45.058274 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.877s	user 0.006s	sys 0.027s
W20260809 23:21:45.058483 24498 connection.cc:232] RPC call timeout handler was delayed by -3.6030495930463076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
W20260809 23:21:45.058836 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.006s	sys 0.027s
W20260809 23:21:45.059046 24498 connection.cc:232] RPC call timeout handler was delayed by -3.5814495929516852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2178s for a response
W20260809 23:21:45.074891 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 899316us
I20260809 23:21:45.063566 25363 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:45.030500  3563 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.065688 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:45.011673  3815 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.004s
I20260809 23:21:45.040295  3923 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:45.070858 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:45.098017 24311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.020s	sys 0.031s
I20260809 23:21:45.061735 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:45.099392 24541 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:45.100844 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:45.102105  3788 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:45.102254 24311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.020s	sys 0.031s
I20260809 23:21:45.102571  3788 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:45.077561 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.005s	sys 0.025s
I20260809 23:21:45.078601 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:45.006843  3958 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.019s	sys 0.004s
W20260809 23:21:45.010063  3892 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.075s	user 0.034s	sys 0.055s
W20260809 23:21:45.044818  3789 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.008s	sys 0.026s
I20260809 23:21:45.047662  3963 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:45.041818  3932 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:45.006654  3272 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.047967  3877 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.081004 24534 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
W20260809 23:21:45.075095 24498 connection.cc:232] RPC call timeout handler was delayed by -3.5490495930425823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
I20260809 23:21:45.125105 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:45.125526 24498 connection.cc:232] RPC call timeout handler was delayed by -3.5490495930425823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260809 23:21:45.106743  3958 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.019s	sys 0.005s
I20260809 23:21:45.136684 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:45.105090 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.006s	sys 0.025s
I20260809 23:21:45.050097  3885 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:45.106452 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.022s	sys 0.003s
I20260809 23:21:45.039920  3843 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.146152  3843 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.146876 25375 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:45.109782 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.003s	sys 0.022s
W20260809 23:21:45.147480 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.003s	sys 0.022s
I20260809 23:21:45.120286 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:45.106276  3218 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260809 23:21:45.149144 25365 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:45.149751  3218 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:45.150316  3218 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:45.149904  3923 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.150547 25365 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:45.151041  3923 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:45.123492 24424 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:45.125837 24498 connection.cc:232] RPC call timeout handler was delayed by -3.4830272330436856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
W20260809 23:21:45.151898 24498 connection.cc:232] RPC call timeout handler was delayed by -3.3903272331226617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20260809 23:21:45.103051  3788 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:45.152241 24498 connection.cc:232] RPC call timeout handler was delayed by -3.3426272331271321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
W20260809 23:21:45.152580 24498 connection.cc:232] RPC call timeout handler was delayed by -3.3102272329851985s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260809 23:21:45.152837 24498 connection.cc:232] RPC call timeout handler was delayed by -2.9943272331729531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20260809 23:21:45.153092 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8346320660784841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
W20260809 23:21:45.153360 24498 connection.cc:232] RPC call timeout handler was delayed by -2.8305272331926972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260809 23:21:45.153422  3272 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:45.153688 24498 connection.cc:232] RPC call timeout handler was delayed by -2.7255495931021869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3129s for a response
W20260809 23:21:45.153993 24498 connection.cc:232] RPC call timeout handler was delayed by -2.5590495930518955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33140000000000003s for a response
W20260809 23:21:45.154247 24498 connection.cc:232] RPC call timeout handler was delayed by -2.4962320660706609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20260809 23:21:45.154516 24498 connection.cc:232] RPC call timeout handler was delayed by -2.2154320660047233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
I20260809 23:21:45.097826 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:45.155023 25365 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260809 23:21:45.155298  3885 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:45.139524 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.022s	sys 0.003s
W20260809 23:21:45.155682 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58085.2us
I20260809 23:21:45.156179  3349 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:45.159619 25366 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:45.160504 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.143723 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.161209  3259 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:21:45.162374  3259 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
W20260809 23:21:45.118669  3896 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.018s	sys 0.031s
I20260809 23:21:45.161532 24541 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:45.162731  3259 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:45.163802  3259 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 20.19, Last appended by leader: 19, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.164603 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3666) took 1140 ms. Trace:
I20260809 23:21:45.165009  3259 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:45.165777 24427 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:45.165967  3788 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.167094  3431 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:45.168349  3788 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.164969 24541 rpcz_store.cc:276] 0809 23:21:44.024607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.163773 (+1139166us) service_pool.cc:227] Handling call
0809 23:21:45.164156 (+   383us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.164325 (+   169us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.164350 (+    25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.164389 (+    39us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:45.164420 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.164468 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.164557 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.174052 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.168845 24427 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:45.047391  3683 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.174961 24541 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:45.162678 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.175490 25370 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:45.176522  3683 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.176687 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3554) took 1151 ms. Trace:
I20260809 23:21:45.176877 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:45.169039  3896 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.132s	user 0.019s	sys 0.031s
I20260809 23:21:45.171676 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.177155  3239 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:45.177278 24541 rpcz_store.cc:276] 0809 23:21:44.025741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.174421 (+1148680us) service_pool.cc:227] Handling call
0809 23:21:45.174760 (+   339us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.176431 (+  1671us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.176614 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.171676  3633 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:45.179862  3154 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:45.180290  3154 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:45.180471  3870 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.173002 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.098634  3556 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260809 23:21:45.181496  3870 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:21:45.180505 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3556) took 1127 ms. Trace:
I20260809 23:21:45.181475 24429 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:45.182654  3870 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:45.174602  3810 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.762s	user 0.008s	sys 0.004s
I20260809 23:21:45.169915 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.068876  3316 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.185637  3964 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:45.186561  2995 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:45.186283  3870 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.187208  3431 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:45.188050  3431 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:45.189033  3431 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:45.189457  3431 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:45.185287  3810 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.190775 24316 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260809 23:21:45.171455 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.192615  3810 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.193259  3557 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.193913 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3553) took 1283 ms. Trace:
I20260809 23:21:45.194285  3557 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:45.199054  3932 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:45.160779  3895 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.689s	user 0.004s	sys 0.004s
W20260809 23:21:45.212175 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.004s	sys 0.041s
W20260809 23:21:45.212589 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.004s	sys 0.041s
I20260809 23:21:45.144404  3963 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:45.194494 24543 rpcz_store.cc:276] 0809 23:21:43.910509 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.192587 (+1282078us) service_pool.cc:227] Handling call
0809 23:21:45.193167 (+   580us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.193501 (+   334us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.193541 (+    40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.193599 (+    58us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.193638 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.193726 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.193848 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.217875 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:45.185187 24541 rpcz_store.cc:276] 0809 23:21:44.053014 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.179745 (+1126731us) service_pool.cc:227] Handling call
0809 23:21:45.180053 (+   308us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.180219 (+   166us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.180242 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.180282 (+    40us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.180316 (+    34us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.180365 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.180459 (+    94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.172499  3407 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.226287 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.227440  3316 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.228083 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3653) took 1984 ms. Trace:
I20260809 23:21:45.228461 24546 rpcz_store.cc:276] 0809 23:21:43.243998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.061544 (+817546us) service_pool.cc:227] Handling call
0809 23:21:44.061843 (+   299us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.087034 (+ 25191us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.087064 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.119713 (+ 32649us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.853403 (+733690us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:44.853456 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.853555 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.227887 (+374332us) raft_consensus.cc:1671] finished
0809 23:21:45.227908 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.228027 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24762,"raft.queue_time_us":25193,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":26792,"thread_start_us":725814,"threads_started":1,"wal-append.queue_time_us":1085105}
I20260809 23:21:45.190436  3431 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.230526 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.231002  3431 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:45.202668  3789 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:45.235503  3789 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:45.236368  3789 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.239171 24167 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 16, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 16 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:21:45.241827  3911 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.985s	user 0.007s	sys 0.004s
I20260809 23:21:45.242167 24167 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 9 to 11, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 11 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:45.229094  3677 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:45.245837  3677 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:45.141430 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3548) took 1372 ms. Trace:
I20260809 23:21:45.229061 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:45.246498 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3651) took 2083 ms. Trace:
I20260809 23:21:45.246862 24543 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:45.151348  3218 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.210136 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.246384 24534 rpcz_store.cc:276] 0809 23:21:43.769237 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.139102 (+1369865us) service_pool.cc:227] Handling call
0809 23:21:45.140777 (+  1675us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.141026 (+   249us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.141062 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.141115 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:45.141169 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.141235 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.141363 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.247236 24540 rpcz_store.cc:276] 0809 23:21:43.162841 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.047301 (+884460us) service_pool.cc:227] Handling call
0809 23:21:44.047671 (+   370us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.069720 (+ 22049us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.069752 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.119997 (+ 50245us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.866020 (+746023us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:21:44.866079 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.866169 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.246285 (+380116us) raft_consensus.cc:1671] finished
0809 23:21:45.246305 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.246442 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19163,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":11716,"thread_start_us":683377,"threads_started":1,"wal-append.queue_time_us":1093877}
I20260809 23:21:45.248885 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.216271 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3664) took 1601 ms. Trace:
I20260809 23:21:45.250121 24539 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:45.251075 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:45.249711 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3562) took 1057 ms. Trace:
I20260809 23:21:45.252097 24545 rpcz_store.cc:276] 0809 23:21:44.192363 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.211956 (+ 19593us) service_pool.cc:227] Handling call
0809 23:21:45.249526 (+1037570us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:45.249651 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366475520}
I20260809 23:21:45.251966 24167 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 14, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 14 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:21:45.254873 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.213438 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1784) took 1704 ms. Trace:
I20260809 23:21:45.242388  3911 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.265496 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3565) took 1019 ms. Trace:
I20260809 23:21:45.265973 24545 rpcz_store.cc:276] 0809 23:21:44.245991 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.253281 (+1007290us) service_pool.cc:227] Handling call
0809 23:21:45.264907 (+ 11626us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.265120 (+   213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.265155 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.265205 (+    50us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:45.265247 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.265311 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.265433 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:45.249694 24435 rpcz_store.cc:276] 0809 23:21:43.615143 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.681703 (+ 66560us) service_pool.cc:227] Handling call
0809 23:21:43.742839 (+ 61136us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.982164 (+239325us) raft_consensus.cc:1520] Early marking committed up to index 11
0809 23:21:44.030882 (+ 48718us) log.cc:842] Serialized 14 byte log entry
0809 23:21:44.832638 (+801756us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:44.832695 (+    57us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:44.832742 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.216005 (+383263us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54002f94d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:45.216066 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.216206 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23600,"prepare.queue_time_us":53627,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":97879,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13006,"spinlock_wait_cycles":549779200,"thread_start_us":777776,"threads_started":1,"wal-append.queue_time_us":1094514}
I20260809 23:21:45.271291 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:45.276495 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.277338 24543 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:45.256811  3920 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.519s	user 0.008s	sys 0.001s
W20260809 23:21:45.254439  3930 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.008s	user 0.012s	sys 0.003s
W20260809 23:21:45.280067 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 62290.9us
I20260809 23:21:45.285781  3563 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.280010 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.184998  3556 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
I20260809 23:21:45.287297 24541 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:45.293346  3218 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:45.295678 25372 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:45.300781 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.300931 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3650) took 2156 ms. Trace:
I20260809 23:21:45.214094  3557 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.278857  3683 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:45.309890 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1733) took 1795 ms. Trace:
I20260809 23:21:45.313161  3889 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:45.316956  3836 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.321192 24316 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:45.287068  3895 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:45.304294 24535 rpcz_store.cc:276] 0809 23:21:43.144514 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.044384 (+899870us) service_pool.cc:227] Handling call
0809 23:21:44.072569 (+ 28185us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.072947 (+   378us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.072979 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.074901 (+  1922us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.272832 (+197931us) spinlock_profiling.cc:243] Waited 7.87 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:44.870296 (+597464us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:44.870351 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:44.878837 (+  8486us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.286516 (+407679us) raft_consensus.cc:1671] finished
0809 23:21:45.286543 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.300815 (+ 14272us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":791,"raft.run_cpu_time_us":2364,"raft.run_wall_time_us":197912,"spinlock_wait_cycles":18107776,"thread_start_us":725465,"threads_started":1,"wal-append.queue_time_us":1193148}
I20260809 23:21:45.247946  3810 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.046s
W20260809 23:21:45.331074 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.007s	sys 0.030s
I20260809 23:21:45.260728  3834 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.279322  3878 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:45.279655  3633 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.331489 24539 rpcz_store.cc:276] 0809 23:21:43.514073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.438473 (+924400us) service_pool.cc:227] Handling call
0809 23:21:45.249792 (+811319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.250579 (+   787us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.309678 (+ 59099us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b5400115650. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:45.309829 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1864969344}
W20260809 23:21:45.343480 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.007s	sys 0.031s
I20260809 23:21:45.343433 24167 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 8 to 13, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 13 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:45.358534  3878 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:45.361682  3878 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.362181  3154 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.365869 24545 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260809 23:21:45.305609 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:45.356341 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.013s	sys 0.025s
I20260809 23:21:45.331442  3683 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:45.367219  3683 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:45.343165  3272 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.348019  3895 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:45.366881 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.013s	sys 0.025s
I20260809 23:21:45.293717  3920 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:45.373240 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 67744.2us
I20260809 23:21:45.371769  3239 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:45.376646  3843 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
I20260809 23:21:45.377817  4010 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:45.383874  3895 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.367054  3633 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:45.384119  3763 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.020s	user 0.011s	sys 0.000s
W20260809 23:21:45.407209 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.012s	sys 0.021s
W20260809 23:21:45.407637 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.013s	sys 0.021s
W20260809 23:21:45.396818 24155 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 1.242s	user 0.013s	sys 0.009s
I20260809 23:21:45.382647  3892 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.384543 24543 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:45.413949 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:45.414899 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.005s	sys 0.017s
W20260809 23:21:45.415237 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.005s	sys 0.023s
W20260809 23:21:45.415453 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.006s	sys 0.017s
W20260809 23:21:45.382108  3947 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.872s	user 0.010s	sys 0.003s
I20260809 23:21:45.405416 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.382472  3407 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:45.385732  3587 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.335837  3677 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:45.415656 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.005s	sys 0.023s
I20260809 23:21:45.359529  3963 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.405634  3964 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:45.390236 24541 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:21:45.430624 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:45.418812 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.417092  3154 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.383206  3272 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.431618  3964 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:45.416766 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1785) took 1561 ms. Trace:
I20260809 23:21:45.427383  3889 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:45.409432  3763 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:45.434207 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:45.407795  4010 consensus_queue.cc:1225] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260809 23:21:45.451025  4022 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260809 23:21:45.426069  3885 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
W20260809 23:21:45.452473  4024 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:45.436832  3772 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:45.453408  3772 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:45.454370  3772 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.455754  3772 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.457446 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1842) took 1132 ms. Trace:
I20260809 23:21:45.259594 25368 rpcz_store.cc:276] 0809 23:21:43.509060 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.746395 (+237335us) service_pool.cc:227] Handling call
0809 23:21:43.776902 (+ 30507us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.777107 (+   205us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:44.588330 (+811223us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.213127 (+624797us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.213177 (+    50us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:45.213223 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.213282 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.213388 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":960700,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":3580,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":24553,"thread_start_us":1398192,"threads_started":2}
I20260809 23:21:45.439515  3529 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:45.458956  3529 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:45.457837 24320 rpcz_store.cc:276] 0809 23:21:44.324546 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.325087 (+   541us) service_pool.cc:227] Handling call
0809 23:21:44.349509 (+ 24422us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.358842 (+  9333us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.358877 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.359367 (+   490us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.138192 (+778825us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.138248 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.257343 (+119095us) spinlock_profiling.cc:243] Waited 8.94 ms on lock 0x7b54003c23d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:45.257388 (+    45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.457057 (+199669us) raft_consensus.cc:1671] finished
0809 23:21:45.457087 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.457353 (+   266us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1783,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"spinlock_wait_cycles":20568448,"thread_start_us":778438,"threads_started":1,"wal-append.queue_time_us":1074688}
I20260809 23:21:45.459812  3529 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.433238  3349 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.414111  3892 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.450388 24541 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:45.426745  3407 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.452412  3885 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:45.438199  3239 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260809 23:21:45.427332 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 673919us
I20260809 23:21:45.460307 24320 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:45.463665 24317 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:45.434273  3911 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.396332 24167 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 9 to 10, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 10 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:21:45.422542 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 759618us
I20260809 23:21:45.463166  3154 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:45.465292 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.008s	sys 0.014s
I20260809 23:21:45.485690 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:45.486053 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.008s	sys 0.015s
I20260809 23:21:45.486425 24429 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:45.489634 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3656) took 2119 ms. Trace:
I20260809 23:21:45.450516 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.490090 24542 rpcz_store.cc:276] 0809 23:21:43.370605 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.106400 (+735795us) service_pool.cc:227] Handling call
0809 23:21:44.254663 (+148263us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.255073 (+   410us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.255104 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.255528 (+   424us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.016329 (+760801us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:45.016399 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.016492 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.455073 (+438581us) raft_consensus.cc:1671] finished
0809 23:21:45.455095 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.472709 (+ 17614us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54001fc050. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:45.489551 (+ 16842us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4588,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":340154112,"thread_start_us":760375,"threads_started":1,"wal-append.queue_time_us":1150389}
I20260809 23:21:45.440693 25369 rpcz_store.cc:276] 0809 23:21:43.855087 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.872573 (+ 17486us) service_pool.cc:227] Handling call
0809 23:21:43.872820 (+   247us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:43.873007 (+   187us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:44.748747 (+875740us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.416370 (+667623us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.416437 (+    67us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.416498 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.416573 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.416709 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4770,"prepare.queue_time_us":953194,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":43,"raft.queue_time_us":99,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":9726,"thread_start_us":1496498,"threads_started":2}
I20260809 23:21:45.479945 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:45.442780 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.024s	sys 0.006s
W20260809 23:21:45.481067 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.007s	sys 0.016s
I20260809 23:21:45.429782  3587 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.484177 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.472312 24545 pending_rounds.cc:76] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
I20260809 23:21:45.495661 24545 pending_rounds.cc:97] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260809 23:21:45.495998 24545 raft_consensus.cc:2866] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:21:45.510149 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60104.3us
W20260809 23:21:45.494791 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.007s	sys 0.016s
W20260809 23:21:45.494501 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.024s	sys 0.006s
I20260809 23:21:45.487504 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1838) took 1469 ms. Trace:
I20260809 23:21:45.466720  3349 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:45.452569  3272 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:45.522907 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.523077 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3605) took 1040 ms. Trace:
I20260809 23:21:45.543617 24317 rpcz_store.cc:276] 0809 23:21:44.487274 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.496134 (+  8860us) service_pool.cc:227] Handling call
0809 23:21:44.496508 (+   374us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:45.487263 (+990755us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.522817 (+ 35554us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b5400224a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:45.523024 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2223708416}
I20260809 23:21:45.524339 24431 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:45.523705 24314 rpcz_store.cc:276] 0809 23:21:44.017502 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.089110 (+ 71608us) service_pool.cc:227] Handling call
0809 23:21:44.089455 (+   345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:44.089651 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:44.823497 (+733846us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.486766 (+663269us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:45.486833 (+    67us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:45.486903 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.487000 (+    97us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.487183 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1159215,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":2788,"raft.queue_time_us":1067,"thread_start_us":1396406,"threads_started":2}
I20260809 23:21:45.527459 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.529366 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.530920 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.546548 24430 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.548092 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.533713 24541 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
W20260809 23:21:45.488431  3369 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.011s	sys 0.010s
W20260809 23:21:45.489110  3253 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
I20260809 23:21:45.557008 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:45.561123 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3596) took 1781 ms. Trace:
I20260809 23:21:45.561975 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.562644 24314 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:21:45.563766 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.562322 24167 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 12 to 14, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 14 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:45.571334 24320 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:45.572163 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.572953 24431 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:45.576303 24167 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 15 to 18, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 18 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:21:45.578979 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.007s	sys 0.023s
I20260809 23:21:45.561795 24313 rpcz_store.cc:276] 0809 23:21:43.779462 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.785951 (+  6489us) service_pool.cc:227] Handling call
0809 23:21:43.786277 (+   326us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:43.850529 (+ 64252us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:43.850570 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:43.851043 (+   473us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.738885 (+887842us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.520274 (+781389us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.560587 (+ 40313us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.560776 (+   189us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.560926 (+   150us) raft_consensus.cc:1671] finished
0809 23:21:45.560943 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.561068 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117632,"prepare.queue_time_us":524422,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":64639,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":8900,"thread_start_us":1586853,"threads_started":2,"wal-append.queue_time_us":1243903}
I20260809 23:21:45.580461 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.522264  3911 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.572111 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.525853  3834 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:45.584586 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61814.2us
I20260809 23:21:45.586709 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.587649 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:45.587850 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.546774  3239 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260809 23:21:45.562727  3369 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.012s	sys 0.010s
W20260809 23:21:45.579484 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.007s	sys 0.023s
I20260809 23:21:45.591240 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:45.595745 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.000s	sys 0.036s
W20260809 23:21:45.596212 24309 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.006s	sys 0.019s
I20260809 23:21:45.597028 24313 raft_consensus.cc:2354] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22.
I20260809 23:21:45.542593  2995 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.581360 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.565692  2820 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:45.572960  3557 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.585224  3431 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:45.620147  4000 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260809 23:21:45.522097  3154 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.592433  3633 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:45.679220 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88027.7us
I20260809 23:21:45.597281  3759 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:45.596576  3911 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.687852 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.703332 24167 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 12 to 14, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 14 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:45.600615 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.569681  3253 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.596758  3889 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:45.597880  3791 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.647406  3920 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:45.644217  3316 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:45.653278 24431 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:45.681700  3239 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:21:45.681989  3259 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.005s
W20260809 23:21:45.682868  4029 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
W20260809 23:21:45.683362  4025 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
W20260809 23:21:45.684276  4027 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260809 23:21:45.589720  3676 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.008s	sys 0.017s
I20260809 23:21:45.670902  2995 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.673868  3823 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.697497 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:45.757959 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60639.5us
W20260809 23:21:45.706022 24309 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.008s	sys 0.020s
I20260809 23:21:45.705757 24320 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:45.682683 24161 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 242) took 27743 ms (client timeout 15000 ms). Trace:
I20260809 23:21:45.763921 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1840) took 1686 ms. Trace:
W20260809 23:21:45.725909 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.911s	user 0.005s	sys 0.036s
I20260809 23:21:45.727058  3677 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:45.731535 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.732779 24546 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:45.726364  3660 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.013s	sys 0.023s
I20260809 23:21:45.780608 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:45.781777 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:45.748494 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:45.745324 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.646689  3958 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:45.785297 25372 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:45.764381 24309 rpcz_store.cc:276] 0809 23:21:44.077696 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.094431 (+ 16735us) service_pool.cc:227] Handling call
0809 23:21:44.094721 (+   290us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.121854 (+ 27133us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.121891 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.132644 (+ 10753us) log.cc:842] Serialized 26 byte log entry
0809 23:21:44.852341 (+719697us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:45.759888 (+907547us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.763500 (+  3612us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.763614 (+   114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.763671 (+    57us) raft_consensus.cc:1671] finished
0809 23:21:45.763688 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.763844 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5652,"prepare.queue_time_us":411223,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":3328,"raft.queue_time_us":5321,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":123,"thread_start_us":1555552,"threads_started":2,"wal-append.queue_time_us":1056083}
I20260809 23:21:45.786398 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:45.712577  3948 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.006s	sys 0.022s
W20260809 23:21:45.792472 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44075.6us
W20260809 23:21:45.726312  3689 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.019s	sys 0.022s
W20260809 23:21:45.811077  3689 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s	user 0.019s	sys 0.022s
I20260809 23:21:45.646996  3885 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:45.773304 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.814483 24431 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:45.815317 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.815847 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:45.816381 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.817063 24431 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:45.817714 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.818387 24431 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:45.774107 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.821735 24430 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:45.786356  3793 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.023s	user 0.043s	sys 0.072s
W20260809 23:21:45.710870 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.000s	sys 0.027s
I20260809 23:21:45.735033  3316 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:45.754511  3676 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.009s	sys 0.018s
I20260809 23:21:45.777032 24313 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:45.848351 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.848627 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.849093 24431 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:45.752535 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.789309 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:45.795054 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.795768 24309 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:45.778302  3660 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.013s	sys 0.023s
I20260809 23:21:45.796765 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:45.797778  3154 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:45.799477  4031 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
W20260809 23:21:45.850301 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 97968.8us
W20260809 23:21:45.827589  3948 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.021s	user 0.006s	sys 0.022s
W20260809 23:21:45.802587  4034 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260809 23:21:45.803107  3407 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.763132  3823 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.778512  3815 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:45.868914 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:45.807026  3958 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.808578 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.809407 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.810034  3253 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:45.776023  3828 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:45.810150 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.810664  3587 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:45.824450 24429 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:45.824080 24317 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
W20260809 23:21:45.824479 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.002s	sys 0.027s
I20260809 23:21:45.819432 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3607) took 1331 ms. Trace:
I20260809 23:21:45.833002 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:45.879488 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46579.8us
I20260809 23:21:45.878954 24321 rpcz_store.cc:276] 0809 23:21:44.488042 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.491043 (+  3001us) service_pool.cc:227] Handling call
0809 23:21:44.491428 (+   385us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.521153 (+ 29725us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.521199 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.521702 (+   503us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.147836 (+626134us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.147889 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.147957 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.819207 (+671250us) raft_consensus.cc:1671] finished
0809 23:21:45.819227 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.819351 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29269,"raft.queue_time_us":29525,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":625779,"threads_started":1,"wal-append.queue_time_us":927929}
I20260809 23:21:45.880949 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.881634 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:45.847494 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:45.756765  3810 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
I20260809 23:21:45.726720  3791 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.854050 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.854431 24313 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:45.763157  3896 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.012s	sys 0.020s
I20260809 23:21:45.857782 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.898183 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:45.801385  4032 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260809 23:21:45.850627 24428 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:45.834048  3793 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.044s	sys 0.072s
W20260809 23:21:45.851250 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56322.9us
I20260809 23:21:45.859849 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:45.863253 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:45.835968  3870 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.021s	sys 0.085s
I20260809 23:21:45.864989 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1843) took 1539 ms. Trace:
I20260809 23:21:45.870476 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:45.868153 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3614) took 1363 ms. Trace:
W20260809 23:21:45.864142  4021 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260809 23:21:45.848097  3676 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:45.850310 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:21:45.925817 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.926836 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:45.884136 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:45.849689  4041 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:45.835786  3843 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260809 23:21:45.889138 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.886132 24321 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:45.887622 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.013s	sys 0.019s
W20260809 23:21:45.886183 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.018s	sys 0.024s
W20260809 23:21:45.889381 24544 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.010s	sys 0.026s
W20260809 23:21:45.939405 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.018s	sys 0.024s
W20260809 23:21:45.939765 24544 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.010s	sys 0.026s
W20260809 23:21:45.939430 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.014s	sys 0.019s
W20260809 23:21:45.895725 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64363.1us
W20260809 23:21:45.899750 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.014s	sys 0.022s
I20260809 23:21:45.864820  3892 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.867520  2995 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.866654  3407 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.904690 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.948402 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:45.902276 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.011s	sys 0.015s
W20260809 23:21:45.949276 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.011s	sys 0.015s
I20260809 23:21:45.910212 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:45.911233 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.015s	sys 0.012s
I20260809 23:21:45.907115 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:45.907883 24312 rpcz_store.cc:276] 0809 23:21:44.504638 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.516596 (+ 11958us) service_pool.cc:227] Handling call
0809 23:21:44.516953 (+   357us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.517178 (+   225us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.517201 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.517645 (+   444us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.155969 (+638324us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.156007 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.158974 (+  2967us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.867953 (+708979us) raft_consensus.cc:1671] finished
0809 23:21:45.867975 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.868102 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":637949,"threads_started":1,"wal-append.queue_time_us":933168}
I20260809 23:21:45.907657 24311 rpcz_store.cc:276] 0809 23:21:44.325270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.325396 (+   126us) service_pool.cc:227] Handling call
0809 23:21:44.341425 (+ 16029us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.341777 (+   352us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.341805 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.342376 (+   571us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.102905 (+760529us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.102985 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.103076 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:45.864765 (+761689us) raft_consensus.cc:1671] finished
0809 23:21:45.864786 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:45.864920 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1148,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":1721,"thread_start_us":760028,"threads_started":1,"wal-append.queue_time_us":1096797}
W20260809 23:21:45.899549  3789 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.018s	sys 0.017s
I20260809 23:21:45.908717  3948 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:45.930858 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:45.942082 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.015s	sys 0.023s
W20260809 23:21:45.916042  3870 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.021s	sys 0.086s
W20260809 23:21:45.907794  3896 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.013s	sys 0.020s
I20260809 23:21:45.947917 24541 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260809 23:21:45.905601 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:45.885893  4055 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260809 23:21:45.951301 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.015s	sys 0.012s
I20260809 23:21:45.927561  3920 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:45.889398  3154 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:45.953667 24378 leader_election.cc:398] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22.
I20260809 23:21:45.954484 24309 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:45.949326 24321 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:46.007719 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:45.871955  3895 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.874171  3815 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:45.955799 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71764.6us
I20260809 23:21:46.013945 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:46.014530 24311 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:46.014698 24317 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:45.956284 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46170.8us
I20260809 23:21:46.015947 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.918869  3930 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:45.955435  3889 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:45.959175  3660 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:45.959703 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:46.048859 24312 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:45.891956  3563 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
W20260809 23:21:45.950954 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43955.3us
I20260809 23:21:46.051738 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:45.953274  3789 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.019s	sys 0.018s
I20260809 23:21:45.929761  3877 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:46.063371 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:46.004912 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:46.064047 24435 raft_consensus.cc:2354] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
I20260809 23:21:46.064105 25369 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:46.065078  3633 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:46.065604  3633 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:45.958125 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:46.067579 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 109605us
I20260809 23:21:46.068910 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.070194 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.071323  4046 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:46.071739  4046 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:46.072328  4053 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260809 23:21:46.072613  4046 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:46.073589  4045 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260809 23:21:46.073915  4046 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.051494  3660 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:46.075596  3660 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:46.078455  3979 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Stepping down as leader of term 16
I20260809 23:21:46.007009 24378 leader_election.cc:401] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:46.007826 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:45.954627 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:46.083904 24320 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:45.948197  3239 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:46.003017 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 72265.3us
I20260809 23:21:46.019435 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:46.017423 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.092432 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.083312 24321 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:46.008744  4034 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.077s	user 0.006s	sys 0.000s
I20260809 23:21:46.043139  3676 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.046226 24311 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:46.047240 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:46.034391 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:46.054518 24542 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:46.059563  3253 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:46.063210 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:46.009654  3885 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:46.085486 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.019809  3772 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
W20260809 23:21:46.041291 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.219s	user 0.020s	sys 0.032s
I20260809 23:21:45.929675  4054 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.130s
I20260809 23:21:45.917414  3316 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.035079  3963 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:46.006610  3920 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:46.079047  3979 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.123883 24313 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:46.122637 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:46.158959 24312 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:46.094007  4010 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:46.092612 24309 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:46.107213  3930 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.127610 24266 leader_election.cc:398] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
W20260809 23:21:46.099225 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 145671us
I20260809 23:21:46.119932 24541 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260809 23:21:46.168104 24266 leader_election.cc:401] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260809 23:21:46.134595 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.165822 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3617) took 1660 ms. Trace:
I20260809 23:21:46.096837 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.008584  3911 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:45.953719  3689 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
I20260809 23:21:46.147943  3529 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:46.144903 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 63315.8us
W20260809 23:21:46.145924 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.361s	user 0.020s	sys 0.032s
I20260809 23:21:46.147647 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:46.096012  3968 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:46.091954  2995 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:46.134271  4032 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260809 23:21:46.209237 25366 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:46.211818  3815 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:46.214522  4056 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260809 23:21:46.096971 24321 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:46.216244  4070 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260809 23:21:46.216347 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.133981  3983 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260809 23:21:46.218066  4058 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260809 23:21:46.219803  3755 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:46.220208  3755 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.221058  3755 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.222034  3755 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:46.164932 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.128772  4044 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.054s
I20260809 23:21:46.218086 24321 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:21:46.224454 24321 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260809 23:21:46.130064  3218 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22.
I20260809 23:21:46.215240 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.227190 24314 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260809 23:21:46.056418  3843 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.095s
I20260809 23:21:46.228231  3930 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:45.763037 24161 rpcz_store.cc:269] 0809 23:21:17.732271 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:17.732585 (+   314us) service_pool.cc:227] Handling call
0809 23:21:45.466002 (+27733417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:21:33.779316 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:21:33.779898 (+   582us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:21:33.779952 (+    54us) write_op.cc:435] Acquired schema lock
0809 23:21:33.779972 (+    20us) tablet.cc:660] Decoding operations
0809 23:21:33.788526 (+  8554us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:21:33.788625 (+    99us) write_op.cc:641] Partition lock acquired for write op
0809 23:21:33.788664 (+    39us) tablet.cc:683] Acquiring locks for 18 operations
0809 23:21:33.790103 (+  1439us) tablet.cc:699] Row locks acquired
0809 23:21:33.790141 (+    38us) write_op.cc:260] PREPARE: finished
0809 23:21:33.790303 (+   162us) write_op.cc:270] Start()
0809 23:21:33.790469 (+   166us) write_op.cc:276] Timestamp: P: 1786317693790247 usec, L: 0
0809 23:21:33.790506 (+    37us) op_driver.cc:331] REPLICATION: starting
0809 23:21:33.879616 (+ 89110us) log.cc:842] Serialized 6990 byte log entry
0809 23:21:34.829091 (+949475us) spinlock_profiling.cc:243] Waited 9.49 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:36.068506 (+1239415us) op_driver.cc:444] REPLICATION: finished
0809 23:21:37.831352 (+1762846us) write_op.cc:301] APPLY: starting
0809 23:21:37.854859 (+ 23507us) tablet.cc:1365] starting BulkCheckPresence
0809 23:21:39.037090 (+1182231us) tablet.cc:1368] finished BulkCheckPresence
0809 23:21:39.037169 (+    79us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:21:39.761794 (+724625us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:21:39.762204 (+   410us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=21,mrs_lookups=0
0809 23:21:39.762281 (+    77us) write_op.cc:312] APPLY: finished
0809 23:21:39.762890 (+   609us) log.cc:842] Serialized 164 byte log entry
0809 23:21:40.120280 (+357390us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:40.120706 (+   426us) write_op.cc:454] Released schema lock
0809 23:21:40.120843 (+   137us) write_op.cc:341] FINISH: Updating metrics
0809 23:21:40.121391 (+   548us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:40.121423 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8145,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":1614025,"apply.run_cpu_time_us":126660,"apply.run_wall_time_us":2304685,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79295,"num_ops":18,"prepare.queue_time_us":133095,"prepare.run_cpu_time_us":21459,"prepare.run_wall_time_us":433377,"raft.queue_time_us":1071509,"raft.run_cpu_time_us":48376,"raft.run_wall_time_us":1369467,"replication_time_us":2259697,"spinlock_wait_cycles":22143616,"thread_start_us":2346985,"threads_started":5,"wal-append.queue_time_us":895066,"wal-append.run_cpu_time_us":41419,"wal-append.run_wall_time_us":3198021}]]}
I20260809 23:21:45.955698  3407 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.230479 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.231117 24435 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:46.231329  3407 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:46.232183  3633 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:46.013459  3154 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.169144 24430 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:46.170518 24323 rpcz_store.cc:276] 0809 23:21:44.505573 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.560148 (+ 54575us) service_pool.cc:227] Handling call
0809 23:21:44.560562 (+   414us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:44.627283 (+ 66721us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.627549 (+   266us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:44.628773 (+  1224us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.404863 (+776090us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.427581 (+ 22718us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.427822 (+   241us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.460866 (+ 33044us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b18001210c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:45.482378 (+ 21512us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:45.482463 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.165515 (+683052us) raft_consensus.cc:1671] finished
0809 23:21:46.165545 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.165729 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":22897,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":1758,"raft.queue_time_us":1615,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":10641,"spinlock_wait_cycles":133198848,"thread_start_us":716983,"threads_started":1,"wal-append.queue_time_us":1038574}
I20260809 23:21:46.207631 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:46.140712 24320 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:46.241729 24311 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 24. Ops: []
I20260809 23:21:46.096206  3759 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
W20260809 23:21:46.172837 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47678.7us
I20260809 23:21:46.161080  3772 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.151299  3885 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.247980 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.248737 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:46.248894 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.245473 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3658) took 2778 ms. Trace:
I20260809 23:21:46.246057 25366 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:46.227730  3895 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:46.208058 24313 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:46.202775 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:46.217996  4067 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260809 23:21:46.227820 24312 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:21:46.271942 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:46.161126  3259 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:46.235972 25369 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:46.236042 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.162478  3920 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:46.259912  4033 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:46.215183  3791 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:46.128096  3431 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260809 23:21:46.245117 24323 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260809 23:21:46.251264 24544 rpcz_store.cc:276] 0809 23:21:43.467178 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.366274 (+899096us) service_pool.cc:227] Handling call
0809 23:21:44.442572 (+ 76298us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:44.443053 (+   481us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:44.443093 (+    40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:44.580261 (+137168us) log.cc:842] Serialized 52 byte log entry
0809 23:21:45.359797 (+779536us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.940327 (+580530us) log.cc:842] Serialized 14 byte log entry
0809 23:21:45.940633 (+   306us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.940723 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.047917 (+107194us) raft_consensus.cc:1671] finished
0809 23:21:46.047947 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.245292 (+197345us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x7b54001d6850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:46.245417 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":649073,"prepare.run_cpu_time_us":187,"prepare.run_wall_time_us":2289,"raft.queue_time_us":3179,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":8695,"spinlock_wait_cycles":174679296,"thread_start_us":1267236,"threads_started":2,"wal-append.queue_time_us":1046518}
I20260809 23:21:46.251053  4054 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:46.239980  3529 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.016s
W20260809 23:21:46.256549 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49006.7us
I20260809 23:21:46.257393 24430 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:46.306090 24430 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.306257 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:46.205235 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 57677.6us
I20260809 23:21:46.239167  3823 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:46.233242  3958 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:46.276793 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3581) took 1048 ms. Trace:
I20260809 23:21:46.315078 24546 rpcz_store.cc:276] 0809 23:21:45.228573 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.230319 (+  1746us) service_pool.cc:227] Handling call
0809 23:21:46.276600 (+1046281us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b540029ca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:46.276729 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1143627136}
I20260809 23:21:46.316177 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1746) took 1661 ms. Trace:
I20260809 23:21:46.316457 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.317023  4089 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:46.317405 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1795) took 1562 ms. Trace:
I20260809 23:21:46.247447  4010 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:46.162789  3963 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.290220 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.238121 24429 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:46.332304 24429 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.295631  4033 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:46.253630  4044 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:46.273401  2820 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:46.300566 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 203797us
I20260809 23:21:46.372131 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.273614  3633 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:46.304240 24544 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:46.307477 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:46.373553 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66211.6us
I20260809 23:21:46.314781  3947 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:46.291132  3878 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.929s	user 0.012s	sys 0.001s
I20260809 23:21:46.317842 25371 rpcz_store.cc:276] 0809 23:21:44.755352 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:44.757414 (+  2062us) service_pool.cc:227] Handling call
0809 23:21:45.065971 (+308557us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.066464 (+   493us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.066508 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.067019 (+   511us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.941577 (+874558us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.941636 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.941722 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.217510 (+275788us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:46.314361 (+ 96851us) raft_consensus.cc:1671] finished
0809 23:21:46.314386 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.317240 (+  2854us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54006d4cd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:46.317338 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":10308,"spinlock_wait_cycles":648372352,"thread_start_us":822725,"threads_started":1,"wal-append.queue_time_us":956683}
I20260809 23:21:46.329547  3870 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:46.317721  4089 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.260854  3810 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
I20260809 23:21:46.337901 24546 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:46.331385 24539 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:46.318693 24430 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.293893  3154 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:46.338244 24428 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:21:46.301630  3843 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:46.333118 24429 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.415943 24429 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.300005  3759 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:46.354625 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 151916us
I20260809 23:21:46.427711 24378 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.307749 25321 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.389690  2820 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:46.389474  3633 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:46.374984 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:46.370326  4044 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:46.405578 24430 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.406010 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.400451 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.408962 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:46.352276  4033 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:46.353914  4010 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:46.425047 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.312892  4054 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:46.429443 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 121802us
I20260809 23:21:46.466782 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.316609 24534 rpcz_store.cc:276] 0809 23:21:44.654972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.280237 (+625265us) service_pool.cc:227] Handling call
0809 23:21:45.280569 (+   332us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.283633 (+  3064us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.283670 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.284111 (+   441us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.939841 (+655730us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.939878 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.939952 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.315981 (+376029us) raft_consensus.cc:1671] finished
0809 23:21:46.316001 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.316119 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2640,"raft.queue_time_us":4374,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"thread_start_us":655315,"threads_started":1,"wal-append.queue_time_us":717267}
I20260809 23:21:46.467923 24430 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:46.468086 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.388047  3677 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:46.473047 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.433189 24429 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260809 23:21:46.395645 24321 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260809 23:21:46.387079  3407 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:46.455026 24546 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:46.487875 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 78991.1us
I20260809 23:21:46.489399 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.456845 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.463450 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.450964  3272 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:46.484017 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.494804  3968 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.497078 24546 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:46.498136 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.480950 24429 pending_rounds.cc:76] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 12
I20260809 23:21:46.331105  3772 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.415514  4096 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:46.485873 24534 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:46.486898 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 112002us
I20260809 23:21:46.397127  3870 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:46.511086  3870 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:46.484704 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.456611  3154 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:21:46.453923  2820 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:46.499735 24429 pending_rounds.cc:97] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260809 23:21:46.428265  3676 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:46.461766  3633 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:46.499854 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.462714  4044 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:46.501431 24544 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:46.503103  3815 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:46.456073 24155 maintenance_manager.cc:643] P e3357d0fe4ef46db9a4739044fa527d5: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 5.208s	user 0.110s	sys 0.065s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":24720,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":419591,"dirs.run_cpu_time_us":358,"dirs.run_wall_time_us":39683,"lbm_read_time_us":63303,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":19,"lbm_write_time_us":122656,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":8,"reinsert_count":0,"spinlock_wait_cycles":7807232,"thread_start_us":1151194,"threads_started":3,"update_count":160,"wal-append.queue_time_us":1213998,"wal-append.run_cpu_time_us":928,"wal-append.run_wall_time_us":1040453}
W20260809 23:21:46.503984 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 255141us
I20260809 23:21:46.518343 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:46.487911  3683 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.108s	user 0.009s	sys 0.005s
I20260809 23:21:46.507730 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:46.399716  3878 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.354825  3885 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.492158  3529 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:46.436883  3759 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.427902  3843 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.502066  4084 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:46.557534 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.513980 24542 raft_consensus.cc:2362] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24. Re-sending same reply.
W20260809 23:21:46.519723 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 147696us
I20260809 23:21:46.471764  3983 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.543805 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:46.566418 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.566843 24546 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260809 23:21:46.547598  3950 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:46.571185  3950 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.571964  4054 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:46.572083  3950 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:46.573112  4069 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260809 23:21:46.573470  3950 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:46.561096  3154 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.577503  3878 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:46.578902  4089 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:46.578495  3154 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.579213  3963 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:46.580003  4075 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.376s	user 0.000s	sys 0.006s
I20260809 23:21:46.515288  4096 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.581444 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3619) took 2075 ms. Trace:
W20260809 23:21:46.532732  3349 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.971s	user 0.009s	sys 0.003s
I20260809 23:21:46.584223 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.544782 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.532655 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:46.587662 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55097.5us
W20260809 23:21:46.587736 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 93616.9us
I20260809 23:21:46.535019 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.535463 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.532262 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.532115 24544 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:46.523285 24429 raft_consensus.cc:2866] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:46.519330 24534 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:46.487439  3677 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.562001 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:46.519970  4079 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
I20260809 23:21:46.581852 24316 rpcz_store.cc:276] 0809 23:21:44.506227 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.335304 (+829077us) service_pool.cc:227] Handling call
0809 23:21:45.362149 (+ 26845us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.362489 (+   340us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.362541 (+    52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.362970 (+   429us) log.cc:842] Serialized 26 byte log entry
0809 23:21:46.003133 (+640163us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:46.021765 (+ 18632us) log.cc:842] Serialized 14 byte log entry
0809 23:21:46.054387 (+ 32622us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:46.054457 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.572337 (+517880us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:46.581112 (+  8775us) raft_consensus.cc:1671] finished
0809 23:21:46.581230 (+   118us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.581371 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68527,"prepare.queue_time_us":43385,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":34183,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":9315,"spinlock_wait_cycles":9944832,"thread_start_us":639792,"threads_started":1,"wal-append.queue_time_us":880712}
I20260809 23:21:46.583595 24546 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:46.539788  4099 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
W20260809 23:21:46.521267  4068 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260809 23:21:46.591132 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:46.592427 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:46.595705 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.605103 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1755) took 1294 ms. Trace:
I20260809 23:21:46.531490  3272 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:46.597504 24423 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260809 23:21:46.598687 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.620540 24544 raft_consensus.cc:2396] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260809 23:21:46.600023 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.623063 24546 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:46.600348 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:46.532259  4035 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.748s	user 0.002s	sys 0.001s
I20260809 23:21:46.610049 24535 rpcz_store.cc:276] 0809 23:21:45.310792 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.356068 (+ 45276us) service_pool.cc:227] Handling call
0809 23:21:45.369961 (+ 13893us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.370221 (+   260us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.370258 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.374417 (+  4159us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.957785 (+583368us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.957828 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.957892 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.579444 (+621552us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:46.583446 (+  4002us) raft_consensus.cc:1671] finished
0809 23:21:46.583467 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.605040 (+ 21573us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27635072,"thread_start_us":583086,"threads_started":1,"wal-append.queue_time_us":779094}
I20260809 23:21:46.558254  4084 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:46.631692  4039 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:46.632023  4084 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:46.635507 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:46.637037 24541 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:46.637807 25366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:46.638639  3892 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:46.639038  3892 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:46.639925  3892 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.641608  3892 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:46.643993 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1745) took 2131 ms. Trace:
I20260809 23:21:46.644398 24540 rpcz_store.cc:276] 0809 23:21:44.512507 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.257043 (+744536us) service_pool.cc:227] Handling call
0809 23:21:45.257385 (+   342us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.264243 (+  6858us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.264287 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.281506 (+ 17219us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.949891 (+668385us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:45.949948 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.950039 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.643789 (+693750us) raft_consensus.cc:1671] finished
0809 23:21:46.643815 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.643938 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21219,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":3902,"thread_start_us":605914,"threads_started":1,"wal-append.queue_time_us":826176}
I20260809 23:21:46.590441  3529 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:46.632117  4039 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:46.649544 24534 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260809 23:21:46.630127 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.652380 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:46.654563 24266 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:46.609084  4078 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
I20260809 23:21:46.603358  4109 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
W20260809 23:21:46.662191 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61956.8us
I20260809 23:21:46.601962  3963 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:46.554010  3683 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.656450 24544 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:46.606182  4089 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:46.565466  3763 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
W20260809 23:21:46.648317  3964 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.186s	user 0.007s	sys 0.007s
I20260809 23:21:46.626238  4054 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:46.552513  3676 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.666025 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:46.654695  4082 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
I20260809 23:21:46.531270  4010 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:46.678148 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 170473us
W20260809 23:21:46.728698 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45405.8us
W20260809 23:21:46.674268 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 182762us
W20260809 23:21:46.662611  3889 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.601s	user 0.011s	sys 0.000s
I20260809 23:21:46.627002  4099 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.728667 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:46.654995  4093 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260809 23:21:46.648530  4046 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.004s
I20260809 23:21:46.596913  3349 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.685724 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:46.646442  4115 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:46.665518  4041 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:46.658604  3431 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:46.731386 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.736202 24317 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:46.670368  4116 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:46.747824 25368 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:46.748202 25365 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:46.622079  3878 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260809 23:21:46.681843 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:46.753049  3689 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:46.754112 25373 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:46.755313 25363 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:46.756999 25375 raft_consensus.cc:2354] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
W20260809 23:21:46.759167 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.565s (SENT) [suppressed 7 similar messages]
I20260809 23:21:46.762310 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.763268 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.757927 25369 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:46.769840 24544 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260809 23:21:46.769752 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:46.755754  4108 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260809 23:21:46.770864 25371 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:46.768862 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1756) took 1327 ms. Trace:
I20260809 23:21:46.789186 24543 rpcz_store.cc:276] 0809 23:21:45.441595 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.442476 (+   881us) service_pool.cc:227] Handling call
0809 23:21:45.442949 (+   473us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:45.474625 (+ 31676us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.474657 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:45.483133 (+  8476us) log.cc:842] Serialized 26 byte log entry
0809 23:21:45.935495 (+452362us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:45.935535 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:45.935598 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.768649 (+833051us) raft_consensus.cc:1671] finished
0809 23:21:46.768669 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.768793 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1712,"raft.queue_time_us":31259,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":12029,"thread_start_us":369433,"threads_started":1,"wal-append.queue_time_us":712184}
I20260809 23:21:46.756524  3791 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:46.792273 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:46.759503  3889 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:46.788156 24535 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260809 23:21:46.794793 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 168617us
W20260809 23:21:46.769446  4032 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260809 23:21:46.671748  3815 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.094s
W20260809 23:21:46.795673 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 64422.3us
I20260809 23:21:46.802610 24312 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:46.819343 24423 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:46.682346  3964 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.821266 24541 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:46.813138 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:46.816507  4119 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
W20260809 23:21:46.796473  4098 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
W20260809 23:21:46.797202  3932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.234s	user 0.037s	sys 0.021s
W20260809 23:21:46.823375  4048 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.000s	sys 0.007s
I20260809 23:21:46.791495  3431 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:46.892249 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1757) took 1429 ms. Trace:
I20260809 23:21:46.894920 24320 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:46.817873  4046 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:46.838244 24313 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:46.904608 24545 rpcz_store.cc:276] 0809 23:21:45.463186 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.463388 (+   202us) service_pool.cc:227] Handling call
0809 23:21:45.471965 (+  8577us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:45.579641 (+107676us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:45.579672 (+    31us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0809 23:21:45.580403 (+   731us) log.cc:842] Serialized 78 byte log entry
0809 23:21:45.924820 (+344417us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:46.158315 (+233495us) log.cc:842] Serialized 14 byte log entry
0809 23:21:46.158561 (+   246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:46.158631 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:46.891954 (+733323us) raft_consensus.cc:1671] finished
0809 23:21:46.891984 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:46.892173 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22648,"prepare.queue_time_us":173377,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":125699,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":18800,"thread_start_us":469793,"threads_started":2,"wal-append.queue_time_us":531237}
W20260809 23:21:46.942719 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 118321us
I20260809 23:21:46.951928 24424 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260809 23:21:46.809641  3828 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
W20260809 23:21:46.923173  3932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.368s	user 0.038s	sys 0.021s
W20260809 23:21:46.923192  4048 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.000s	sys 0.008s
I20260809 23:21:46.801687  4099 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:46.791671  4093 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.141s
I20260809 23:21:46.894870  3958 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:46.886998  4054 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:46.942137  4063 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.870s	user 0.001s	sys 0.002s
I20260809 23:21:46.967183 24422 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:46.927614  3759 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:46.940953  3563 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260809 23:21:46.941757  4096 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:46.939757  4109 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:46.766691  4039 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:46.902861  4119 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260809 23:21:46.951236  3677 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:47.007302 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 248119us
W20260809 23:21:47.016075 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 213876us
W20260809 23:21:46.983417  4122 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260809 23:21:47.008081  4089 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:47.003500  4048 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:21:47.039840 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:47.041239 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:47.041354  3431 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.051355  4054 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:47.053246  4131 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260809 23:21:47.038347  4130 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260809 23:21:47.081699 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:47.091173 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:47.047508 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:47.063225 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:47.068657 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:47.047894 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 394299us
I20260809 23:21:47.030465  3878 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:47.043612  3683 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:47.059757  3968 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.564s	user 0.007s	sys 0.003s
I20260809 23:21:46.997895  3958 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.096813 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:47.061597  4143 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260809 23:21:47.060353  4135 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
W20260809 23:21:47.045853  3660 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.970s	user 0.009s	sys 0.006s
I20260809 23:21:47.100380 24423 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:47.049161  3349 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:47.053921  4133 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260809 23:21:47.053388  4129 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
W20260809 23:21:47.057452  3633 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.529s	user 0.004s	sys 0.004s
W20260809 23:21:47.066435  4080 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.847s	user 0.005s	sys 0.003s
I20260809 23:21:47.058223  3983 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:47.046528  4152 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:47.052513  3870 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.004s
I20260809 23:21:47.087001  4041 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
I20260809 23:21:47.174252 24423 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:47.065976  4084 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.309882 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:47.183048 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:47.190912 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123076us
I20260809 23:21:47.241070 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1863) took 1460 ms. Trace:
I20260809 23:21:47.320683 25368 rpcz_store.cc:276] 0809 23:21:45.780566 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.780703 (+   137us) service_pool.cc:227] Handling call
0809 23:21:45.815421 (+ 34718us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:46.772044 (+956623us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:47.240788 (+468744us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:47.241004 (+   216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2098193280}
W20260809 23:21:47.190294 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 108709us
W20260809 23:21:47.179232 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 232401us
I20260809 23:21:47.090330  3272 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:47.323158 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.336895 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:47.297585  3950 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.724s	user 0.012s	sys 0.002s
I20260809 23:21:47.326668 24423 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.601787 24423 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:47.206094  3755 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.984s	user 0.014s	sys 0.000s
W20260809 23:21:47.177547  3920 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.876s	user 0.009s	sys 0.003s
W20260809 23:21:47.188349  4052 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.000s	sys 0.021s
W20260809 23:21:47.461848 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59419us
W20260809 23:21:47.461203 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 124426us
I20260809 23:21:47.610570 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:47.610625 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:47.214474  4112 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260809 23:21:47.117903  3676 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.434576  4124 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.180976  4046 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:47.173887  3968 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:47.511772 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 111030us
W20260809 23:21:47.512359 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 343021us
I20260809 23:21:47.512636 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.123172  4157 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.208098  3964 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.178261  4048 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:47.260546  4159 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:47.223151  4156 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:47.569435 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56832.4us
I20260809 23:21:47.338542  3239 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:47.307508  4134 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:47.635829  4169 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260809 23:21:47.174288  2992 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.390s
I20260809 23:21:47.200932  3683 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:47.519706  3930 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:47.177610  3660 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.519285  3689 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:47.587157  4042 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.362s	user 0.020s	sys 0.059s
I20260809 23:21:47.326097  3870 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.338961  3633 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.489168  3815 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.095s
I20260809 23:21:47.398756 24542 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:47.428056 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:47.585381  4138 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.384s	user 0.001s	sys 0.000s
I20260809 23:21:47.663906  3239 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:47.177732  2820 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.630s	user 0.007s	sys 0.002s
I20260809 23:21:47.237877  3979 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.665436  4134 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.287976  4089 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.589653  4165 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:47.604096  3895 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:47.606336  4084 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:47.570067  3983 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260809 23:21:47.201470  4109 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.619398 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:47.575512  3789 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:21:47.570612  2995 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.036s
W20260809 23:21:47.607966  4033 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.132s	user 0.010s	sys 0.004s
W20260809 23:21:47.610204  4052 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.956s	user 0.003s	sys 0.036s
I20260809 23:21:47.615537  3920 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.397472  4152 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.659879 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:47.724104 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64341.6us
I20260809 23:21:47.678910  3815 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:47.647046  4119 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.019s	sys 0.071s
I20260809 23:21:47.626302  3676 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:47.618028  4164 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:47.626227  3950 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:47.637960  3563 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.652328  4157 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.680006 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:47.680716 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 252790us
I20260809 23:21:47.686990 24546 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:47.681155 25370 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
W20260809 23:21:47.675927  4042 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.451s	user 0.020s	sys 0.059s
I20260809 23:21:47.665443  3958 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:47.665196  4096 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:47.630229  4124 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.724915 24269 leader_election.cc:398] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
W20260809 23:21:47.815557 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90778.8us
I20260809 23:21:47.661669  4159 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.688225  4134 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.667619  4163 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:47.680405  3633 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:47.678324  3870 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.684216  3239 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.675333  3660 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:47.693183  3895 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:47.769526 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.698693  3885 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:21:47.707917  4140 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:47.673445  3930 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.723316  4165 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:47.675688  3689 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.682564  2820 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:47.727020  4174 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260809 23:21:47.810884 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:47.768865  4119 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.020s	sys 0.080s
I20260809 23:21:47.767825  3948 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:47.725250  4046 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.858445 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:47.817487 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:47.805195  3950 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.768008  4148 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.819103 24431 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:47.720963  4033 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.674223  4048 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.836109 24427 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:47.814366  4052 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.761097  3815 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:47.847714 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 232770us
I20260809 23:21:47.804949  3563 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.853984 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.834905  4010 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:47.174201  3878 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.863791 24321 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:47.855607 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:47.880424 25372 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:47.888556 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:47.915131 25363 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:47.932442 24423 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260809 23:21:47.918555 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.837251  4163 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.934252 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:47.934852 24427 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:47.935169  3878 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:47.935819  4163 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:47.919325 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:47.939498 25368 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 17: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:47.844528  4140 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.941758 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:47.943508 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55074us
I20260809 23:21:47.817991  4096 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.948125  4096 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.932456  3895 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.952039  3895 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:47.956171 24430 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:47.956516  4033 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.957304  4033 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:47.959355  4197 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:21:47.961014  4197 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:47.961469  4197 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:47.963003  4197 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.964833  4197 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:47.971356 24423 pending_rounds.cc:76] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:21:47.971763 24423 pending_rounds.cc:97] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260809 23:21:47.972150 24423 raft_consensus.cc:2866] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:47.916304  4152 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:47.972558 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:47.858119  3948 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:47.980476 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:47.978210 24427 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:47.986999 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:47.987625 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:47.987422 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:47.976459 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:47.989403 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:47.991063  3836 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260809 23:21:47.991197  4096 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:47.992426  4096 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:47.992611  3870 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:47.993467  4096 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:47.995839  3563 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:47.996258  3689 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:47.996869  3689 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:47.998239  3815 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:47.925201  4041 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:47.999066  4124 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.001237 24435 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:47.919057  3676 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.017164 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:47.938361  4010 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.018002 25373 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:48.020177 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:48.021082 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.024006 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:48.024282 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:48.025007 25372 raft_consensus.cc:2379] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:48.009837  3239 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:48.040249 25366 raft_consensus.cc:595] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:47.853322  4165 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.011919  4152 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:48.038863 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:48.041857 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.034686 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.035987 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:48.047070 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:48.032263  4195 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.035739 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.043452 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:48.043131 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 172832us
I20260809 23:21:48.047395 25368 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:48.037756 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:48.018025  3920 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.048547 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.059774 24311 raft_consensus.cc:2379] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:48.055343 24314 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:48.033806  3529 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:48.057924 24309 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:21:48.065073 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:48.068563 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.068872 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.034832  4124 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:48.060793 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.069154 24309 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:48.076516 24311 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:48.077006 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:48.035457  4182 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.067168 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:47.970002  4052 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:47.879739  4148 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.081298 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.066905 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.073689 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:48.066915 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:48.067384  4187 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.082613 24535 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:48.018458  3676 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.133255 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.134133 24314 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:48.212339 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.132776 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.045718  3930 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.138015 25372 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:48.084007  4163 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:48.081290  4164 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.042641  3239 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:48.178248 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 95498.6us
W20260809 23:21:48.181898 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108310us
I20260809 23:21:48.192874 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:48.193666 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.008s	sys 0.014s
I20260809 23:21:48.132861  4119 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.215166 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.064951  3633 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:48.150625  4044 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.611s	user 0.008s	sys 0.008s
I20260809 23:21:48.084007  3529 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:48.132787 24311 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:48.044540  3948 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.214151 24309 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:48.078894  4157 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.192131  3889 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.246099 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:48.271832 24427 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:48.243848 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.209168  4187 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.079241  4195 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.151170  4201 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.080992  3920 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.260385 24423 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:48.192999  4134 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.211079  3983 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:48.217670  4141 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:48.272805 24535 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:48.294210 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.294899 24535 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
I20260809 23:21:48.295892 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:48.274302 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.297336 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:48.298224 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:48.298508 24311 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:48.300263 24311 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
W20260809 23:21:48.259200  4202 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260809 23:21:48.278967 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.280469 24314 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:21:48.242619  2820 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:48.284636 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 216184us
I20260809 23:21:48.257539  4166 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:21:48.223470  4096 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.265676  3239 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:48.252187  3676 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.255571  4148 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.261519  4159 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.133796  4182 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.265407  4044 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:48.291932  3892 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.647s	user 0.003s	sys 0.013s
I20260809 23:21:48.286309  3948 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:48.248425  4163 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:48.302058  4185 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.418s	user 0.001s	sys 0.000s
I20260809 23:21:48.320636 24311 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:48.294508 24427 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:48.315518  4048 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:21:48.340818 24425 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1569) took 3167 ms (client timeout 1565 ms). Trace:
W20260809 23:21:48.372099 24425 rpcz_store.cc:269] 0809 23:21:45.173690 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.173810 (+   120us) service_pool.cc:227] Handling call
0809 23:21:48.335151 (+3161341us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:48.340693 (+  5542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2790021888}
I20260809 23:21:48.270088  3633 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.273923  4119 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.317260  4166 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:48.320963  4152 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:48.326143  3950 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.323993  4096 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.287463  3889 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.374373 24539 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:48.287669  4206 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
W20260809 23:21:48.377238 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 310378us
I20260809 23:21:48.293336  3920 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.416990 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1779) took 1321 ms. Trace:
I20260809 23:21:48.294726  4134 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:48.364719  4191 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.321s	user 0.000s	sys 0.001s
I20260809 23:21:48.373772  4225 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.262009  4164 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.411142  4141 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.339782  4218 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.288275  4201 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.414108 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3600) took 1334 ms. Trace:
I20260809 23:21:48.416298 24311 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:48.416649 24425 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260809 23:21:48.380191 24427 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:48.448920 24427 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:48.376870  4048 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:48.395598 24535 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:48.422830 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108126us
I20260809 23:21:48.427052  4230 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.340375  3529 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.440819 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:48.438383 24543 rpcz_store.cc:276] 0809 23:21:47.079777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.338396 (+258619us) service_pool.cc:227] Handling call
0809 23:21:47.519451 (+181055us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:47.519841 (+   390us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.520009 (+   168us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:47.576735 (+ 56726us) log.cc:842] Serialized 26 byte log entry
0809 23:21:47.859639 (+282904us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:47.859676 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:47.859742 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:48.413589 (+553847us) raft_consensus.cc:1671] finished
0809 23:21:48.413620 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:48.414050 (+   430us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10895,"raft.queue_time_us":41960,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":12171,"thread_start_us":248509,"threads_started":1,"wal-append.queue_time_us":451012}
I20260809 23:21:48.437613 24539 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:21:48.413933  4152 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:48.462751 24311 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:48.412062  3950 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260809 23:21:48.449963 24427 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.423095 24544 rpcz_store.cc:276] 0809 23:21:47.095647 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.579411 (+483764us) service_pool.cc:227] Handling call
0809 23:21:47.579762 (+   351us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:47.580011 (+   249us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:21:47.666712 (+ 86701us) log.cc:842] Serialized 14 byte log entry
0809 23:21:48.416668 (+749956us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:48.416718 (+    50us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:48.416765 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.416825 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:48.416940 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":231495,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":63453,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":8451,"thread_start_us":828119,"threads_started":3,"wal-append.queue_time_us":697083}
I20260809 23:21:48.427475  4052 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.475028 24427 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:48.470052 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:48.452946 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.475873 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:48.474339  4083 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.242s	user 0.001s	sys 0.010s
I20260809 23:21:48.452157  4048 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:48.475000  4197 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.510s	user 0.013s	sys 0.000s
I20260809 23:21:48.485721 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:48.488387 24544 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260809 23:21:48.508073 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.508496 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3691) took 1183 ms. Trace:
I20260809 23:21:48.459520  4010 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.485740 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1774) took 1444 ms. Trace:
I20260809 23:21:48.509328 24544 rpcz_store.cc:276] 0809 23:21:47.325531 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.487683 (+1162152us) service_pool.cc:227] Handling call
0809 23:21:48.488095 (+   412us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:48.508136 (+ 20041us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.508426 (+   290us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:48.492101 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 282553us
I20260809 23:21:48.508046 24425 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:48.476183  4152 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:48.439500  4141 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:48.512763 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61411.9us
I20260809 23:21:48.481467  4206 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.459959  4165 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.510599 24534 rpcz_store.cc:276] 0809 23:21:47.041442 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.103828 (+ 62386us) service_pool.cc:227] Handling call
0809 23:21:47.104243 (+   415us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:47.397774 (+293531us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.397818 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:47.470127 (+ 72309us) log.cc:842] Serialized 52 byte log entry
0809 23:21:47.704106 (+233979us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:47.704144 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:47.704214 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:48.485503 (+781289us) raft_consensus.cc:1671] finished
0809 23:21:48.485528 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:48.485682 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":217988,"raft.queue_time_us":220197,"spinlock_wait_cycles":84527488,"thread_start_us":116506,"threads_started":1,"wal-append.queue_time_us":261845}
I20260809 23:21:48.529260 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:48.522675 24427 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:21:48.481299  3983 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.482764  4258 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:48.512341  4211 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
I20260809 23:21:48.531164 25369 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:48.507381  3563 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:48.550743 24543 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:48.551179 24543 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:48.551311 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3603) took 1124 ms. Trace:
I20260809 23:21:48.525130  4195 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.436677  4225 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.523969  4246 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:48.514670  4187 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:48.583695 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54541.9us
I20260809 23:21:48.551960 24543 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.442531  4218 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.475152  4230 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.452116  4166 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.532577  4206 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.773684 24543 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:48.555408 24544 rpcz_store.cc:276] 0809 23:21:47.426853 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.512236 (+1085383us) service_pool.cc:227] Handling call
0809 23:21:48.550677 (+ 38441us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:48.550891 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:48.550947 (+    56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:48.551006 (+    59us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:48.551045 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.551110 (+    65us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:48.551248 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3712}
I20260809 23:21:48.776984  4099 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:48.693977 24316 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:48.616417 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.552937  3563 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:48.616003 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:48.782119 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 353006us
W20260809 23:21:48.589156  4219 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260809 23:21:48.811446 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.812183 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:48.654336  4213 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
W20260809 23:21:48.588579  4241 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
W20260809 23:21:48.619561  4254 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260809 23:21:48.805902 24311 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:48.781821  4246 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.621824  4251 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:48.655252  4201 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:48.598387  3895 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.646s	user 0.014s	sys 0.000s
I20260809 23:21:48.795954 24535 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:21:48.652386  4221 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260809 23:21:48.581332  3889 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.666000  4134 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.648197  3843 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.837154 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.804973  3563 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:48.584518  4044 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.647826  4182 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.623888  3253 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260809 23:21:48.839578 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 401017us
I20260809 23:21:48.653923  4096 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.589561  4232 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:48.654138  4163 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.651548  4249 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:48.588131  4234 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:48.660367  3633 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.648552  4119 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.665066  4141 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.878463  3920 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.881295  3253 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:48.857354 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:48.870870 24543 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:48.548478  3983 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:48.877009 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.004s	sys 0.009s
W20260809 23:21:48.918453 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.004s	sys 0.009s
I20260809 23:21:48.917738 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3604) took 1490 ms. Trace:
W20260809 23:21:48.879940 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 409941us
I20260809 23:21:48.919230 24543 rpcz_store.cc:276] 0809 23:21:47.427244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.514131 (+1086887us) service_pool.cc:227] Handling call
0809 23:21:48.550613 (+ 36482us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:48.917362 (+366749us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.917663 (+   301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:48.919945 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:48.880957 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:48.619695  4273 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.255s
I20260809 23:21:48.805034  4099 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.877233  3895 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.553254  4258 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.928332  4099 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.928926  3889 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:48.929313  3889 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:48.929651  3889 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:48.932593  3983 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:48.933015  4033 consensus_queue.cc:1159] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 17 status { last_received { term: 16 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:21:48.849992 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.935837  3895 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.937839  3895 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.899770  4234 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.905941  4206 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.881079  4267 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:48.908691  4230 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:48.908689  4212 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.005s	sys 0.007s
I20260809 23:21:48.797606  3836 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:48.909691  3239 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.576s	user 0.007s	sys 0.004s
I20260809 23:21:48.964046  3239 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:48.966495  3948 consensus_meta.cc:287] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.595s	user 0.005s	sys 0.005s
I20260809 23:21:48.954232  4267 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:48.969643  4198 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.776s	user 0.001s	sys 0.001s
I20260809 23:21:48.899959  4279 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:48.912405  4225 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.890892  4232 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:48.970887  4279 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.972143  4232 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.973151  4279 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.973337  4232 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:48.955065  4212 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.005s	sys 0.007s
I20260809 23:21:48.974645 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1770) took 2377 ms. Trace:
I20260809 23:21:48.893052  4274 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:48.975327  4274 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:48.976020 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3689) took 1667 ms (client timeout 1972 ms). Trace:
I20260809 23:21:48.975632  4152 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:48.976397 24546 rpcz_store.cc:269] 0809 23:21:47.308514 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.309595 (+  1081us) service_pool.cc:227] Handling call
0809 23:21:48.975849 (+1666254us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:48.975963 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":848514816}
I20260809 23:21:48.977591 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.977967  4274 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.898057  4249 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.980104  4249 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.909765  4218 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.915441  4246 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.916925  3253 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:48.920245 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:48.982743 24543 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:48.937849 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:48.987195  3932 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:48.987614  3253 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:48.988725  3968 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:48.989135  3968 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:48.989574  3253 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:48.989871  3968 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.990660  3968 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:48.963321  4258 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.972548  4234 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.956403  3836 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.000830  3836 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.991080  3239 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.970588  4267 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:48.978292  4152 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:49.003032  4214 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.677s	user 0.002s	sys 0.000s
I20260809 23:21:49.004673 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3690) took 1695 ms. Trace:
I20260809 23:21:49.005017 24542 rpcz_store.cc:276] 0809 23:21:47.308811 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.806511 (+497700us) service_pool.cc:227] Handling call
0809 23:21:47.806849 (+   338us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:47.807324 (+   475us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.807360 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:47.815018 (+  7658us) log.cc:842] Serialized 26 byte log entry
0809 23:21:48.074387 (+259369us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:48.074443 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.074537 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.002416 (+927879us) spinlock_profiling.cc:243] Waited 6.72 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:49.004341 (+  1925us) raft_consensus.cc:1671] finished
0809 23:21:49.004365 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.004589 (+   224us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":557,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":3839,"spinlock_wait_cycles":15446784,"thread_start_us":226359,"threads_started":1,"wal-append.queue_time_us":443136}
I20260809 23:21:49.002358  3889 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.008049  4166 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:21:49.013166  4161 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.419s	user 0.002s	sys 0.000s
W20260809 23:21:48.993767  4220 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.571s	user 0.001s	sys 0.000s
I20260809 23:21:49.014241 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1756) took 1173 ms. Trace:
I20260809 23:21:48.993963  3563 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.016412  3563 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:48.992667 24543 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:48.975047 24541 rpcz_store.cc:276] 0809 23:21:46.597328 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.665928 (+ 68600us) service_pool.cc:227] Handling call
0809 23:21:46.683904 (+ 17976us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:46.821804 (+137900us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:48.974404 (+2152600us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:48.974580 (+   176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315312512}
I20260809 23:21:49.017874 24541 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:49.018590 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.019013  2995 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.019515  2995 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.987618  3932 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:49.019529 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1799) took 3019 ms. Trace:
I20260809 23:21:49.022707 25370 rpcz_store.cc:276] 0809 23:21:46.000436 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.000922 (+   486us) service_pool.cc:227] Handling call
0809 23:21:46.022664 (+ 21742us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:47.799609 (+1776945us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.019251 (+1219642us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.019462 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3813581184}
I20260809 23:21:49.023247  3892 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:49.023738  3892 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.021729  4042 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.014658 25371 rpcz_store.cc:276] 0809 23:21:47.840719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.841440 (+   721us) service_pool.cc:227] Handling call
0809 23:21:47.856076 (+ 14636us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:47.858332 (+  2256us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.858360 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:47.868326 (+  9966us) log.cc:842] Serialized 52 byte log entry
0809 23:21:48.083152 (+214826us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:21:48.140935 (+ 57783us) log.cc:842] Serialized 14 byte log entry
0809 23:21:48.179378 (+ 38443us) log.cc:842] Serialized 14 byte log entry
0809 23:21:49.013614 (+834236us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b1800438d00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:49.013826 (+   212us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.013911 (+    85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.014039 (+   128us) raft_consensus.cc:1671] finished
0809 23:21:49.014058 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.014185 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110540,"prepare.queue_time_us":128946,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":12817,"raft.queue_time_us":892210,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":15491,"spinlock_wait_cycles":48742016,"thread_start_us":214451,"threads_started":1,"wal-append.queue_time_us":634842}
I20260809 23:21:49.024917  4042 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.025622  3892 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:49.026062  4197 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.026532  4197 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:48.961383  4048 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.031795 25375 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:49.035975 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:49.036782 25375 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:49.037628  4177 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.122s	user 0.000s	sys 0.001s
W20260809 23:21:49.040380  4176 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.280s	user 0.002s	sys 0.000s
I20260809 23:21:49.044018 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:49.020861  3947 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:49.047354 25375 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:49.036262  4197 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:49.050971 25363 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:49.027652  4042 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:49.036345  4204 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.970s	user 0.001s	sys 0.001s
I20260809 23:21:49.042738  4010 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:49.057003 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.040395  2995 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:49.063875  3889 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.065701  4048 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:49.069105  4208 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.745s	user 0.000s	sys 0.001s
I20260809 23:21:49.060094  3932 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.072206  3932 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:49.073314  4179 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.105s	user 0.001s	sys 0.000s
I20260809 23:21:49.078229  4164 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:49.097003  4010 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:49.097848  4140 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.104216  4147 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:49.104743  4147 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:49.105782  4147 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.174995 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:49.183722 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1864) took 2412 ms. Trace:
I20260809 23:21:49.184361 24313 rpcz_store.cc:276] 0809 23:21:46.770829 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.771204 (+   375us) service_pool.cc:227] Handling call
0809 23:21:46.771565 (+   361us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:46.923843 (+152278us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.182968 (+2259125us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b54003868d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.183457 (+   489us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92627328}
I20260809 23:21:49.187431 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:49.189250 24313 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:49.191974 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1866) took 1578 ms. Trace:
I20260809 23:21:49.192467 24312 rpcz_store.cc:276] 0809 23:21:47.613392 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.614035 (+   643us) service_pool.cc:227] Handling call
0809 23:21:47.631989 (+ 17954us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:47.632274 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:47.632302 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:47.638374 (+  6072us) log.cc:842] Serialized 26 byte log entry
0809 23:21:47.921256 (+282882us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:47.921371 (+   115us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:47.921861 (+   490us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:48.969099 (+1047238us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:49.191218 (+222119us) raft_consensus.cc:1671] finished
0809 23:21:49.191267 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.191725 (+   458us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8203136,"thread_start_us":253769,"threads_started":1,"wal-append.queue_time_us":410887}
I20260809 23:21:48.879185 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:48.908149 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:48.918771 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 574614us
I20260809 23:21:49.198124 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:48.940688 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 141545us
I20260809 23:21:48.921717 24428 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:48.942277 24425 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:49.203888 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.205302 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.021142 24542 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:49.207871 24546 raft_consensus.cc:2396] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260809 23:21:49.208714 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3694) took 1437 ms. Trace:
I20260809 23:21:49.209165 24542 rpcz_store.cc:276] 0809 23:21:47.771122 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.006568 (+1235446us) service_pool.cc:227] Handling call
0809 23:21:49.006872 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.208035 (+201163us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.208450 (+   415us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:48.924566 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:49.212325 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3596) took 2285 ms. Trace:
W20260809 23:21:49.213500 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 288862us
I20260809 23:21:49.216456 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.202667 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.217332 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:49.210717 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:49.234611 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3694) took 2643 ms (client timeout 2002 ms). Trace:
W20260809 23:21:49.235316 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3695) took 1913 ms (client timeout 1712 ms). Trace:
I20260809 23:21:49.235702  4137 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:49.235836 24431 rpcz_store.cc:269] 0809 23:21:47.322184 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.322787 (+   603us) service_pool.cc:227] Handling call
0809 23:21:49.234981 (+1912194us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.235125 (+   144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":823747840}
I20260809 23:21:49.237439  4140 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:49.243525  4235 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:49.244160  4145 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:49.244401  4262 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:49.245398 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.246439 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.246731  4116 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:49.247274  4089 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:49.247352  3889 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:49.235152 24429 rpcz_store.cc:269] 0809 23:21:46.591050 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.592152 (+  1102us) service_pool.cc:227] Handling call
0809 23:21:49.234101 (+2641949us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.234355 (+   254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1311035648}
I20260809 23:21:49.237453 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.259545 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1782) took 1269 ms. Trace:
I20260809 23:21:49.260305 24535 rpcz_store.cc:276] 0809 23:21:47.989449 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.295653 (+306204us) service_pool.cc:227] Handling call
0809 23:21:49.258997 (+963344us) spinlock_profiling.cc:243] Waited 99.1 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.259187 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":227842816}
I20260809 23:21:49.261862 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.262120 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:49.263561  4145 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.250458  4235 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:49.267896  4235 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:49.268144 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3701) took 1492 ms. Trace:
I20260809 23:21:49.268467 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:49.269146 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.271257 24544 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:49.272210 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1811) took 1240 ms. Trace:
I20260809 23:21:49.272832 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3693) took 1502 ms. Trace:
I20260809 23:21:49.272789 25372 rpcz_store.cc:276] 0809 23:21:48.031499 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.031642 (+   143us) service_pool.cc:227] Handling call
0809 23:21:48.031973 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:48.223703 (+191730us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.271559 (+1047856us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.271979 (+   420us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140124544}
I20260809 23:21:49.274178  4251 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.276948 24535 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260809 23:21:49.236919  4137 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.270501 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.256824 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.270414 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:49.273381 24544 rpcz_store.cc:276] 0809 23:21:47.770156 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.986843 (+1216687us) service_pool.cc:227] Handling call
0809 23:21:49.270695 (+283852us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.271676 (+   981us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.272625 (+   949us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b540029d450. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.272748 (+   123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":651753216}
I20260809 23:21:49.268517 24435 rpcz_store.cc:276] 0809 23:21:47.775410 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.775532 (+   122us) service_pool.cc:227] Handling call
0809 23:21:47.820146 (+ 44614us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:48.010893 (+190747us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.267643 (+1256750us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400062554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.267941 (+   298us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415897344}
I20260809 23:21:49.288000 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:49.288910 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 409683us
I20260809 23:21:49.289615 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:49.290596 24424 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:21:49.291338 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:49.291426  4187 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:49.292501 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:49.292382  4187 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.293817 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.295297 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3017) took 3751 ms. Trace:
I20260809 23:21:49.296651  4217 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.297700 24496 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:49.296288  4166 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:49.275110  4251 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.294607 24431 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:21:49.304488  4140 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:49.304999  4140 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:49.305490  4140 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:49.307310  4297 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:49.308039  4207 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:21:49.309600 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 27) took 2672 ms. Trace:
I20260809 23:21:49.212790 24545 rpcz_store.cc:276] 0809 23:21:46.926628 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.926956 (+   328us) service_pool.cc:227] Handling call
0809 23:21:46.956956 (+ 30000us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:47.310503 (+353547us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.310542 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:47.312380 (+  1838us) log.cc:842] Serialized 26 byte log entry
0809 23:21:47.762533 (+450153us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:47.762577 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:47.762671 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.041823 (+1279152us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:49.211576 (+169753us) raft_consensus.cc:1671] finished
0809 23:21:49.211622 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.212061 (+   439us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":353120,"raft.queue_time_us":354061,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"spinlock_wait_cycles":81847680,"thread_start_us":449808,"threads_started":1,"wal-append.queue_time_us":550393}
I20260809 23:21:49.309979 25366 rpcz_store.cc:276] 0809 23:21:46.637433 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.637655 (+   222us) service_pool.cc:227] Handling call
0809 23:21:49.309246 (+2671591us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400403b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.309387 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3224006912}
I20260809 23:21:49.297762  4217 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.311439 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:49.312033  4251 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.294752 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3696) took 1424 ms. Trace:
I20260809 23:21:49.312577 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:49.306566 24269 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:49.295801 24317 rpcz_store.cc:276] 0809 23:21:45.543572 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.059872 (+516300us) service_pool.cc:227] Handling call
0809 23:21:46.060249 (+   377us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:46.766974 (+706725us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.232242 (+2465268us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b540019d7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.295059 (+ 62817us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1553965312}
I20260809 23:21:49.315529  4089 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.316325  4147 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:49.317204 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.318214 24429 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:49.312853 24546 rpcz_store.cc:276] 0809 23:21:47.870422 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.293463 (+1423041us) service_pool.cc:227] Handling call
0809 23:21:49.293990 (+   527us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.294354 (+   364us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.294378 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.294437 (+    59us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:49.294519 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.294574 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.294680 (+   106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:49.318552  4089 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.319684 24428 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:49.320175  3889 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.318125 25370 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:49.311962 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1868) took 1693 ms (client timeout 1515 ms). Trace:
W20260809 23:21:49.328091 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44887.9us
I20260809 23:21:49.331723  3788 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.002s
I20260809 23:21:49.300428  4116 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:49.332721  3788 consensus_queue.cc:1225] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260809 23:21:49.327710 24321 rpcz_store.cc:269] 0809 23:21:47.618741 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.618883 (+   142us) service_pool.cc:227] Handling call
0809 23:21:49.311535 (+1692652us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54000dc0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.311733 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":371190656}
I20260809 23:21:49.335479 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:49.335805 24311 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:21:49.336341 24321 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:49.337759 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.338389  4039 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:49.334131  4116 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.332257 24543 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260809 23:21:49.330914 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.345140 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1758) took 1448 ms. Trace:
I20260809 23:21:49.345679 25365 rpcz_store.cc:276] 0809 23:21:47.853242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.853390 (+   148us) service_pool.cc:227] Handling call
0809 23:21:47.853725 (+   335us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:47.854086 (+   361us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:47.854118 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:47.862304 (+  8186us) log.cc:842] Serialized 26 byte log entry
0809 23:21:48.626030 (+763726us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b1800490420. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:48.893552 (+267522us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:49.218561 (+325009us) log.cc:842] Serialized 14 byte log entry
0809 23:21:49.229524 (+ 10963us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.249069 (+ 19545us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.252415 (+  3346us) raft_consensus.cc:1671] finished
0809 23:21:49.257320 (+  4905us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.301571 (+ 44251us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":170016,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":718,"raft.run_cpu_time_us":767,"raft.run_wall_time_us":90386,"spinlock_wait_cycles":3885952,"thread_start_us":413902,"threads_started":2,"wal-append.queue_time_us":1044898}
I20260809 23:21:49.347415 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3685) took 4131 ms. Trace:
I20260809 23:21:49.347808  3889 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.348954  4157 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:49.349790 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3703) took 1371 ms. Trace:
I20260809 23:21:49.350263 24423 rpcz_store.cc:276] 0809 23:21:47.978085 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.980153 (+  2068us) service_pool.cc:227] Handling call
0809 23:21:49.349187 (+1369034us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.349574 (+   387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":629249280}
I20260809 23:21:49.351456 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.351884 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.352907 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:49.354310 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3674) took 3043 ms (client timeout 3159 ms). Trace:
I20260809 23:21:49.349195  3154 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260809 23:21:49.354882 24539 rpcz_store.cc:269] 0809 23:21:46.310799 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.483742 (+172943us) service_pool.cc:227] Handling call
0809 23:21:49.353842 (+2870100us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.354070 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4324491008}
I20260809 23:21:49.355410 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:49.355604  3154 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:49.356050  3154 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:49.356359 24312 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:49.357002  4157 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:49.357447  4157 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:49.357605 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.358669 24423 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260809 23:21:49.360882  4137 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.361688 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:49.347919 24422 rpcz_store.cc:276] 0809 23:21:45.216133 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:45.216528 (+   395us) service_pool.cc:227] Handling call
0809 23:21:45.256394 (+ 39866us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:47.025167 (+1768773us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.346856 (+2321689us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b540018c254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.347203 (+   347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3934024576}
I20260809 23:21:49.363508 24428 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:49.363945 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "974355bb93274291bb6be768b656cc67"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:49.363797  3154 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.364789 24428 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: []
I20260809 23:21:49.364923 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.365806  3154 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:49.356274 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.367465 24539 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:49.367915 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1783) took 1376 ms. Trace:
I20260809 23:21:49.368294 24546 rpcz_store.cc:276] 0809 23:21:47.991822 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.319048 (+1327226us) service_pool.cc:227] Handling call
0809 23:21:49.367127 (+ 48079us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.367519 (+   392us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.367543 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.367596 (+    53us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:49.367662 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.367723 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.367835 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:49.369033  4304 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:49.369666 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.355010 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:49.369974  4304 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.340880 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.373387  4217 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.374215 24542 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:49.358681  4157 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.364799 25365 raft_consensus.cc:595] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:49.379047 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:49.379488 25365 raft_consensus.cc:603] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:49.365842 24544 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:21:49.383448 25366 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:49.378211 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3695) took 1606 ms. Trace:
I20260809 23:21:49.384349 24544 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:49.380318  4157 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:49.389317 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.392058  4217 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:21:49.399814 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.2068e+06us
I20260809 23:21:49.402856  4140 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.405369  4140 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:49.405651  3979 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 16 FOLLOWER]: Advancing to term 17
I20260809 23:21:49.408663  3789 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.409413  3789 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.393555 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3641) took 1330 ms. Trace:
I20260809 23:21:49.410768 24314 rpcz_store.cc:276] 0809 23:21:48.063467 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.322505 (+259038us) service_pool.cc:227] Handling call
0809 23:21:49.315327 (+992822us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.315906 (+   579us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.315943 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.316019 (+    76us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:49.316085 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.316178 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.393219 (+ 77041us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54000dc0d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.393348 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2277048576}
I20260809 23:21:49.389143  3788 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:49.413187  3789 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:49.385695 24543 rpcz_store.cc:276] 0809 23:21:47.771821 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.017247 (+1245426us) service_pool.cc:227] Handling call
0809 23:21:49.330222 (+312975us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.339707 (+  9485us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.377978 (+ 38271us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400147b50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.378142 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647591296}
I20260809 23:21:49.397122 24535 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260809 23:21:49.339804  4251 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.420058 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:49.414682 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 101276us
I20260809 23:21:49.373425  4304 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:49.386096  4145 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.385378  4089 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.416601  4187 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.444569 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.445457 25370 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:49.449928 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.451380 24543 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:49.452487 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.455529 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:49.456295 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:49.457309 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3700) took 1287 ms. Trace:
I20260809 23:21:49.459642 24539 rpcz_store.cc:276] 0809 23:21:48.166282 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.384852 (+1218570us) service_pool.cc:227] Handling call
0809 23:21:49.389159 (+  4307us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.452976 (+ 63817us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.453017 (+    41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.453079 (+    62us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:49.453123 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.453202 (+    79us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.453341 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:49.458811 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:49.461535 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 155112us
W20260809 23:21:49.463923 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 85204us
I20260809 23:21:49.463371 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.468206 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.467228 24428 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260809 23:21:49.472635  4276 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260809 23:21:49.477267 25363 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 17.
I20260809 23:21:49.477156  4089 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.480136  4089 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:49.481542 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.486536 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.488382 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.486765 24544 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:49.490438 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:49.457433  4251 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.494786  4249 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260809 23:21:49.501703 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 110100us
I20260809 23:21:49.502524 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:49.500799 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 397277us
I20260809 23:21:49.503372 24542 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:49.499637 24317 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:49.504334 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.506110 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.511051 24539 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:49.500020  4249 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:49.526585 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:49.528551 24543 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:49.527549 24544 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:49.544685 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.551184 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3605) took 1723 ms. Trace:
I20260809 23:21:49.552876 25370 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:49.547516 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:49.545290 24542 raft_consensus.cc:2362] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27. Re-sending same reply.
I20260809 23:21:49.555266 24539 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260809 23:21:49.562310 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 78790.2us
I20260809 23:21:49.562501  4271 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:49.563118  4271 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:49.563583  4271 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:49.564092  3950 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:49.564744  3950 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:49.565963  3950 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.551687 24541 rpcz_store.cc:276] 0809 23:21:47.776208 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.017498 (+1241290us) service_pool.cc:227] Handling call
0809 23:21:49.017758 (+   260us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.491299 (+473541us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.491363 (+    64us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.491453 (+    90us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:49.491498 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.491583 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.499701 (+  8118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16469,"raft.queue_time_us":24067,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":9871}
I20260809 23:21:49.525389 24424 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:49.587133 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:49.529940  4249 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:49.592131 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:49.566660  4310 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.001s	sys 0.001s
I20260809 23:21:49.597432 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1769) took 2969 ms. Trace:
I20260809 23:21:49.651535 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3270) took 1783 ms. Trace:
I20260809 23:21:49.605396 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:49.670981  4278 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.671650  4278 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:49.673125 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 129016us
I20260809 23:21:49.637018  4089 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:49.683416 24544 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 21
I20260809 23:21:49.678447 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1781) took 1689 ms. Trace:
W20260809 23:21:49.691257 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111979us
W20260809 23:21:49.692288 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 188258us
I20260809 23:21:49.690910 24545 rpcz_store.cc:276] 0809 23:21:47.989084 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.310843 (+1321759us) service_pool.cc:227] Handling call
0809 23:21:49.587055 (+276212us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:21:49.598803 (+ 11748us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.663106 (+ 64303us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.663138 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.678162 (+ 15024us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:21:49.678212 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.678267 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.678384 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4561152}
I20260809 23:21:49.691102 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:49.704311 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:49.705018 25365 raft_consensus.cc:595] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:49.718160 24543 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:49.719736 24543 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:49.720746 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46838.1us
I20260809 23:21:49.732717 25365 raft_consensus.cc:603] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:49.733798 24541 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:21:49.652084 24430 rpcz_store.cc:276] 0809 23:21:47.831249 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:47.958101 (+126852us) service_pool.cc:227] Handling call
0809 23:21:47.958527 (+   426us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:48.210706 (+252179us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:48.210752 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:48.211358 (+   606us) log.cc:842] Serialized 26 byte log entry
0809 23:21:48.332377 (+121019us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:48.332430 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.586656 (+1254226us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540023da54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.598391 (+ 11735us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.614304 (+ 15913us) raft_consensus.cc:1671] finished
0809 23:21:49.614334 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.614674 (+   340us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17355,"raft.queue_time_us":56839,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4240,"spinlock_wait_cycles":539042176,"thread_start_us":48794,"threads_started":1,"wal-append.queue_time_us":130564}
W20260809 23:21:49.763231 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 239530us
I20260809 23:21:49.640491 24540 rpcz_store.cc:276] 0809 23:21:46.489998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.646642 (+156644us) service_pool.cc:227] Handling call
0809 23:21:46.657233 (+ 10591us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:46.734523 (+ 77290us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:46.734575 (+    52us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:46.735406 (+   831us) log.cc:842] Serialized 52 byte log entry
0809 23:21:46.747634 (+ 12228us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:46.818747 (+ 71113us) log.cc:842] Serialized 14 byte log entry
0809 23:21:47.305523 (+486776us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b180023e9c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac27708bcf 00007fac209387b6 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d
0809 23:21:49.291861 (+1986338us) spinlock_profiling.cc:243] Waited 1.36 ms on lock 0x7b180023e9c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:49.427519 (+135658us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.434420 (+  6901us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:49.447570 (+ 13150us) raft_consensus.cc:1671] finished
0809 23:21:49.447589 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.459113 (+ 11524us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":870,"prepare.queue_time_us":64048,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":2765,"raft.queue_time_us":624891,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":13942,"spinlock_wait_cycles":102201344,"thread_start_us":121062,"threads_started":3,"wal-append.queue_time_us":347454,"wal-append.run_cpu_time_us":3744,"wal-append.run_wall_time_us":1594670}
I20260809 23:21:49.752197  4262 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260809 23:21:49.759126  4145 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:49.799211  4311 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260809 23:21:49.801366 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:49.739549  3763 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.803252  4262 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:49.784870  4276 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:49.785961  4052 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.788374  4109 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.789651  3958 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:49.789829  3932 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.009s
I20260809 23:21:49.751696  3789 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.791657 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:49.803472  3763 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.764751  4251 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.805012 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:49.860090 24542 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:49.772781 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:49.816069 24543 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260809 23:21:49.823827  4052 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:49.825316 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 77163.6us
I20260809 23:21:49.824483  4276 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.831550  4109 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.799238  4145 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:49.878017 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86477.6us
W20260809 23:21:49.890707 24313 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.700s	user 0.012s	sys 0.005s
I20260809 23:21:49.891197 24313 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:49.931624  4297 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.931524  3878 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:49.932824  3878 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.932699  4297 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.935393  3878 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.935504  4297 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.940555  4276 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.942503  4276 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:49.944384  4145 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:49.893162  4251 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:49.946475  4251 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:49.946873  4251 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:49.948264 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:49.949137 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61899.3us
I20260809 23:21:49.949399  4297 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:21:49.950456  4166 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:49.951267  4166 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.950201  4297 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:49.952708  4297 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:21:49.957525  4278 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:49.958315 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3029) took 1922 ms (client timeout 1667 ms). Trace:
W20260809 23:21:49.958734 24316 rpcz_store.cc:269] 0809 23:21:48.036266 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.036422 (+   156us) service_pool.cc:227] Handling call
0809 23:21:49.909548 (+1873126us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54000dcfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:49.958140 (+ 48592us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":744519936}
I20260809 23:21:49.962499 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:49.965546  4052 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:49.968410  4235 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.700s	user 0.008s	sys 0.006s
I20260809 23:21:49.968894  4235 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:49.970191 24422 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:49.971525 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3692) took 2202 ms. Trace:
I20260809 23:21:49.971922 24534 rpcz_store.cc:276] 0809 23:21:47.769151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.527514 (+758363us) service_pool.cc:227] Handling call
0809 23:21:48.527873 (+   359us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:49.215909 (+688036us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.215980 (+    71us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:49.216062 (+    82us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:49.216140 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.971088 (+754948us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:49.971139 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:49.971327 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1580583168}
I20260809 23:21:49.869056  3889 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:49.976900  3889 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.990561 24542 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:49.931936 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:49.953711  4166 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:49.838052  3789 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.029925  3932 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:50.030659  3932 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:49.892993  3958 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.032240  4109 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.032779  3932 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.037304  3789 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.051643 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.055404 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.040323 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:50.064277 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46074.9us
I20260809 23:21:49.982595  4271 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.069128 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.040395 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.034737  3958 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:50.014890  4276 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:50.043255  4147 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.726s	user 0.006s	sys 0.013s
I20260809 23:21:50.074461  4308 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:50.075309  3763 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:50.076053 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 82563.6us
I20260809 23:21:50.076221 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.083518 24317 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:50.070657 24312 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:50.084851 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.087203  4145 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:50.088297  4145 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:50.088814  4145 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:50.081413 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:50.074844 24161 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 13 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 22 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:50.082096 25366 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:50.057984 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.082589  3889 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.088724  4166 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:50.081643 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:50.052096  4304 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:50.086588 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.113402 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3606) took 1562 ms. Trace:
I20260809 23:21:50.053071  4235 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.131459 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.135495 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.135563 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.136287 25371 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:50.140966  4271 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:50.141635  4157 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.750s	user 0.011s	sys 0.006s
I20260809 23:21:50.113898 24546 rpcz_store.cc:276] 0809 23:21:48.533025 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.369299 (+836274us) service_pool.cc:227] Handling call
0809 23:21:49.369895 (+   596us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:49.391165 (+ 21270us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.391211 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:49.447465 (+ 56254us) log.cc:842] Serialized 26 byte log entry
0809 23:21:49.728713 (+281248us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:49.728766 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.728873 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:50.094966 (+366093us) raft_consensus.cc:1671] finished
0809 23:21:50.094987 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:50.095117 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12066,"raft.queue_time_us":16428,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":43235,"thread_start_us":247314,"threads_started":1,"wal-append.queue_time_us":450652}
W20260809 23:21:50.143455 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 623502us
W20260809 23:21:50.143724  3154 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.777s	user 0.017s	sys 0.005s
I20260809 23:21:50.144404 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:50.143759  4276 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.147550  4137 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:50.147970  4137 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:21:50.148902 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 67423.7us
I20260809 23:21:50.150915 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:50.151670 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.667s (SENT)
I20260809 23:21:50.152418 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.152560 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:50.153434 24423 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:50.154220  4262 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.155117 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.155896 24427 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
W20260809 23:21:50.155948 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 224224us
I20260809 23:21:50.156782  3950 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:50.157608  3950 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:50.118836  4140 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:50.160509  4140 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:50.161708 24542 pending_rounds.cc:76] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
I20260809 23:21:50.162241 24542 pending_rounds.cc:97] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260809 23:21:50.161660  4140 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.162535 24542 raft_consensus.cc:2866] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:50.165673 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.167294 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.146564 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:50.154848  4262 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.166492 25375 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:50.150966 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.158595 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:50.181309 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.182700 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.183589 24543 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:50.168287 25365 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:50.202059 24545 raft_consensus.cc:2396] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260809 23:21:50.203820 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45574.6us
W20260809 23:21:50.210471 24544 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.512s	user 0.011s	sys 0.003s
I20260809 23:21:50.210880 24544 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:50.127147  4304 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:50.231763 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:50.229518 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.237742 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.239795 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.240752 24545 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:50.243023 24541 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 16. Current term is 17. Ops: []
I20260809 23:21:50.239840 24543 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:50.245435 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.174973 24161 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 24, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:21:50.251542 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46976.7us
I20260809 23:21:50.205261  3154 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:50.224974  4089 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.253407 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.253499 24540 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
W20260809 23:21:50.253899 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 107461us
I20260809 23:21:50.254156  3958 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:50.257913 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.258539 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:50.260190  4304 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.263140 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:50.261600 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:50.262312  3920 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:50.241685  4147 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:50.228832  3878 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:50.281019 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:50.284121 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:50.254473  4309 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.094s	user 0.000s	sys 0.007s
I20260809 23:21:50.291023 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:50.291337  3920 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:50.291729  3920 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.292105  3920 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:50.295704 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:50.296634  4052 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:21:50.297636  3791 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:50.298273  4052 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:21:50.299664  4109 raft_consensus.cc:2741] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260809 23:21:50.301951  4109 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260809 23:21:50.303225  3772 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:50.305510  4052 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:50.308877  4052 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:50.308435  4145 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 20.19, Last appended by leader: 19, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.310009  4052 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.311518  4052 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:50.311714 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 59467.9us
I20260809 23:21:50.279112 24316 raft_consensus.cc:595] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:50.305927 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.307065 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:50.264293  4089 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:50.284035 25365 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:50.262878  3958 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:50.327426  3958 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:50.283707  4147 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.229465  4308 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.329818 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.288335  4140 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:50.333814 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:50.285635  4271 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.325006  3154 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:50.333123 24161 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.329824  4235 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:50.331579  3763 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:50.333051  4147 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:50.338124  3791 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:50.335361  3789 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:50.360663 25375 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:50.356158  3793 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:50.363916  4271 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.356563  4137 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.366292  4271 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.359136  3791 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.367460 24540 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260809 23:21:50.367748  3791 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:50.371357 25365 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:21:50.373694 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.376443  4109 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:50.376922  4109 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:50.377446  4109 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:50.377029  3791 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.379148  4157 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:50.380393  3920 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.363865  3793 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.358052  3763 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:50.392951 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 664880us
I20260809 23:21:50.381145  4157 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:50.335213  3878 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.405570 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.401486 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:50.402112 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.404847  3793 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.406675 25365 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:50.409597  3789 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.409989  3878 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.405980 24161 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:50.401746  3763 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:50.401305  4157 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:50.410051  3789 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:50.417387 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3282) took 1559 ms. Trace:
W20260809 23:21:50.428260 24167 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 343) took 73487 ms (client timeout 15000 ms). Trace:
I20260809 23:21:50.429630 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.431082 24540 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:50.435624  4157 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.437273  4157 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:50.440649  4308 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.443147 24161 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 18. New cstate: current_term: 18 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.444945 24161 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 19, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:50.413373  4262 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:50.447954  3979 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.042s	user 0.015s	sys 0.000s
I20260809 23:21:50.449216 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.400725  4271 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260809 23:21:50.450449 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.449993 24161 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 17 to 19, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 19 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.427829  4317 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:21:50.453469  3885 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.453933 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:50.419744 24425 rpcz_store.cc:276] 0809 23:21:48.857988 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.206238 (+348250us) service_pool.cc:227] Handling call
0809 23:21:49.206748 (+   510us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:49.300424 (+ 93676us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.300458 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:49.359339 (+ 58881us) log.cc:842] Serialized 26 byte log entry
0809 23:21:49.481647 (+122308us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:49.481687 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:49.481750 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:50.417125 (+935375us) raft_consensus.cc:1671] finished
0809 23:21:50.417154 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:50.417316 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8212,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"thread_start_us":121868,"threads_started":1,"wal-append.queue_time_us":788187}
I20260809 23:21:50.458520 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.459133 24543 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 20
I20260809 23:21:50.407222  4297 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.464363 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:21:50.465020 24316 raft_consensus.cc:595] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:50.465708 24316 raft_consensus.cc:603] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:50.469306 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.470024 24540 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:50.466045  3791 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.470132  4235 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:50.481804 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:21:50.478353 24161 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:21:50.483137 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60732.3us
I20260809 23:21:50.484566  3920 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:50.479259  3810 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.025s	sys 0.008s
I20260809 23:21:50.490362 24161 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 18 to 21, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:21:50.487196  3810 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.026s	sys 0.008s
I20260809 23:21:50.460088  3885 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:50.505517  4251 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.005s
I20260809 23:21:50.503903  4317 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:50.428687 24167 rpcz_store.cc:269] 0809 23:20:36.932746 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:36.959033 (+ 26287us) service_pool.cc:227] Handling call
0809 23:20:37.889681 (+930648us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b5400002dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac297bad6c 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec 00007fac27bc60a2 00007fac21c8a94c 00007fac21c89d86
0809 23:21:50.416478 (+72526797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:21:46.396311 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:21:46.412915 (+ 16604us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:21:46.412968 (+    53us) write_op.cc:435] Acquired schema lock
0809 23:21:46.412987 (+    19us) tablet.cc:660] Decoding operations
0809 23:21:46.414320 (+  1333us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:21:46.414451 (+   131us) write_op.cc:641] Partition lock acquired for write op
0809 23:21:46.414493 (+    42us) tablet.cc:683] Acquiring locks for 10 operations
0809 23:21:46.563729 (+149236us) tablet.cc:699] Row locks acquired
0809 23:21:46.574355 (+ 10626us) write_op.cc:260] PREPARE: finished
0809 23:21:46.574532 (+   177us) write_op.cc:270] Start()
0809 23:21:46.574695 (+   163us) write_op.cc:276] Timestamp: P: 1786317706574464 usec, L: 0
0809 23:21:46.574719 (+    24us) op_driver.cc:331] REPLICATION: starting
0809 23:21:46.575452 (+   733us) log.cc:842] Serialized 4056 byte log entry
0809 23:21:48.259727 (+1684275us) op_driver.cc:444] REPLICATION: finished
0809 23:21:48.885568 (+625841us) write_op.cc:301] APPLY: starting
0809 23:21:48.885719 (+   151us) tablet.cc:1365] starting BulkCheckPresence
0809 23:21:48.962535 (+ 76816us) tablet.cc:1368] finished BulkCheckPresence
0809 23:21:48.962660 (+   125us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:21:48.969428 (+  6768us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:21:48.969691 (+   263us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0809 23:21:48.969742 (+    51us) write_op.cc:312] APPLY: finished
0809 23:21:48.970419 (+   677us) log.cc:842] Serialized 97 byte log entry
0809 23:21:48.987841 (+ 17422us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:48.987986 (+   145us) write_op.cc:454] Released schema lock
0809 23:21:48.988034 (+    48us) write_op.cc:341] FINISH: Updating metrics
0809 23:21:48.988395 (+   361us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:48.988429 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":928222,"reported_tablets":47,"spinlock_wait_cycles":99347328,"child_traces":[["op",{"apply.queue_time_us":625426,"apply.run_cpu_time_us":15442,"apply.run_wall_time_us":103263,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45887,"cfile_init":1,"lbm_read_time_us":561,"lbm_reads_lt_1ms":4,"num_ops":10,"prepare.queue_time_us":230189,"prepare.run_cpu_time_us":13325,"prepare.run_wall_time_us":351821,"raft.queue_time_us":542134,"raft.run_cpu_time_us":13835,"raft.run_wall_time_us":729504,"replication_time_us":1684861,"thread_start_us":1303194,"threads_started":4,"wal-append.queue_time_us":192601,"wal-append.run_cpu_time_us":42885,"wal-append.run_wall_time_us":3201481}]]}
W20260809 23:21:50.499336  4249 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.795s	user 0.013s	sys 0.001s
I20260809 23:21:50.502096 24161 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 23. New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:50.518484 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 17.
I20260809 23:21:50.534552 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:21:50.520097  4317 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:50.518488  4249 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.514724  4251 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:50.546203 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 393139us
I20260809 23:21:50.544932 24161 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 12 to 25, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:50.549196 25365 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:50.542730  4297 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.564208 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:50.566660 24535 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:21:50.566653 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1861) took 3799 ms (client timeout 1849 ms). Trace:
W20260809 23:21:50.583957 24323 rpcz_store.cc:269] 0809 23:21:46.761235 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.762966 (+  1731us) service_pool.cc:227] Handling call
0809 23:21:50.542211 (+3779245us) spinlock_profiling.cc:243] Waited 3.74 s on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:50.561059 (+ 18848us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8597282560}
W20260809 23:21:50.589651 25323 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.011s	sys 0.038s
W20260809 23:21:50.590102 25323 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.011s	sys 0.039s
W20260809 23:21:50.591372 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.022s	sys 0.014s
I20260809 23:21:50.591677 24534 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
W20260809 23:21:50.591809 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.023s	sys 0.014s
I20260809 23:21:50.592495 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3708) took 1111 ms. Trace:
I20260809 23:21:50.592957 24535 rpcz_store.cc:276] 0809 23:21:49.475852 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.493129 (+ 17277us) service_pool.cc:227] Handling call
0809 23:21:50.586718 (+1093589us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:50.586915 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2429452032}
I20260809 23:21:50.569001  4052 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:50.590781 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 553614us
W20260809 23:21:50.595507 25370 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.012s	sys 0.031s
W20260809 23:21:50.604449 25370 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.012s	sys 0.032s
I20260809 23:21:50.604722 24161 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 15 to 20, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 20 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.566164  3763 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.643440 24161 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 17 to 20, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 20 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.646334  4317 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.664528 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.667392 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.669503  4297 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:50.673970  4271 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:50.676945 24161 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.674666  4271 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:50.680140  4271 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.682229  4271 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:50.699452 24534 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 19.
W20260809 23:21:50.702332 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88320.5us
I20260809 23:21:50.704540 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.590121  4109 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:50.700477 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.013s	sys 0.033s
W20260809 23:21:50.717609 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.014s	sys 0.034s
W20260809 23:21:50.700484 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.022s	sys 0.042s
W20260809 23:21:50.719997 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.022s	sys 0.042s
W20260809 23:21:50.720366 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 634483us
I20260809 23:21:50.763828 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.715788 24161 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 17. New cstate: current_term: 17 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.765617 24534 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:50.775408 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.779842 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.776260 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:50.797091 24424 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260809 23:21:50.797024  3885 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.798213 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.800577  3763 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.801831 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.802611 24423 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:50.804637  4109 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.805614  4109 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:50.759686  3958 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.807219 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:50.813077 24543 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:50.815723 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.816473 24543 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:21:50.817812  4251 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:50.818629  4249 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:50.818728  3958 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:50.826335  3885 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260809 23:21:50.826558 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.827314 24534 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:21:50.841035 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.840705  3885 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:50.811956  4308 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.844213  3885 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:50.853605  4308 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.854380  4322 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.856395  4308 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:50.856787  4089 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.525s	user 0.012s	sys 0.000s
I20260809 23:21:50.857220  4089 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.857646  4137 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:50.851596 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:50.859337 24425 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:50.852386 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.859712 24161 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 11 to 21. New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:21:50.846657 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.872702 24540 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:50.881600 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.882725  4251 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:50.883579  4251 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:50.884709  4251 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.887423  4308 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:50.888113  4308 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:50.890570  4308 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:50.895423 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.896418 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:50.897181 24540 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 23
W20260809 23:21:50.902483 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56363.5us
I20260809 23:21:50.872216  4297 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:50.896412 24430 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:21:50.892889 24543 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:50.904031 24161 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 13 to 20, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 20 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:50.822408  4147 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.913935 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.915314 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:50.843238  3789 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.917924 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.907747  4308 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:50.924574  4326 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:50.905076  3763 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:50.925570  3885 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.926445  4089 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:50.872393  4322 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:50.931551  3979 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:50.944607  3791 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:50.933883 24161 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 14 to 22, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:50.940395 24425 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:50.946026 24534 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:21:50.913785  4297 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:50.968562 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:50.913286  4140 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260809 23:21:50.980811 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:50.979247 24543 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: []
I20260809 23:21:50.949915  4326 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:50.966948  4249 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:50.947981  4308 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:50.980445 24534 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:21:50.999850  4308 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:51.000159 24496 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:51.001194 24534 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260809 23:21:51.001801  4336 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:51.002365  4336 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:50.995146 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:50.995023 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:51.003376 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.013s	sys 0.021s
W20260809 23:21:51.003878 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.013s	sys 0.021s
W20260809 23:21:51.004303 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 701398us
I20260809 23:21:50.997025 24161 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 12 to 21. New cstate: current_term: 21 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:50.995308  3789 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:51.006807 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.004773 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.008572 24427 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:51.033569 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.013s	sys 0.025s
I20260809 23:21:50.983747  3979 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:51.034024 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.013s	sys 0.025s
W20260809 23:21:51.028347 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.908s	user 0.032s	sys 0.043s
W20260809 23:21:51.034775 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s	user 0.032s	sys 0.043s
I20260809 23:21:51.004154  4089 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:51.043782 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.044440 24425 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260809 23:21:51.045189 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:51.034313 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 720773us
W20260809 23:21:51.007568  4304 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.006s	sys 0.025s
I20260809 23:21:50.995814  4249 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:51.010739  3932 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.008s	sys 0.030s
I20260809 23:21:50.969815  3763 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:51.026093  3950 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.009s	sys 0.028s
I20260809 23:21:51.069285 24427 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:51.070148  3789 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:21:51.070538 24425 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:51.086805 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.087577 24425 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:21:51.064193  4304 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.007s	sys 0.025s
W20260809 23:21:51.066366  3932 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.008s	sys 0.030s
I20260809 23:21:51.007354  3791 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.092908 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.093559 24161 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 10 to 22, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 22 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:51.047142  4089 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:51.103085 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.002s	sys 0.027s
I20260809 23:21:51.098019 24540 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
W20260809 23:21:51.107206 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.003s	sys 0.028s
W20260809 23:21:51.085439  3950 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.009s	sys 0.029s
I20260809 23:21:51.065572  4249 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:51.121052 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.014s	sys 0.053s
W20260809 23:21:51.129603 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.014s	sys 0.053s
I20260809 23:21:51.101850 24161 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 7 to 24, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:51.090999  4262 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:51.150462 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:51.151237 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.155987 24425 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:21:51.156050 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:51.157469 24546 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.017s	sys 0.033s
I20260809 23:21:51.119318  4089 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:51.157943 24546 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.018s	sys 0.033s
I20260809 23:21:51.155736 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.164095 24429 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:21:51.045935  4297 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.165984 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:51.154511  4262 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:51.167848 24269 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.801s	user 0.012s	sys 0.019s
W20260809 23:21:51.176779 24269 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.012s	sys 0.020s
W20260809 23:21:51.177155 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 810228us
W20260809 23:21:51.168493  4276 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.010s	sys 0.034s
I20260809 23:21:51.161293  4147 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:51.176002 24161 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:21:51.202674 24161 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 10 to 19. New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:21:51.191694  4276 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.011s	sys 0.035s
I20260809 23:21:51.204196 24161 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 13 to 20. New cstate: current_term: 20 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:21:51.205541 24161 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 7 to 21, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:21:51.207324 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 446982us
I20260809 23:21:51.179173  4262 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:51.215780  4329 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260809 23:21:51.217348  3791 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:51.175418  3877 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:21:51.218334  4334 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260809 23:21:51.219486 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.219595 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.220299 25365 raft_consensus.cc:2354] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:51.220480 25373 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:51.221274 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:51.221889 25375 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:51.227453  3892 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.014s	sys 0.054s
W20260809 23:21:51.227797  3892 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.014s	sys 0.054s
I20260809 23:21:51.229502 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3622) took 1069 ms. Trace:
I20260809 23:21:51.229846 24542 rpcz_store.cc:276] 0809 23:21:50.159787 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.160883 (+  1096us) service_pool.cc:227] Handling call
0809 23:21:50.161295 (+   412us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:50.163493 (+  2198us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:50.163526 (+    33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:50.164134 (+   608us) log.cc:842] Serialized 52 byte log entry
0809 23:21:50.584803 (+420669us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:50.584862 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:50.584952 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.229261 (+644309us) raft_consensus.cc:1671] finished
0809 23:21:51.229287 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.229443 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98439,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":420116,"threads_started":1,"wal-append.queue_time_us":817787}
I20260809 23:21:51.228909 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3618) took 1897 ms. Trace:
I20260809 23:21:51.189224  3950 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:21:51.244768  3950 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:51.234333  4322 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.247196 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.247411 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.247423 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.239919 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:51.248304 25363 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:51.248905 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.249619 25372 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:51.246186  3950 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.241245  4368 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.250892  4368 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.089404  4336 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.253373 25372 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260809 23:21:51.242774 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:51.243889  4331 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260809 23:21:51.243306 24161 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 21, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:51.244778 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:51.277652 25373 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:51.189107  3885 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.248345 25371 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:51.281556 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.814s	user 0.026s	sys 0.036s
I20260809 23:21:51.160635  4326 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.233086 25375 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:51.278517 24541 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.897s	user 0.019s	sys 0.022s
W20260809 23:21:51.281899 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.026s	sys 0.036s
I20260809 23:21:51.233757 24539 rpcz_store.cc:276] 0809 23:21:49.331818 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:49.676486 (+344668us) service_pool.cc:227] Handling call
0809 23:21:49.712392 (+ 35906us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:49.762991 (+ 50599us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.763056 (+    65us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:49.892314 (+129258us) log.cc:842] Serialized 52 byte log entry
0809 23:21:50.592412 (+700098us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:50.592471 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:50.592562 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.228678 (+636116us) raft_consensus.cc:1671] finished
0809 23:21:51.228699 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.228842 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7660,"raft.queue_time_us":12948,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":15653,"thread_start_us":575360,"threads_started":1,"wal-append.queue_time_us":1110248}
I20260809 23:21:51.301445 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.291677 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:51.284636 24541 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.019s	sys 0.022s
W20260809 23:21:51.307843 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.015s	sys 0.015s
I20260809 23:21:51.296422 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:51.316771 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.016s	sys 0.015s
I20260809 23:21:51.290830 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:51.285068 24161 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 9 to 18, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 18 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:51.253634  4368 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:51.328370 24542 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
W20260809 23:21:51.277920  3154 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.023s	sys 0.039s
W20260809 23:21:51.354154 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 828280us
I20260809 23:21:51.358731 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.368489 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.370174 24424 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
W20260809 23:21:51.283347  4145 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.017s	sys 0.032s
W20260809 23:21:51.295969  4235 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.020s	sys 0.029s
I20260809 23:21:51.356725 25366 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:51.315624  3810 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.022s	sys 0.034s
I20260809 23:21:51.367296 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:51.394919 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:51.190392  3979 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:51.394963  3810 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.022s	sys 0.040s
I20260809 23:21:51.419260  4147 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:51.420281  4039 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:51.420631 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.420918  4039 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.424962 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1896) took 1339 ms. Trace:
I20260809 23:21:51.380381 25375 raft_consensus.cc:2396] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260809 23:21:51.323422  4278 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.005s	sys 0.019s
I20260809 23:21:51.425433 25370 rpcz_store.cc:276] 0809 23:21:50.085437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.085587 (+   150us) service_pool.cc:227] Handling call
0809 23:21:50.085958 (+   371us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:50.086478 (+   520us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:50.086504 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:50.087062 (+   558us) log.cc:842] Serialized 26 byte log entry
0809 23:21:50.604873 (+517811us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:50.604908 (+    35us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:50.604973 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.424707 (+819734us) raft_consensus.cc:1671] finished
0809 23:21:51.424736 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.424896 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1607,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":517342,"threads_started":1,"wal-append.queue_time_us":1001860}
I20260809 23:21:51.383467 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:51.395087 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:51.436583 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.395762 24430 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:51.437196 24539 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:51.381698 24161 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 15 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:21:51.404462 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.013s	sys 0.032s
W20260809 23:21:51.408469 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100485us
W20260809 23:21:51.352599  4317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.010s	sys 0.016s
I20260809 23:21:51.379586  3772 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:51.372394  3154 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.027s	sys 0.042s
I20260809 23:21:51.439090 24542 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:51.368968  4157 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:51.437885 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.095s	user 0.013s	sys 0.033s
W20260809 23:21:51.441244 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.018s	sys 0.018s
I20260809 23:21:51.443771 24161 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 8 to 19, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 19 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:21:51.384181  4145 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.955s	user 0.021s	sys 0.032s
W20260809 23:21:51.384107  4235 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.023s	sys 0.029s
W20260809 23:21:51.439427  4317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.010s	sys 0.016s
I20260809 23:21:51.479482  4235 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:51.480685  4235 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.481971  4308 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.483500  4157 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.485567  4157 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.451673 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:51.451870 24323 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:51.453233 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.453896  4322 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:21:51.460878 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.019s	sys 0.019s
I20260809 23:21:51.463766 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.490073 24313 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:51.464231 24161 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 15 to 16, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 16 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:21:51.491771 24161 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 12 to 19, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 19 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:21:51.493151  3885 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:51.465337  4326 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:51.465782 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92328.5us
I20260809 23:21:51.466166 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3631) took 1202 ms. Trace:
W20260809 23:21:51.496918 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45364.1us
I20260809 23:21:51.497215 24544 rpcz_store.cc:276] 0809 23:21:50.263507 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.359217 (+ 95710us) service_pool.cc:227] Handling call
0809 23:21:50.359568 (+   351us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:50.359800 (+   232us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:50.359824 (+    24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:50.403369 (+ 43545us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.130056 (+726687us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:51.130110 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.130199 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.465890 (+335691us) raft_consensus.cc:1671] finished
0809 23:21:51.465919 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.466087 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":726300,"threads_started":1,"wal-append.queue_time_us":1038452}
I20260809 23:21:51.499114 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:51.499822 24544 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:51.500895 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:51.501756 24544 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:51.467237 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.023s	sys 0.027s
W20260809 23:21:51.504031 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.023s	sys 0.027s
W20260809 23:21:51.469228  4271 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.000s
I20260809 23:21:51.468734  3772 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.283599  4262 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.472069 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3034) took 3426 ms. Trace:
I20260809 23:21:51.439180  4039 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.477018 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:51.433706  4278 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.006s	sys 0.020s
I20260809 23:21:51.505913 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.493956  3885 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.509023 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:51.442359  3789 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:21:51.515251 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.007s	sys 0.011s
I20260809 23:21:51.526257 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:51.524462 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.007s	sys 0.011s
I20260809 23:21:51.526984 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3633) took 1172 ms. Trace:
W20260809 23:21:51.530913 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 550207us
W20260809 23:21:51.535898 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 387123us
I20260809 23:21:51.530969 24545 rpcz_store.cc:276] 0809 23:21:50.354470 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.354621 (+   151us) service_pool.cc:227] Handling call
0809 23:21:50.365501 (+ 10880us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:50.365894 (+   393us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:50.365917 (+    23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:50.366314 (+   397us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.107683 (+741369us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:51.526320 (+418637us) log.cc:842] Serialized 14 byte log entry
0809 23:21:51.526649 (+   329us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.526729 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.526774 (+    45us) raft_consensus.cc:1671] finished
0809 23:21:51.526785 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.526902 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":75688,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"thread_start_us":1099861,"threads_started":2,"wal-append.queue_time_us":1063411}
I20260809 23:21:51.524045 24539 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:51.526012 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.507195  4317 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:51.541138 24323 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:51.510958 24309 rpcz_store.cc:276] 0809 23:21:48.045366 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:48.278828 (+233462us) service_pool.cc:227] Handling call
0809 23:21:49.959907 (+1681079us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:49.960761 (+   854us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:49.960792 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:49.961341 (+   549us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.035236 (+1073895us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:51.085048 (+ 49812us) log.cc:842] Serialized 14 byte log entry
0809 23:21:51.092011 (+  6963us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.099625 (+  7614us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:51.471000 (+371375us) raft_consensus.cc:1671] finished
0809 23:21:51.471023 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.471907 (+   884us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54000dcfd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:51.472014 (+   107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10083,"prepare.queue_time_us":51701,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":9503,"raft.queue_time_us":129977,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":232,"spinlock_wait_cycles":3865023104,"thread_start_us":1073432,"threads_started":1,"wal-append.queue_time_us":1406559}
I20260809 23:21:51.552305 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.524653  3958 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:51.557049 24498 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.009s	sys 0.016s
W20260809 23:21:51.557401 24498 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.009s	sys 0.016s
I20260809 23:21:51.542429 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.524381  3789 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:51.553921 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3626) took 1386 ms. Trace:
W20260809 23:21:51.557708 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 764288us
I20260809 23:21:51.570356 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:51.572785 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:51.574111 24545 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:51.573172 24539 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:51.582626 25370 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:51.580310 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58631.1us
I20260809 23:21:51.584111 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3632) took 1230 ms. Trace:
I20260809 23:21:51.584506 24541 rpcz_store.cc:276] 0809 23:21:50.354110 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.354240 (+   130us) service_pool.cc:227] Handling call
0809 23:21:50.379523 (+ 25283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:50.380444 (+   921us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:51.315875 (+935431us) log.cc:842] Serialized 14 byte log entry
0809 23:21:51.583770 (+267895us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:51.583823 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:51.583875 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.583942 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.584059 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1202155,"threads_started":2}
I20260809 23:21:51.526036  4157 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260809 23:21:51.542207  4317 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:51.594310 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:51.566982  4251 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.011s	sys 0.016s
I20260809 23:21:51.556892  3958 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:51.519651  4322 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.562875  3789 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:51.601462 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:51.595137 24542 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:51.603978 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:51.606415 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:51.557564  4271 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.608255 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.624887 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.609449 24542 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
W20260809 23:21:51.628316 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119366us
I20260809 23:21:51.571338  4356 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:51.629216 24542 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 17. Current term is 18. Ops: []
I20260809 23:21:51.611155 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:51.531607  4359 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.069s
I20260809 23:21:51.627024 25363 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:51.536149  4358 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
W20260809 23:21:51.603225  4251 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.011s	sys 0.016s
I20260809 23:21:51.593451  4317 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:51.599610  4137 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.741s	user 0.007s	sys 0.007s
I20260809 23:21:51.637398 24314 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: []
I20260809 23:21:51.579319  4344 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:51.609012  3958 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:51.659376 24316 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:51.627312  3810 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:51.646394 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45047.5us
I20260809 23:21:51.626740  4109 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:51.653704 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:51.612515  3877 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.592141  4147 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.053s
I20260809 23:21:51.656562 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.669446 24535 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260809 23:21:51.671043 24320 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:51.675519 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 88338.3us
I20260809 23:21:51.677275 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:51.655680  4344 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:51.657696  4368 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.642998  4089 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.706709 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.659555  4377 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:51.663190  4109 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:51.669040  4304 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:51.723876 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:51.650774  4271 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:51.703476  4147 consensus_queue.cc:1225] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260809 23:21:51.733664 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:51.747301 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:51.752918 24424 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:51.749519 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:51.666738  4157 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.061s
I20260809 23:21:51.704349  4344 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:51.704056  4308 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:51.736817 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.728904  3810 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:51.771202 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 131918us
I20260809 23:21:51.775123  4360 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.733637  4336 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:51.737319  4262 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:51.777423 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 100231us
I20260809 23:21:51.715830  4371 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:51.783901  4308 raft_consensus.cc:2741] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260809 23:21:51.771615  3892 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:51.808265 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.817378 24309 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:21:51.796241 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71610.8us
I20260809 23:21:51.802373  4366 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:21:51.731084  4368 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:51.803426  4374 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260809 23:21:51.804112 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:51.805160  4372 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:21:51.755546  4377 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:51.838160  4377 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.839781 25366 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260809 23:21:51.840282 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 96372.2us
W20260809 23:21:51.779567 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60592.2us
I20260809 23:21:51.773442  3885 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.841650 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.816133  3772 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.842373 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:51.843327 24498 leader_election.cc:398] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:51.843735 24498 leader_election.cc:401] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:51.844403 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.822408 24431 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:21:51.845517 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:51.821965  4364 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260809 23:21:51.841738  3885 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:51.822871  4370 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:21:51.707971  3877 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.852385 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:51.852790  3877 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:51.852984 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3722) took 1391 ms (client timeout 1720 ms). Trace:
W20260809 23:21:51.853498 24535 rpcz_store.cc:269] 0809 23:21:50.462043 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.640650 (+178607us) service_pool.cc:227] Handling call
0809 23:21:51.852775 (+1212125us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:51.852907 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2193256832}
W20260809 23:21:51.843418  4357 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260809 23:21:51.567160 24546 rpcz_store.cc:276] 0809 23:21:50.167570 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.181633 (+ 14063us) service_pool.cc:227] Handling call
0809 23:21:50.225710 (+ 44077us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:50.288630 (+ 62920us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:51.164729 (+876099us) log.cc:842] Serialized 14 byte log entry
0809 23:21:51.553577 (+388848us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:51.553630 (+    53us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:51.553673 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.553733 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:51.553861 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3275,"prepare.queue_time_us":1273149,"raft.queue_time_us":10916,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":11316,"thread_start_us":1195482,"threads_started":2}
I20260809 23:21:51.853008 24313 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:51.902118 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 152691us
I20260809 23:21:51.903199 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:51.904425  4371 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:51.904803  4371 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:51.905115  4371 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:51.905202 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.844507 25375 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:21:51.860404 24545 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:51.907361  4359 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.908070 24545 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:51.839999 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.806334  4039 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:51.854660 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:51.908794 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68862.2us
I20260809 23:21:51.771935  4304 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.055s
I20260809 23:21:51.931983 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:51.898865 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.933686 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:51.934254 24311 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:51.935155 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.935895 25372 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:51.936890 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.826330  4366 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:51.913522 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 109489us
I20260809 23:21:51.938223 24546 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:51.866919  3932 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:21:51.939702 24424 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:21:51.899636 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49444us
I20260809 23:21:51.781177  4346 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:51.909560 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.911165 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:51.964402 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 54919us
I20260809 23:21:51.929162  3979 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:51.971222 24546 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:21:51.954540 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.957758 24545 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260809 23:21:51.908560  4140 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:51.967594 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:51.922968  4039 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:51.805788  4336 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.784119  4360 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:51.974452 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63370.6us
I20260809 23:21:51.911865 24313 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:51.928162  4145 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:51.967152  4346 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:51.943367  4308 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:51.908003  4359 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:51.937261  4137 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.017s
I20260809 23:21:51.962617 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:51.976805 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:51.955485  4262 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:51.943876  3932 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:51.973886  3950 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:52.017866 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50429.8us
I20260809 23:21:51.974397  3979 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:51.954839  4322 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:51.940929  3889 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:21:52.046860 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 204606us
I20260809 23:21:52.046895 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.003072 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:51.985754  4039 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:52.051651 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:51.979825  4251 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:52.021252 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:51.896770  4109 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.044005 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3725) took 1566 ms. Trace:
W20260809 23:21:52.048877 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45877us
I20260809 23:21:52.048013 25371 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:52.005964  4356 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:52.091715 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:52.013653  4346 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.045372  3979 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:52.084329 24534 rpcz_store.cc:276] 0809 23:21:50.477360 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.001987 (+524627us) service_pool.cc:227] Handling call
0809 23:21:51.002293 (+   306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:51.002605 (+   312us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.002635 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:51.024575 (+ 21940us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.572625 (+548050us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:51.572667 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.572728 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.023257 (+450529us) raft_consensus.cc:1671] finished
0809 23:21:52.023279 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.043943 (+ 20664us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21954,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":14967,"thread_start_us":479063,"threads_started":1,"wal-append.queue_time_us":589165}
I20260809 23:21:52.076831  4271 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:52.126952 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:52.018363  4308 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.047261  3889 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:52.113744 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 146214us
W20260809 23:21:52.146446 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67014us
W20260809 23:21:52.206374 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 79566.4us
I20260809 23:21:52.199295 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.207677 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.208410 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:52.208580 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:52.209256  4365 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.341s	user 0.001s	sys 0.000s
I20260809 23:21:52.209816  4332 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:52.210187  4411 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:21:52.210433  3877 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:52.210733  3932 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.045012  4262 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.051249  4278 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.098s
I20260809 23:21:52.081965  4322 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.079s
I20260809 23:21:52.050143  4418 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:52.208779 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.115250  4089 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.195091 25371 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:52.210258 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.239639 24311 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:52.240080  4271 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:52.240386 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:52.241742 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.098007  3892 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:21:52.201653  4401 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260809 23:21:52.210246  4332 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.081723  4251 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:52.128060  3154 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260809 23:21:52.212502  4392 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.241s	user 0.000s	sys 0.000s
W20260809 23:21:52.212435  4317 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.550s	user 0.004s	sys 0.005s
I20260809 23:21:52.236464  4326 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:52.260008 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.288532 24431 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:52.145987  4379 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260809 23:21:52.200156  4419 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20260809 23:21:52.200963  4406 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
W20260809 23:21:52.214370  4389 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.443s	user 0.000s	sys 0.001s
W20260809 23:21:52.214978  4390 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
W20260809 23:21:52.213146  4393 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.233s	user 0.000s	sys 0.000s
I20260809 23:21:52.263312 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:52.210911  3877 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.266569 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.211199  4343 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:52.215687  4278 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Stepping down as leader of term 17
I20260809 23:21:52.212038  3932 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:52.211238  4303 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:52.214193  4262 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.216630  3889 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:52.158100  4362 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:52.241176  4271 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:52.213785  3763 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.326912  4308 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.210325  4360 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.327215  4439 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:21:52.306533  3877 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.328043 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.328907 24535 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:52.329805  4383 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.558s	user 0.005s	sys 0.000s
I20260809 23:21:52.331212 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3640) took 1846 ms. Trace:
I20260809 23:21:52.331589 24540 rpcz_store.cc:276] 0809 23:21:50.485112 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.129348 (+644236us) service_pool.cc:227] Handling call
0809 23:21:51.129719 (+   371us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:51.159956 (+ 30237us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.159987 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:51.189911 (+ 29924us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.549079 (+359168us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:51.549147 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.549240 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.329412 (+780172us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b24003c0308. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:21:52.330911 (+  1499us) raft_consensus.cc:1671] finished
0809 23:21:52.330952 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.331158 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7659,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3921,"spinlock_wait_cycles":52240768,"thread_start_us":331655,"threads_started":1,"wal-append.queue_time_us":460824}
I20260809 23:21:52.304134  4326 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.215271  4368 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:52.260066  4410 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260809 23:21:52.374378 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:52.211606  4249 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.054s	user 0.014s	sys 0.002s
I20260809 23:21:52.234892  4418 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:52.236167  4408 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260809 23:21:52.267091  4421 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260809 23:21:52.211344  3810 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:52.233376  4359 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:52.290254 24320 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.619s	user 0.009s	sys 0.002s
I20260809 23:21:52.394412 24320 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:21:52.394703 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.241575  3950 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260809 23:21:52.291199 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.396438 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.396801 24311 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:52.397804 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.270397  4251 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:52.398896 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.400055 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.400581 24314 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:52.401306 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.401922 24314 raft_consensus.cc:2354] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:52.402623 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.403685 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.404260 24314 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:21:52.292316 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 348652us
I20260809 23:21:52.405112 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.405247 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.406028 24314 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:52.406316  3789 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.762s	user 0.008s	sys 0.000s
I20260809 23:21:52.406847 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.406860 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.408939  4360 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:52.409049 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.409674  4360 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:21:52.412398  4395 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.284s	user 0.002s	sys 0.000s
I20260809 23:21:52.264367  3892 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:52.268249  4332 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:52.271709  3154 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:52.291208  4304 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:52.431393  3154 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:52.439116  3892 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:52.441810  4360 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:52.442684  4360 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:52.443276  4431 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:52.443482  4360 consensus_queue.cc:1225] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 28.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260809 23:21:52.443996  4431 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:52.305011 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 406378us
I20260809 23:21:52.313767 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.316082 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1794) took 1191 ms. Trace:
I20260809 23:21:52.317891 24546 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:52.209306  4356 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:52.326543  4422 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260809 23:21:52.326433  4435 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:52.126319  3772 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election won for term 18
I20260809 23:21:52.335786  4336 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.338764 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3301) took 1208 ms. Trace:
I20260809 23:21:52.340065 24540 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:21:52.341583 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.502120 25373 raft_consensus.cc:2379] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:52.342979 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.503259 25365 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 17.
I20260809 23:21:52.343464 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.345171 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.347368  3877 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.331442  4371 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.321285  4137 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:52.323650  4271 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.307509  4278 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:52.529511  4271 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:52.530093  4278 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.531697  4278 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:52.351912  4341 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.062s
W20260809 23:21:52.271314 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 270140us
I20260809 23:21:52.537636  4346 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:52.325002 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 116871us
I20260809 23:21:52.539143 25370 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:52.539139 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.541158  4371 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.541925  4371 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:52.542343  4371 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:52.354231 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.505798  4435 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.543465  4304 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:52.308159  4343 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.544339  4343 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:52.545681  4435 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.332928  4420 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:52.547101  4420 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:52.548022  4420 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:52.368513 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 150512us
I20260809 23:21:52.291712  4317 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.560815 24430 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:52.561550  4317 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.562273 24429 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:21:52.563191 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.563827 24435 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:52.564131 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.564810 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:52.263718  4089 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:52.565692  4089 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:52.566632 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:52.410698  3958 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.740s	user 0.008s	sys 0.000s
I20260809 23:21:52.423043  4430 raft_consensus.cc:2741] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260809 23:21:52.422966 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.568701  3958 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.569279 24320 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:52.569904 25363 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:52.570065 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.406718  3789 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.568181  4251 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.572115 24431 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260809 23:21:52.572875 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.424492 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.562320 24542 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260809 23:21:52.573784  4251 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.574095 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.574326 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.425693 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3726) took 1947 ms. Trace:
I20260809 23:21:52.575304 24543 rpcz_store.cc:276] 0809 23:21:50.477713 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.003573 (+525860us) service_pool.cc:227] Handling call
0809 23:21:51.003881 (+   308us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:51.004131 (+   250us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.004165 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:51.004700 (+   535us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.525072 (+520372us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:51.525123 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.525226 (+   103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.404162 (+878936us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b24003c0308. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c740 00007fac2350f578 00007fac2350fcd8 00007fac234e76e8 00007fac234e2aea 00007fac234e1f36
0809 23:21:52.413258 (+  9096us) raft_consensus.cc:1671] finished
0809 23:21:52.413279 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.425515 (+ 12236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30732544,"thread_start_us":457880,"threads_started":1,"wal-append.queue_time_us":570309}
I20260809 23:21:52.429853 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:52.577859 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148113us
I20260809 23:21:52.574745 25363 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:52.579499 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.580500  4303 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.580955  4303 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:52.584298  3920 raft_consensus.cc:979] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
W20260809 23:21:52.486356 24309 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.668s	user 0.012s	sys 0.000s
I20260809 23:21:52.492727 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:52.584939  4251 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:21:52.585268 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 92616.7us
I20260809 23:21:52.585559  4251 raft_consensus.cc:1067] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:21:52.585786 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.586256  4437 raft_consensus.cc:2741] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:52.449911 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3727) took 1072 ms. Trace:
I20260809 23:21:52.587126 24323 raft_consensus.cc:1203] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Deduplicated request from leader. Original: 16.10->[18.11-18.11]   Dedup: 18.11->[]
I20260809 23:21:52.587705 24323 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260809 23:21:52.587294 24541 rpcz_store.cc:276] 0809 23:21:51.377506 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.585616 (+208110us) service_pool.cc:227] Handling call
0809 23:21:51.607896 (+ 22280us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:51.642302 (+ 34406us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.642350 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:51.666238 (+ 23888us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.757721 (+ 91483us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:51.757776 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.757877 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.449651 (+691774us) raft_consensus.cc:1671] finished
0809 23:21:52.449680 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.449841 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5983,"thread_start_us":32310,"threads_started":1,"wal-append.queue_time_us":104010}
I20260809 23:21:52.588996  4420 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:52.587150 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.590363  4438 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:52.590467 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:52.491254 24544 rpcz_store.cc:276] 0809 23:21:51.124441 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.531228 (+406787us) service_pool.cc:227] Handling call
0809 23:21:51.548938 (+ 17710us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:51.595824 (+ 46886us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:51.716631 (+120807us) log.cc:842] Serialized 14 byte log entry
0809 23:21:51.968512 (+251881us) log.cc:842] Serialized 14 byte log entry
0809 23:21:52.301380 (+332868us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:52.301496 (+   116us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:52.315733 (+ 14237us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:52.315835 (+   102us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.316007 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30786,"prepare.queue_time_us":69027,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":19033,"raft.queue_time_us":98110,"spinlock_wait_cycles":32178304,"thread_start_us":593919,"threads_started":3,"wal-append.queue_time_us":245303}
I20260809 23:21:52.591816 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1819) took 1055 ms. Trace:
I20260809 23:21:52.592657 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.592178 25375 rpcz_store.cc:276] 0809 23:21:51.536671 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.536945 (+   274us) service_pool.cc:227] Handling call
0809 23:21:51.553698 (+ 16753us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:51.844343 (+290645us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54006cffa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:21:51.907220 (+ 62877us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:52.591566 (+684346us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b54006cffa0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23863b42 00007fac2385ff26 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:52.591759 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514610816}
I20260809 23:21:52.384982  4408 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.593739  4435 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:52.594211  4408 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.595410  4437 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260809 23:21:52.396600  4381 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260809 23:21:52.497731 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:52.600514 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102901us
I20260809 23:21:52.601820 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:52.499118 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 264954us
W20260809 23:21:52.450883  4432 log.cc:924] Time spent T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:21:52.504312 25368 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:52.566745  4089 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.604871 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3741) took 1502 ms. Trace:
I20260809 23:21:52.605261 24427 rpcz_store.cc:276] 0809 23:21:51.102948 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.103098 (+   150us) service_pool.cc:227] Handling call
0809 23:21:51.148628 (+ 45530us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:51.153610 (+  4982us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.153644 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:51.165668 (+ 12024us) log.cc:842] Serialized 26 byte log entry
0809 23:21:51.613338 (+447670us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:51.613376 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.613440 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.604660 (+991220us) raft_consensus.cc:1671] finished
0809 23:21:52.604682 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.604811 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5055,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":8150,"thread_start_us":406762,"threads_started":1,"wal-append.queue_time_us":1088012}
I20260809 23:21:52.498835  3979 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.385358  4418 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.505759 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.446807  4431 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.607985  4431 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:52.608191  3979 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:52.348783  4428 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260809 23:21:52.608712  4431 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:52.508817 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1801) took 1086 ms. Trace:
I20260809 23:21:52.509622 25369 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:52.447628  3885 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.601s	user 0.011s	sys 0.001s
I20260809 23:21:52.501073 24428 rpcz_store.cc:276] 0809 23:21:51.130264 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.148595 (+ 18331us) service_pool.cc:227] Handling call
0809 23:21:51.160213 (+ 11618us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:51.165779 (+  5566us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:51.165807 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:51.201750 (+ 35943us) log.cc:842] Serialized 52 byte log entry
0809 23:21:51.567348 (+365598us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:51.567389 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:51.567446 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:52.338498 (+771052us) raft_consensus.cc:1671] finished
0809 23:21:52.338526 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.338695 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3735,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":5490,"thread_start_us":349735,"threads_started":1,"wal-append.queue_time_us":1094679}
I20260809 23:21:52.383199  4249 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.609467 24539 rpcz_store.cc:276] 0809 23:21:51.422652 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.607178 (+184526us) service_pool.cc:227] Handling call
0809 23:21:51.638760 (+ 31582us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:51.638977 (+   217us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:21:51.659056 (+ 20079us) log.cc:842] Serialized 14 byte log entry
0809 23:21:52.479209 (+820153us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b18004e8b00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:21:52.508383 (+ 29174us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:52.508459 (+    76us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:52.508525 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:52.508611 (+    86us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:52.508750 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4444,"prepare.queue_time_us":27249,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":4619,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":22741,"spinlock_wait_cycles":5398272,"thread_start_us":24257,"threads_started":1}
I20260809 23:21:52.584956 24309 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:21:52.611392 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3726) took 1848 ms (client timeout 1822 ms). Trace:
I20260809 23:21:52.611541  4249 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.513751  4426 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
W20260809 23:21:52.611793 24422 rpcz_store.cc:269] 0809 23:21:50.762804 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:50.775301 (+ 12497us) service_pool.cc:227] Handling call
0809 23:21:52.611207 (+1835906us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540017f7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:52.611328 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2590717312}
I20260809 23:21:52.612295  4426 consensus_queue.cc:1225] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 25.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:52.518421  4326 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:52.523996 24313 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:52.524994  3877 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.509455  3772 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:52.613078 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.613772 24309 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.525566  3154 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.614967 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:52.616526  3154 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.526302 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.527148 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.449877  4358 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:52.617249 24544 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:21:52.617654  4358 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:52.617357  4366 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.614231  3772 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.618517  4358 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.619593  4358 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.608011 24546 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260809 23:21:52.609936  3885 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:52.618865 25365 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:52.594237  4435 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:52.622754  4435 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:52.626142 25373 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 17 FOLLOWER]: Advancing to term 18
I20260809 23:21:52.618484 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:52.627779 24541 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 24
I20260809 23:21:52.630719  3920 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260809 23:21:52.631760 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.632390 24313 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:52.633227 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.633913 24313 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:52.634286  4109 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:52.636070 24321 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:52.637058 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.637630 24321 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:52.573226  3789 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:52.640370 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
W20260809 23:21:52.640897 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.578s (SENT) [suppressed 11 similar messages]
I20260809 23:21:52.641167 24311 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:52.642091  4137 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:52.587041 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.644002 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.644706 24312 raft_consensus.cc:2379] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:52.645527 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.646100 24312 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:52.555635  4375 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260809 23:21:52.491470 24534 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:52.496570 24316 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:52.642630  4137 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.558108 24496 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:52.568172 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.647238 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.651008 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.677055 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.650811 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3039) took 1093 ms. Trace:
I20260809 23:21:52.652809 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:52.678167 24309 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:52.677716 24323 rpcz_store.cc:276] 0809 23:21:51.557392 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.896446 (+339054us) service_pool.cc:227] Handling call
0809 23:21:52.587017 (+690571us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:52.588289 (+  1272us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:52.650626 (+ 62337us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b540019d2d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:52.650757 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1483648640}
I20260809 23:21:52.678821 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.680893 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:52.649091 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.646899 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.657547 24541 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:52.663161 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.517040  3889 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.634647 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:52.687140 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84239.9us
I20260809 23:21:52.676246 24429 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:52.683377 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:52.688724 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:52.667554 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.702790 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.703516 25372 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260809 23:21:52.710088 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63224.5us
I20260809 23:21:52.711234 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:52.702594 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 144564us
I20260809 23:21:52.713124 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:52.677304  4039 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.582s	user 0.005s	sys 0.006s
I20260809 23:21:52.721333 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.721611 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:52.724952 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.713980 25366 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:52.720934 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:52.726227 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 462717us
I20260809 23:21:52.724725 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.726419 25369 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.727442 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.733150 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:52.674302  4358 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:52.734362 24427 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:52.725598 24428 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:21:52.736368 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:52.736524 24309 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260809 23:21:52.738152 24543 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260809 23:21:52.685539  4366 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:21:52.676749  3958 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:52.701048  4297 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:52.722043  4375 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:52.753584 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.739647 24431 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:52.741595 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51197.4us
I20260809 23:21:52.757035 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.756626 24309 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260809 23:21:52.760433 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:52.756609  4145 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:52.755548 24423 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 16
I20260809 23:21:52.721115  3885 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:52.824879 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.766026 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:52.762756 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:52.714567  4377 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:21:52.825697 25372 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:52.785252 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:52.715163  3932 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:52.790835 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.738811  3979 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:52.764544  4437 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:52.765327 24541 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:21:52.763653 24320 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:52.815725 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:52.832769 24424 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:52.780478  4140 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:52.817345  4137 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.817446 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:52.818630 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:52.834908 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:52.818835 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 82932us
I20260809 23:21:52.820729 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:52.822803 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59473.5us
I20260809 23:21:52.822830 24423 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260809 23:21:52.739809  4368 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260809 23:21:52.813416 24543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:21:52.840591 24543 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:21:52.841037 24543 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:52.791169 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:52.826956 24435 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:52.825485 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:52.786473  4304 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:52.821885  4375 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.833321 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:52.838038 24423 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:52.735271  4039 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.839445 24534 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:52.827461 24323 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:52.837442 24498 leader_election.cc:398] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:52.787863  3763 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:52.849361 24427 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:52.839706 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:52.862442 24498 leader_election.cc:401] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260809 23:21:52.763860  4358 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.839821  4437 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.828691  4377 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.830257  4145 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.834709  4140 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:52.864540 24539 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260809 23:21:52.863327 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:52.875496 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 148138us
I20260809 23:21:52.870683 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:21:52.839082  3932 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:52.835898  4147 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260809 23:21:52.860095  4375 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:52.891582 24317 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:52.884317 24543 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.893669 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:52.882001 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44740.1us
I20260809 23:21:52.912631 24543 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.917936 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:52.787504  4371 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:52.906049 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:52.924067 24498 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:52.930691 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:52.937187 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:52.942083 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:52.880136  4377 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:52.944331  4445 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:52.944362 24534 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:52.884840  3810 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:52.892611  4308 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:52.892769  4140 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:52.935593 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
W20260809 23:21:52.947830  4395 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260809 23:21:52.989166 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.012034 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.006806 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.008328 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.014200 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.939005  4359 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:52.990168  4452 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:21:52.928462  4089 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:53.015278 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:52.958315 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.008924  3810 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:52.937837  4346 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:52.859378  4304 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:52.948300  4249 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:53.018349 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:53.087574 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69334.6us
I20260809 23:21:53.041031 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:53.084187 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 299005us
I20260809 23:21:52.941382  4336 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:53.044332 24435 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.044461 24309 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 21: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:52.865227  4457 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.143s
W20260809 23:21:53.042057 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 178821us
W20260809 23:21:53.144522 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56254.3us
I20260809 23:21:52.989146  4459 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:53.047217 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:53.015602  4396 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:53.085162 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.046579 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:53.046567  4343 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.000s
I20260809 23:21:53.080742  3885 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:53.145241 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:53.145951 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 56298.9us
W20260809 23:21:53.146575 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56013.8us
I20260809 23:21:53.017612  3889 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:52.948792  3772 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.098s
I20260809 23:21:53.154726 24534 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:53.154645 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.182780 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:53.046386  4089 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:53.085112  3810 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:53.170090 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:53.043534  4344 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.228s	user 0.013s	sys 0.008s
I20260809 23:21:53.199940 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:53.178409 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 260615us
I20260809 23:21:53.179586 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.154198  4459 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:53.160419  4396 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:53.091312  4371 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.047123  4359 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:53.166622  4403 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.119s	user 0.002s	sys 0.004s
I20260809 23:21:53.195559  4089 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:53.159217  4445 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:53.194720 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.195981 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:53.198940  4366 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:53.087098  4249 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:53.202432 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.213073  4459 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:53.259305 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56966.2us
I20260809 23:21:53.180279  4343 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.212533 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:53.213256 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:53.214300  4464 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
W20260809 23:21:53.201179  4278 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.669s	user 0.011s	sys 0.000s
I20260809 23:21:53.224423 25372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:53.226718 24534 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:53.231784 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.158759  4341 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:53.238679 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:53.171690  3877 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260809 23:21:53.287349  4431 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:21:53.306525  4472 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260809 23:21:53.237972  4039 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.255488 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.256043  4445 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.257333  4366 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.174295 25369 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:53.171844  4453 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:53.279878  4249 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:53.212320  4344 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.215440  4439 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:53.235037  4396 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:53.282248 24429 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:21:53.240743  4358 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.261194  4251 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260809 23:21:53.299892  4454 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:53.300634  4359 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.301177  4437 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.301679  4474 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.087s
I20260809 23:21:53.303372  4145 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:53.303514  4467 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260809 23:21:53.304100  4341 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
W20260809 23:21:53.304427  4332 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.873s	user 0.008s	sys 0.003s
I20260809 23:21:53.304956  4304 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:53.261021  4109 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.617s	user 0.012s	sys 0.000s
W20260809 23:21:53.305190  4435 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.000s
I20260809 23:21:53.419513  4435 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.421339 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.421681  4435 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:53.305620  4468 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260809 23:21:53.422518 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.422966  4485 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:53.423309 24545 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:53.316812 25370 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
W20260809 23:21:53.391837  4408 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.749s	user 0.007s	sys 0.004s
I20260809 23:21:53.318032 24311 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:53.265014 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:53.427263  4408 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.429580  4408 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.430840 25368 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:53.435606 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3671) took 1575 ms. Trace:
I20260809 23:21:53.435997 24317 rpcz_store.cc:276] 0809 23:21:51.860222 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.860361 (+   139us) service_pool.cc:227] Handling call
0809 23:21:51.930748 (+ 70387us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:52.911927 (+981179us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.435370 (+523443us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:53.435543 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2199657216}
I20260809 23:21:53.437196 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:53.439677  4462 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.485s	user 0.004s	sys 0.000s
I20260809 23:21:53.441108 25372 raft_consensus.cc:595] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:53.441614 25372 raft_consensus.cc:603] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:53.322600 25363 raft_consensus.cc:2354] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:53.162680  3932 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.342787  4439 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:53.410615 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 206141us
I20260809 23:21:53.322213  4453 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:53.445901 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 214185us
W20260809 23:21:53.463683 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 139575us
I20260809 23:21:53.429487 24424 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260809 23:21:53.446520 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 181611us
I20260809 23:21:53.489885 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:53.473781 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:53.438786 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.455619 24545 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:53.345494  4454 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:53.418215 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.473935 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:53.444298  4303 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.862s	user 0.019s	sys 0.002s
I20260809 23:21:53.491041 24317 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:21:53.490160 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.498083 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:53.499321 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.441005  4343 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.502240 24429 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:53.501729 24424 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260809 23:21:53.445573  4278 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:53.508426 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:53.490185  3979 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:53.464918  4498 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:53.473686  4039 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:53.473404  4453 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:53.517879 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.558540 24317 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:53.473239  4497 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:53.490238  4341 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:53.518671  3979 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:53.524777 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:53.524040  4140 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.000s
I20260809 23:21:53.510186  4430 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.525251  4437 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:53.495177  4454 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:53.558897 24317 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:53.418288  4332 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:53.611840 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 87125us
I20260809 23:21:53.507937  4435 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:53.507027  4362 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:53.340636  3885 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.505676  4109 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.099s
I20260809 23:21:53.573788 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.506004  3877 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:53.605279 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 106041us
I20260809 23:21:53.620643  3979 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.559271  4336 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.573402  4497 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:53.664834  4497 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:21:53.575892 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48620.6us
I20260809 23:21:53.668094  4341 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.670979  4344 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.670717 24541 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260809 23:21:53.671548  4439 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.673174 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:53.674904 24542 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260809 23:21:53.557514  4278 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.676533 24540 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:21:53.506068  4303 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.514801  4426 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.098s
W20260809 23:21:53.612236  4381 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260809 23:21:53.625847  4517 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260809 23:21:53.604640  4308 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260809 23:21:53.604847  4140 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.688058  4140 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:53.634052  4482 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260809 23:21:53.617532  4276 raft_consensus.cc:2741] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:53.629868  4362 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.622056  4438 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:53.635491  4359 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.689973  4359 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:53.636545  4510 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
W20260809 23:21:53.636878  4495 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260809 23:21:53.569044  4516 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:53.640753 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3675) took 1667 ms. Trace:
W20260809 23:21:53.655697  4487 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.317s	user 0.000s	sys 0.001s
I20260809 23:21:53.655525 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.656164  4317 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:53.657068 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.695243 24543 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:53.696491 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.697300  4317 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:21:53.697427 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1896) took 1050 ms. Trace:
I20260809 23:21:53.697688  4317 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:53.698100  4317 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:21:53.644815  4358 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.699234  4358 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.617102 24317 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.703922 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1922) took 1054 ms. Trace:
I20260809 23:21:53.704327 25368 rpcz_store.cc:276] 0809 23:21:52.649591 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.676790 (+ 27199us) service_pool.cc:227] Handling call
0809 23:21:53.703667 (+1026877us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:53.703826 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1731684736}
I20260809 23:21:53.705947 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1887) took 1061 ms. Trace:
I20260809 23:21:53.706341 24311 rpcz_store.cc:276] 0809 23:21:52.644091 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.681575 (+ 37484us) service_pool.cc:227] Handling call
0809 23:21:53.705770 (+1024195us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b5400385ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:53.705888 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1294645504}
I20260809 23:21:53.645802 24321 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:21:53.707631 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.708609 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.708709 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.709227 24321 raft_consensus.cc:2362] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25. Re-sending same reply.
I20260809 23:21:53.573478  4039 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.622628  4430 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.663637  4366 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.716885  4358 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.720782  4516 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:53.721377  4516 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:53.723042 24317 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:53.722467  4516 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.723578  4039 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.724356  4516 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.727460  4332 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.663968  4459 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.709290 24311 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:53.697840 24316 rpcz_store.cc:276] 0809 23:21:52.646609 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.789610 (+143001us) service_pool.cc:227] Handling call
0809 23:21:52.790008 (+   398us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:52.799982 (+  9974us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:52.800024 (+    42us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:52.812883 (+ 12859us) log.cc:842] Serialized 52 byte log entry
0809 23:21:53.147164 (+334281us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:53.147221 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.147310 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:53.697195 (+549885us) raft_consensus.cc:1671] finished
0809 23:21:53.697217 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:53.697369 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31075,"raft.queue_time_us":17694,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":9996,"thread_start_us":238676,"threads_started":1,"wal-append.queue_time_us":412774}
I20260809 23:21:53.694680 24312 rpcz_store.cc:276] 0809 23:21:51.977215 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.709847 (+732632us) service_pool.cc:227] Handling call
0809 23:21:52.784216 (+ 74369us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:52.883325 (+ 99109us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:52.883367 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:52.883435 (+    68us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:52.902892 (+ 19457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.640537 (+737645us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:53.640585 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:53.640694 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212872064}
W20260809 23:21:53.747007 24435 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.004s
I20260809 23:21:53.747501 24435 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:21:53.641796  4271 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.111s	user 0.009s	sys 0.004s
W20260809 23:21:53.637375  4297 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.851s	user 0.009s	sys 0.003s
I20260809 23:21:53.748837 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:53.759479 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.754904 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.755661 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.762138 24316 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.763902 24546 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:53.764129 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.739725  3889 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.739039  4308 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:53.769134 24539 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:53.786017 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.787424 24311 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:53.749604 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1893) took 1097 ms. Trace:
I20260809 23:21:53.793242 24543 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:21:53.795552 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3316) took 1019 ms. Trace:
I20260809 23:21:53.799198 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.803773 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.804414 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.796037 24425 rpcz_store.cc:276] 0809 23:21:52.776569 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.835820 (+ 59251us) service_pool.cc:227] Handling call
0809 23:21:52.836114 (+   294us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:52.836300 (+   186us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:53.170007 (+333707us) log.cc:842] Serialized 14 byte log entry
0809 23:21:53.795093 (+625086us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:53.795165 (+    72us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:53.795225 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.795330 (+   105us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:53.795482 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":266430,"prepare.queue_time_us":28066,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":322383,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":4450,"thread_start_us":684587,"threads_started":3,"wal-append.queue_time_us":134655}
I20260809 23:21:53.763788  3979 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:53.793442 24313 rpcz_store.cc:276] 0809 23:21:52.645772 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.735201 (+ 89429us) service_pool.cc:227] Handling call
0809 23:21:52.735489 (+   288us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:52.735831 (+   342us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:52.735859 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:52.812669 (+ 76810us) log.cc:842] Serialized 26 byte log entry
0809 23:21:53.321037 (+508368us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:53.606351 (+285314us) log.cc:842] Serialized 14 byte log entry
0809 23:21:53.609260 (+  2909us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.609378 (+   118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:53.651024 (+ 41646us) raft_consensus.cc:1671] finished
0809 23:21:53.651065 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:53.651259 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4701,"prepare.queue_time_us":313456,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":49576,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":11457,"thread_start_us":723633,"threads_started":2,"wal-append.queue_time_us":675810}
I20260809 23:21:53.855278 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.810493 24317 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:53.820294  4493 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:53.856812  4493 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.857080 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.857885 24317 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:53.858728 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:53.859040  4493 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.859786 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:53.758164  4396 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.860584 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.861128 24317 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260809 23:21:53.861856  4396 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.821734  4453 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.863049 24317 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:53.825563  4459 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:53.864636  4453 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.743067  4336 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.872520  4336 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.791065  3763 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260809 23:21:53.875458 24311 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:21:53.880033 24539 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:53.881873  4109 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:53.770555  4140 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:53.826988 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.766943  4278 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.792111  3889 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.804072  4235 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:53.804203  4359 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.794848  4308 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:53.836169  4371 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.599s	user 0.007s	sys 0.003s
I20260809 23:21:53.765350  4297 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:21:53.839635  4039 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:53.841367 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:53.843470  4517 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:53.843902  4474 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:21:53.844457 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:21:53.853567  4089 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.574s	user 0.009s	sys 0.000s
I20260809 23:21:53.853981  4491 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:53.822979 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.899969 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.765769  4303 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:53.916890  4140 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:53.945583 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:53.941452 24543 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:21:53.946236 24546 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:53.911919 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 438030us
W20260809 23:21:53.901041  3892 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.461s	user 0.016s	sys 0.004s
I20260809 23:21:53.764513  4271 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.866302  4497 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.919145  4359 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:53.934036 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 92845us
I20260809 23:21:53.936024 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1787) took 1251 ms. Trace:
I20260809 23:21:53.985436 25375 rpcz_store.cc:276] 0809 23:21:52.684137 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.737905 (+ 53768us) service_pool.cc:227] Handling call
0809 23:21:52.783473 (+ 45568us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:52.783795 (+   322us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:52.783831 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:52.784357 (+   526us) log.cc:842] Serialized 26 byte log entry
0809 23:21:53.187401 (+403044us) raft_consensus.cc:1631] Marking committed up to 20
0809 23:21:53.207038 (+ 19637us) log.cc:842] Serialized 14 byte log entry
0809 23:21:53.930797 (+723759us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b18003419e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:21:53.931007 (+   210us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:53.931104 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:53.931164 (+    60us) raft_consensus.cc:1671] finished
0809 23:21:53.935824 (+  4660us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:53.935967 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5930,"prepare.queue_time_us":19440,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":728560,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7730,"spinlock_wait_cycles":2699648,"thread_start_us":334026,"threads_started":1,"wal-append.queue_time_us":433966}
W20260809 23:21:53.937074 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 92720.6us
I20260809 23:21:53.940132 24316 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:21:53.791934  4418 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.185s	user 0.008s	sys 0.004s
I20260809 23:21:53.949231 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:53.980026 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.991250 24425 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:53.939040  4371 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:21:53.993239 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 44087.1us
I20260809 23:21:53.988087 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3075) took 1029 ms. Trace:
I20260809 23:21:53.996261 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:53.996945 24316 rpcz_store.cc:276] 0809 23:21:52.958668 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.755424 (+796756us) service_pool.cc:227] Handling call
0809 23:21:53.988020 (+232596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:53.988880 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:53.996733 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:53.997905 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49657.8us
I20260809 23:21:53.998641 24422 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.989630  4343 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:54.000464  4343 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.918031  4454 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.949388  4278 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:53.983757  3892 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:53.996107  4430 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.988566  4359 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.978456  4491 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.014974 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.003206  4529 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:54.019773 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1783) took 1345 ms. Trace:
I20260809 23:21:54.059284 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.059396 25370 rpcz_store.cc:276] 0809 23:21:52.674383 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.674580 (+   197us) service_pool.cc:227] Handling call
0809 23:21:54.014273 (+1339693us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:54.019711 (+  5438us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":372587264}
I20260809 23:21:53.994294  4140 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:53.995710  4532 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
W20260809 23:21:54.059405  3810 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.848s	user 0.010s	sys 0.000s
I20260809 23:21:54.061004  4532 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:54.061450  4532 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:54.061349  3810 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.013263 24309 raft_consensus.cc:2354] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:53.978223  4444 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260809 23:21:54.062575  4532 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.020339 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:53.997995  4528 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:54.063866  4430 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260809 23:21:54.064000 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43748.7us
I20260809 23:21:54.064311  4430 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:54.064733  4430 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:54.065136  4308 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.020517  4431 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:54.065855  4431 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:54.005745 24431 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:54.066947  4109 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:21:54.068360  4109 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:54.040727 25375 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:21:54.039142 24427 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
W20260809 23:21:54.048834 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.21237e+06us
I20260809 23:21:54.090839 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:54.091979 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:54.093642 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:53.994047  4418 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.047176 24425 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:54.097502  4418 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.052263  4278 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:54.099324  4278 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.103130 24314 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:54.052673  4377 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.108s	user 0.013s	sys 0.000s
I20260809 23:21:54.104650  4377 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.053190  4343 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.107851  4343 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:54.108435  4343 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:54.109580  4343 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.111267 24320 raft_consensus.cc:2354] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27.
I20260809 23:21:54.112152 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.113082 25375 raft_consensus.cc:2354] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23.
I20260809 23:21:54.053341 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.114112 24424 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:54.053613 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.120322 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:54.122349 24269 leader_election.cc:398] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23.
I20260809 23:21:54.122761 24269 leader_election.cc:401] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260809 23:21:54.053890  3892 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:54.030802  4249 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.693s	user 0.010s	sys 0.003s
I20260809 23:21:54.054476 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:54.125957 24544 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:54.055213 25371 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:54.055382  4359 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.060122 25373 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:54.063140 24323 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:54.068781 24431 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260809 23:21:54.069339  4531 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260809 23:21:54.070243  4460 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.251s	user 0.000s	sys 0.003s
I20260809 23:21:54.070818 25363 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260809 23:21:54.030121  4251 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.009s
I20260809 23:21:54.134280 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.050451 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3310) took 1341 ms. Trace:
I20260809 23:21:54.032932 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:54.071885 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 859346us
I20260809 23:21:54.137483 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:54.072075 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:54.077078  4235 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:54.138774 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:54.077148  3889 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:54.053340  4375 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.125s	user 0.008s	sys 0.003s
I20260809 23:21:54.079731  4271 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.082429  4454 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:53.995373  4497 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.138269  4418 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:54.087126 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44984.6us
I20260809 23:21:54.088670 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.186143 25369 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:54.128139 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.508s (SENT)
I20260809 23:21:54.193928 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:54.136953 24430 rpcz_store.cc:276] 0809 23:21:52.709043 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.709210 (+   167us) service_pool.cc:227] Handling call
0809 23:21:52.757079 (+ 47869us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:53.502674 (+745595us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:53.502722 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:53.502762 (+    40us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:53.502792 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.050172 (+547380us) spinlock_profiling.cc:243] Waited 665 ms on lock 0x7b5400287dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:54.050232 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:54.050382 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":127630,"raft.queue_time_us":96525,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":43713,"spinlock_wait_cycles":1529491712}
I20260809 23:21:54.144501  4438 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:21:54.147846 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76842.7us
I20260809 23:21:54.149605  4235 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:53.945148  4089 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.130820  4308 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:54.042629  4317 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.136580  4147 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:54.145583  4435 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:54.141429  4377 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.211885 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:54.209250 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:54.158234 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 125604us
I20260809 23:21:54.224179 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:54.214761  4109 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:54.128849  3772 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260809 23:21:54.168937  4359 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:54.242616 24321 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:54.246657 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:54.245110 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:54.130467  4249 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.216758 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.250123 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:21:54.250396 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122295us
W20260809 23:21:54.227738  4341 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.054s	sys 0.072s
I20260809 23:21:54.171550  4528 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:21:54.149421  4457 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:54.263278 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.363021 25375 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:54.264263 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.271490 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.277088 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.277333 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.162902  4375 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.247265  4308 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:54.247788  4336 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:54.308707 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.302850 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.232486  4454 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:54.248159  3789 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:54.209605  4420 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260809 23:21:54.248111  3877 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:54.426808 24535 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 27
I20260809 23:21:54.312947  4332 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:54.217772  4408 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.058s
I20260809 23:21:54.266557  4109 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:54.317972 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:54.357494  3810 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.309094  4435 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:54.262595  4497 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:54.327250 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.410719  4308 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:54.336422  4362 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.335829  4453 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:54.338385  4532 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:54.270860  4439 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:54.338752 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.344517 24534 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:54.347255 24422 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:54.349611 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.278508 24425 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:54.355487 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1897) took 1421 ms (client timeout 1547 ms). Trace:
W20260809 23:21:54.481451 24323 rpcz_store.cc:269] 0809 23:21:52.933920 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.934135 (+   215us) service_pool.cc:227] Handling call
0809 23:21:54.355232 (+1421097us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:54.355415 (+   183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2414234368}
I20260809 23:21:54.430128  3892 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:54.358920 24544 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:21:54.410763 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:54.411685 24430 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:54.336822  4317 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.415134 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:54.499128 24427 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:21:54.415087 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60223us
W20260809 23:21:54.356383  4341 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.054s	sys 0.072s
I20260809 23:21:54.346699  3950 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:54.420331 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:54.356518  4528 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:54.318753  4530 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:54.530673 24422 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260809 23:21:54.531116 24422 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 10
I20260809 23:21:54.531390 24422 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260809 23:21:54.531669 24422 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:54.436555 24435 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:54.334829 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:54.471119 24311 raft_consensus.cc:2354] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:54.471814 25375 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:21:54.534004 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:54.534600 25375 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:54.154845 24424 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:21:54.489780 24425 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:54.501895 25365 raft_consensus.cc:2354] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:54.502628 25369 raft_consensus.cc:2354] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
I20260809 23:21:54.503525 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:21:54.507548 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 249282us
I20260809 23:21:54.524691 24498 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:54.524958 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 104805us
I20260809 23:21:54.545583 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:54.525436 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3067) took 1569 ms (client timeout 1508 ms). Trace:
I20260809 23:21:54.525823 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:54.546454 24320 rpcz_store.cc:269] 0809 23:21:52.955825 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.008584 (+ 52759us) service_pool.cc:227] Handling call
0809 23:21:54.525236 (+1516652us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003c32d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:54.525370 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2350947584}
W20260809 23:21:54.527752 24314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3048) took 2178 ms (client timeout 2749 ms). Trace:
I20260809 23:21:54.435878 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.529805 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.547027 24544 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:21:54.547740 24314 rpcz_store.cc:269] 0809 23:21:52.349672 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.406586 (+ 56914us) service_pool.cc:227] Handling call
0809 23:21:54.527552 (+2120966us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:54.527685 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3898987264}
I20260809 23:21:54.469880  4474 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:54.472473  4527 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:54.476635  3810 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:54.581617 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:54.548009 24312 raft_consensus.cc:2362] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25. Re-sending same reply.
I20260809 23:21:54.426452  4454 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.429699  3789 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.474889  4497 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.433655  3877 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.584117 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:54.586061 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.595461 24320 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 18: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:54.564802 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.587697 25370 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:54.585842 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:54.590291 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.597472 24544 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:54.590880 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.440651  4332 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.602237 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.534819  4483 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:54.529662  4528 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:21:54.601522  4543 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260809 23:21:54.578599  4359 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:21:54.611627 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 350875us
W20260809 23:21:54.594522  4516 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.866s	user 0.014s	sys 0.000s
I20260809 23:21:54.483906  4453 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.587782  4491 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.621423 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.622586 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.625033 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:54.593221  3920 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:54.586867  4445 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:21:54.602762  4420 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:54.602595  4439 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:54.655519 24313 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:54.655423 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.604852  4408 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:54.594072  3810 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.621413 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:54.681440 24323 raft_consensus.cc:2354] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25.
I20260809 23:21:54.676995 25373 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:54.705088 25321 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:54.706485 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.529106  3950 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.608282  4089 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:54.729182 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:54.637104 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53081.2us
I20260809 23:21:54.732524 24314 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:54.741240 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:54.620477  4358 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:54.676421  4396 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:54.747828 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:54.747954 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.749884 25323 leader_election.cc:398] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25.
I20260809 23:21:54.750202 25323 leader_election.cc:401] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260809 23:21:54.627285  4249 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.676972  4341 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:54.788369 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:54.779464 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:54.662915  4359 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:21:54.730149  4344 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.039s	sys 0.108s
I20260809 23:21:54.780413 24311 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:21:54.780934 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.026s	sys 0.010s
W20260809 23:21:54.887872 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.026s	sys 0.010s
W20260809 23:21:54.781677 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 536632us
W20260809 23:21:54.799062  4418 consensus_meta.cc:287] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.000s
W20260809 23:21:54.837783  4344 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.039s	sys 0.112s
W20260809 23:21:54.748733  4540 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.406s	user 0.000s	sys 0.001s
I20260809 23:21:54.865274 24422 pending_rounds.cc:76] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:21:54.914490 24422 pending_rounds.cc:97] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260809 23:21:54.914772 24422 raft_consensus.cc:2866] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:54.712172  3889 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:54.742722  4445 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:54.779784  3772 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:54.774164  4326 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:54.888900 24378 leader_election.cc:398] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:54.794296 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:54.788316  4485 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:21:54.836445  4538 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.482s	user 0.000s	sys 0.001s
W20260809 23:21:54.776495  4541 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.460s	user 0.000s	sys 0.001s
W20260809 23:21:54.850922  4543 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260809 23:21:54.710964  4343 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.126s
I20260809 23:21:54.901436  4483 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:54.902130 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:54.903400  3892 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:21:54.904844  4457 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
W20260809 23:21:54.909027  4546 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.484s	user 0.000s	sys 0.001s
W20260809 23:21:54.909612  4431 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.843s	user 0.006s	sys 0.004s
W20260809 23:21:54.910620  4430 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.845s	user 0.010s	sys 0.001s
I20260809 23:21:54.705224  4435 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.911089  4356 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:54.911585  4346 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:54.911876  4453 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.912595  4359 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.916517 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:54.788558  4375 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:54.731901  4420 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.987816  4457 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:54.747921  4516 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.143s
I20260809 23:21:54.736972  4439 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.928328 24378 leader_election.cc:401] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260809 23:21:54.928794 24312 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260809 23:21:54.745311  4408 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.747200  4358 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:54.668373  4109 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.946638 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:54.944355 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 359226us
I20260809 23:21:54.949463  3950 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.972309 24544 raft_consensus.cc:2354] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23.
I20260809 23:21:54.917960  4418 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23.
I20260809 23:21:54.996404 24323 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260809 23:21:54.983135  4303 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.007s	sys 0.034s
I20260809 23:21:54.958645  4326 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:54.929759  3889 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.001760  3154 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:55.097110  4561 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
I20260809 23:21:55.019511  4332 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:54.993335  4431 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.020179  4497 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:54.997275  4430 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.010509  4356 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:54.965126  4485 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:21:54.994969  4562 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.317s	user 0.000s	sys 0.002s
I20260809 23:21:55.037590 24314 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
W20260809 23:21:55.039826 24535 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.608s	user 0.009s	sys 0.002s
W20260809 23:21:55.040961 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1917) took 2685 ms (client timeout 3372 ms). Trace:
I20260809 23:21:54.988077  4483 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:55.082749 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 193999us
W20260809 23:21:55.156370 25371 rpcz_store.cc:269] 0809 23:21:52.346733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.574064 (+227331us) service_pool.cc:227] Handling call
0809 23:21:55.029561 (+2455497us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54006d4cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:55.031619 (+  2058us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3403659648}
I20260809 23:21:55.077358  3789 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:55.077512 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.095255 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.161216 24431 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:55.157183 24535 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:21:55.008114  3892 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:55.097101 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 150582us
I20260809 23:21:55.174009 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:55.175648 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 98207.9us
I20260809 23:21:55.176594 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:55.177784 24498 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:55.178174 24498 leader_election.cc:401] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260809 23:21:55.179471 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.180519 24314 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:54.950429  4308 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.034657  4535 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:54.586457  4474 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.108297 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:55.097458  4551 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.157s	user 0.001s	sys 0.001s
W20260809 23:21:55.103958  4548 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.542s	user 0.000s	sys 0.002s
W20260809 23:21:55.100358  4556 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.488s	user 0.002s	sys 0.000s
I20260809 23:21:55.226069 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1621) took 2135 ms. Trace:
I20260809 23:21:55.096616  4530 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:55.226665  4483 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.227495 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3677) took 1207 ms. Trace:
I20260809 23:21:55.227658 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3682) took 1141 ms. Trace:
I20260809 23:21:55.228634 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3680) took 1143 ms. Trace:
I20260809 23:21:55.228121 24541 rpcz_store.cc:276] 0809 23:21:54.085912 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.086062 (+   150us) service_pool.cc:227] Handling call
0809 23:21:54.086360 (+   298us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.086727 (+   367us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.086753 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.120279 (+ 33526us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.413181 (+292902us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:54.413237 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.413384 (+   147us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.227447 (+814063us) raft_consensus.cc:1671] finished
0809 23:21:55.227465 (+    18us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.227597 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7906,"raft.queue_time_us":1605,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":8830,"spinlock_wait_cycles":1526016,"thread_start_us":269987,"threads_started":1,"wal-append.queue_time_us":467582}
I20260809 23:21:55.229139 24539 rpcz_store.cc:276] 0809 23:21:54.085272 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.085403 (+   131us) service_pool.cc:227] Handling call
0809 23:21:54.085738 (+   335us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.086073 (+   335us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.086105 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.086552 (+   447us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.900660 (+814108us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:54.900718 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.900807 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.228448 (+327641us) raft_consensus.cc:1671] finished
0809 23:21:55.228469 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.228583 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1109,"spinlock_wait_cycles":20095744,"thread_start_us":801444,"threads_started":1,"wal-append.queue_time_us":1075973}
I20260809 23:21:55.228338 24543 rpcz_store.cc:276] 0809 23:21:54.020099 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.020357 (+   258us) service_pool.cc:227] Handling call
0809 23:21:54.042621 (+ 22264us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.043057 (+   436us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.043086 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.080288 (+ 37202us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.268679 (+188391us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:54.268733 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.268825 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:54.916075 (+647250us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:55.227104 (+311029us) raft_consensus.cc:1671] finished
0809 23:21:55.227127 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.227427 (+   300us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6384,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":9752320,"thread_start_us":187930,"threads_started":1,"wal-append.queue_time_us":448378}
I20260809 23:21:55.098832  4343 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:55.125990 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:55.151324 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.034s	sys 0.034s
I20260809 23:21:55.136423 25363 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260809 23:21:55.235446 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.034s	sys 0.034s
W20260809 23:21:55.052376  4563 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260809 23:21:55.050803  4552 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.504s	user 0.000s	sys 0.001s
I20260809 23:21:55.154556 25365 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260809 23:21:55.239650 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 489890us
W20260809 23:21:55.051909  4564 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.178s	user 0.001s	sys 0.001s
I20260809 23:21:55.157626 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3747) took 1476 ms. Trace:
I20260809 23:21:55.118882  4109 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:54.592542  4527 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.169703 24270 leader_election.cc:398] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23.
W20260809 23:21:55.244765 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81018.3us
I20260809 23:21:55.240491 24540 rpcz_store.cc:276] 0809 23:21:53.681219 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.707216 (+ 25997us) service_pool.cc:227] Handling call
0809 23:21:53.707560 (+   344us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:53.707950 (+   390us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:53.707978 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:53.708410 (+   432us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.044124 (+335714us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:54.044673 (+   549us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.044753 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:54.905418 (+860665us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b1800556240. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:55.124276 (+218858us) raft_consensus.cc:1671] finished
0809 23:21:55.124313 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.157566 (+ 33253us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3375,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":57670912,"thread_start_us":307607,"threads_started":1,"wal-append.queue_time_us":507049}
I20260809 23:21:55.170574 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3772) took 1479 ms. Trace:
I20260809 23:21:55.183441 24498 leader_election.cc:398] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
W20260809 23:21:55.247519 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68619.3us
I20260809 23:21:55.248432 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.124615  3877 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.050978  4454 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.267390 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:55.268404 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1817) took 1022 ms. Trace:
I20260809 23:21:55.268705  4039 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.269352  4039 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.270603  4346 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25.
I20260809 23:21:55.268874 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:55.153796  3950 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.155726  3810 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.192521  4358 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.193009  4356 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:55.194160  4235 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.945s	user 0.012s	sys 0.000s
I20260809 23:21:55.187332  4332 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:55.246006 24429 rpcz_store.cc:276] 0809 23:21:53.691394 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.691559 (+   165us) service_pool.cc:227] Handling call
0809 23:21:53.717939 (+ 26380us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b5800004598. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:21:53.718118 (+   179us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:53.718498 (+   380us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:53.718859 (+   361us) log.cc:842] Serialized 14 byte log entry
0809 23:21:54.526096 (+807237us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.582259 (+ 56163us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.582678 (+   419us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:54.582744 (+    66us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.582837 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.152185 (+569348us) raft_consensus.cc:1671] finished
0809 23:21:55.170324 (+ 18139us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.170506 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28119,"prepare.queue_time_us":983,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":54,"raft.queue_time_us":53166,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":13794,"spinlock_wait_cycles":56543488,"thread_start_us":741258,"threads_started":2,"wal-append.queue_time_us":662734}
I20260809 23:21:55.196360  4408 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:55.198263 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.304430 24314 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:55.193109  4308 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.198349  4326 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.199801  4439 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.200708  4453 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:55.201494  4368 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:55.202164  3889 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.202214  4497 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.202911 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.203135  4572 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:55.204080 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.207376  4431 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.348258 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3756) took 1088 ms. Trace:
I20260809 23:21:55.348707 24534 rpcz_store.cc:276] 0809 23:21:54.260097 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.186423 (+926326us) service_pool.cc:227] Handling call
0809 23:21:55.348178 (+161755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.350023 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.207875  3892 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.352528 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.353130 24534 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:55.354056  4566 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:21:55.354278 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:55.354506  4566 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:55.354875  4566 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:55.355556 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.356187 24534 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:21:55.356607 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1818) took 1093 ms. Trace:
I20260809 23:21:55.356877 24534 rpcz_store.cc:276] 0809 23:21:54.263220 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.355300 (+1092080us) service_pool.cc:227] Handling call
0809 23:21:55.356561 (+  1261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.358006 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.187994  4535 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:55.359659  4535 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.361054  4535 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:55.211372 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3679) took 1126 ms. Trace:
I20260809 23:21:55.363469 24546 rpcz_store.cc:276] 0809 23:21:54.084911 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.087313 (+  2402us) service_pool.cc:227] Handling call
0809 23:21:54.130986 (+ 43673us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.141050 (+ 10064us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.141096 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.141643 (+   547us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.380694 (+239051us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:54.380731 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.380813 (+    82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.211137 (+830324us) raft_consensus.cc:1671] finished
0809 23:21:55.211159 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.211309 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9608,"raft.queue_time_us":11590,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":390,"thread_start_us":235697,"threads_started":1,"wal-append.queue_time_us":433428}
I20260809 23:21:55.366101 24546 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260809 23:21:55.211756  4474 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.212021  4571 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:55.213953 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.367476  4453 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:55.367995  4453 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:55.368708 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.369061  4453 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.366752 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1813) took 1226 ms. Trace:
I20260809 23:21:55.370220 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.189740  4491 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:55.219053 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.184199  4485 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:55.223158 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3084) took 1418 ms (client timeout 1626 ms). Trace:
I20260809 23:21:55.223547 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.191671 24320 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:21:55.235818 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 667848us
I20260809 23:21:55.226444 24423 rpcz_store.cc:276] 0809 23:21:53.090238 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.423525 (+333287us) service_pool.cc:227] Handling call
0809 23:21:53.423848 (+   323us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:53.790157 (+366309us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:53.790202 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:53.799158 (+  8956us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.227126 (+427968us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:54.227182 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.227249 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.225848 (+998599us) raft_consensus.cc:1671] finished
0809 23:21:55.225874 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.226010 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98579,"raft.queue_time_us":133565,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":7274,"thread_start_us":529714,"threads_started":2,"wal-append.queue_time_us":493087}
W20260809 23:21:55.375211 24313 rpcz_store.cc:269] 0809 23:21:53.804785 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.854876 (+ 50091us) service_pool.cc:227] Handling call
0809 23:21:55.223019 (+1368143us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400253fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:55.223110 (+    91us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1838567296}
I20260809 23:21:55.227658 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3672) took 1472 ms. Trace:
W20260809 23:21:55.120601  4532 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.002s
W20260809 23:21:55.241189 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.053s	sys 0.095s
W20260809 23:21:55.379189 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.053s	sys 0.096s
I20260809 23:21:55.242504 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.380455 24541 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:55.242815 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.381963 25365 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:55.383466 25365 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [22.15-23.16]
I20260809 23:21:55.243211 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.384546 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:55.243592 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3670) took 1516 ms. Trace:
I20260809 23:21:55.255112 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.127183  3154 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.259291 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.259907 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.391454 24539 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:55.391906 25369 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:55.261005  4444 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.261649 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.262722  4420 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.271551  4039 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.273571  4343 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:55.187958 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:55.275979  4567 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260809 23:21:55.268776 24543 rpcz_store.cc:276] 0809 23:21:54.246115 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.238715 (+992600us) service_pool.cc:227] Handling call
0809 23:21:55.268320 (+ 29605us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.190275  4579 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:55.109025  4445 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.238453  4581 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:55.294214 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.417886 24540 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:55.296407 24498 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.302184 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 30) took 2090 ms. Trace:
I20260809 23:21:55.419049 25372 rpcz_store.cc:276] 0809 23:21:53.199260 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.224254 (+ 24994us) service_pool.cc:227] Handling call
0809 23:21:55.289358 (+2065104us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:55.289504 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":358325504}
I20260809 23:21:55.419416 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.116374  4457 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.319564  3810 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.080s
I20260809 23:21:55.389051  4431 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:55.315461 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.316308 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.293677  4336 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:55.433971  4336 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.422343  4408 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.435333  4408 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.436110  4336 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.306402  4089 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.436786  4089 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.437359  4408 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.425169  4308 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:55.345578 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.439051 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1631) took 1172 ms. Trace:
I20260809 23:21:55.439251 24314 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:55.347033 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 143020us
I20260809 23:21:55.370585 24546 rpcz_store.cc:276] 0809 23:21:54.139934 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.365327 (+1225393us) service_pool.cc:227] Handling call
0809 23:21:55.365683 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.366486 (+   803us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.366693 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.443398  3154 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.376514 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.444773 24313 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:55.444911 24546 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:55.310118  4507 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:55.386324 24545 rpcz_store.cc:276] 0809 23:21:53.727457 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.739685 (+ 12228us) service_pool.cc:227] Handling call
0809 23:21:53.740024 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:53.935907 (+195883us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:53.935936 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:53.985588 (+ 49652us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.359249 (+373661us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:54.359294 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.222410 (+863116us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400148554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:55.222463 (+    53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.230085 (+  7622us) raft_consensus.cc:1671] finished
0809 23:21:55.230106 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.243523 (+ 13417us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":461,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7615,"spinlock_wait_cycles":441029504,"thread_start_us":337171,"threads_started":1,"wal-append.queue_time_us":756082}
I20260809 23:21:55.447836 24545 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260809 23:21:55.448171  4527 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.376072 24542 rpcz_store.cc:276] 0809 23:21:53.755273 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.755570 (+   297us) service_pool.cc:227] Handling call
0809 23:21:53.755884 (+   314us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:53.871581 (+115697us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:53.871614 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:53.872118 (+   504us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.131054 (+258936us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:54.131092 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.908437 (+777345us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:55.227378 (+318941us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:55.227444 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.227490 (+    46us) raft_consensus.cc:1671] finished
0809 23:21:55.227501 (+    11us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.227597 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1057,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":274602624,"thread_start_us":258501,"threads_started":1,"wal-append.queue_time_us":430290}
I20260809 23:21:55.450752 24542 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:55.452711 24314 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:55.293200  3950 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.453718 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.402293  3920 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.454299 25371 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260809 23:21:55.329079  4326 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:55.454677  3920 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.456187  4457 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:55.456944  3920 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.394637 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.407989  4579 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:55.459513 24311 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:21:55.408524  4444 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.460891 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.395190 24541 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:55.396158 25365 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:55.347718  4497 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.418051  4343 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.372272  4572 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:55.408751 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 147189us
I20260809 23:21:55.410128 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.410702 25369 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
W20260809 23:21:55.419950 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123647us
I20260809 23:21:55.310765 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.486198 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.462596  4444 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:55.315742  4516 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.084s
I20260809 23:21:55.285629  4375 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:21:55.426658 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:55.429886 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 114519us
I20260809 23:21:55.422708  4420 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:21:55.433311 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.436220  4039 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:55.438800  4089 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.443781 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.459831  4579 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:55.392426  4491 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:55.407126 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1819) took 1094 ms. Trace:
I20260809 23:21:55.301626  4235 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.396986  4474 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.439468 24431 rpcz_store.cc:276] 0809 23:21:54.266283 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.417108 (+150825us) service_pool.cc:227] Handling call
0809 23:21:54.417440 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.162494 (+745054us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.438775 (+276281us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b5400061154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:55.438985 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1705001344}
I20260809 23:21:55.554150 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:55.554370 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.463460 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3755) took 1203 ms. Trace:
I20260809 23:21:55.555585 24541 rpcz_store.cc:276] 0809 23:21:54.259762 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.381154 (+1121392us) service_pool.cc:227] Handling call
0809 23:21:55.381481 (+   327us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.463216 (+ 81735us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.463408 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.467135 24544 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:55.462878 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.561056 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:55.519747  4408 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
W20260809 23:21:55.423168 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.006s	sys 0.024s
W20260809 23:21:55.569509 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.007s	sys 0.024s
I20260809 23:21:55.454274 24314 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:55.504123 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:55.504200 24309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3686) took 2322 ms (client timeout 1509 ms). Trace:
I20260809 23:21:55.508487 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.431875 24316 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:21:55.516247 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.426875 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1820) took 1095 ms. Trace:
I20260809 23:21:55.574020 24543 rpcz_store.cc:276] 0809 23:21:54.331472 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.403041 (+1071569us) service_pool.cc:227] Handling call
0809 23:21:55.426386 (+ 23345us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.426563 (+   177us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.426586 (+    23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.426623 (+    37us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.426653 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.426713 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.426824 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:55.517850 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:55.524772 24313 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:55.577762 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.525089 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.429116  4431 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:55.454742  3950 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:55.532194 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88482.5us
I20260809 23:21:55.446482  4507 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.465605  4497 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.533520 24539 rpcz_store.cc:276] 0809 23:21:54.313038 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.387520 (+1074482us) service_pool.cc:227] Handling call
0809 23:21:55.391278 (+  3758us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.406793 (+ 15515us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.407053 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:21:55.517143  4584 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260809 23:21:55.421679  3889 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.101s
I20260809 23:21:55.398597  4430 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.557559 24311 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
I20260809 23:21:55.565586  4474 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.566442  4588 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:55.527469  4532 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:55.570708 24309 rpcz_store.cc:269] 0809 23:21:53.181300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.255208 (+ 73908us) service_pool.cc:227] Handling call
0809 23:21:55.491339 (+2236131us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:55.504124 (+ 12785us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1589723008}
I20260809 23:21:55.573287 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.581640 24424 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:55.546904  4530 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:21:55.584527 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76136.3us
I20260809 23:21:55.591785  4431 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:55.585423 24435 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260809 23:21:55.585309 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:55.588470 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63527.2us
I20260809 23:21:55.591255 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:55.596544 24423 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260809 23:21:55.572549  4527 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:55.608155 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:55.640589 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122833us
I20260809 23:21:55.641237 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.701787 24313 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:55.702647 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.642457 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.703433 24311 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:21:55.704201 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.704716 24311 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:21:55.705412 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.705926 24311 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:55.580796  3877 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:55.518865  4475 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
W20260809 23:21:55.712643 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.005s	sys 0.043s
I20260809 23:21:55.664029 24431 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260809 23:21:55.660694 24435 pending_rounds.cc:76] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 15
W20260809 23:21:55.713102 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.006s	sys 0.043s
I20260809 23:21:55.713264 24435 pending_rounds.cc:97] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 27.16
W20260809 23:21:55.713452 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 527182us
I20260809 23:21:55.713732 24435 raft_consensus.cc:2866] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:55.661044 24541 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 18.
I20260809 23:21:55.700897 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:55.665800  4359 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:55.661904 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:55.662352 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71185.2us
I20260809 23:21:55.715745 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:55.659569  4577 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260809 23:21:55.668334 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.700003  4507 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:55.721451 24423 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:55.669016 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:55.526242  4586 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.144s
I20260809 23:21:55.686261 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.687031  4491 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:55.599364  4396 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.006s	sys 0.028s
W20260809 23:21:55.689493 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3066) took 2734 ms (client timeout 2041 ms). Trace:
W20260809 23:21:55.730840 24321 rpcz_store.cc:269] 0809 23:21:52.955331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.731471 (+776140us) service_pool.cc:227] Handling call
0809 23:21:55.689298 (+1957827us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:55.689423 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381101440}
W20260809 23:21:55.731256 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62328us
I20260809 23:21:55.731947 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.690416 24317 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260809 23:21:55.691905 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.733438 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
W20260809 23:21:55.696786  4558 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.951s	user 0.008s	sys 0.000s
I20260809 23:21:55.427884  4581 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.706609 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.656082 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.720033 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:55.723671 24378 leader_election.cc:398] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 20 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21.
I20260809 23:21:55.746233 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.747063 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:55.598016  4459 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:21:55.604732  3950 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:55.599987  4408 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:55.709170  4528 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.053s	user 0.008s	sys 0.003s
I20260809 23:21:55.672061 24314 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:55.741251 24498 leader_election.cc:398] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:21:55.762830 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.763747 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.655020  4420 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:21:55.743012 24431 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260809 23:21:55.749264 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 87459.6us
I20260809 23:21:55.768908 24314 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:21:55.749241 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:55.758205 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:55.770987 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:55.730769  4396 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.006s	sys 0.028s
I20260809 23:21:55.723299  4359 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:55.726121  4530 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:21:55.785077 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.747936  4493 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:55.790904 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.791064 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:55.701113  4532 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.801101 24321 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:55.799854 24498 leader_election.cc:398] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:55.803078 24498 leader_election.cc:401] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:21:55.721546  4527 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.790529 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.724730  3877 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.806905 24498 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.774233 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:55.808679 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3332) took 1439 ms. Trace:
I20260809 23:21:55.810613 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:55.812269 24314 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:55.767505  4420 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:21:55.755982 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 401748us
I20260809 23:21:55.786911  4396 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:55.791596  4359 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:55.844066 24313 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:55.798455  4571 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:21:55.847805 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.850082 24321 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:55.853057 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.797722  3885 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:55.849083 24542 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
W20260809 23:21:55.855454 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48655.1us
I20260809 23:21:55.856004 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.856360 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.856658 24431 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
W20260809 23:21:55.863606 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89476us
I20260809 23:21:55.857504 24427 rpcz_store.cc:276] 0809 23:21:54.369090 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.499635 (+130545us) service_pool.cc:227] Handling call
0809 23:21:54.529506 (+ 29871us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.561952 (+ 32446us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.561989 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.581245 (+ 19256us) log.cc:842] Serialized 26 byte log entry
0809 23:21:55.379637 (+798392us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:55.792159 (+412522us) log.cc:842] Serialized 14 byte log entry
0809 23:21:55.808302 (+ 16143us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.808392 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.808451 (+    59us) raft_consensus.cc:1671] finished
0809 23:21:55.808468 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.808615 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2388,"raft.queue_time_us":55719,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":37702,"thread_start_us":1066564,"threads_started":2,"wal-append.queue_time_us":810151}
I20260809 23:21:55.798677  4454 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:55.809908  4530 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:55.809235  3889 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
W20260809 23:21:55.870090 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 282769us
I20260809 23:21:55.869781 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.904232 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.874166 24321 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:55.810366  4493 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:55.910056 24314 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260809 23:21:55.911859 24314 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:55.791440  4089 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260809 23:21:55.843175  4420 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:55.814165  4598 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:55.782519  4528 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:55.870175  4439 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:21:55.908110 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:55.882462  4454 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:55.906164 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44545.2us
I20260809 23:21:55.907521 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:55.863579  3885 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:55.911485 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:55.893095 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 173159us
I20260809 23:21:55.867102  4343 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:55.872674  3892 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:21:55.952119 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46907.2us
I20260809 23:21:55.915480 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1932) took 1834 ms. Trace:
W20260809 23:21:55.920091  3932 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.014s	sys 0.028s
I20260809 23:21:55.952905 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:55.905431  4530 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:55.966698 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.908947 24435 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
W20260809 23:21:55.916908  4561 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:55.903482  4438 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:21:56.009708 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56923us
I20260809 23:21:55.972996 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:55.860004  4400 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.098s
I20260809 23:21:55.903764  4582 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:55.912750  4375 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:55.903261  4235 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.895761  4444 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260809 23:21:55.975052 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:55.953171  4507 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:21:55.986639 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:55.908352  4572 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:55.960634 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:55.895617  4579 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:55.981873 25368 rpcz_store.cc:276] 0809 23:21:54.080701 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.080858 (+   157us) service_pool.cc:227] Handling call
0809 23:21:54.081235 (+   377us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.081615 (+   380us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.081644 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.082435 (+   791us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.425103 (+342668us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:54.425159 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.425236 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:55.695858 (+1270622us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:55.905621 (+209763us) raft_consensus.cc:1671] finished
0809 23:21:55.905642 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:55.915397 (+  9755us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1681,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":28497280,"thread_start_us":342297,"threads_started":1,"wal-append.queue_time_us":529305}
W20260809 23:21:55.987649 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 194612us
I20260809 23:21:56.008324 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.057132 24314 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:21:56.057744 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71177.7us
I20260809 23:21:56.058130 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:55.976639  4454 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:55.992755  3885 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:56.019446 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3684) took 1874 ms (client timeout 1571 ms). Trace:
W20260809 23:21:56.062327 24535 rpcz_store.cc:269] 0809 23:21:54.144534 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.159692 (+ 15158us) service_pool.cc:227] Handling call
0809 23:21:56.019214 (+1859522us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54001d6854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:56.019362 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":389619968}
W20260809 23:21:56.056869 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81915.7us
W20260809 23:21:55.986245  3932 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.017s	sys 0.043s
I20260809 23:21:56.063597 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.068424 24539 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260809 23:21:56.025483 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52595.8us
I20260809 23:21:56.070277 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:55.985577  3892 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:56.027678  3889 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:21:56.032150  4592 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260809 23:21:55.993624  4530 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:56.083559 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:56.084615 24542 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:56.050637 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 76709.6us
I20260809 23:21:55.896580  4408 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.054522 24321 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:21:56.055344 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44757.4us
I20260809 23:21:56.086545 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.087992 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.058765 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.053468  4400 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:21:56.065366 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 157923us
I20260809 23:21:56.072799 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.094218 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.094612 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:56.075273 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.095744 25325 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:55.969538  4598 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:56.067788  4543 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260809 23:21:56.089846 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:56.010030  4251 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:56.066748  4572 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:56.066684  4475 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:56.098861 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:56.059588  4474 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.077921  4235 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:56.074278  4303 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:56.105705 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.078344  3892 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:56.067130  4639 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:55.984649  4396 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.108304 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:56.109598 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.066574  4308 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.026s
I20260809 23:21:56.113147 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.079553  3889 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:56.077754  4485 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260809 23:21:56.092003  4566 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.737s	user 0.012s	sys 0.000s
I20260809 23:21:56.151527 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.084013  4581 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:56.127379 24427 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260809 23:21:56.130887  4535 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.769s	user 0.013s	sys 0.003s
I20260809 23:21:56.120980  4400 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:56.098748  4445 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:56.101099  3932 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:56.134021 24539 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:56.134747 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:56.108093  4375 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:56.108304  3885 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:56.113633 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.152711 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:56.062958  4332 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20260809 23:21:56.123082  4344 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:21:56.153764 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82244us
I20260809 23:21:56.151935  3889 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:56.155144  4566 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:56.157371  4608 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260809 23:21:56.158457  4616 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260809 23:21:56.053313  4493 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.121685  4475 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:56.115188  4532 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.123096  4089 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
W20260809 23:21:56.161967 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66361.5us
I20260809 23:21:56.225503 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:56.163200 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53667us
I20260809 23:21:56.227685 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:56.229383 24498 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.172716 24431 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:56.133374  4527 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.130599  4303 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:56.175944 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96092.4us
I20260809 23:21:56.155665  4641 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260809 23:21:56.112522  4366 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.048s
W20260809 23:21:56.160843  4326 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.705s	user 0.011s	sys 0.000s
W20260809 23:21:56.171154  4593 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260809 23:21:56.414494 24540 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:56.197816  4579 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:56.196774  4618 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260809 23:21:56.197798 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:56.416338 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 218607us
I20260809 23:21:56.198568  4611 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
I20260809 23:21:56.198990  4598 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:56.182278  4599 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260809 23:21:56.204880 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.133814  4459 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260809 23:21:56.178982  4581 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:56.430078 24314 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:56.211599 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:56.170442  4445 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:56.432413 24539 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:21:56.134238  4235 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.182349  4375 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:56.203707  4595 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260809 23:21:56.236652 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3688) took 1073 ms. Trace:
W20260809 23:21:56.209738  4559 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.102s	user 0.005s	sys 0.000s
W20260809 23:21:56.380821 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 153880us
I20260809 23:21:56.414011  4332 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:21:56.187337  4400 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:56.382130 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 285585us
W20260809 23:21:56.411499 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 316141us
I20260809 23:21:56.178133  4439 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260809 23:21:56.204456  4528 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:56.215803  4475 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:56.344388  4278 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.023s	sys 0.019s
I20260809 23:21:56.239408  4568 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:56.413800  4336 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:56.205641  4535 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.223s
I20260809 23:21:56.445320 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:56.451812 24311 rpcz_store.cc:276] 0809 23:21:55.162773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.759706 (+596933us) service_pool.cc:227] Handling call
0809 23:21:55.759971 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.760170 (+   199us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:21:55.857314 (+ 97144us) log.cc:842] Serialized 14 byte log entry
0809 23:21:56.236184 (+378870us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:56.236256 (+    72us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:56.236336 (+    80us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:56.236423 (+    87us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:56.236568 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":724,"raft.queue_time_us":623,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":34311,"thread_start_us":474672,"threads_started":2}
I20260809 23:21:56.476200 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:56.414842  4089 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:56.447685  4491 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.489528 24427 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:56.457813  4641 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.419112  4396 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:56.495172 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:56.434345  4445 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:56.488915 24541 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:56.458593  4375 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:56.538806 25368 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:56.527499 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.429382  4532 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:21:56.464978  4400 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:21:56.485985  4278 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.027s	sys 0.023s
I20260809 23:21:56.533190 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:56.547417 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 116067us
I20260809 23:21:56.548418 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:56.561121 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:21:56.515076  4541 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260809 23:21:56.489225  4581 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:56.508876  4336 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:56.547369 24539 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:21:56.495944  4528 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.531301  4600 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:21:56.617455 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 378484us
I20260809 23:21:56.686249 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1936) took 1572 ms. Trace:
I20260809 23:21:56.693195 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.693370 25370 rpcz_store.cc:276] 0809 23:21:55.113918 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.114103 (+   185us) service_pool.cc:227] Handling call
0809 23:21:55.150314 (+ 36211us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:55.150738 (+   424us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.150777 (+    39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:55.157310 (+  6533us) log.cc:842] Serialized 26 byte log entry
0809 23:21:55.513194 (+355884us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.513237 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.513309 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:56.686044 (+1172735us) raft_consensus.cc:1671] finished
0809 23:21:56.686065 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:56.686192 (+   127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":49583,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":34560,"thread_start_us":312832,"threads_started":1,"wal-append.queue_time_us":668394}
I20260809 23:21:56.695670 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.695911 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:56.617043 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 142298us
I20260809 23:21:56.618356 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:56.569407  4400 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:56.559177  4532 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:56.620872 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:56.621685 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68063.4us
I20260809 23:21:56.548942  3950 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.537602  4587 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:56.677618 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:56.495069  4568 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:56.671821 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1908) took 2061 ms. Trace:
I20260809 23:21:56.549177  4535 consensus_queue.cc:1225] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:56.684566  4485 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:56.684746 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.027s	sys 0.055s
I20260809 23:21:56.560794 24543 raft_consensus.cc:2362] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27. Re-sending same reply.
I20260809 23:21:56.697695 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:56.742703 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 65203.7us
I20260809 23:21:56.620456 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:56.696744 25370 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:21:56.705106  4640 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.616791  4344 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.558439  4641 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:56.725067  3950 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.726974  4664 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:21:56.727591  4587 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:56.729576 24541 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260809 23:21:56.615099  4586 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.120s
I20260809 23:21:56.698881  4582 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:56.674693  4655 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260809 23:21:56.739399 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 249391us
I20260809 23:21:56.681318  4336 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:56.737891 24323 rpcz_store.cc:276] 0809 23:21:54.610541 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.102518 (+491977us) service_pool.cc:227] Handling call
0809 23:21:55.116905 (+ 14387us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:55.131029 (+ 14124us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.131076 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:55.162467 (+ 31391us) log.cc:842] Serialized 26 byte log entry
0809 23:21:55.593695 (+431228us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.593737 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.593805 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:56.629930 (+1036125us) raft_consensus.cc:1671] finished
0809 23:21:56.671582 (+ 41652us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:56.671757 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3300,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":23875,"thread_start_us":373170,"threads_started":1,"wal-append.queue_time_us":608160}
W20260809 23:21:56.745782 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.027s	sys 0.055s
W20260809 23:21:56.767050  4558 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260809 23:21:56.550338  4491 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.691202  4600 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:56.756512 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:56.736600  4109 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.456s	user 0.014s	sys 0.000s
I20260809 23:21:56.548085  4639 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.196s
I20260809 23:21:56.765326 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:56.767544 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:56.768599 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44012.5us
I20260809 23:21:56.736776  4611 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:56.698084  4528 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.736461  4649 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260809 23:21:56.803928 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.781483 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:56.736126  4457 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.279s	user 0.012s	sys 0.000s
I20260809 23:21:56.764326  4459 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.028s
I20260809 23:21:56.567008  4303 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.764575  4566 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.794570 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.750773  4326 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260809 23:21:56.801846 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:56.782862 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:56.801215 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:56.793124  4420 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.826s	user 0.009s	sys 0.003s
I20260809 23:21:56.759488  4485 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:56.815956 24427 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260809 23:21:56.796927  4600 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:56.764052  4344 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.806151  4535 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:21:56.828583  4672 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260809 23:21:56.807945  4653 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.828778  4420 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.792208  4568 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.803421  4611 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:56.838371 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:56.821085  4620 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.405s	user 0.000s	sys 0.001s
W20260809 23:21:56.869227  3889 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.666s	user 0.003s	sys 0.006s
W20260809 23:21:56.869760  3892 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.710s	user 0.011s	sys 0.000s
I20260809 23:21:56.870239  3950 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.808292  4655 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:56.871447 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48918us
I20260809 23:21:56.763705  4089 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:56.819577  4527 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:56.876444 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:56.832646  4665 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
W20260809 23:21:56.835556  4671 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260809 23:21:56.830852  4648 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260809 23:21:56.831707  4666 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260809 23:21:56.884281 25370 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:56.827056  4249 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:56.760350  4640 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:56.832449  4652 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:21:56.777254  4532 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:56.869760  4572 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.747s	user 0.004s	sys 0.004s
I20260809 23:21:56.786634  4581 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:56.832113  4641 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:56.837817  4235 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.833668  4675 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:21:56.833237  4681 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:21:56.877743  4566 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:56.912452 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 129671us
W20260809 23:21:56.941603 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 380462us
I20260809 23:21:56.984020 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:56.899936  4474 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:56.909976  3885 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.003s
I20260809 23:21:56.908901  4485 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:56.990873 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1948) took 1702 ms. Trace:
I20260809 23:21:56.874390  4535 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:56.958751 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 155965us
I20260809 23:21:56.909260  4528 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.001461 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:56.971392 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:56.910874  4657 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:57.005081  4657 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.005649  4657 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:56.992537  4474 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.006842  4579 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
W20260809 23:21:57.007233 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1910) took 1754 ms (client timeout 1958 ms). Trace:
I20260809 23:21:57.007737  4474 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:57.007651 24313 rpcz_store.cc:269] 0809 23:21:55.253114 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.702386 (+449272us) service_pool.cc:227] Handling call
0809 23:21:57.007040 (+1304654us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.007180 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":323493632}
I20260809 23:21:56.910200  4655 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.009040 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:57.009644 24313 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:21:57.010013  4655 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.894300  3889 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.010691 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.977883  4249 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:56.969586  4652 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:57.011881 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.011744  4249 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.012439 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3337) took 1764 ms. Trace:
I20260809 23:21:57.012328  4652 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.012874 24424 rpcz_store.cc:276] 0809 23:21:55.247589 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.251874 (+  4285us) service_pool.cc:227] Handling call
0809 23:21:55.252207 (+   333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.662784 (+410577us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.012031 (+1349247us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.012336 (+   305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":756832640}
I20260809 23:21:57.013739  4652 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:57.003998  4628 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260809 23:21:57.015571 24423 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:56.992164 25365 rpcz_store.cc:276] 0809 23:21:55.288750 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.465071 (+176321us) service_pool.cc:227] Handling call
0809 23:21:55.465400 (+   329us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:55.465736 (+   336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.465764 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:55.466195 (+   431us) log.cc:842] Serialized 26 byte log entry
0809 23:21:55.858961 (+392766us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:55.859017 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:55.859103 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:56.990589 (+1131486us) raft_consensus.cc:1671] finished
0809 23:21:56.990620 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:56.990801 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4810,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":21153,"thread_start_us":392363,"threads_started":1,"wal-append.queue_time_us":717220}
I20260809 23:21:57.017108 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.017603 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.017899 25365 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:57.018253 24423 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:57.000115  4459 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260809 23:21:57.007257  4579 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:57.021178  4579 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.022259  4579 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:57.023483  4686 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:21:57.024255  4686 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:57.025267 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.015676 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1853) took 1776 ms. Trace:
I20260809 23:21:57.025198  4686 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.025978 25371 rpcz_store.cc:276] 0809 23:21:55.238811 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.455070 (+216259us) service_pool.cc:227] Handling call
0809 23:21:55.456141 (+  1071us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.518911 (+ 62770us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.518959 (+    48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.519002 (+    43us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.519032 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.015423 (+1496391us) spinlock_profiling.cc:243] Waited 62.6 ms on lock 0x7b540073e454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.015478 (+    55us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.015615 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":143906688}
I20260809 23:21:57.027345 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.027931 25371 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:57.028676 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:56.989344  4566 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.029985 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:21:57.030689  4624 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.868s	user 0.000s	sys 0.002s
I20260809 23:21:56.880234  4653 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.032822  4344 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:56.983896  4572 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:21:57.033557  4636 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.960s	user 0.001s	sys 0.000s
I20260809 23:21:57.035986 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3761) took 2240 ms. Trace:
I20260809 23:21:57.036424 24540 rpcz_store.cc:276] 0809 23:21:54.795607 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.440709 (+645102us) service_pool.cc:227] Handling call
0809 23:21:55.440993 (+   284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:56.419240 (+978247us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.035702 (+616462us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.035916 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2238294272}
I20260809 23:21:56.901935  3892 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:56.991309  4431 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.263s	user 0.010s	sys 0.004s
I20260809 23:21:57.005115  4535 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:57.049480  4530 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.950s	user 0.006s	sys 0.005s
I20260809 23:21:57.042364 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:57.018716 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.050586 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:57.052992  4574 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.923s	user 0.007s	sys 0.000s
I20260809 23:21:57.053671 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3344) took 1750 ms. Trace:
I20260809 23:21:57.054603 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3701) took 1387 ms. Trace:
I20260809 23:21:57.082453 24429 rpcz_store.cc:276] 0809 23:21:55.303677 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.580987 (+277310us) service_pool.cc:227] Handling call
0809 23:21:55.642148 (+ 61161us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.716894 (+ 74746us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.716941 (+    47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.716997 (+    56us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.717042 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.053434 (+1336392us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54003eb2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.053491 (+    57us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.053615 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":171330432}
I20260809 23:21:57.083236 24542 rpcz_store.cc:276] 0809 23:21:55.667377 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.987223 (+319846us) service_pool.cc:227] Handling call
0809 23:21:55.987582 (+   359us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:56.090329 (+102747us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.054348 (+964019us) spinlock_profiling.cc:243] Waited 96.8 ms on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.054538 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":222560000}
I20260809 23:21:57.056269  4251 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:57.057582 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1949) took 1542 ms. Trace:
I20260809 23:21:57.000820  4675 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:56.965095  4332 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:56.917834  4527 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.047245 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.093420 25375 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:56.988852  4235 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:21:57.079629 24424 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:57.079819 25370 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:57.080401 24313 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
W20260809 23:21:57.102093 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 100726us
I20260809 23:21:57.102780 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:57.079021 25365 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:57.091462 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.085494 25369 rpcz_store.cc:276] 0809 23:21:55.515395 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.516845 (+  1450us) service_pool.cc:227] Handling call
0809 23:21:55.523410 (+  6565us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.536079 (+ 12669us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.536131 (+    52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.536196 (+    65us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:55.536255 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.057336 (+1521081us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.057390 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.057518 (+   128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":863,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":13487,"spinlock_wait_cycles":11818752}
I20260809 23:21:57.104753 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:57.106714 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.099325 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.108040 24429 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:57.060784  3877 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.109481 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:21:57.108361 24542 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:57.112401 24424 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:21:57.106889  4377 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:21:57.116674 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:57.121909 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:57.089032  4251 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:57.128069 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:57.128280 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3336) took 1881 ms. Trace:
I20260809 23:21:57.094395  4459 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:57.131578 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.131692 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:57.132932 24425 rpcz_store.cc:276] 0809 23:21:55.247301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.251433 (+  4132us) service_pool.cc:227] Handling call
0809 23:21:55.251713 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.292522 (+ 40809us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.292567 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.292620 (+    53us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:55.292661 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.128052 (+1835391us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b5400062554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.128099 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.128228 (+   129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":93325184}
I20260809 23:21:57.143769 24425 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260809 23:21:57.143014 24542 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260809 23:21:57.130501 25369 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:57.134714  4420 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:57.136229  4454 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.058s	user 0.010s	sys 0.006s
I20260809 23:21:57.149675 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:57.136870  4516 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:57.137369 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:57.139096 24539 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 16. (term mismatch)
I20260809 23:21:57.156328 24539 pending_rounds.cc:76] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 15
I20260809 23:21:57.156635 24539 pending_rounds.cc:97] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260809 23:21:57.156919 24539 raft_consensus.cc:2866] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:21:57.142683 24429 raft_consensus.cc:2396] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260809 23:21:57.136612  3892 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.158309 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.158944 25369 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:21:57.082913  4431 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.163938 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:57.159415 24423 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:21:57.166610 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.169018 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:57.119637  4332 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:57.196698 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3314) took 4438 ms (client timeout 2879 ms). Trace:
I20260809 23:21:57.132001  4140 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
W20260809 23:21:57.197113 24428 rpcz_store.cc:269] 0809 23:21:52.758612 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:52.821614 (+ 63002us) service_pool.cc:227] Handling call
0809 23:21:57.196370 (+4374756us) spinlock_profiling.cc:243] Waited 3.24 s on lock 0x7b5400061654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.196522 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7457201536}
I20260809 23:21:56.993139  4611 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.202116 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1812) took 3069 ms. Trace:
I20260809 23:21:57.202642 24534 rpcz_store.cc:276] 0809 23:21:54.132282 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.359145 (+1226863us) service_pool.cc:227] Handling call
0809 23:21:55.359513 (+   368us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.443297 (+ 83784us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.443329 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.443368 (+    39us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:55.443400 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.201894 (+1758494us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.201944 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.202057 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":192230144}
I20260809 23:21:57.186865  4639 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:21:57.182777  4377 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260809 23:21:57.163168  4454 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.188802  3979 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:57.194351 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1640) took 1881 ms. Trace:
I20260809 23:21:57.140031  4459 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:57.138700  4491 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.183969 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:57.084975  4530 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.206408 24422 rpcz_store.cc:276] 0809 23:21:55.315019 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.733984 (+418965us) service_pool.cc:227] Handling call
0809 23:21:55.758511 (+ 24527us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.860302 (+101791us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.860335 (+    33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.860381 (+    46us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:55.860417 (+    36us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.194128 (+1333711us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.194187 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.194298 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":233650432}
I20260809 23:21:57.155851  4516 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.162223  4566 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.223074 24314 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:21:57.223556 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.223227 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1788) took 2969 ms. Trace:
I20260809 23:21:57.228881 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:57.231079 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3088) took 2954 ms (client timeout 1919 ms). Trace:
W20260809 23:21:57.231799 24316 rpcz_store.cc:269] 0809 23:21:54.276489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.276820 (+   331us) service_pool.cc:227] Handling call
0809 23:21:57.230689 (+2953869us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.230900 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2366867328}
I20260809 23:21:57.213063  4459 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:57.208909  4639 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.206877  4640 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.144455  4699 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.097s
I20260809 23:21:57.254789 24498 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:57.237320  4667 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:57.256546 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.244822 24430 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260809 23:21:57.256628 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1907) took 2650 ms. Trace:
I20260809 23:21:57.223858  4516 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:21:57.239259  4359 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.367s	user 0.015s	sys 0.000s
I20260809 23:21:57.165299  4704 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260809 23:21:57.221215  4528 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:57.227201 25366 rpcz_store.cc:276] 0809 23:21:54.246280 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.246393 (+   113us) service_pool.cc:227] Handling call
0809 23:21:54.273715 (+ 27322us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:54.311537 (+ 37822us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:54.311587 (+    50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:54.339922 (+ 28335us) log.cc:842] Serialized 26 byte log entry
0809 23:21:54.883058 (+543136us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:54.883130 (+    72us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:54.883237 (+   107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:57.052295 (+2169058us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:57.210587 (+158292us) raft_consensus.cc:1671] finished
0809 23:21:57.210627 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.215540 (+  4913us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37208,"raft.queue_time_us":52582,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":6986496,"thread_start_us":404616,"threads_started":1,"wal-append.queue_time_us":703330}
I20260809 23:21:57.225538  4707 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:21:57.229610  4249 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.242551  4377 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:57.354079 24323 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:57.239022  4491 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:57.254595  4445 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.645s	user 0.012s	sys 0.000s
I20260809 23:21:57.291440 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:57.263818  4493 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:21:57.263607  4667 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.302556 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:57.263437  4431 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.312268  4530 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.357280  3154 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:57.383278  3154 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:57.374686  4667 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:21:57.315121  4698 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260809 23:21:57.257809  4659 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:57.323727 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:57.342671  4528 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:57.328958  4475 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.820s	user 0.011s	sys 0.000s
I20260809 23:21:57.211464  4675 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:57.330435  4336 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.536s	user 0.008s	sys 0.000s
I20260809 23:21:57.336735  4657 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.339475  4454 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.340477 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:21:57.347489 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1903) took 2761 ms (client timeout 1710 ms). Trace:
I20260809 23:21:57.410494 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3764) took 2215 ms. Trace:
W20260809 23:21:57.410701 24320 rpcz_store.cc:269] 0809 23:21:54.586463 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:54.596372 (+  9909us) service_pool.cc:227] Handling call
0809 23:21:57.347282 (+2750910us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54003a43d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.347430 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1040331008}
I20260809 23:21:57.410945 24545 rpcz_store.cc:276] 0809 23:21:55.195344 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.448547 (+253203us) service_pool.cc:227] Handling call
0809 23:21:55.448812 (+   265us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.660514 (+211702us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.660779 (+   265us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.660884 (+   105us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:55.660977 (+    93us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.410151 (+1749174us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:57.410251 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.410421 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":486289280}
I20260809 23:21:57.287660 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:57.351228 24544 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1821) took 2404 ms (client timeout 1958 ms). Trace:
W20260809 23:21:57.412535 24544 rpcz_store.cc:269] 0809 23:21:54.947240 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.223275 (+276035us) service_pool.cc:227] Handling call
0809 23:21:57.350920 (+2127645us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.351143 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210116992}
I20260809 23:21:57.259984  4639 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:57.347321  4359 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.238716  3979 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.115s
W20260809 23:21:57.355748  4375 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.746s	user 0.009s	sys 0.008s
W20260809 23:21:57.374472 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 179328us
I20260809 23:21:57.281680  4710 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.068s
I20260809 23:21:57.409515 24541 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:57.367964 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:21:57.416602 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 129053us
I20260809 23:21:57.422111 24316 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
W20260809 23:21:57.423048 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57608.1us
I20260809 23:21:57.400321  4659 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.427111  4724 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:57.427593  4724 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.427958  4724 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:57.404637  4377 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:57.431397  4377 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.433538  4377 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.421980  4375 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.434787  4660 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:21:57.436969  4375 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:57.437237 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 740271us
I20260809 23:21:57.420354  4454 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:57.437917  4598 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:21:57.438246 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:57.438894 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 663809us
I20260809 23:21:57.409334  4336 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.439193  4651 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260809 23:21:57.440578 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 659100us
I20260809 23:21:57.440985  4336 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.441622 24535 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:21:57.442031  4147 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:57.407743  4475 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.442639  4147 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:21:57.443635  4457 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:57.443920  4475 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.444108  4408 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:57.444608 24321 raft_consensus.cc:2354] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:21:57.444576  4408 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:57.445142  4420 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.446062  3892 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.445693  4408 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.446525  3892 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.367195  4491 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.448437 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:57.448796  3892 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.448980  4408 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:21:57.449265  4491 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.445727  4420 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.451396  4359 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.448599 24424 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:21:57.451993  4420 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:57.452762 24540 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:21:57.453061 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.455792 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:57.430773  4527 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.449203 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:57.461339 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.465622  4657 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.444139  4457 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.481616  4457 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:57.488793  4600 consensus_meta.cc:287] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.611s	user 0.008s	sys 0.002s
I20260809 23:21:57.489243  4600 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.490293 25370 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:21:57.491572  4660 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:21:57.491693 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53553.6us
I20260809 23:21:57.493258 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:57.495002  4408 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:57.495494  4408 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:21:57.496376  4408 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.496992  4598 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.497488  4598 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.499879  4598 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:57.467420  4528 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.370075  4445 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.468111 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3083) took 3664 ms. Trace:
I20260809 23:21:57.470196 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:57.513262 24543 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:57.513818 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:21:57.512809 24317 rpcz_store.cc:276] 0809 23:21:53.803530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:53.867480 (+ 63950us) service_pool.cc:227] Handling call
0809 23:21:55.690083 (+1822603us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.733003 (+ 42920us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.467879 (+1734876us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b5400252bd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:57.468046 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4190751104}
I20260809 23:21:57.475423 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:57.515810 24424 raft_consensus.cc:2354] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:57.528271 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.530100 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:57.534097 24496 leader_election.cc:398] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:57.534539 24496 leader_election.cc:401] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260809 23:21:57.537989 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:57.541499 24544 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:57.542215 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:57.512974  4400 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.761s	user 0.014s	sys 0.002s
I20260809 23:21:57.544989 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:57.546229 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.550149 24424 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:21:57.557921 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:57.559240 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.552738 24543 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:57.413194  4251 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.564512 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.562280 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:21:57.557214 24540 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:57.562650 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:57.570765 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.532490  4527 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:57.539530  4420 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:57.578042  4704 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:21:57.470767  4516 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.559805 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.546738  4431 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:21:57.580559 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.581466 24429 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:21:57.590073 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.590919 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:57.559677  3932 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:57.580343 25369 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:57.596081 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:57.601997 24424 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:57.589700  4454 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.603488  4454 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.603780 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.605926  4454 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.569168  4332 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.038s
I20260809 23:21:57.607683  4659 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.568552  4530 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:21:57.560585  4400 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.563437  4528 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:57.615244 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.616072 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:57.621905 25369 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:57.602126 24317 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:57.625231 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:57.572041  4493 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:21:57.626184 24424 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:57.627051 24313 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:21:57.632933 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 190436us
I20260809 23:21:57.592435  3920 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:57.585773  4431 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.669749 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:57.672318 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:57.660573 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.676560 25370 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:57.612661  3932 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:57.667824 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.678243 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:57.679955 25369 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:21:57.681629 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:21:57.668542 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119720us
I20260809 23:21:57.602833  4706 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:57.682591 25369 raft_consensus.cc:2354] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
I20260809 23:21:57.635300  4572 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:57.687734 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.689256 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.635409  4659 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.590575  4420 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.890780 24498 leader_election.cc:398] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
W20260809 23:21:57.900393 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 185170us
I20260809 23:21:57.901060 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.630981  4586 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
W20260809 23:21:57.736260 24378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.725s (SENT) [suppressed 16 similar messages]
I20260809 23:21:57.890048 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.890533 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:57.891673 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:57.692343  4439 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:21:57.896131  4706 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:57.663293  4528 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.902894 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:57.938902 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:57.692760  4485 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.963383  4485 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.966295  4528 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:57.664315  4660 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:21:57.891585 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:57.692543  4704 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:57.697489  3920 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:57.902719  4527 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:21:57.938867 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:57.693632  4431 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:57.943892 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:57.944162  4586 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:21:57.692502  4438 consensus_queue.cc:1046] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.253s
I20260809 23:21:57.913817  4377 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:21:57.945389 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:57.946411  4368 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:57.947039  4109 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:21:57.956488  4366 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:57.960202  4445 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:57.934170  4707 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:57.906435  4659 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:57.990155 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:57.891458 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 202323us
I20260809 23:21:57.674342  4710 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.269s
I20260809 23:21:57.969199  4528 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.124524  4572 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.053298  4704 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:57.690673  3932 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.179741 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.182056 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:58.182885 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.072609  4600 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:58.186872  4731 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.494s	user 0.000s	sys 0.000s
I20260809 23:21:58.198493 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:58.208623  3920 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.166451  4439 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:58.139495  4586 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:21:58.056820  4408 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.027s	sys 0.020s
I20260809 23:21:57.687696  4651 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.501s
I20260809 23:21:58.370339  4704 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:21:58.413655  4724 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.985s	user 0.006s	sys 0.006s
I20260809 23:21:58.413133  4516 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.464413 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.404865  4528 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:58.177301  4109 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:58.367646 25372 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:21:58.184753  4485 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.403435 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.409350 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:57.693890  4664 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.518s
I20260809 23:21:58.183318  4368 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.462342  4530 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.426244 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:21:58.291314  4667 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.904s	user 0.012s	sys 0.000s
I20260809 23:21:58.428293 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3710) took 1432 ms. Trace:
I20260809 23:21:58.289840  4572 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:58.187681  4366 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.398416  4251 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.501694 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:21:58.429404  4408 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.028s	sys 0.020s
I20260809 23:21:58.427198  4600 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:57.669442  4400 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:21:58.203179  4707 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.512382 24543 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:21:58.511866 25370 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:21:58.425118  4639 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.003s	user 0.008s	sys 0.004s
I20260809 23:21:58.520592  4368 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.522821 25372 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260809 23:21:58.523641  4109 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.526837 24323 rpcz_store.cc:276] 0809 23:21:56.995865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.327454 (+331589us) service_pool.cc:227] Handling call
0809 23:21:57.327806 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.354567 (+ 26761us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.428039 (+1073472us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.428227 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48731008}
I20260809 23:21:58.526741 25369 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:58.562896 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:21:58.515640  4736 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260809 23:21:58.199106  4706 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.429582  4439 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.563866 24317 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
W20260809 23:21:58.591960 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.000s	sys 0.018s
I20260809 23:21:58.566978 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.592440 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3720) took 1025 ms. Trace:
W20260809 23:21:58.592896 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.000s	sys 0.018s
I20260809 23:21:58.510710  4528 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:58.593271 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 856979us
I20260809 23:21:58.595098  4528 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:58.572853 25366 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.952s	user 0.000s	sys 0.018s
W20260809 23:21:58.596338 25366 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.000s	sys 0.018s
I20260809 23:21:58.592921 24317 rpcz_store.cc:276] 0809 23:21:57.567117 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.540128 (+973011us) service_pool.cc:227] Handling call
0809 23:21:58.592379 (+ 52251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.597594 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.598346 24317 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:58.607966  4642 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 2.456s	user 0.019s	sys 0.034s
I20260809 23:21:58.609771 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3690) took 3446 ms. Trace:
I20260809 23:21:58.610199 24309 rpcz_store.cc:276] 0809 23:21:55.163422 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.767861 (+604439us) service_pool.cc:227] Handling call
0809 23:21:55.768219 (+   358us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:55.768471 (+   252us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.768506 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:55.787205 (+ 18699us) log.cc:842] Serialized 26 byte log entry
0809 23:21:56.002585 (+215380us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:56.002635 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:56.002729 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:58.607288 (+2604559us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:58.609400 (+  2112us) raft_consensus.cc:1671] finished
0809 23:21:58.609428 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.609700 (+   272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45196032,"thread_start_us":159033,"threads_started":1,"wal-append.queue_time_us":295134}
I20260809 23:21:57.258509 24312 rpcz_store.cc:276] 0809 23:21:54.606316 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.076466 (+470150us) service_pool.cc:227] Handling call
0809 23:21:55.076797 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.077019 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:21:55.549663 (+472644us) log.cc:842] Serialized 14 byte log entry
0809 23:21:57.030026 (+1480363us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:57.244111 (+214085us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b180020d460. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:21:57.256088 (+ 11977us) log.cc:842] Serialized 14 byte log entry
0809 23:21:57.256285 (+   197us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:57.256338 (+    53us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:21:57.256396 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.256456 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:57.256572 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44301,"prepare.queue_time_us":732621,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":4142,"raft.queue_time_us":774,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":24530,"spinlock_wait_cycles":53977216,"thread_start_us":763699,"threads_started":2,"wal-append.queue_time_us":571861}
I20260809 23:21:58.612207 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.613166 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:21:58.500349  4724 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:58.548545  4729 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.641s	user 0.001s	sys 0.000s
I20260809 23:21:58.592519 24323 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.532608  4667 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.576089  4683 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:58.614607 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3113) took 1793 ms. Trace:
I20260809 23:21:58.636234 24316 rpcz_store.cc:276] 0809 23:21:56.821292 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.233121 (+411829us) service_pool.cc:227] Handling call
0809 23:21:57.233521 (+   400us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.422671 (+189150us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.614327 (+1191656us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540043b7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.614548 (+   221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433191296}
I20260809 23:21:58.637789 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.638537 24316 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:58.578342  4614 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.569s	user 0.008s	sys 0.018s
W20260809 23:21:58.579346 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.025s	sys 0.035s
W20260809 23:21:58.650533 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.026s	sys 0.036s
I20260809 23:21:58.616137 25370 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260809 23:21:58.651602 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1963) took 1077 ms. Trace:
I20260809 23:21:58.652016 25370 rpcz_store.cc:276] 0809 23:21:57.573716 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.553719 (+980003us) service_pool.cc:227] Handling call
0809 23:21:58.591579 (+ 37860us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:58.651193 (+ 59614us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.651525 (+   332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.548468  4420 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.619594 24312 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:21:58.653570 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1926) took 1138 ms. Trace:
I20260809 23:21:58.653970 24312 rpcz_store.cc:276] 0809 23:21:57.515295 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.612826 (+1097531us) service_pool.cc:227] Handling call
0809 23:21:58.653494 (+ 40668us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.655061 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:21:58.533327  4530 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.613597 24309 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.616113  4109 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:21:58.541626  4600 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.561451  4639 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.682570 24317 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:58.658816 24435 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:58.673870 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3688) took 3514 ms. Trace:
I20260809 23:21:58.688993 24546 rpcz_store.cc:276] 0809 23:21:55.159489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.445759 (+286270us) service_pool.cc:227] Handling call
0809 23:21:55.446076 (+   317us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:55.784510 (+338434us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.784569 (+    59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:55.784624 (+    55us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:55.784667 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.673317 (+2888650us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54001fc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.673389 (+    72us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.673784 (+   395us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777696128}
I20260809 23:21:58.660820 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1923) took 1147 ms. Trace:
W20260809 23:21:58.657009 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 754151us
I20260809 23:21:58.685739  4659 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.686398  3154 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.529943  4586 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.688297 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1830) took 2228 ms. Trace:
I20260809 23:21:58.691303  3154 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:58.696558  4530 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:58.691339  4732 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.005s	user 0.007s	sys 0.007s
W20260809 23:21:58.697199 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.012s	sys 0.033s
W20260809 23:21:58.697523 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.012s	sys 0.034s
I20260809 23:21:58.697961 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3788) took 1314 ms. Trace:
I20260809 23:21:58.698305 24541 rpcz_store.cc:276] 0809 23:21:57.383194 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.396777 (+ 13583us) service_pool.cc:227] Handling call
0809 23:21:57.397184 (+   407us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.421640 (+ 24456us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.697715 (+1276075us) spinlock_profiling.cc:243] Waited 8.44 ms on lock 0x7b5400115b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.697905 (+   190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19401984}
I20260809 23:21:58.691707 24535 rpcz_store.cc:276] 0809 23:21:56.459908 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.557180 (+ 97272us) service_pool.cc:227] Handling call
0809 23:21:56.559266 (+  2086us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.441978 (+882712us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.688042 (+1246064us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54000b5254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.688236 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2028700032}
I20260809 23:21:58.701732 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3347) took 1884 ms. Trace:
I20260809 23:21:58.702121 24427 rpcz_store.cc:276] 0809 23:21:56.817822 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.090721 (+272899us) service_pool.cc:227] Handling call
0809 23:21:57.110072 (+ 19351us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.110286 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.110311 (+    25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.130680 (+ 20369us) log.cc:842] Serialized 26 byte log entry
0809 23:21:57.420431 (+289751us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:57.420462 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.420531 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:58.690688 (+1270157us) spinlock_profiling.cc:243] Waited 8.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:21:58.701415 (+ 10727us) raft_consensus.cc:1671] finished
0809 23:21:58.701439 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.701672 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18628864,"thread_start_us":214131,"threads_started":1,"wal-append.queue_time_us":429557}
I20260809 23:21:58.689720 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3721) took 1117 ms. Trace:
I20260809 23:21:58.695205  4438 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:58.693241 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3117) took 1588 ms. Trace:
I20260809 23:21:58.704041 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:58.704918  4438 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.705084 24313 rpcz_store.cc:276] 0809 23:21:57.105264 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.149407 (+ 44143us) service_pool.cc:227] Handling call
0809 23:21:58.693072 (+1543665us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.693187 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1095633152}
I20260809 23:21:58.706529  4600 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:58.705569 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1651) took 1116 ms. Trace:
I20260809 23:21:58.693545 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:58.708884  4493 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:58.708995  4667 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.694042 24312 rpcz_store.cc:276] 0809 23:21:57.513265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.654720 (+1141455us) service_pool.cc:227] Handling call
0809 23:21:58.660742 (+  6022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.710280  4707 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.707485 24427 rpcz_store.cc:276] 0809 23:21:57.589473 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.703650 (+1114177us) service_pool.cc:227] Handling call
0809 23:21:58.705514 (+  1864us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.711496 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3812) took 1483 ms. Trace:
I20260809 23:21:58.709157  4438 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.711829 24422 rpcz_store.cc:276] 0809 23:21:57.228252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.233791 (+  5539us) service_pool.cc:227] Handling call
0809 23:21:57.234194 (+   403us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.234598 (+   404us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.234630 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.235242 (+   612us) log.cc:842] Serialized 26 byte log entry
0809 23:21:57.536204 (+300962us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:57.536267 (+    63us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.536358 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:58.711294 (+1174936us) raft_consensus.cc:1671] finished
0809 23:21:58.711314 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.711440 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1222,"raft.queue_time_us":7328,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":14369,"thread_start_us":300572,"threads_started":1,"wal-append.queue_time_us":403825}
I20260809 23:21:58.715924  4532 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:58.716295 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3364) took 1201 ms. Trace:
I20260809 23:21:58.633296  4675 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:58.716971  4675 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.717733  4366 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.704721  4453 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:58.720927 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3115) took 1617 ms. Trace:
I20260809 23:21:58.721406 24320 rpcz_store.cc:276] 0809 23:21:57.103840 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.411781 (+307941us) service_pool.cc:227] Handling call
0809 23:21:57.412100 (+   319us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:57.414868 (+  2768us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.414899 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:57.423521 (+  8622us) log.cc:842] Serialized 52 byte log entry
0809 23:21:57.426459 (+  2938us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:57.636507 (+210048us) raft_consensus.cc:1631] Marking committed up to 11
0809 23:21:57.636584 (+    77us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.636676 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:58.720484 (+1083808us) raft_consensus.cc:1671] finished
0809 23:21:58.720524 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.720827 (+   303us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":76462,"raft.queue_time_us":2553,"raft.run_cpu_time_us":861,"raft.run_wall_time_us":10726,"spinlock_wait_cycles":6196096,"thread_start_us":188434,"threads_started":1,"wal-append.queue_time_us":764212}
I20260809 23:21:58.723131 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3784) took 1457 ms. Trace:
I20260809 23:21:58.723723  4366 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.724022  3810 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:58.724529  3810 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:21:58.724459  4366 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.725677  3810 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:58.727921 24321 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3708) took 2701 ms (client timeout 1725 ms). Trace:
W20260809 23:21:58.728413 24321 rpcz_store.cc:269] 0809 23:21:56.026087 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.093098 (+ 67011us) service_pool.cc:227] Handling call
0809 23:21:58.727656 (+2634558us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54003a48d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.727823 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2903214464}
I20260809 23:21:58.703729 24317 rpcz_store.cc:276] 0809 23:21:57.572618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.597269 (+1024651us) service_pool.cc:227] Handling call
0809 23:21:58.689661 (+ 92392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.730495 24317 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:58.731019 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3355) took 1730 ms. Trace:
I20260809 23:21:58.731383 24425 rpcz_store.cc:276] 0809 23:21:57.002281 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.144675 (+142394us) service_pool.cc:227] Handling call
0809 23:21:57.144973 (+   298us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.161879 (+ 16906us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.161925 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.165493 (+  3568us) log.cc:842] Serialized 26 byte log entry
0809 23:21:57.419397 (+253904us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:57.528440 (+109043us) log.cc:842] Serialized 14 byte log entry
0809 23:21:57.528824 (+   384us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.528955 (+   131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:58.718893 (+1189938us) spinlock_profiling.cc:243] Waited 8.62 ms on lock 0x7b18003a5d00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:21:58.730691 (+ 11798us) raft_consensus.cc:1671] finished
0809 23:21:58.730803 (+   112us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.730961 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7005,"prepare.queue_time_us":108705,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":6859,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":19607,"spinlock_wait_cycles":19816320,"thread_start_us":207037,"threads_started":1,"wal-append.queue_time_us":372101}
I20260809 23:21:58.723526 24540 rpcz_store.cc:276] 0809 23:21:57.265588 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.291046 (+ 25458us) service_pool.cc:227] Handling call
0809 23:21:58.722952 (+1431906us) spinlock_profiling.cc:243] Waited 84.9 ms on lock 0x7b54000eecd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.723070 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195174144}
W20260809 23:21:58.723508 24431 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3794) took 2847 ms (client timeout 2085 ms). Trace:
W20260809 23:21:58.735034 24431 rpcz_store.cc:269] 0809 23:21:55.876143 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.903979 (+ 27836us) service_pool.cc:227] Handling call
0809 23:21:58.723349 (+2819370us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b5400130454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.723447 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615754496}
I20260809 23:21:58.736270 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1920) took 2530 ms. Trace:
I20260809 23:21:58.736696 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3821) took 1116 ms. Trace:
I20260809 23:21:58.736617 24314 rpcz_store.cc:276] 0809 23:21:56.205735 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.437544 (+231809us) service_pool.cc:227] Handling call
0809 23:21:56.437916 (+   372us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.223856 (+785940us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.736033 (+1512177us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b5400385ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.736214 (+   181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1804836096}
I20260809 23:21:58.737074 24431 rpcz_store.cc:276] 0809 23:21:57.619877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.735961 (+1116084us) service_pool.cc:227] Handling call
0809 23:21:58.736216 (+   255us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.736399 (+   183us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.736426 (+    27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:58.736462 (+    36us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:58.736491 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.736545 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.736646 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.737969 24314 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:58.738544  4582 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.739078  4493 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:58.739773 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.740509 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.741390 24535 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260809 23:21:58.742249 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.742504 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.743414 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:58.745551 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.746346  4639 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:58.678306  4730 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.501s	user 0.001s	sys 0.001s
I20260809 23:21:58.739600  4493 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.707213 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3116) took 1603 ms. Trace:
I20260809 23:21:58.630852  4343 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.756455 24313 rpcz_store.cc:276] 0809 23:21:57.104296 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.706118 (+1601822us) service_pool.cc:227] Handling call
0809 23:21:58.706507 (+   389us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.706843 (+   336us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.706882 (+    39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:58.706954 (+    72us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:58.706993 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.707052 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.707162 (+   110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.750489 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.751976 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3716) took 1187 ms. Trace:
I20260809 23:21:58.716676 24422 rpcz_store.cc:276] 0809 23:21:57.515148 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.713656 (+1198508us) service_pool.cc:227] Handling call
0809 23:21:58.714065 (+   409us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.714291 (+   226us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.714326 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:58.714396 (+    70us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:58.714449 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.714515 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.716242 (+  1727us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.758529 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3799) took 2880 ms. Trace:
I20260809 23:21:58.753414 24321 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:21:58.760097 24430 rpcz_store.cc:276] 0809 23:21:55.877747 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.917774 (+ 40027us) service_pool.cc:227] Handling call
0809 23:21:55.918104 (+   330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.263942 (+1345838us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.758206 (+1494264us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.758455 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3044617984}
I20260809 23:21:58.738811 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3717) took 1173 ms. Trace:
I20260809 23:21:58.762496 24314 rpcz_store.cc:276] 0809 23:21:57.565030 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.737549 (+1172519us) service_pool.cc:227] Handling call
0809 23:21:58.737806 (+   257us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.738303 (+   497us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.738724 (+   421us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.764276 24316 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:21:58.764946 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3718) took 1199 ms. Trace:
I20260809 23:21:58.627974  4527 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.760324 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1656) took 1133 ms. Trace:
I20260809 23:21:58.760790 24317 rpcz_store.cc:276] 0809 23:21:57.564301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.729981 (+1165680us) service_pool.cc:227] Handling call
0809 23:21:58.730306 (+   325us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.730836 (+   530us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.751919 (+ 21083us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.770148 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3723) took 1179 ms. Trace:
I20260809 23:21:58.738734 24431 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260809 23:21:58.767006 24323 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260809 23:21:58.770375 24422 rpcz_store.cc:276] 0809 23:21:57.626557 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.759558 (+1133001us) service_pool.cc:227] Handling call
0809 23:21:58.759862 (+   304us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.760036 (+   174us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.760060 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:58.760102 (+    42us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:58.760133 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.760182 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.760274 (+    92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.772817 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3719) took 1206 ms. Trace:
I20260809 23:21:58.773289 24323 rpcz_store.cc:276] 0809 23:21:57.566487 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.566637 (+1000150us) service_pool.cc:227] Handling call
0809 23:21:58.772761 (+206124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.774842 24323 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:21:58.770696 24320 rpcz_store.cc:276] 0809 23:21:57.591159 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.751981 (+1160822us) service_pool.cc:227] Handling call
0809 23:21:58.769393 (+ 17412us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.769606 (+   213us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.769733 (+   127us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:58.769795 (+    62us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:58.769850 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.769943 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:58.770080 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.765362 24314 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:58.777508 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.771203 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1655) took 1145 ms. Trace:
I20260809 23:21:58.779345 24431 rpcz_store.cc:276] 0809 23:21:57.626106 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.737904 (+1111798us) service_pool.cc:227] Handling call
0809 23:21:58.738173 (+   269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.770976 (+ 32803us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.771154 (+   178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.780586 24431 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:21:58.765362 24316 rpcz_store.cc:276] 0809 23:21:57.565699 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.637455 (+1071756us) service_pool.cc:227] Handling call
0809 23:21:58.764877 (+127422us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.781052 24431 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:58.778154 25370 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:58.766426  4576 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.783001  4724 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.783310  4576 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.784586 24430 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:21:58.785497  4576 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:58.788010  4644 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 2.053s	user 0.005s	sys 0.034s
W20260809 23:21:58.791931 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.023s	sys 0.024s
W20260809 23:21:58.792364 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.023s	sys 0.024s
W20260809 23:21:58.792704 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 805324us
I20260809 23:21:58.794929 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:58.782810 25370 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:21:58.756247  4322 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.798643 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1817) took 1132 ms. Trace:
I20260809 23:21:58.739086  4582 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.802116 24309 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:58.816533  4664 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:58.816911  4140 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.825711  4140 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.825940  4582 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.827214 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1952) took 2750 ms. Trace:
I20260809 23:21:58.827875  4140 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.828056 24543 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:21:58.828897  4664 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.830315 25373 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:58.830785  4678 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.830914  4664 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:58.831594 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1944) took 3562 ms (client timeout 1942 ms). Trace:
I20260809 23:21:58.825987  4653 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.832389  4109 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:21:58.832141 25373 rpcz_store.cc:269] 0809 23:21:55.269618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.369938 (+100320us) service_pool.cc:227] Handling call
0809 23:21:58.831357 (+3461419us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.831492 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7955780992}
I20260809 23:21:58.833127  4336 raft_consensus.cc:3032] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:21:58.833550  4336 raft_consensus.cc:729] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:58.833864 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:58.834677 25373 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:58.837769  4572 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.839879  4572 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.800974 25370 rpcz_store.cc:276] 0809 23:21:57.665893 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.782326 (+1116433us) service_pool.cc:227] Handling call
0809 23:21:58.782627 (+   301us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.798237 (+ 15610us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.798576 (+   339us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.818814 24313 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:58.813849  4438 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:58.822791  4366 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.823776  4439 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.841107 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:58.831322  4678 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.841318  4438 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.842208 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1927) took 1181 ms. Trace:
I20260809 23:21:58.841845  3932 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.842563 24313 rpcz_store.cc:276] 0809 23:21:57.660973 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.757378 (+1096405us) service_pool.cc:227] Handling call
0809 23:21:58.818624 (+ 61246us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.841162 (+ 22538us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.841999 (+   837us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b54003c2dd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:21:58.842147 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":140082432}
I20260809 23:21:58.843508  4438 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.843706  4678 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.767139  4343 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.844645 24425 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:58.845691  4343 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.847177 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.832962  4109 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.847321 24431 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:58.827991  4459 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.544s	user 0.010s	sys 0.003s
I20260809 23:21:58.848335 24431 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.848367  4459 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.850625  4459 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:58.665843  4683 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.851434 25373 raft_consensus.cc:2379] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:58.852846  4683 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:21:58.853410  4738 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.340s	user 0.001s	sys 0.002s
I20260809 23:21:58.849046  4109 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.792330  4326 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.854341  4326 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.856132  4739 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260809 23:21:58.857493  4326 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.858623  3932 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.801257  4576 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:21:58.859485  4576 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:21:58.846354 25370 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:58.860261  4576 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:21:58.846037  4431 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.673s	user 0.009s	sys 0.001s
I20260809 23:21:58.853106 25373 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:58.863402 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1868) took 1188 ms. Trace:
I20260809 23:21:58.863826 25373 rpcz_store.cc:276] 0809 23:21:57.674577 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.852488 (+1177911us) service_pool.cc:227] Handling call
0809 23:21:58.852861 (+   373us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.863063 (+ 10202us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.863321 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.820552 24317 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:58.827569 25375 rpcz_store.cc:276] 0809 23:21:56.076925 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.077054 (+   129us) service_pool.cc:227] Handling call
0809 23:21:56.077345 (+   291us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.107248 (+1029903us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.826973 (+1719725us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400550854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:58.827144 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2336174464}
I20260809 23:21:58.866801 25375 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:21:58.867553 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1820) took 1171 ms. Trace:
I20260809 23:21:58.867918 25375 rpcz_store.cc:276] 0809 23:21:57.696813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.866040 (+1169227us) service_pool.cc:227] Handling call
0809 23:21:58.866395 (+   355us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.867286 (+   891us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:58.867495 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:58.792181  4675 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.862224  4431 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.911252 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:58.911929 25373 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.914558  4326 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.834323  4336 consensus_queue.cc:260] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.915131  4326 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.917313  4326 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:58.918687 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.011s	sys 0.022s
W20260809 23:21:58.919317 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.011s	sys 0.022s
I20260809 23:21:58.778950  4527 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.921065 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:58.923473 24543 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:21:58.931280 24427 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:21:58.932266 24543 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:21:58.932628 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:21:58.933943 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:58.907668  4683 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:21:58.936743 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.008s	sys 0.025s
I20260809 23:21:58.817912  4322 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:58.938396 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.008s	sys 0.025s
I20260809 23:21:58.941738 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:58.943631 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.943753 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:58.947284 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.943236 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:58.947999 24540 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:21:58.949156 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.950179 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.950737 24540 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:21:58.952847 25373 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:58.954082 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:58.955070  4572 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:58.955495  4572 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.955869  4572 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:58.959501  4366 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:58.960004  4366 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:58.962023  4366 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.963045  4366 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.963555  4366 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.964764 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.965436 24546 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:21:58.968515  4366 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:58.972406  4745 raft_consensus.cc:2741] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:21:58.947480  4326 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:21:58.973691  4739 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:21:58.974397  4326 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:58.974977  4582 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:21:58.975358  4739 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:21:58.975919  4739 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:21:58.975668  4326 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.977015  4739 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:58.978538  4739 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:58.980000  4303 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:58.907703  3932 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260809 23:21:58.922380  4675 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:58.986467 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:58.995128 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:58.996331 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:58.996878 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:21:58.998085 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.873s	user 0.021s	sys 0.013s
I20260809 23:21:58.998819 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:21:58.959697 25373 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:21:58.999598 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:59.006129 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:21:58.998829 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.021s	sys 0.013s
I20260809 23:21:59.016650 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:59.018373 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:21:59.019158 24541 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:59.019817 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:58.990044  4493 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:59.041841 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:59.046257 24424 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:21:58.963896 24431 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:21:59.047544 24546 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:21:59.048214 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3823) took 1359 ms. Trace:
I20260809 23:21:59.048672 24431 rpcz_store.cc:276] 0809 23:21:57.689018 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.780010 (+1090992us) service_pool.cc:227] Handling call
0809 23:21:58.780474 (+   464us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.047865 (+267391us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.048147 (+   282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:59.051824 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:59.053200 24431 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:21:59.052621 25370 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:21:59.054457 25375 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: [22.15-24.17]
I20260809 23:21:59.056954  4322 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:59.057492 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3366) took 1157 ms. Trace:
I20260809 23:21:59.060241 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:33980 (request call id 39) took 1384 ms. Trace:
I20260809 23:21:59.057844 24431 rpcz_store.cc:276] 0809 23:21:57.900587 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.056787 (+1156200us) service_pool.cc:227] Handling call
0809 23:21:59.057043 (+   256us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.057209 (+   166us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:59.057233 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.057271 (+    38us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:59.057310 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.057358 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.057443 (+    85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:59.061605 24541 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:21:59.060621 25370 rpcz_store.cc:276] 0809 23:21:57.676038 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.059210 (+1383172us) service_pool.cc:227] Handling call
0809 23:21:59.060160 (+   950us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:59.062863 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:21:59.063688  4527 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:58.947543  4683 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.064687 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:21:59.066140  4683 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:59.070200  4737 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.126s	user 0.010s	sys 0.004s
I20260809 23:21:58.999133  4459 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.073990 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:21:59.003518  4582 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:59.077159 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:59.078043 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:21:59.081306 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:59.082104 25373 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:21:58.990346  4675 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:21:59.087118 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:59.072893  4704 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.000s
I20260809 23:21:59.096875 24424 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:21:59.084697  4493 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:21:59.102627  4493 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:59.103013  4493 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:59.103711 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3822) took 1475 ms. Trace:
I20260809 23:21:59.103820  4678 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.104105 24430 rpcz_store.cc:276] 0809 23:21:57.628345 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.761748 (+1133403us) service_pool.cc:227] Handling call
0809 23:21:58.762077 (+   329us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.784924 (+ 22847us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.103475 (+318551us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.103654 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50918272}
I20260809 23:21:59.104465  4678 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.105400 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:21:59.106016 24430 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:21:59.085515  4459 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.097842 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:21:59.100323 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:59.100452 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1650) took 1640 ms. Trace:
I20260809 23:21:59.108892 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3368) took 1029 ms. Trace:
I20260809 23:21:59.110734 24430 rpcz_store.cc:276] 0809 23:21:58.079930 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.106554 (+1026624us) service_pool.cc:227] Handling call
0809 23:21:59.106831 (+   277us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.108118 (+  1287us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:59.108582 (+   464us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.108628 (+    46us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:59.108666 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.108724 (+    58us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.108840 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:59.079928  4582 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.109534 24428 rpcz_store.cc:276] 0809 23:21:57.459811 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.461982 (+  2171us) service_pool.cc:227] Handling call
0809 23:21:57.462269 (+   287us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.471862 (+  9593us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.471906 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.472476 (+   570us) log.cc:842] Serialized 26 byte log entry
0809 23:21:57.897330 (+424854us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:57.897409 (+    79us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:57.897682 (+   273us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:59.069863 (+1172181us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b2400415958. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20790cf4 00007fac2078edfb 00007fac2081d559 00007fac2081d1e5 00007fac2081c82d 00007fac234cb46e 00007fac234e2c0c 00007fac234e1f36 00007fac234e1b58 00007fac234eb551
0809 23:21:59.096405 (+ 26542us) raft_consensus.cc:1671] finished
0809 23:21:59.096435 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.100382 (+  3947us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9228,"raft.queue_time_us":10809,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"spinlock_wait_cycles":87860992,"thread_start_us":424313,"threads_started":1,"wal-append.queue_time_us":471298}
I20260809 23:21:59.120997 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:21:59.085397  4527 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:21:59.112978 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:21:59.125304 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3367) took 1055 ms. Trace:
I20260809 23:21:59.126439  4453 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:21:59.089607  4140 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.086843  4651 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:59.129621 24424 rpcz_store.cc:276] 0809 23:21:58.069758 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.084360 (+1014602us) service_pool.cc:227] Handling call
0809 23:21:59.089048 (+  4688us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.124777 (+ 35729us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.125195 (+   418us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:21:59.084510  4439 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:59.127362  4322 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:21:59.155440 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.102s	user 0.046s	sys 0.124s
I20260809 23:21:59.097613  3932 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:21:59.160562 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.107s	user 0.046s	sys 0.124s
I20260809 23:21:59.209018 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:21:59.106823  4678 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:59.181733  4439 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.147806  4527 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:59.131116  4582 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:21:59.086047  4683 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.053s
I20260809 23:21:59.182265  4651 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.148072  4653 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:21:59.249207 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.019s	sys 0.009s
W20260809 23:21:59.409366 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.019s	sys 0.010s
I20260809 23:21:59.101277  4704 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.286901  4589 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.620s	user 0.017s	sys 0.046s
I20260809 23:21:59.086493  4400 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
W20260809 23:21:59.247771 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.012s	sys 0.013s
I20260809 23:21:59.163244 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:59.425405 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.013s	sys 0.017s
I20260809 23:21:59.147640  4140 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:21:59.285318 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.008s	sys 0.017s
W20260809 23:21:59.427166 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.009s	sys 0.017s
I20260809 23:21:59.344805 25373 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:21:59.448606 25373 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
W20260809 23:21:59.345515 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.011s	sys 0.028s
I20260809 23:21:59.129832  4572 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.352679  3932 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:21:59.153468  4431 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:59.294994  4554 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.000s	sys 0.029s
I20260809 23:21:59.468885 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:21:59.475886  4554 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.000s	sys 0.030s
I20260809 23:21:59.474777  3932 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:21:59.478080 24431 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:59.238102 25372 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.009s	sys 0.017s
W20260809 23:21:59.474316 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.012s	sys 0.028s
W20260809 23:21:59.479328 25372 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.009s	sys 0.020s
I20260809 23:21:59.480198  4444 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:21:59.402981 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:21:59.411275 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.004s	sys 0.033s
W20260809 23:21:59.483335 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.005s	sys 0.033s
I20260809 23:21:59.484051  4431 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:21:59.413172 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.012s	sys 0.027s
W20260809 23:21:59.485195 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.013s	sys 0.027s
I20260809 23:21:59.485319  4431 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:59.331871  4739 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.034s
I20260809 23:21:59.487155  4739 consensus_queue.cc:1225] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 21.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260809 23:21:59.414634 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1959) took 1870 ms. Trace:
W20260809 23:21:59.422086 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.926s	user 0.003s	sys 0.037s
W20260809 23:21:59.489188 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.317s	user 0.003s	sys 0.037s
I20260809 23:21:59.489029 25366 rpcz_store.cc:276] 0809 23:21:57.544011 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.544155 (+   144us) service_pool.cc:227] Handling call
0809 23:21:57.565998 (+ 21843us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.566234 (+   236us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:21:58.596965 (+1030731us) log.cc:842] Serialized 14 byte log entry
0809 23:21:59.413945 (+816980us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.414054 (+   109us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:21:59.414142 (+    88us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.414279 (+   137us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.414521 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1216701,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":9652,"thread_start_us":1794728,"threads_started":2}
I20260809 23:21:59.488363  4431 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.535542 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:21:59.360368  3920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.015s	sys 0.025s
W20260809 23:21:59.552799  3920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s	user 0.016s	sys 0.026s
I20260809 23:21:59.406560  4683 consensus_queue.cc:1225] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260809 23:21:59.414408  4753 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:59.556685  4753 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.559441  4527 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.364385  4459 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:59.424952  4438 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.018s	sys 0.012s
I20260809 23:21:59.566331  4651 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:59.577155 24429 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:21:59.416024  4589 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.749s	user 0.018s	sys 0.046s
I20260809 23:21:59.417730  4453 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.545516  4742 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:21:59.575901  4438 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.018s	sys 0.012s
I20260809 23:21:59.548063  4741 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:21:59.576031  4750 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:21:59.588848  4528 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:21:59.588920  4653 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:21:59.596303  4400 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:21:59.595660  4675 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:21:59.614501  4653 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:21:59.601266  4453 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.629837  4706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s	user 0.033s	sys 0.092s
I20260809 23:21:59.627799  4528 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:21:59.666729 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1967) took 2087 ms. Trace:
I20260809 23:21:59.606041  4741 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.662031  4706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.033s	sys 0.092s
I20260809 23:21:59.667793 25372 rpcz_store.cc:276] 0809 23:21:57.581979 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.561231 (+979252us) service_pool.cc:227] Handling call
0809 23:21:58.561551 (+   320us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.561765 (+   214us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:21:59.479975 (+918210us) log.cc:842] Serialized 14 byte log entry
0809 23:21:59.665818 (+185843us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.665915 (+    97us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:21:59.666270 (+   355us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.666413 (+   143us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.666623 (+   210us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":229811,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":1046187,"threads_started":2}
I20260809 23:21:59.657923  4653 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:21:59.613343  4750 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.702348  4758 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
W20260809 23:21:59.726605  4763 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260809 23:21:59.703758  4753 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:59.706911  4568 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:21:59.720577  4765 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.135s	user 0.000s	sys 0.002s
I20260809 23:21:59.692173  3932 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.724682  4704 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:21:59.762636  4493 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.659s	user 0.016s	sys 0.000s
I20260809 23:21:59.772971  4493 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.733340  4439 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:21:59.757041  4759 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.091s	user 0.002s	sys 0.000s
W20260809 23:21:59.779953  4576 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.919s	user 0.021s	sys 0.000s
W20260809 23:21:59.815563  4773 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260809 23:21:59.816529  4769 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.115s	user 0.001s	sys 0.001s
W20260809 23:21:59.815687  4772 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260809 23:21:59.819643  4767 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.118s	user 0.002s	sys 0.000s
W20260809 23:21:59.880499 24428 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.003s	sys 0.027s
W20260809 23:21:59.882331 24428 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.003s	sys 0.028s
I20260809 23:21:59.890275 24544 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:21:59.891387 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.023s	sys 0.081s
W20260809 23:21:59.892272 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.023s	sys 0.081s
W20260809 23:21:59.895586 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.016s	sys 0.024s
I20260809 23:21:59.896710 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1834) took 2456 ms. Trace:
W20260809 23:21:59.897286 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.012s	sys 0.028s
I20260809 23:21:59.903471 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1847) took 4737 ms. Trace:
W20260809 23:21:59.903718 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 740259us
I20260809 23:21:59.897789 24542 rpcz_store.cc:276] 0809 23:21:57.440255 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.451040 (+ 10785us) service_pool.cc:227] Handling call
0809 23:21:57.451365 (+   325us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.451695 (+   330us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.451723 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.465597 (+ 13874us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:58.747835 (+1282238us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:58.759488 (+ 11653us) log.cc:842] Serialized 26 byte log entry
0809 23:21:59.484282 (+724794us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:59.484400 (+   118us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.484693 (+   293us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:59.895332 (+410639us) raft_consensus.cc:1671] finished
0809 23:21:59.895497 (+   165us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.896191 (+   694us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":313,"raft.run_cpu_time_us":798,"raft.run_wall_time_us":13740,"spinlock_wait_cycles":34796800,"thread_start_us":683531,"threads_started":1,"wal-append.queue_time_us":855269}
W20260809 23:21:59.896733 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.017s	sys 0.025s
I20260809 23:21:59.916484 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 1968) took 2327 ms. Trace:
I20260809 23:21:59.921924 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3824) took 2006 ms. Trace:
I20260809 23:21:59.910395 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3820) took 2383 ms. Trace:
W20260809 23:21:59.941888 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 538888us
W20260809 23:21:59.943109 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.013s	sys 0.029s
I20260809 23:21:59.759256  4140 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:21:59.944948 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3363) took 2437 ms. Trace:
I20260809 23:21:59.769726  4528 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.957926 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51434.8us
I20260809 23:21:59.904808 25363 rpcz_store.cc:276] 0809 23:21:55.165476 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:55.385304 (+219828us) service_pool.cc:227] Handling call
0809 23:21:55.385660 (+   356us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:55.408677 (+ 23017us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:55.408708 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:55.409135 (+   427us) log.cc:842] Serialized 26 byte log entry
0809 23:21:58.606794 (+3197659us) spinlock_profiling.cc:243] Waited 7.55 ms on lock 0x7b18001f6c00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:59.479856 (+873062us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:59.479918 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.487004 (+  7086us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:59.901459 (+414455us) raft_consensus.cc:1671] finished
0809 23:21:59.901566 (+   107us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.902742 (+  1176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7873,"raft.queue_time_us":17952,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7480,"spinlock_wait_cycles":17357184,"thread_start_us":872777,"threads_started":1,"wal-append.queue_time_us":4198131}
W20260809 23:21:59.951900 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45441.8us
W20260809 23:21:59.973242  4630 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 3.814s	user 0.007s	sys 0.020s
I20260809 23:21:59.973891  4322 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:59.975581  4322 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:21:59.977058 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 869220us
I20260809 23:21:59.922761 24431 rpcz_store.cc:276] 0809 23:21:57.915155 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.062300 (+1147145us) service_pool.cc:227] Handling call
0809 23:21:59.062574 (+   274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.479128 (+416554us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.920831 (+441703us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54002f85d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.921533 (+   702us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":954615552}
I20260809 23:21:59.940874 24425 rpcz_store.cc:276] 0809 23:21:57.526774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.733223 (+1206449us) service_pool.cc:227] Handling call
0809 23:21:58.733536 (+   313us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.845050 (+111514us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.909026 (+1063976us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54002f8fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.909872 (+   846us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":255123200}
W20260809 23:21:59.770483  4303 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.790s	user 0.011s	sys 0.007s
I20260809 23:21:59.986018 25365 raft_consensus.cc:595] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Received request to transfer leadership to TS ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:21:59.987044  4786 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:21:59.988734 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3726) took 2504 ms. Trace:
I20260809 23:21:59.991647  4493 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:59.987406 25365 raft_consensus.cc:603] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:21:59.997989 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3787) took 2655 ms. Trace:
I20260809 23:21:59.807165  4576 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:21:59.949793 24427 rpcz_store.cc:276] 0809 23:21:57.507082 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.711412 (+1204330us) service_pool.cc:227] Handling call
0809 23:21:58.711708 (+   296us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:58.934757 (+223049us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.943552 (+1008795us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540012fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.944534 (+   982us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":483805568}
I20260809 23:22:00.006690  3932 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:21:59.917318 25369 rpcz_store.cc:276] 0809 23:21:57.588660 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.563570 (+974910us) service_pool.cc:227] Handling call
0809 23:21:58.572623 (+  9053us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:58.572908 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:58.572943 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:58.573442 (+   499us) log.cc:842] Serialized 26 byte log entry
0809 23:21:59.427994 (+854552us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:21:59.428088 (+    94us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.428237 (+   149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:59.915537 (+487300us) raft_consensus.cc:1671] finished
0809 23:21:59.915603 (+    66us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.916101 (+   498us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":794049,"threads_started":1,"wal-append.queue_time_us":1005572}
W20260809 23:22:00.002127 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 466510us
I20260809 23:21:59.989696 24544 rpcz_store.cc:276] 0809 23:21:57.484733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.570610 (+ 85877us) service_pool.cc:227] Handling call
0809 23:21:57.570884 (+   274us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.892037 (+2321153us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.987830 (+ 95793us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.988601 (+   771us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5331080960}
I20260809 23:22:00.020390  4774 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.021399 24431 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:00.027489  4140 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.028460  4780 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:00.031714 25370 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:00.046917 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.050019 25366 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:21:59.979851 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.055924  4109 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:22:00.017822 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.029481  4140 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.057332  4109 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:00.022248  4774 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.070639 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3828) took 1265 ms. Trace:
I20260809 23:22:00.071462 24425 rpcz_store.cc:276] 0809 23:21:58.805056 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.985117 (+1180061us) service_pool.cc:227] Handling call
0809 23:22:00.058287 (+ 73170us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:00.069890 (+ 11603us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:00.069956 (+    66us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:00.070080 (+   124us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:00.070149 (+    69us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.070278 (+   129us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.070492 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:00.081583  4774 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.083437  3889 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.090735 25372 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:21:59.989711  4786 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.100198  4742 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.102097  4742 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.074239  4572 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.123481 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1831) took 3644 ms. Trace:
I20260809 23:22:00.124225 24539 rpcz_store.cc:276] 0809 23:21:56.475406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.671288 (+195882us) service_pool.cc:227] Handling call
0809 23:21:56.671664 (+   376us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:57.157436 (+485772us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.118644 (+2961208us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54003dc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:00.119336 (+   692us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1074070400}
I20260809 23:22:00.116604 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3836) took 1194 ms. Trace:
I20260809 23:22:00.128378 24427 rpcz_store.cc:276] 0809 23:21:58.911779 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:00.064840 (+1153061us) service_pool.cc:227] Handling call
0809 23:22:00.104192 (+ 39352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:00.105173 (+   981us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:00.105253 (+    80us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:00.105432 (+   179us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:00.106152 (+   720us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.106275 (+   123us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.106530 (+   255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:00.030366  4780 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.003172 24534 rpcz_store.cc:276] 0809 23:21:57.342223 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.354645 (+ 12422us) service_pool.cc:227] Handling call
0809 23:21:57.372419 (+ 17774us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.449661 (+ 77242us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.449693 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:58.617133 (+1167440us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:58.624561 (+  7428us) log.cc:842] Serialized 26 byte log entry
0809 23:21:59.486238 (+861677us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:21:59.486306 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.996786 (+510480us) spinlock_profiling.cc:243] Waited 76.9 ms on lock 0x7b540029d454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:21:59.996955 (+   169us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:21:59.997095 (+   140us) raft_consensus.cc:1671] finished
0809 23:21:59.997121 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:21:59.997617 (+   496us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":160,"spinlock_wait_cycles":326131072,"thread_start_us":711212,"threads_started":1,"wal-append.queue_time_us":947433}
I20260809 23:22:00.146023  4706 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:00.148104  4706 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:00.151533 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 108681us
I20260809 23:22:00.120260  4109 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.153484  4109 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:00.154407  4576 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.157187  3772 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.159722 25370 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: [20.17-20.17]
I20260809 23:22:00.160566 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3826) took 1358 ms. Trace:
I20260809 23:22:00.161283 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3735) took 1146 ms. Trace:
I20260809 23:22:00.158869  4742 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.162209 24540 rpcz_store.cc:276] 0809 23:21:59.014700 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.014884 (+   184us) service_pool.cc:227] Handling call
0809 23:21:59.015302 (+   418us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.075981 (+ 60679us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:59.076026 (+    45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.076088 (+    62us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:59.076132 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.160501 (+1084369us) spinlock_profiling.cc:243] Waited 60.4 ms on lock 0x7b5400148054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:00.160607 (+   106us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.160979 (+   372us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":139001216}
I20260809 23:22:00.167454  4493 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:00.158818  3772 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.206670 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.161365 24428 rpcz_store.cc:276] 0809 23:21:58.801679 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.122144 (+320465us) service_pool.cc:227] Handling call
0809 23:21:59.124036 (+  1892us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.124424 (+   388us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:21:59.883908 (+759484us) log.cc:842] Serialized 14 byte log entry
0809 23:22:00.153301 (+269393us) log.cc:842] Serialized 14 byte log entry
0809 23:22:00.159411 (+  6110us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:00.159554 (+   143us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:00.159778 (+   224us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.160006 (+   228us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.160434 (+   428us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5505,"prepare.queue_time_us":634857,"prepare.run_cpu_time_us":189,"prepare.run_wall_time_us":5463,"raft.queue_time_us":10227,"thread_start_us":894581,"threads_started":2,"wal-append.queue_time_us":269290}
I20260809 23:22:00.225488 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.214411  4786 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.231372  4787 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.234467  4400 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.240840  3920 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.228729  4322 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.242390  3920 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.256070  3889 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:00.258224  4823 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:22:00.259421 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43692us
I20260809 23:22:00.240887  4303 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:00.249742  4735 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.621s	user 0.006s	sys 0.002s
I20260809 23:22:00.232890  4787 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.263577 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.258502  3772 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.275103  4706 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.273221  4493 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.236975  4400 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.301033  4589 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:00.303576  4823 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:00.308158 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.18723e+06us
I20260809 23:22:00.314713  4611 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:00.302205  4589 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:00.283730  4303 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.295693  3889 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.325371 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3713) took 3297 ms. Trace:
I20260809 23:22:00.327425  4787 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.332522  4493 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.335614 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.336516 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.358078  4812 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.312839  4823 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.299283  4780 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.361809 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.367947 25366 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:22:00.360940  4812 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.372275  4820 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.372709  4639 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:00.375443  4435 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.352133 24311 rpcz_store.cc:276] 0809 23:21:56.996587 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.394340 (+397753us) service_pool.cc:227] Handling call
0809 23:21:57.402743 (+  8403us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:21:57.409241 (+  6498us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.409284 (+    43us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:21:57.413257 (+  3973us) log.cc:842] Serialized 52 byte log entry
0809 23:21:57.496785 (+ 83528us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b18003d9ba0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:21:57.664100 (+167315us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:21:57.698024 (+ 33924us) log.cc:842] Serialized 14 byte log entry
0809 23:21:58.067126 (+369102us) log.cc:842] Serialized 14 byte log entry
0809 23:21:58.852826 (+785700us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:00.104800 (+1251974us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b18003d9ba0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac20773990 00007fac20935864 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7 00007fac2387b058
0809 23:22:00.251006 (+146206us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.266920 (+ 15914us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:00.276724 (+  9804us) raft_consensus.cc:1671] finished
0809 23:22:00.276762 (+    38us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.293595 (+ 16833us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2694653,"prepare.queue_time_us":103546,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":6620,"raft.queue_time_us":2545532,"raft.run_cpu_time_us":905,"raft.run_wall_time_us":25529,"spinlock_wait_cycles":79891584,"thread_start_us":273107,"threads_started":2,"wal-append.queue_time_us":955444,"wal-append.run_cpu_time_us":6910,"wal-append.run_wall_time_us":1584619}
I20260809 23:22:00.350868  4400 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.380297 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.380707  4589 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.363747  4303 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.376729  4435 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.390597 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:00.391113 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:00.392697 24540 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:00.392885  4787 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:00.394188  4435 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:00.393546 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 76346.1us
I20260809 23:22:00.395452 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.400836  4400 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:00.404007 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 2.20518e+06us
I20260809 23:22:00.373469  3920 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:00.411701  4766 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.782s	user 0.004s	sys 0.002s
I20260809 23:22:00.413520 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.373554  4820 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.403939 24270 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:00.399565  3889 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.403847 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:00.418735  4811 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260809 23:22:00.429194 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3833) took 1528 ms. Trace:
I20260809 23:22:00.421689 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.431476 25370 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:00.436084 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.439755  4657 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.424008 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3721) took 2994 ms. Trace:
I20260809 23:22:00.441200  4657 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:00.443789  4806 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.289s	user 0.004s	sys 0.000s
I20260809 23:22:00.425544 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1924) took 2911 ms. Trace:
I20260809 23:22:00.446204 24321 rpcz_store.cc:276] 0809 23:21:57.513539 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.729342 (+1215803us) service_pool.cc:227] Handling call
0809 23:21:58.729691 (+   349us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:58.767856 (+ 38165us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.424596 (+1656740us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b54003c3cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:00.425170 (+   574us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53940992}
I20260809 23:22:00.400249 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.447765 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.397497 24534 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:00.448814 24539 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:00.426285 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.419972  4639 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.449311  4435 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:00.450644 25369 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:00.429906 24431 rpcz_store.cc:276] 0809 23:21:58.901068 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.988779 (+1087711us) service_pool.cc:227] Handling call
0809 23:21:59.990058 (+  1279us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:00.022385 (+ 32327us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.428791 (+406406us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:00.429066 (+   275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70463872}
I20260809 23:22:00.453673  4639 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.454470  4589 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:00.454519 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:00.436441 24311 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:00.423898 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:00.470791 24546 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:00.458539  4820 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.478999 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:00.480803 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.481069 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.440887 24545 rpcz_store.cc:276] 0809 23:21:57.429481 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.440759 (+ 11278us) service_pool.cc:227] Handling call
0809 23:21:57.441150 (+   391us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:21:57.441654 (+   504us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:57.441683 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:21:57.463104 (+ 21421us) spinlock_profiling.cc:243] Waited 8.33 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:21:58.714073 (+1250969us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b2c000bdd88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:21:58.714657 (+   584us) log.cc:842] Serialized 26 byte log entry
0809 23:21:59.426267 (+711610us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:21:59.426371 (+   104us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:21:59.426534 (+   163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:00.384781 (+958247us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:00.423077 (+ 38296us) raft_consensus.cc:1671] finished
0809 23:22:00.423123 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.423705 (+   582us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6019,"raft.run_cpu_time_us":888,"raft.run_wall_time_us":15738,"spinlock_wait_cycles":99102080,"thread_start_us":710956,"threads_started":1,"wal-append.queue_time_us":873850}
I20260809 23:22:00.480037 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:00.459754  4639 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:00.483529 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.481400 24539 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:00.482425 24541 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:00.397254 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:00.483973  4657 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.483173 24546 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:00.502799  4639 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:00.508526  4653 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.796s	user 0.021s	sys 0.012s
W20260809 23:22:00.526095 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102500us
W20260809 23:22:00.529757 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 132858us
I20260809 23:22:00.530889 24540 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
W20260809 23:22:00.531352 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 51856.4us
I20260809 23:22:00.529310 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.535792  4491 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.536960  4491 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.539062 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:00.541239 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:00.541782 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1854) took 3929 ms (client timeout 2053 ms). Trace:
I20260809 23:22:00.543624 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:00.531319  4639 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:00.545687  4820 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:00.546756  4820 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:00.549115  4820 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.551301  4457 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.551545  4745 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:00.552464  4745 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:00.553471  4745 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:00.552392  4457 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.555614 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.556774 24313 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:00.557803 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:00.559782 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.560978 24321 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:00.561589 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:00.529247  4812 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.563372 24309 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:00.561046  4780 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:00.542469 25368 rpcz_store.cc:269] 0809 23:21:56.612444 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:56.750655 (+138211us) service_pool.cc:227] Handling call
0809 23:22:00.541240 (+3790585us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:00.541637 (+   397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7354521344}
W20260809 23:22:00.565186 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 111008us
W20260809 23:22:00.578548 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89540.3us
I20260809 23:22:00.571398  4493 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260809 23:22:00.592581 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.601315 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.606889 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:00.611889 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.609928 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.625707 25366 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:00.625837 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64554.1us
I20260809 23:22:00.627338 24313 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
W20260809 23:22:00.633106  4582 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.219s	user 0.014s	sys 0.023s
I20260809 23:22:00.635908 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.636937  4491 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.573189  4653 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.640187 24309 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:00.641462  4653 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.616093 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:00.607131  4787 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:00.653519 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.654719 24309 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:00.665541 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:00.662798 25369 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:00.673864 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.677979 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:00.687543 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.689414 24427 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:00.678166  4823 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.692895 24422 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:00.695713  4823 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.700606  4140 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:00.702004 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 111991us
I20260809 23:22:00.689136 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.634127  4582 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.642791  4653 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:00.706401 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.709017 24311 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:00.706156 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:00.694638  4359 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.659682  4812 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:00.659444  4774 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:00.730028 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 81965.5us
W20260809 23:22:00.743634 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91190.7us
I20260809 23:22:00.748095 25366 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:22:00.747624 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.744249 24425 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:00.747416 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.715338  4787 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:00.757464 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.757941 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.749529 24313 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:00.760667  3920 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:00.789292 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:00.790235  3920 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.791046  3920 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.791337  4842 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260809 23:22:00.793445  4640 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:00.793849  4566 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:00.794349  4566 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:00.794130  4640 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.794781  4566 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:00.794690  4812 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.796445  4640 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.797428  4812 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.753327  4657 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:00.799252  4657 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:00.799974  4582 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:00.800452  3950 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:00.800287  4657 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.763247  4653 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:00.744125  4359 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.769338  4841 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:00.770653 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:00.772543  4568 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.927s	user 0.019s	sys 0.012s
I20260809 23:22:00.773136 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.774032 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.774353 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.775782 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.778978 25370 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:00.779605  4588 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.783929 25366 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:00.785394 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.788642  4686 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:00.826234 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.793411  3920 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:00.836026 24311 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:22:00.833295 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:00.836019 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 45201.9us
W20260809 23:22:00.849690  4826 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.191s	user 0.000s	sys 0.006s
I20260809 23:22:00.850276 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.851706 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:00.852303 24428 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:00.853356  4823 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.853086  4588 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.854717  4787 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:00.855507  4588 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:00.855798 25371 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260809 23:22:00.856204 25371 pending_rounds.cc:76] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 17
I20260809 23:22:00.856130 24427 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:00.856608 25371 pending_rounds.cc:97] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260809 23:22:00.856951 25371 raft_consensus.cc:2866] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:00.858601  4343 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.859133  4789 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.859870  4789 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:00.861263  4834 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.060s	user 0.002s	sys 0.006s
I20260809 23:22:00.862239 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3827) took 2058 ms. Trace:
I20260809 23:22:00.863683  4444 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:00.864323  4493 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:00.867184 24314 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:00.867511 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:00.872316 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.876901 25370 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:22:00.874091 24424 rpcz_store.cc:276] 0809 23:21:58.803344 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.150889 (+347545us) service_pool.cc:227] Handling call
0809 23:21:59.155158 (+  4269us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.350993 (+195835us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:59.351064 (+    71us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.351160 (+    96us) raft_consensus.cc:1631] Marking committed up to 20
0809 23:21:59.351242 (+    82us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.861808 (+1510566us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003bb354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:00.861886 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.862090 (+   204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449334016}
I20260809 23:22:00.942379 24539 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260809 23:22:00.942658 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:00.859287  4343 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:00.958436 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1921) took 3356 ms. Trace:
I20260809 23:22:00.968425 24546 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:00.988842 24545 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:00.991787 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.012219 25363 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.012444  4792 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:01.012835 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3734) took 1998 ms (client timeout 1746 ms). Trace:
I20260809 23:22:01.013275 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.013466 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:01.014107 24544 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:22:00.867913  4686 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.015491 24309 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:00.945997  4491 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:00.911845  4444 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:01.027385 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.031731 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.060010 24546 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:01.043493 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.032831 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.056234 25366 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:01.025945 24543 rpcz_store.cc:269] 0809 23:21:59.014011 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.016383 (+  2372us) service_pool.cc:227] Handling call
0809 23:22:01.012464 (+1996081us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.012629 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108821888}
I20260809 23:22:00.974820 24312 rpcz_store.cc:276] 0809 23:21:57.512653 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.709843 (+1197190us) service_pool.cc:227] Handling call
0809 23:21:58.710190 (+   347us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:00.221438 (+1511248us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:00.225475 (+  4037us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:00.267339 (+ 41864us) log.cc:842] Serialized 26 byte log entry
0809 23:22:00.511005 (+243666us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:00.511637 (+   632us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:00.527187 (+ 15550us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:00.862735 (+335548us) raft_consensus.cc:1671] finished
0809 23:22:00.862762 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:00.868678 (+  5916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1438011,"raft.queue_time_us":1532867,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":4254,"thread_start_us":276791,"threads_started":2,"wal-append.queue_time_us":314591}
I20260809 23:22:01.058598 24316 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260809 23:22:01.075973 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:00.961061  4322 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:00.834398  4774 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.079422 24425 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:01.097640 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:00.961771  4140 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:01.097856 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 56323.8us
W20260809 23:22:01.110201 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 167386us
W20260809 23:22:01.111086 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 356886us
I20260809 23:22:01.109819  4109 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:01.118733  4847 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260809 23:22:01.118392  4109 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.120117 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.120847  4109 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.121501 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.123383 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.123718  4843 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260809 23:22:01.124213 24323 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:22:01.124684  4843 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.125231  4843 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.127040  4792 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.128316 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.129737 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:01.130081 25373 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:01.122676 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.116569 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:01.002022  4780 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.096s
I20260809 23:22:01.136621 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.076520  4611 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.190752 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.178791 24539 raft_consensus.cc:2379] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:01.196076 24311 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:01.200700  4843 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.123267  4147 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260809 23:22:01.140718  4792 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:01.187973 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73771.1us
I20260809 23:22:01.203611  4750 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:01.129294  4803 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.933s	user 0.000s	sys 0.005s
I20260809 23:22:01.196350 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.207553 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.224807 25366 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
W20260809 23:22:01.224772 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 168081us
I20260809 23:22:01.229288 25368 raft_consensus.cc:2396] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260809 23:22:01.233436 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:01.229084 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 239381us
I20260809 23:22:01.237249 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.237423 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:01.239302 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 150511us
I20260809 23:22:01.238032 25366 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:01.123149  4322 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.244169 24543 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:22:01.254273 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.257579 25368 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:01.226475  4582 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260809 23:22:01.267117 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.268113 25369 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:01.268888  4322 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.271584 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.272464 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.277930 24541 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:22:01.286634 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:01.202052  4846 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.198s
I20260809 23:22:01.227790  4491 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.296583 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3825) took 2502 ms. Trace:
I20260809 23:22:01.301072 25368 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:01.269483  4322 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260809 23:22:01.306558 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.297684  4820 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:01.307497  4812 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.307701  4820 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:01.308247  4820 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:01.247570  4140 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.311188  4789 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:01.312664  4140 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.307271 24428 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:01.298920 24430 rpcz_store.cc:276] 0809 23:21:58.794156 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:59.112734 (+318578us) service_pool.cc:227] Handling call
0809 23:21:59.113272 (+   538us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:21:59.131072 (+ 17800us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:21:59.131128 (+    56us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:21:59.131207 (+    79us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:21:59.131268 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:01.296078 (+2164810us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:01.296171 (+    93us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:01.296377 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40119552}
I20260809 23:22:01.307403 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.317071 24546 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:01.314679 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.322659 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.323403 24541 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 24
W20260809 23:22:01.329829 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 129371us
I20260809 23:22:01.330129 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.287102  4359 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.249895  4611 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.331079 25370 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
W20260809 23:22:01.335662 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 81894.3us
I20260809 23:22:01.325616 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.341547 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.342415 25369 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:22:01.349409 24425 raft_consensus.cc:2396] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260809 23:22:01.346674 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.312706  4846 raft_consensus.cc:3032] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:01.285919  4493 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260809 23:22:01.358062 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.359474 24425 raft_consensus.cc:2396] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260809 23:22:01.356173 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.361848 24546 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:01.355309 24430 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:01.363327 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.345516  4792 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.367901 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.308221  4812 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.376348 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:01.353458  4846 raft_consensus.cc:729] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:01.366061  4322 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:01.391506 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 263290us
I20260809 23:22:01.351339  4147 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260809 23:22:01.396481 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:01.364995 24269 connection.cc:232] RPC call timeout handler was delayed by -1.0302621158771217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
I20260809 23:22:01.399158 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.387815 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.392261  4657 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:01.391584  4792 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.409273 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.411337  4582 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:01.412150  4582 consensus_queue.cc:1225] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 25.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260809 23:22:01.412863  4457 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:01.413338  4757 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.413926  4757 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:01.415045 24435 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1649) took 3904 ms (client timeout 3527 ms). Trace:
W20260809 23:22:01.415689 24435 rpcz_store.cc:269] 0809 23:21:57.447981 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.448971 (+   990us) service_pool.cc:227] Handling call
0809 23:22:01.297433 (+3848462us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.352113 (+ 54680us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2734278528}
W20260809 23:22:01.416584  4639 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.870s	user 0.011s	sys 0.005s
I20260809 23:22:01.416936 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.418183  4611 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.353642  4843 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260809 23:22:01.419945 24546 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:01.408861  4657 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.418568 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.427477 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.421271 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.429277 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.380195  4841 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.379846  4653 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.417035  4757 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.437335 24309 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:01.412335  4812 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:01.440426 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52741.1us
I20260809 23:22:01.441567 24423 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260809 23:22:01.418532  4741 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:01.442183  4566 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.647s	user 0.012s	sys 0.001s
I20260809 23:22:01.442883 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.444743  4653 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.445549 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.446102  4855 consensus_queue.cc:1046] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.108s
I20260809 23:22:01.448487  4336 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:01.450228 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 31) took 4764 ms. Trace:
I20260809 23:22:01.452335 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.479147 24545 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:01.454258 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1821) took 3744 ms. Trace:
W20260809 23:22:01.456990  4745 consensus_meta.cc:287] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.903s	user 0.014s	sys 0.000s
I20260809 23:22:01.460996 24542 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:01.466857 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.431156  4657 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:01.429920  4792 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:01.470242  4774 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.471071 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.472131 24546 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:01.417043  4639 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.475687 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.483363 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.485713 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:01.487506 24542 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:01.488925 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.478366 25365 rpcz_store.cc:276] 0809 23:21:56.685527 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.109335 (+423808us) service_pool.cc:227] Handling call
0809 23:22:01.450004 (+4340669us) spinlock_profiling.cc:243] Waited 2.88 s on lock 0x7b5400634554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.450151 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6613233024}
I20260809 23:22:01.480142 25375 rpcz_store.cc:276] 0809 23:21:57.709678 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.868698 (+1159020us) service_pool.cc:227] Handling call
0809 23:21:59.054194 (+185496us) raft_consensus.cc:1457] Updating replica for 3 ops
0809 23:21:59.055495 (+  1301us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:01.453902 (+2398407us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000c4750. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.454086 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":425556352}
I20260809 23:22:01.493476 24321 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:01.495617 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.485778 24424 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:01.488118 25373 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:01.500213 24545 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:22:01.500273  4660 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:01.500833  4660 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:01.501895  4660 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.503640  4874 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.102s
I20260809 23:22:01.504470  4653 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.471441  4147 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260809 23:22:01.495005  4359 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:01.508606 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.496286 24435 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:01.509809 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.509897 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.510576 24435 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:01.511297 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.511461 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.511907 24314 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:01.505111  3889 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:01.513301 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.514374 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.533011 24425 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:01.514266 25369 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 18.
W20260809 23:22:01.492321  4589 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.037s	user 0.011s	sys 0.006s
I20260809 23:22:01.538287 24546 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260809 23:22:01.539482 24320 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:01.540431 24546 pending_rounds.cc:76] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 17
I20260809 23:22:01.540706 24546 pending_rounds.cc:97] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260809 23:22:01.542389 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.439718  4841 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.478251  4566 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.555673 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.548357 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.540968 24546 raft_consensus.cc:2866] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:01.541298 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47132.1us
I20260809 23:22:01.551591  4343 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.544835  4588 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260809 23:22:01.553351  4686 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.484815  4812 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.052s
I20260809 23:22:01.538378 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.538285 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.561260 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.546344 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.556592 24323 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:01.548496 24542 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.521812  3889 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:01.477670  4140 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.072s
I20260809 23:22:01.579823 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.581796 24431 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260809 23:22:01.573001 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.534884  4875 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:01.572399 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.587518 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:01.576197 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.590747 24430 raft_consensus.cc:2354] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:01.589427 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:01.592367 24542 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:01.547956  4359 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:01.591290 24320 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:01.492095  4745 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:01.597370 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 102148us
I20260809 23:22:01.593142 24321 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:01.599192 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:01.601406 24424 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:01.594877 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.606251  4251 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:01.606613  4589 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:01.607270 24535 raft_consensus.cc:2396] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 21 because replica has last-logged OpId of term: 20 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260809 23:22:01.608528 24435 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:01.609087  4531 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:01.609246  4812 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:01.609493 24435 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:22:01.610278 24435 raft_consensus.cc:595] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:01.610616 24435 raft_consensus.cc:603] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:01.611665 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1657) took 3943 ms. Trace:
I20260809 23:22:01.607156  4589 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:01.571415  4660 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:01.606683  4251 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.612619 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.616739 24546 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 18.
I20260809 23:22:01.492746  4444 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.612010  4683 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.609522  4531 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.620604 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.612072 24422 rpcz_store.cc:276] 0809 23:21:57.668187 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.772280 (+1104093us) service_pool.cc:227] Handling call
0809 23:21:58.772822 (+   542us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:00.693754 (+1920932us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:01.611364 (+917610us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b5400332cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:01.611607 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4313371392}
I20260809 23:22:01.635002 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.621038 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:01.620330 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.625953 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:01.608872 24535 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1841) took 2667 ms (client timeout 2045 ms). Trace:
I20260809 23:22:01.620645  4251 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:01.598533  4491 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.602921  4741 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:01.612111  3810 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.631953 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.589298  4875 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.639519 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:01.637121 24535 rpcz_store.cc:269] 0809 23:21:58.941230 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:58.941406 (+   176us) service_pool.cc:227] Handling call
0809 23:22:01.608634 (+2667228us) spinlock_profiling.cc:243] Waited 2.62 s on lock 0x7b54003dd454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.608788 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6029979904}
I20260809 23:22:01.643231 24545 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:01.632694  4435 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:01.649396 24422 pending_rounds.cc:76] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:22:01.649802 24422 pending_rounds.cc:97] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260809 23:22:01.650110 24422 raft_consensus.cc:2866] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:01.640471  4531 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:01.655314 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.634750  4683 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.656875 24430 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260809 23:22:01.646373  4336 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:01.661708 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.658942 24545 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260809 23:22:01.667476 24320 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:01.656883 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:01.669711 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 169165us
I20260809 23:22:01.644934  4858 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260809 23:22:01.672170 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 275584us
I20260809 23:22:01.644726  4875 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:01.619874  4589 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.656913  4375 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:01.689796 24535 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:01.700310 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:22:01.691900 24425 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:01.650645  4493 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:01.641537  3979 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260809 23:22:01.689893 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.703245 24320 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:01.704737 25365 raft_consensus.cc:595] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:01.661994  4336 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.708544 24314 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:01.705705 24435 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:01.707098 25365 raft_consensus.cc:603] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:01.702805  4589 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:01.714641  4739 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:01.715090  4739 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:01.715668  4739 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:01.699899  4862 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
W20260809 23:22:01.718214  4863 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260809 23:22:01.710390 24430 pending_rounds.cc:76] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 12
I20260809 23:22:01.720261 24430 pending_rounds.cc:97] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260809 23:22:01.678630  4639 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.721585  4743 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:01.722884 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.720551 24430 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:01.723173 24380 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.706660  4375 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.661942  4877 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.195s
I20260809 23:22:01.712436  4336 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:01.680648  4858 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.728129  4743 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.706532  4435 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260809 23:22:01.748965 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.702250  3889 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.706001  4359 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 22.20, Last appended by leader: 20, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.759738  4435 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:01.760164  4435 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:01.761237 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1654) took 4149 ms (client timeout 2826 ms). Trace:
I20260809 23:22:01.761067  4435 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 22.20, Last appended by leader: 20, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:01.761643 24429 rpcz_store.cc:269] 0809 23:21:57.612044 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.903528 (+291484us) service_pool.cc:227] Handling call
0809 23:22:01.760865 (+3857337us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540023f354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.761072 (+   207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3754227456}
I20260809 23:22:01.763546 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1857) took 5150 ms. Trace:
I20260809 23:22:01.763983 25371 rpcz_store.cc:276] 0809 23:21:56.613543 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.115655 (+502112us) service_pool.cc:227] Handling call
0809 23:21:57.115941 (+   286us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:00.857511 (+3741570us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:01.763293 (+905782us) spinlock_profiling.cc:243] Waited 3.71 s on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:01.763486 (+   193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8533614464}
I20260809 23:22:01.711663  4820 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.767237  3889 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.768522  4820 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.770994 25365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:22:01.773056 24496 leader_election.cc:398] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:01.773509 24496 leader_election.cc:401] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:01.747970  4375 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:01.800981 25375 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:01.802062  4899 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:01.752940  4743 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:01.802969 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.803894 25370 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:01.805229 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.805837 25368 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:22:01.806805 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.775537 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.783392 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.783841  4435 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.750720  4858 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:22:01.784567  4678 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:22:01.787324 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.816383 24320 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:01.788054 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.818179 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1850) took 1416 ms. Trace:
I20260809 23:22:01.788242  4757 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:01.788790 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.789517 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.828382 24542 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:01.792721 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.794119 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.797641 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:01.750439  4493 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.802289  4865 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:01.808286 24539 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:01.783013  4881 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260809 23:22:01.807291  4874 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:01.818564 24544 rpcz_store.cc:276] 0809 23:22:00.401625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:00.401933 (+   308us) service_pool.cc:227] Handling call
0809 23:22:01.013712 (+611779us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:01.026783 (+ 13071us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:01.814226 (+787443us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b5400116a50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.818122 (+  3896us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1404121856}
I20260809 23:22:01.813279 24424 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:01.814318 24541 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:01.818461 24323 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:01.781872  4745 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.774013  4566 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.831112 24311 raft_consensus.cc:2354] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:22:01.837524 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.789952  4846 consensus_queue.cc:260] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.838801 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:01.846320 24428 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.001s
I20260809 23:22:01.806676 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.848096 24543 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:01.803067  4841 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:01.801204  4875 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.806480  4657 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.857122 24316 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:01.857733 24317 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:01.857790 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.858592  4493 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.859386 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:01.859589 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62091.4us
I20260809 23:22:01.860549  4336 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.852137 24428 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:01.862308  3979 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:01.863306 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56782.1us
I20260809 23:22:01.863984 24546 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:01.866509 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.867312 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.868237 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.819866  4322 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:01.872920 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 191960us
I20260809 23:22:01.817701  4678 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:01.891995  4657 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.891872  4678 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.892315  4858 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:01.835361  4582 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260809 23:22:01.831179  4865 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:01.893649  4865 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:01.893952 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3727) took 1476 ms (client timeout 1718 ms). Trace:
I20260809 23:22:01.894338  4858 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.828068  4757 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:01.894312 24317 rpcz_store.cc:269] 0809 23:22:00.417941 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:00.447054 (+ 29113us) service_pool.cc:227] Handling call
0809 23:22:01.893750 (+1446696us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540043c1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:01.893869 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3239812864}
I20260809 23:22:01.895155 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.895841 24312 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:01.895689  4757 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.896107  4860 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:01.897079  4780 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:01.897579  4780 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:01.898505  4780 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.848670  4774 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:01.900120  4780 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.880725 24313 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:01.903692  4568 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:01.904109  4568 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:01.905014  4453 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.905089  4568 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.906243  4568 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:01.907435 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.897097  4757 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:01.908179  4875 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:01.847937  4874 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:01.892444  4586 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:01.883282 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 148196us
I20260809 23:22:01.890651 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:01.911489 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:01.916025 25371 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 22
W20260809 23:22:01.915753 25363 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.802s	user 0.011s	sys 0.000s
I20260809 23:22:01.920753 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:01.921581 25368 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:01.923794 24545 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260809 23:22:01.916767 25363 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:01.927750 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.928736 24539 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:22:01.930254 24270 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:01.932013 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1884) took 1109 ms. Trace:
I20260809 23:22:01.932533 25363 rpcz_store.cc:276] 0809 23:22:00.815796 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:00.824527 (+  8731us) service_pool.cc:227] Handling call
0809 23:22:01.924926 (+1100399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:01.933769 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.934167 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.885389  4359 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:01.936431 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:01.936472 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.939563 24543 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:01.922011  4874 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:01.942337 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:01.944151 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.948704  4743 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:01.937667  4639 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.949746  4639 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.951838  4639 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:01.944509 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:01.958045 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44671us
I20260809 23:22:01.966408 24427 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:01.971052 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:01.971809 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:01.944078  4657 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:01.941556  4589 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:01.941723  3979 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:01.957165  4741 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:01.950786  4743 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:01.975275 24380 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:01.983348 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:01.994472 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:01.995075 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50650.4us
I20260809 23:22:01.996038  4745 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:01.999190  4907 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:01.999827  4444 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:02.074713  4745 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.000599  4869 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:02.075312  4869 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:02.075723  4869 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:02.029107 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.769s (SENT)
I20260809 23:22:02.079906 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:02.075757  4745 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:02.081046 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51989.7us
I20260809 23:22:02.081497 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.081787  4444 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:02.082144 24431 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:02.042778 24313 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:02.019795  4359 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.087371  4359 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.087632 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.064858  4741 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:02.089372  4741 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:02.089663  4359 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.047084 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3360) took 4540 ms. Trace:
I20260809 23:22:02.090085  4787 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:22:02.090515  4787 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:02.090371 24423 rpcz_store.cc:276] 0809 23:21:57.504259 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:57.504907 (+   648us) service_pool.cc:227] Handling call
0809 23:22:01.353274 (+3848367us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:01.471224 (+117950us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:02.043865 (+572641us) spinlock_profiling.cc:243] Waited 3.81 s on lock 0x7b54003ea8d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:02.044667 (+   802us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8758859648}
I20260809 23:22:02.074365  3979 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.091468  4787 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:01.974337  4774 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.067s
I20260809 23:22:02.092758  3979 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.095011  3889 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:02.095558  3889 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:02.095963  3889 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.092808  4787 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 18 FOLLOWER]: Advancing to term 19
W20260809 23:22:02.101347  4859 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.899s	user 0.004s	sys 0.000s
I20260809 23:22:02.087568 24422 raft_consensus.cc:1226] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:02.103356  4657 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.104457  4657 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.108450 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:02.108583  4589 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:02.109272 24546 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:02.109812  4589 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Stepping down as leader of term 26
W20260809 23:22:02.109870 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 142040us
I20260809 23:22:02.110325  4589 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.111100  4566 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:02.107386  4869 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.114002 24546 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:22:02.115461  4895 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.254s	user 0.004s	sys 0.000s
I20260809 23:22:02.115574 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:02.116500  3920 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:02.117151  3920 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.117527  3920 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:02.117455  4589 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.119058  4589 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.060268 24311 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:02.074494  4907 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:02.084779 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110335us
W20260809 23:22:02.068252 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73902.3us
I20260809 23:22:02.131459 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.073423 24539 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:22:02.121115 24428 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:02.121992 24535 raft_consensus.cc:1226] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:02.122973  4678 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:22:02.124387 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3746) took 1350 ms. Trace:
I20260809 23:22:02.144009 24545 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 18. Current term is 19. Ops: []
I20260809 23:22:02.131263 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.140674 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.144963 24435 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:22:02.054790 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 147460us
I20260809 23:22:02.143783 24540 rpcz_store.cc:276] 0809 23:22:00.773623 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:00.872054 (+ 98431us) service_pool.cc:227] Handling call
0809 23:22:00.959635 (+ 87581us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:01.203262 (+243627us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:01.203304 (+    42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:01.203471 (+   167us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:01.203528 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:02.124108 (+920580us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b5400115654. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:02.124159 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:02.124304 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":527926144}
I20260809 23:22:02.099126  4875 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:22:02.161532 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50902.8us
I20260809 23:22:02.162307 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.161226 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.163913  4862 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election won for term 19
I20260809 23:22:02.130596  4336 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.122269  4869 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.132391  4683 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.143666  4907 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:02.187402 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.518s (SENT)
I20260809 23:22:02.189651  4862 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:02.197789 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60774us
I20260809 23:22:02.167639  4875 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.175947  4774 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:02.205823 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.212743 24311 raft_consensus.cc:2354] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 19. Current term is 20.
I20260809 23:22:02.207157 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:02.208578 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.209054 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:02.214475 25369 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:02.216284 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:02.217002 25373 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.213631 25363 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
W20260809 23:22:02.221024 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55848.8us
I20260809 23:22:02.216375 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.231607  4445 raft_consensus.cc:2741] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260809 23:22:02.228228  4787 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.233013  4787 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.232945  4862 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.235268  4787 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.235735  4787 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:02.236204  4787 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.238564  4787 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.211992  4875 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:02.227610 24380 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.211789  4774 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.246572 24543 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:22:02.242278 25373 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:02.228179  4308 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.996s	user 0.027s	sys 0.015s
I20260809 23:22:02.248472 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.250716 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.250424 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.253645  4787 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:02.254089 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:02.254546  4787 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:02.255086  4787 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.255110  4109 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:02.255580  4787 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.256196 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.258428  4322 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:02.260304 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.242655  4359 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:02.266281 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.264688 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:02.251901  4308 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.202s	user 0.027s	sys 0.015s
I20260809 23:22:02.251222  4639 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.249732  4774 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:02.272475 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 85126.4us
I20260809 23:22:02.275455 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.252246  4566 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:02.258805  4909 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:02.281963 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:02.282840 25321 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.045s (SENT)
I20260809 23:22:02.283190 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.283339 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:02.285945 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.262521  4912 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:02.269857  4893 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:02.285040 24321 raft_consensus.cc:2362] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23. Re-sending same reply.
I20260809 23:22:02.267767  4846 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:02.288491 24498 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.888s (SENT)
I20260809 23:22:02.284816 25369 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:02.292491 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.293725 24320 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:02.284536  4531 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.584s	user 0.002s	sys 0.015s
W20260809 23:22:02.302101 25321 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.516s (SENT)
I20260809 23:22:02.307775 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.311291  4445 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:22:02.309234 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:02.314090 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 139266us
I20260809 23:22:02.292495  4435 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:02.318866  4322 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:02.319276 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.320209 24317 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:02.320842 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:02.291463  4915 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260809 23:22:02.328305 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:02.331090 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58252.1us
I20260809 23:22:02.331792 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:02.293941  4841 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:22:02.294553  4893 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.333631 25369 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 26. Ops: []
I20260809 23:22:02.299095  4359 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:02.338603 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.340101 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.340608 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.282174  4375 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.325176  4322 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:02.289932  4566 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.343163 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.343333  4308 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:02.362555 25363 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 18. Current term is 19. Ops: []
I20260809 23:22:02.345309 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.348929 24427 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260809 23:22:02.329175  4912 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:02.351838 24317 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:02.349325  4893 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.370409 24321 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:22:02.371970 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:02.372027 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88849.7us
I20260809 23:22:02.372604 24431 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:02.369426  4912 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.373277 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.373967 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:02.375320 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.380403 25370 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 20. Preceding OpId from leader: term: 23 index: 21. (index mismatch)
I20260809 23:22:02.372038 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:02.350646  4359 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.385977  4359 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:02.386139  4919 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260809 23:22:02.386443 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.387152 24317 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:02.388000 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.313845  4531 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.385991  4659 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 13, Time since last communication: 0.054s
I20260809 23:22:02.391884  4659 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:02.392204  4659 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.392493  4531 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.392942  4659 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.393592 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.394085 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.383811 25363 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:02.385082  4507 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.008s
I20260809 23:22:02.385458  4566 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.383813 24314 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:02.394455 24321 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260809 23:22:02.394793  4912 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:02.395208  3889 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.395783  4375 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:02.385121 24320 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
W20260809 23:22:02.416725 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 84938.6us
I20260809 23:22:02.394270  4659 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:02.398416 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.334096  4812 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260809 23:22:02.398844 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.393870  4741 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.362964  3810 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:22:02.439596 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.441329 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:02.441653  4865 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.548s	user 0.013s	sys 0.002s
I20260809 23:22:02.442041 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.442829 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:02.443117  4566 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:02.444201 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.444873  4741 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.445089  4566 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:02.445721  4566 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:02.447126  4909 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:02.448053  4909 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:02.448380  4909 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.448688  4909 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.451097  4109 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:02.442061  4865 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.453922  4865 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:02.455695 24167 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 18 to 20, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.457970 24167 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 19, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:02.362434  4308 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.411720  4493 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.377723  4322 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.467291  4493 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.467582  4308 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.468545  4909 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.402861 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.403818  4611 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:02.469434  4493 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:02.469527  4611 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:02.470250 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.470604  4611 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.471578 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.471184 24167 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 20, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.436830 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.474315  4909 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.475389 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.475662  3810 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.476050  4812 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Stepping down as leader of term 18
I20260809 23:22:02.476467  4812 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.476641  3810 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.437541 25372 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:02.477228  4812 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.478545  4812 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:22:02.479132  4790 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.479485 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.479643  4790 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.482199  3979 raft_consensus.cc:2741] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260809 23:22:02.482812  3979 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:02.478844 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1980) took 1186 ms. Trace:
I20260809 23:22:02.483698 25372 rpcz_store.cc:276] 0809 23:22:01.292011 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:01.292269 (+   258us) service_pool.cc:227] Handling call
0809 23:22:02.478686 (+1186417us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:02.478789 (+   103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2577550720}
I20260809 23:22:02.484225  3979 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:02.484602  3979 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.446631  4566 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.468947  4308 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:02.401473 24498 leader_election.cc:334] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.024s (SENT)
I20260809 23:22:02.497222 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:02.404918  4916 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260809 23:22:02.472086  4611 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:02.413386 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.472121 24309 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:02.473192  4929 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260809 23:22:02.484622  4739 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.768s	user 0.009s	sys 0.009s
I20260809 23:22:02.481773 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.516314  4739 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.518399  4739 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:02.519153  4657 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:02.519784 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:02.521076 24161 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 374) took 16015 ms (client timeout 15000 ms). Trace:
I20260809 23:22:02.413358  4531 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:02.523209  4531 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:02.523608  4531 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:02.519179 24167 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 13 to 19, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:02.524982 24167 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:02.500526  4375 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election won for term 29
W20260809 23:22:02.495294 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62679.2us
I20260809 23:22:02.527037 24167 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.484967  4790 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:02.511623 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110203us
I20260809 23:22:02.529698 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.511610  3979 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.514418  4493 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:02.545727 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:02.521474 24161 rpcz_store.cc:269] 0809 23:21:46.519343 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:46.540576 (+ 21233us) service_pool.cc:227] Handling call
0809 23:22:02.520640 (+15980064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:21:52.292572 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:21:52.411505 (+118933us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:21:52.411571 (+    66us) write_op.cc:435] Acquired schema lock
0809 23:21:52.411596 (+    25us) tablet.cc:660] Decoding operations
0809 23:21:52.424826 (+ 13230us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:21:52.424956 (+   130us) write_op.cc:641] Partition lock acquired for write op
0809 23:21:52.425010 (+    54us) tablet.cc:683] Acquiring locks for 31 operations
0809 23:21:52.514838 (+ 89828us) tablet.cc:699] Row locks acquired
0809 23:21:52.514886 (+    48us) write_op.cc:260] PREPARE: finished
0809 23:21:52.515099 (+   213us) write_op.cc:270] Start()
0809 23:21:52.515253 (+   154us) write_op.cc:276] Timestamp: P: 1786317712515032 usec, L: 0
0809 23:21:52.515296 (+    43us) op_driver.cc:331] REPLICATION: starting
0809 23:21:52.516223 (+   927us) log.cc:842] Serialized 12118 byte log entry
0809 23:21:54.068930 (+1552707us) op_driver.cc:444] REPLICATION: finished
0809 23:21:54.618556 (+549626us) write_op.cc:301] APPLY: starting
0809 23:21:54.676861 (+ 58305us) tablet.cc:1365] starting BulkCheckPresence
0809 23:21:56.835054 (+2158193us) tablet.cc:1368] finished BulkCheckPresence
0809 23:21:56.835181 (+   127us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:21:57.639741 (+804560us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:21:58.447462 (+807721us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=62,key_file_lookups=62,delta_file_lookups=0,mrs_lookups=0
0809 23:21:58.507055 (+ 59593us) write_op.cc:312] APPLY: finished
0809 23:21:58.707443 (+200388us) log.cc:842] Serialized 268 byte log entry
0809 23:21:59.476648 (+769205us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:59.477299 (+   651us) write_op.cc:454] Released schema lock
0809 23:21:59.477518 (+   219us) write_op.cc:341] FINISH: Updating metrics
0809 23:21:59.478160 (+   642us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:21:59.478217 (+    57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":821248,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":472330,"apply.run_cpu_time_us":98185,"apply.run_wall_time_us":4893431,"cfile_cache_hit":126,"cfile_cache_hit_bytes":133583,"num_ops":31,"prepare.queue_time_us":462715,"prepare.run_cpu_time_us":38729,"prepare.run_wall_time_us":568792,"raft.queue_time_us":608899,"raft.run_cpu_time_us":22849,"raft.run_wall_time_us":233066,"replication_time_us":1553522,"thread_start_us":1943263,"threads_started":5,"wal-append.queue_time_us":1290840,"wal-append.run_cpu_time_us":61457,"wal-append.run_wall_time_us":3168030}]]}
I20260809 23:22:02.533694 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.549042  4375 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.549232 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.550081 24422 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:02.534327 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.551806  3979 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:02.552232 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.552886 24425 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:02.553735 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.534992 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.535322 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.555523 24423 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:02.535975 24429 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:02.556272 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.535830  4588 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:02.533733  4823 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.536932 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.540614  4531 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.541560 24314 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260809 23:22:02.546085  4639 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:02.545290 24167 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 18 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.547847  3889 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.548053 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.557423 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.554790 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.596261 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:02.596510 24425 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.521106 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.377995 25371 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:22:02.550143  4375 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.599998 24317 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:02.484455  4912 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.557744 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.563241 24309 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:02.572609 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:02.579922  4757 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.671s	user 0.014s	sys 0.000s
I20260809 23:22:02.565197  4739 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:02.565249  4820 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:02.630810 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.582748  4589 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:02.608583 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.609675 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.617616 24423 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:02.506027  4907 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:02.640640 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 196483us
I20260809 23:22:02.646050 24167 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 21, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.651080 24309 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:02.618235 24316 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:02.642289 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.605581 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.609040  3889 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:02.605372  4639 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:02.645318  4589 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:02.624806  4745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.013s	sys 0.008s
W20260809 23:22:02.664779 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59289.8us
W20260809 23:22:02.665159  4745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.013s	sys 0.011s
I20260809 23:22:02.656178 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:02.632372 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 190383us
I20260809 23:22:02.632342  4308 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:02.594417  4823 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.663616 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.665861 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:02.642827  4874 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.680s	user 0.003s	sys 0.010s
I20260809 23:22:02.683753 24543 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:02.684078 24435 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:02.653968  4739 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.675064  4308 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.694599 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:02.679625 24498 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.257s (SENT)
I20260809 23:22:02.655654  4109 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.668383 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.683135  4912 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.656994  4820 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.698681 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:02.663036  4639 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.703279 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:02.699920 24309 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:02.703742 25366 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:02.698050 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 141839us
I20260809 23:22:02.670193  4589 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.706672 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:02.687062  3920 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.569s	user 0.013s	sys 0.000s
I20260809 23:22:02.707716 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:02.709375 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:02.674970  4901 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:02.711230 24425 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:02.673322  3889 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.712643 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:02.702759  4678 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:02.718724 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.720324 25368 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.721868 24534 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:02.697471  4308 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:02.724160 25323 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.516s (SENT)
I20260809 23:22:02.732947 25372 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:02.714174 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:02.740077 25363 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:02.735253 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:02.735831 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:02.739202 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:02.731051 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.746214 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:02.745654 24167 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 17 to 19, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:02.743189 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:02.753334 24541 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 20. Preceding OpId from leader: term: 23 index: 21. (index mismatch)
I20260809 23:22:02.702847  4109 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.755376 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:02.756134 24540 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:02.756875 24543 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260809 23:22:02.757076 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.758585 24535 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:02.765007 24542 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:02.768535 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:02.769166 24534 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260809 23:22:02.689952  4874 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.771608 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:02.736935  4678 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.707767  4774 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:02.775902 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51781.3us
W20260809 23:22:02.772222 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 92639.2us
I20260809 23:22:02.776939 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.713434  4909 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.051s
I20260809 23:22:02.722985  3920 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.775000 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:02.715842  4588 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260809 23:22:02.797778 24535 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:02.762020  4901 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.747910  3810 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:02.805133 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.805754 24317 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:02.714869  4359 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.811486 24425 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:02.811604  4109 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.812417  4359 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.785157 25365 raft_consensus.cc:595] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:02.787696 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:02.813131 25365 raft_consensus.cc:603] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:02.779173 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:02.767931  4787 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260809 23:22:02.814015 24309 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:02.814150 25375 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:02.813848  4787 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.815024 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.815667 24309 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:02.816054  4787 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.816241  4874 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.816419 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.790161 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.817544 24314 raft_consensus.cc:2354] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:02.803840  4862 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:22:02.818964  4862 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:02.819335  4862 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:02.819198 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.820508  4862 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.821317 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.788822 24167 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 14 to 22, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 22 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.822067  4862 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:02.823601 24167 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:02.824174  4893 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.825445 24422 raft_consensus.cc:2354] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27.
I20260809 23:22:02.825418 24167 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 22, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 22 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.790799 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:02.793493 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:02.827543 25372 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
W20260809 23:22:02.795231 24498 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.500s (SENT)
I20260809 23:22:02.795651 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.772940 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:02.794531  4491 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:02.798897 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:02.829443 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44004.6us
I20260809 23:22:02.829530  4491 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.830051 25371 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:22:02.830446 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.137s (SENT) [suppressed 52 similar messages]
I20260809 23:22:02.805545  3810 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:02.831235 24427 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:02.831337  3810 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:02.831822  4683 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 23:22:02.797601 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:02.829353 24546 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 24
I20260809 23:22:02.833942  4901 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.806610  3889 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.839529  4944 consensus_queue.cc:1046] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:02.840875  4944 raft_consensus.cc:3032] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:02.841058 24546 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:02.841332  4944 raft_consensus.cc:729] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:02.784932  4251 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.131s	user 0.000s	sys 0.018s
W20260809 23:22:02.842275  4931 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260809 23:22:02.842587 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:02.801613 24428 raft_consensus.cc:1261] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:02.771301  4789 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:02.804769 24313 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:02.746069  4790 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260809 23:22:02.805261 24431 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:02.783393  4678 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:02.744372  4875 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.808717  4820 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.809125 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:02.809795  4739 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.816680 24430 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:02.826294  4893 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.875864 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48046 (request call id 33) took 1128 ms. Trace:
I20260809 23:22:02.840446  4639 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.780071  4841 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.895149  4841 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.710050  4936 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260809 23:22:02.800068  4531 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:02.845983 24309 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:02.792029 25366 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:02.783174  4865 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:02.903294  4659 consensus_queue.cc:1159] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 26 status { last_received { term: 23 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:02.905592  4865 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.771624  4657 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.036s
I20260809 23:22:02.907776  4657 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:02.908211  4657 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:02.909184  4657 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.852294 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.911180 24320 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:02.842761  4251 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.859196 24496 leader_election.cc:398] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27.
I20260809 23:22:02.870246 24424 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:02.912777 24496 leader_election.cc:401] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260809 23:22:02.875373 24167 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 12 to 20. New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:02.914973 24544 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:02.873277  4948 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:02.828792  4907 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.875140  4912 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.854453  4359 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:02.944025  4359 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:02.944672  4359 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:02.944031  4912 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:02.870481  4938 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260809 23:22:02.842206  4944 consensus_queue.cc:260] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.896421 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.850036  4846 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:02.854620  4683 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.092s
I20260809 23:22:02.947479  4944 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.947317  4846 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.872040  4507 consensus_queue.cc:1046] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.090s
I20260809 23:22:02.949049  4507 raft_consensus.cc:3032] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:02.949484  4507 raft_consensus.cc:729] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:02.892961 25365 rpcz_store.cc:276] 0809 23:22:01.747429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:01.747579 (+   150us) service_pool.cc:227] Handling call
0809 23:22:02.875649 (+1128070us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:02.875768 (+   119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2330533632}
I20260809 23:22:02.871879  4874 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:02.954919 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:22:02.954687  4874 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:02.939608  4823 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.953115 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:02.964785 24320 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:02.926608  4657 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:02.972715 24311 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 29: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:02.973237 25366 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:02.977398  4875 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:02.830685  4491 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.979712  4743 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:02.950534  4507 consensus_queue.cc:260] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:02.980979  4507 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:02.961925 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 106835us
W20260809 23:22:02.964260  4932 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260809 23:22:03.017961 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:03.008468 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:03.006062  4683 consensus_queue.cc:1035] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260809 23:22:03.007746  4823 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260809 23:22:03.042459  4823 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:03.042986  4823 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:03.015842 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:03.044143 24320 raft_consensus.cc:2396] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:03.045161 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:03.045871 24313 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:03.044154  4823 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.034421  3889 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260809 23:22:03.086509 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 314960us
W20260809 23:22:03.086920 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 323740us
I20260809 23:22:02.987203 24167 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 24, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:03.087553 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.100301 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:03.109512 24309 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:03.111470 24309 raft_consensus.cc:1226] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: [21.14-23.15]
I20260809 23:22:03.130393 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:03.130671  4912 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:03.099216  4939 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.221s	user 0.000s	sys 0.005s
I20260809 23:22:03.133841 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.134445 24424 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:03.135509 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.131232 24544 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:03.138592 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:03.131584 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260809 23:22:03.139593 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52140.1us
I20260809 23:22:03.141255 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:03.147434 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 338911us
I20260809 23:22:03.119410  4774 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.146133 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.144502 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:03.151623 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.152245 24544 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:03.153277 24316 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:03.105830  4846 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.155789  4582 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:03.143438  4639 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:03.159451 24424 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
W20260809 23:22:03.160248 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 60038.6us
I20260809 23:22:03.164556 24317 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:03.166043 24496 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:03.136489  3920 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.167987 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:03.168753 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.168936 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.154114  4912 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:03.180262 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.182161 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:03.176139  4659 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:22:03.187924 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:03.188865 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:03.173764 24167 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 9 to 22, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 22 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:03.151836  4823 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:03.149727  4901 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:03.146957  3979 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:03.204880  3979 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:03.205376  3979 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:03.207222 24544 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:03.164052  4251 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.178503 25373 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 22: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:03.175253 24427 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:22:03.179461 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:03.222231 25366 raft_consensus.cc:3037] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:03.164064  4568 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:03.174901  4846 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.172489 24316 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 18. Current term is 20. Ops: [18.15-18.15]
I20260809 23:22:03.163695 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:22:03.190394 24167 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 20, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 20 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:03.193393 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:03.193821 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:03.195555 25321 leader_election.cc:334] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.534s (SENT)
I20260809 23:22:03.176707 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.217911 24429 raft_consensus.cc:1226] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:03.218992 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:03.232681 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.216171 24312 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:03.176455 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:03.189095 24323 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260809 23:22:03.228760  4841 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.172166  4445 raft_consensus.cc:2781] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:03.227703  3920 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260809 23:22:03.235237 24167 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 16 to 17, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 17 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:03.159946  4493 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260809 23:22:03.175609  4741 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.215942  3979 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.175804  4611 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260809 23:22:03.219563  4435 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.177105  4865 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.198501  4874 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.223384  4901 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.269143 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.296646 24423 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:03.296758 24167 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:03.230589  4568 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:03.284881 25366 raft_consensus.cc:2454] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:22:03.236600  4958 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:03.312868 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 130782us
W20260809 23:22:03.295019  4308 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.017s
I20260809 23:22:03.308583 24427 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:03.307986 24430 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
W20260809 23:22:03.293190  4916 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260809 23:22:03.289999  4251 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:03.315490 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:03.331691 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 103140us
I20260809 23:22:03.326361 24270 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:03.334565 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 158183us
I20260809 23:22:03.335503 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:03.338163 24496 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:03.309751  4445 raft_consensus.cc:686] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:03.339349 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:03.340303  4445 consensus_queue.cc:237] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.341325 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3757) took 1006 ms. Trace:
I20260809 23:22:03.313390  4568 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:03.309191  4745 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:03.300873  3889 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260809 23:22:03.247517  4846 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.327162  4493 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:03.342217 24323 rpcz_store.cc:276] 0809 23:22:02.335074 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:02.335205 (+   131us) service_pool.cc:227] Handling call
0809 23:22:03.188876 (+853671us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:03.235332 (+ 46456us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:03.341056 (+105724us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54001afc50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:03.341251 (+   195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944694016}
W20260809 23:22:03.359576 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45798.5us
I20260809 23:22:03.361692 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:03.344604  4659 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:03.367869 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.279594  3810 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.330955  4251 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:03.347868  4865 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.384395 24430 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:03.304395  4435 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.287611  4741 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:03.372783 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 103546us
I20260809 23:22:03.392967 25365 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:03.378445  3889 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260809 23:22:03.372059 25366 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:03.388574 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.393898 24321 raft_consensus.cc:2379] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:03.394513 24313 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:03.396983  3810 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.397400  4841 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:03.398439  4375 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:03.408674 24431 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:03.408591 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.410342 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.369935  4493 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:03.412791 24266 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:03.371129  4780 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:03.334600  4308 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.416700 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.430873 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.326093  4901 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.388182  4251 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
W20260809 23:22:03.440176 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46643.5us
I20260809 23:22:03.393033  4659 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:03.445466 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:03.400663  3889 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:03.452287  3889 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 18 FOLLOWER]: Advancing to term 19
I20260809 23:22:03.454670  3979 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:03.462397  3979 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:03.428892  3810 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:03.465219  3979 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:03.399416  4865 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.428261  4588 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:03.472224  4659 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.477466 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.478368 24429 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:03.479513  4741 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.479658  4948 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:03.482470  4865 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.486160  4774 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:03.486667  4774 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:03.487816  4774 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.489624  4774 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:03.408028  4375 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:03.507483 24546 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:03.429121  4820 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:03.491868 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:03.510450 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 69595.2us
W20260809 23:22:03.512153 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62987.8us
I20260809 23:22:03.515272 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:03.516587 24266 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.518679 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:03.520131 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 343467us
I20260809 23:22:03.523015 25370 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260809 23:22:03.513772 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 318226us
I20260809 23:22:03.515122 24535 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:03.535051 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:03.535946 25366 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:03.537724 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:03.528110 25321 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.802s (SENT)
I20260809 23:22:03.667407 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.574041 24423 raft_consensus.cc:3032] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:03.538717 25373 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:03.545931 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3170) took 1030 ms. Trace:
I20260809 23:22:03.669811 24312 rpcz_store.cc:276] 0809 23:22:02.515707 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:02.605875 (+ 90168us) service_pool.cc:227] Handling call
0809 23:22:03.545680 (+939805us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400225454. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:03.545854 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":313652608}
I20260809 23:22:03.620250 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.542394 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.673374 24430 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:03.577325 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 55735.2us
I20260809 23:22:03.594322 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.674777 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:03.672348 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.675531 24431 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:03.575069 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59866.9us
I20260809 23:22:03.676692 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:03.678040 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.665771 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.722503 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.721649 24422 raft_consensus.cc:2379] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:03.668329 24423 raft_consensus.cc:729] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:03.678408 24378 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:03.620292  4812 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.127s	user 0.014s	sys 0.009s
W20260809 23:22:03.620250  4359 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.675s	user 0.014s	sys 0.002s
W20260809 23:22:03.663928  4967 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260809 23:22:03.527160  3810 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:03.610047  4975 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.594564  4820 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.154s
I20260809 23:22:03.612082  4973 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:03.733666  4978 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.759665  4978 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.762396  4978 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:03.763857  4944 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.816s	user 0.011s	sys 0.004s
I20260809 23:22:03.658701  4958 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.764937  4958 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.765380  4841 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.765887  4841 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.767957  4841 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:03.622750  4493 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:03.768788 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 94143.6us
I20260809 23:22:03.769613  4757 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:03.770139  4757 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:03.737205  4359 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.770977  4757 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.771906  4359 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.772589  4359 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:03.773038  4359 raft_consensus.cc:686] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:03.773824  4359 consensus_queue.cc:237] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.775194 24314 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:22:03.775851 24435 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:03.776974  4874 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:03.780210  4901 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.780495  4790 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:03.780655 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 252653us
I20260809 23:22:03.781530  4790 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:03.781616 25321 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.864s (SENT)
I20260809 23:22:03.783025 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:03.785647  4948 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:03.787111 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:03.771982  4757 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:03.747304 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.723011 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.769640 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:03.814321 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44760.2us
I20260809 23:22:03.816035 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:03.799142 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:03.732585 24427 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:22:03.683255  4678 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.808s	user 0.009s	sys 0.010s
I20260809 23:22:03.864910 24435 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260809 23:22:03.866948 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.566895  4611 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:03.821847 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3156) took 1322 ms. Trace:
W20260809 23:22:03.809085  4743 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.829s	user 0.011s	sys 0.012s
W20260809 23:22:03.737609  4981 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260809 23:22:03.872956 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.874084 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:03.879488 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:03.881529 24430 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:03.881826 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:03.755446  4962 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.427s	user 0.000s	sys 0.003s
I20260809 23:22:03.910068 24535 raft_consensus.cc:2379] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:03.821741  4862 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.999s	user 0.019s	sys 0.001s
I20260809 23:22:03.739233  4971 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.930787 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.925061 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.931757 24430 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:03.863139 24423 consensus_queue.cc:260] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:03.851565  4657 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.880s	user 0.010s	sys 0.006s
I20260809 23:22:03.874771 24314 rpcz_store.cc:276] 0809 23:22:02.499146 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:02.820858 (+321712us) service_pool.cc:227] Handling call
0809 23:22:03.821645 (+1000787us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:03.821780 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2191636352}
I20260809 23:22:03.810410  4790 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.949369  4308 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:03.954975  4993 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:03.955190  4790 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.929431 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.758085  4957 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:03.930233 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.748790  4820 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:03.863353  4841 raft_consensus.cc:2741] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260809 23:22:03.938104 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122142us
I20260809 23:22:03.929922  4984 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:03.931963 24422 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:03.874194  4948 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:03.940032 24423 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:03.916448  4823 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.673287  4375 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:03.948792  4507 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.967s	user 0.016s	sys 0.000s
I20260809 23:22:03.863636  3979 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:03.910332  4435 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.883528  4846 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:03.880440  4985 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260809 23:22:03.951701  4589 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.218s	user 0.027s	sys 0.004s
I20260809 23:22:04.003042  4589 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.003675  3979 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.005215  4589 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.923753  4743 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27.
I20260809 23:22:03.965574 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.006467 24435 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:22:04.008217 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3819) took 1280 ms. Trace:
I20260809 23:22:03.971313 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.009275 24425 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:03.810668  4780 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:04.008559 24545 rpcz_store.cc:276] 0809 23:22:02.728128 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:02.728833 (+   705us) service_pool.cc:227] Handling call
0809 23:22:02.729283 (+   450us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:02.755723 (+ 26440us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:02.774529 (+ 18806us) log.cc:842] Serialized 14 byte log entry
0809 23:22:03.011628 (+237099us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b18004ed4e0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:03.156115 (+144487us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:03.159649 (+  3534us) log.cc:842] Serialized 26 byte log entry
0809 23:22:03.159803 (+   154us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:03.159831 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:03.159921 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:03.190698 (+ 30777us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:22:03.754726 (+564028us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:04.008039 (+253313us) raft_consensus.cc:1671] finished
0809 23:22:04.008061 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:04.008166 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40272,"prepare.queue_time_us":19523,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":3292,"raft.queue_time_us":62353,"raft.run_cpu_time_us":2992,"raft.run_wall_time_us":406234,"spinlock_wait_cycles":114382464,"thread_start_us":132425,"threads_started":1,"wal-append.queue_time_us":439433}
I20260809 23:22:04.010540  4780 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.973831 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:03.975391 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.015188 24431 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:03.976190 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.016185  4790 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:04.021714  4582 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.844s	user 0.008s	sys 0.002s
I20260809 23:22:03.977428 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.754205  4974 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.192s
I20260809 23:22:04.036120 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.035890 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.037478 24317 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 29: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:03.979524 25323 leader_election.cc:398] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:22:04.038126 25323 leader_election.cc:401] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:03.975593 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.873471  4678 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.984272 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.040095 24321 raft_consensus.cc:2396] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260809 23:22:03.942589  4971 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.988899 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.747758  4975 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:03.991873 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:03.992457 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:03.956985  4958 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.976961  4812 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:03.964849  4308 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:03.974629  4820 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:04.001577 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:04.026508  4568 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.683s	user 0.003s	sys 0.006s
I20260809 23:22:04.027055  4611 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:04.027631 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:03.976416 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:03.991897  4823 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.081281  4971 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.041924  4743 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.082477  4743 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.084519  4743 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:04.085002  4975 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.086308 24314 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:04.029158 24539 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:04.029726  4948 consensus_queue.cc:1046] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:03.981401  4683 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.087425  4683 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.089586  4683 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:03.978804  4984 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:04.033424 25365 consensus_meta.cc:287] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.634s	user 0.012s	sys 0.001s
I20260809 23:22:04.033512 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.090178  4984 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:04.010821 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.089898  4493 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.091673 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.092258 24545 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.035878 24542 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:03.988019 24424 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
W20260809 23:22:04.036923  4739 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.014s	sys 0.041s
I20260809 23:22:04.046357 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.097071 24427 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:04.046726 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.047322 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:04.097769 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 419405us
I20260809 23:22:04.048381 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.049083 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.050024 24431 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
W20260809 23:22:04.040318  3889 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.004s
I20260809 23:22:04.000735  4985 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:04.040467  4582 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:04.072556  4568 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.073601  4678 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:04.096915  4739 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.015s	sys 0.041s
I20260809 23:22:04.097203 24317 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:22:04.101892 24544 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.019s	sys 0.046s
W20260809 23:22:04.116673 24544 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.019s	sys 0.047s
I20260809 23:22:04.102274 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:04.102176 24321 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260809 23:22:04.101070 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.104348 24316 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:04.105610 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.108991 24535 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:04.056989  4973 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:04.120733 24542 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:04.104300 24423 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:04.061897  4308 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:04.090620 25365 raft_consensus.cc:2396] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 20 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260809 23:22:04.113663 24546 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
W20260809 23:22:04.114581 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 329435us
I20260809 23:22:04.096918  4790 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22.
I20260809 23:22:04.081326  4823 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.122234 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.123593 24539 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:04.125212 24542 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:04.128667 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.132103 24535 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:22:04.130750 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 158253us
I20260809 23:22:04.133641 24535 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260809 23:22:04.135003 25368 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:04.135771 24423 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260809 23:22:04.135018 24535 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260809 23:22:04.138162 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.092823  4958 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260809 23:22:04.145833 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:04.146175 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.107501  4985 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:04.140759 24535 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:04.140638 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:04.151466 24424 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:04.124306  4639 consensus_meta.cc:287] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.952s	user 0.007s	sys 0.005s
I20260809 23:22:04.155459 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.103741  4611 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260809 23:22:04.147241 24425 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:04.147186 25363 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:04.149997 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.162227 25372 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:04.150424 25366 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:04.134353 25323 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.153080 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:04.163885 24431 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.071689  4812 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:04.131376 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.903s (SENT)
I20260809 23:22:04.166014 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.138833  4678 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:04.139501  4874 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:04.110119  3889 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.151561  4846 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:22:04.182698 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48436.2us
W20260809 23:22:04.171677  4251 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.662s	user 0.010s	sys 0.003s
I20260809 23:22:04.185063 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:04.170516  4745 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.000s	sys 0.025s
I20260809 23:22:04.197839  3920 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:04.198480 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1989) took 1011 ms. Trace:
I20260809 23:22:04.207563 25365 rpcz_store.cc:276] 0809 23:22:03.187147 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.187625 (+   478us) service_pool.cc:227] Handling call
0809 23:22:04.198250 (+1010625us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:04.198415 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470015232}
I20260809 23:22:04.207945 24431 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:04.208132 25373 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260809 23:22:04.199314  4639 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:04.175127  4678 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.201027  4375 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.175179  4841 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:04.159384  4823 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260809 23:22:04.175776  4790 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260809 23:22:04.215348 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 350422us
I20260809 23:22:04.212407 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.195219 24424 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
W20260809 23:22:04.221698 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 207089us
I20260809 23:22:04.228421 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.229337 24425 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:04.230034 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.193650 24430 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:04.233400 24427 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:04.198915  3889 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.180089  4846 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.167160  4973 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260809 23:22:04.208559 25373 pending_rounds.cc:76] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 14
I20260809 23:22:04.244374 25373 pending_rounds.cc:97] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260809 23:22:04.244762 25373 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:04.166718  4308 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.131310  4820 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.236734 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:04.209223  4745 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.000s	sys 0.026s
I20260809 23:22:04.238530 24423 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260809 23:22:04.214962  4957 raft_consensus.cc:2741] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260809 23:22:04.243247 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:04.185133 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53803.8us
I20260809 23:22:04.246548 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:04.251082 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.241784 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:04.219913  4639 consensus_queue.cc:1225] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260809 23:22:04.191653  4251 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.291494 24430 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:22:04.255817 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:04.296962 24431 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.236588  5003 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:04.255033  4874 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.249580  4862 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:04.242086  4657 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:04.312439 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100106us
I20260809 23:22:04.312497  4657 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.313035  4999 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:04.313216 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.313778  4999 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:04.315117  4999 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:04.316257 25321 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 2.005s (SENT)
I20260809 23:22:04.320554 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.321298  4790 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:04.321642  4790 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:04.321949  4790 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:04.331287  4841 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:04.334322 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.335408  4568 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.336143  5010 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:04.336716  5010 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.337069  4812 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.337538 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.338147 24535 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:04.337105  5010 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.346459 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.353168 24535 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:04.350380  5010 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.356508  5010 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.236119  3810 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:04.359596  3810 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.362250  3810 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.363113  5013 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:04.365388 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.367163 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.372900 24429 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:22:04.248097  4846 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:04.335258 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90891.6us
I20260809 23:22:04.212357  3920 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.335172  4790 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.246559  4949 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.404383 24496 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:04.405021  4998 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260809 23:22:04.405738  4739 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:04.407168  4251 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.407439  4739 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:04.407964  4739 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.408351  4739 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.408270 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.409287 25363 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:04.292661  4308 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:04.388672  4862 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:04.247344  4582 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.173s
I20260809 23:22:04.422814  3920 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.424453  4582 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.387300 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3882) took 1254 ms. Trace:
W20260809 23:22:04.427116 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59124us
I20260809 23:22:04.428095 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.314690  4657 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.384156  4639 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.397991 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.400123  4957 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.385270  4568 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:04.376302  4678 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.404158  4949 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.430756 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.370323 24545 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:04.426537 24428 rpcz_store.cc:276] 0809 23:22:03.133297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.135573 (+  2276us) service_pool.cc:227] Handling call
0809 23:22:03.151583 (+ 16010us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:03.156987 (+  5404us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:03.157018 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:03.164477 (+  7459us) log.cc:842] Serialized 26 byte log entry
0809 23:22:03.787627 (+623150us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b18002835a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:04.179170 (+391543us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:04.179207 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:04.179270 (+    63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:04.370838 (+191568us) raft_consensus.cc:1671] finished
0809 23:22:04.370875 (+    37us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:04.387219 (+ 16344us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3567,"raft.queue_time_us":5861,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":38448,"spinlock_wait_cycles":2406144,"thread_start_us":391323,"threads_started":1,"wal-append.queue_time_us":1141824}
I20260809 23:22:04.435854 24429 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260809 23:22:04.427788 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 255040us
I20260809 23:22:04.447497 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:04.469197 24430 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.469878 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.473804 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:04.474078 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 235126us
I20260809 23:22:04.475091 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.452272 25372 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:04.482699 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.425076  4582 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:04.457551 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 234960us
I20260809 23:22:04.492817 25366 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260809 23:22:04.460544 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 314760us
I20260809 23:22:04.455237  4251 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:04.496464  4251 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.496877  4251 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:04.484875 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.487465 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.488446 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.488876 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.493275 25368 raft_consensus.cc:2396] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260809 23:22:04.481846  4974 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:04.472587  4945 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.040s
I20260809 23:22:04.494823 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.505996 25369 raft_consensus.cc:2396] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:04.505007  4973 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:04.377115  4985 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.485810  3810 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:04.500305 24430 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:04.503921 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.504673 25371 raft_consensus.cc:2354] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
I20260809 23:22:04.461457  4957 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.478704  4589 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:04.509049 25366 raft_consensus.cc:1261] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260809 23:22:04.509049 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.455019  4812 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260809 23:22:04.443948 24431 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:04.526476 24269 leader_election.cc:398] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
I20260809 23:22:04.529212 24269 leader_election.cc:401] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:04.506839  4974 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.484930  4657 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260809 23:22:04.529991 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:04.529976 25371 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:04.531580 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.533607 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.534750 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.542884 25363 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260809 23:22:04.543578 25363 pending_rounds.cc:76] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 12
I20260809 23:22:04.543996 25363 pending_rounds.cc:97] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260809 23:22:04.540843 25368 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
W20260809 23:22:04.528440  4757 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.708s	user 0.011s	sys 0.000s
I20260809 23:22:04.540659 24545 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:04.544251 25363 raft_consensus.cc:2866] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:04.548554 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.549126 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.549522 25368 raft_consensus.cc:2379] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 21: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:04.549729 24314 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.551084 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.552838 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.554900 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:04.556169 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.553462 25366 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:04.558125 24317 raft_consensus.cc:1226] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:04.551332 25373 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:04.558640 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.559437 25366 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:22:04.501124  5011 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.050s
I20260809 23:22:04.528455  4589 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.560111 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.563970 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.566432 24313 raft_consensus.cc:2379] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:04.564162 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.568468 25375 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:04.524719  4973 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:04.531741  4976 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:04.537731  4974 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.551697  4945 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 21
I20260809 23:22:04.576398 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.578142  4999 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:04.602888 25375 raft_consensus.cc:2379] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 19: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:04.572939  4589 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.578639 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:04.584667 24314 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:04.587174  4957 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:04.588927  5004 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:04.589623 24312 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
W20260809 23:22:04.588518 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.885s (SENT)
W20260809 23:22:04.590884  5012 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:22:04.594358 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:04.596465 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.617684 24321 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:04.598101 24540 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:04.599812 25373 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:04.606489 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46464us
I20260809 23:22:04.607501 24312 pending_rounds.cc:76] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 14
I20260809 23:22:04.574322 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:04.608485 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.914s (SENT)
I20260809 23:22:04.531059  4984 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.618707 24312 pending_rounds.cc:97] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260809 23:22:04.619123 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.619372 24311 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:04.625216 24312 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:04.607663 24498 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.621680 25369 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:04.637300 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.638597 24378 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.591866  4976 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:04.650761 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.656719 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:04.657146 25375 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:04.607309  5004 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:04.565322  4823 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260809 23:22:04.660606 25370 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.675490 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.558276  5018 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.106s
W20260809 23:22:04.666440  4774 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.176s	user 0.011s	sys 0.006s
I20260809 23:22:04.640043  5003 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:04.712262 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 123755us
I20260809 23:22:04.716374 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.718129 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:04.719897 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 216076us
I20260809 23:22:04.711845 25370 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:04.662134  4678 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:22:04.617236  4996 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
I20260809 23:22:04.674293  5004 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:04.683845  4790 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.731406 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:04.686872  5010 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:04.690917  4862 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:04.707397  4823 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:04.706144  4757 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:04.617009  5026 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.127s
I20260809 23:22:04.728963  4949 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.731206  4774 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:04.743242  4657 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:04.768813 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:04.772101 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.772979 24535 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
W20260809 23:22:04.775014 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 43664.4us
I20260809 23:22:04.776597 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:04.778117 24498 leader_election.cc:398] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:04.778460 24498 leader_election.cc:401] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:04.750510  5039 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:04.753137  5030 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:04.761282  4582 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:04.801889  5010 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24.
I20260809 23:22:04.803620 24496 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:04.762955  3889 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:04.806550  3889 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:04.807205  3889 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:04.807386  5011 raft_consensus.cc:2781] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:04.808097  5011 raft_consensus.cc:686] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:04.809198  5011 consensus_queue.cc:237] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.782912 25325 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:04.811960 25325 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260809 23:22:04.760663  4823 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:04.748139  5029 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260809 23:22:04.772346  4657 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:04.760205  4757 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.772158  4985 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.787086  4774 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:04.731374  4973 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:04.830546 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 101113us
I20260809 23:22:04.799850  4745 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:04.800434  5030 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:04.732774  4976 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.841773 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:04.808300  3889 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.827587  5038 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:04.850135 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 72168.4us
I20260809 23:22:04.851130 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:04.848053 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 116115us
W20260809 23:22:04.856474  5031 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260809 23:22:04.859154  4507 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:04.863428  5021 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260809 23:22:04.868563  4846 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.869928  4251 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.862701  4976 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:04.841822  4774 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:04.819643  5003 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.790949  4739 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.874951 24321 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:04.822852  5036 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260809 23:22:04.885460 25365 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:04.847152  5014 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
W20260809 23:22:04.862706  4493 consensus_meta.cc:287] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.771s	user 0.008s	sys 0.002s
I20260809 23:22:04.826386  5018 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260809 23:22:04.845204  5030 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:04.899922 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48891.2us
I20260809 23:22:04.854071  4984 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:04.901280 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.908532 24321 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:04.898401 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:04.902357 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:04.820290  5039 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:04.904985 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 310744us
I20260809 23:22:04.904244 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:04.891312  4774 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:04.911906 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:04.922767 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:04.871675  4823 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:04.923439 24535 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:04.965265 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 59125.5us
I20260809 23:22:04.841159  4862 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:04.958786  5009 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.050s	user 0.002s	sys 0.004s
W20260809 23:22:04.959761 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 643605us
I20260809 23:22:04.960525  5039 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:04.872011  4657 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:04.911079  4846 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.001197 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:05.217554  5020 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.532s	user 0.001s	sys 0.005s
W20260809 23:22:05.057804  5043 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
W20260809 23:22:05.257887 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 256770us
I20260809 23:22:05.258496 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3185) took 1222 ms. Trace:
I20260809 23:22:05.258955  4841 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:05.007299 24535 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:05.258890 24323 rpcz_store.cc:276] 0809 23:22:04.035775 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.063735 (+ 27960us) service_pool.cc:227] Handling call
0809 23:22:04.064076 (+   341us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:04.092679 (+ 28603us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:04.092710 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:04.107841 (+ 15131us) log.cc:842] Serialized 26 byte log entry
0809 23:22:04.464059 (+356218us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:04.747271 (+283212us) log.cc:842] Serialized 14 byte log entry
0809 23:22:04.747571 (+   300us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:04.747668 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:05.216828 (+469160us) spinlock_profiling.cc:243] Waited 7.99 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:05.258179 (+ 41351us) raft_consensus.cc:1671] finished
0809 23:22:05.258282 (+   103us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:05.258438 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28217,"prepare.queue_time_us":368993,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":123557,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":3896,"spinlock_wait_cycles":24112640,"thread_start_us":593588,"threads_started":2,"wal-append.queue_time_us":487171}
I20260809 23:22:05.231297  4985 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.042059  5063 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.288s
I20260809 23:22:05.235572  5077 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.263761  5077 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.263013  4985 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.262959 25325 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:04.975494  4984 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:05.265487  4984 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:05.265885  4984 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:05.266522 24546 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1877) took 1468 ms (client timeout 1918 ms). Trace:
I20260809 23:22:05.060017 24534 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:05.267246  4985 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:05.266955 24546 rpcz_store.cc:269] 0809 23:22:03.798536 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.799744 (+  1208us) service_pool.cc:227] Handling call
0809 23:22:05.266308 (+1466564us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54000ed3d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:05.266464 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":532851456}
I20260809 23:22:05.268602  4985 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:05.270092  5025 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:05.270563  5025 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:05.271095 24435 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:05.271498 24435 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:05.271435  5025 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.272743 24435 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:05.199651  5051 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.328s	user 0.002s	sys 0.000s
I20260809 23:22:05.273732 24435 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:04.923889  5037 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:05.193620 24542 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
W20260809 23:22:05.132391 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 283035us
I20260809 23:22:05.182144 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3183) took 1147 ms. Trace:
I20260809 23:22:05.275883 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 1997) took 1147 ms. Trace:
W20260809 23:22:05.110288 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 143500us
I20260809 23:22:05.207962 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3832) took 1247 ms. Trace:
I20260809 23:22:05.276402 25370 rpcz_store.cc:276] 0809 23:22:04.128248 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.128380 (+   132us) service_pool.cc:227] Handling call
0809 23:22:05.274886 (+1146506us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54007062d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:05.275827 (+   941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1214194048}
I20260809 23:22:05.281291 25363 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:05.283363 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:05.258984 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:05.049238 24321 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:22:05.287092 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 395600us
W20260809 23:22:04.999954 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 89187.2us
W20260809 23:22:05.288086 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.001s	sys 0.009s
W20260809 23:22:05.288379 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.001s	sys 0.009s
I20260809 23:22:05.001268 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:22:05.109967 24543 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:05.280007 24540 rpcz_store.cc:276] 0809 23:22:03.960281 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.960895 (+   614us) service_pool.cc:227] Handling call
0809 23:22:05.207656 (+1246761us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:05.207867 (+   211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1040165760}
I20260809 23:22:05.276286 24309 rpcz_store.cc:276] 0809 23:22:04.035081 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.035189 (+   108us) service_pool.cc:227] Handling call
0809 23:22:04.047993 (+ 12804us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:04.093233 (+ 45240us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:04.093261 (+    28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:04.099749 (+  6488us) log.cc:842] Serialized 52 byte log entry
0809 23:22:04.415416 (+315667us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:22:04.520115 (+104699us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:04.760705 (+240590us) log.cc:842] Serialized 14 byte log entry
0809 23:22:04.761096 (+   391us) log.cc:842] Serialized 14 byte log entry
0809 23:22:05.174379 (+413283us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b18003e5b40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac238583d0 00007fac2386a576 00007fac2387b0d7
0809 23:22:05.174550 (+   171us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:05.174648 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:05.174708 (+    60us) raft_consensus.cc:1671] finished
0809 23:22:05.174727 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:05.182034 (+  7307us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44836,"prepare.queue_time_us":461131,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":247992,"raft.run_cpu_time_us":3004,"raft.run_wall_time_us":328511,"spinlock_wait_cycles":10988288,"thread_start_us":718146,"threads_started":3,"wal-append.queue_time_us":545576}
W20260809 23:22:05.264571  5055 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260809 23:22:05.047325 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3833) took 1081 ms. Trace:
I20260809 23:22:05.309357 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:05.313413 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:05.312062 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:05.191735  4968 raft_consensus.cc:3032] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
W20260809 23:22:05.055459  5060 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
W20260809 23:22:04.974169  5058 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260809 23:22:05.335474 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:05.347761 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:05.319774 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:05.212554  4820 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:05.189728  4589 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.554s	user 0.005s	sys 0.005s
W20260809 23:22:05.228554  4639 consensus_meta.cc:287] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.774s	user 0.010s	sys 0.001s
W20260809 23:22:05.052186  5041 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260809 23:22:05.349215 24380 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:05.334239 24541 rpcz_store.cc:276] 0809 23:22:03.966073 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.966580 (+   507us) service_pool.cc:227] Handling call
0809 23:22:03.967002 (+   422us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:03.967388 (+   386us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:03.967428 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:03.989576 (+ 22148us) log.cc:842] Serialized 26 byte log entry
0809 23:22:04.464476 (+474900us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:04.464529 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:04.464598 (+    69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:05.047104 (+582506us) raft_consensus.cc:1671] finished
0809 23:22:05.047126 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:05.047265 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8996,"raft.queue_time_us":18965,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":4922,"thread_start_us":427651,"threads_started":1,"wal-append.queue_time_us":569343}
I20260809 23:22:05.272768  5025 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:05.177629  5079 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:05.357239 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:05.291051  4985 raft_consensus.cc:686] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:05.057070  4739 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:05.359347 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.051249  5065 raft_consensus.cc:2781] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:05.354709  4948 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:05.058772  4251 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:05.308746  4944 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.338554  5063 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:05.345306  5057 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
W20260809 23:22:05.348959  4568 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.889s	user 0.013s	sys 0.000s
I20260809 23:22:05.368597 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.333657  4993 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:05.127223 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.345012  3920 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:05.373548 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:05.372375 24541 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:05.372838 25363 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:05.379364 24540 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:05.055779  4757 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.359098  4841 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:05.393497 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2001) took 1047 ms. Trace:
I20260809 23:22:05.338961  4968 raft_consensus.cc:729] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:05.394182 24546 raft_consensus.cc:1226] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:22:05.040599  4678 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.358162  5018 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:05.028970  5042 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.349s
I20260809 23:22:05.366009  4639 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:05.398042 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.012s	sys 0.013s
I20260809 23:22:05.403594 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:05.425545 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.012s	sys 0.013s
I20260809 23:22:05.372547  5079 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:05.406726 25365 rpcz_store.cc:276] 0809 23:22:04.346042 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.346178 (+   136us) service_pool.cc:227] Handling call
0809 23:22:05.393317 (+1047139us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b5400443c54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:05.393452 (+   135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1210201984}
I20260809 23:22:05.383608  5065 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:05.370944  5082 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:05.377381  4588 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:05.360590  4589 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.403756 24430 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:05.425971 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 48202.6us
I20260809 23:22:05.409144  5063 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:05.405544  4568 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:05.426187 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 832565us
I20260809 23:22:05.420459  3920 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:05.396744  4944 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:05.458415 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54904.1us
I20260809 23:22:05.459129  4639 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:05.422158  4678 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.459445  5063 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.461244  4678 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.448628  4745 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:05.400799  4359 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:05.429400  5082 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:05.431528  5065 raft_consensus.cc:3037] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:05.428380  4968 consensus_queue.cc:260] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.452926 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3900) took 1028 ms. Trace:
I20260809 23:22:05.455029  5067 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.455657  5062 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:05.456255  5024 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260809 23:22:05.457126  4974 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.839s	user 0.012s	sys 0.000s
I20260809 23:22:05.451273  3979 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:05.406267  4993 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.513290  3920 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.434023  4588 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.459600  4639 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:05.468706 25363 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:05.437230  5037 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
I20260809 23:22:05.475450 24423 rpcz_store.cc:276] 0809 23:22:04.424422 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.424877 (+   455us) service_pool.cc:227] Handling call
0809 23:22:04.425216 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:04.885675 (+460459us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:04.885710 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:04.886173 (+   463us) log.cc:842] Serialized 26 byte log entry
0809 23:22:05.368027 (+481854us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:22:05.369581 (+  1554us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:05.371876 (+  2295us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:05.452703 (+ 80827us) raft_consensus.cc:1671] finished
0809 23:22:05.452729 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:05.452862 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":151332,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":918739,"threads_started":2,"wal-append.queue_time_us":553125}
I20260809 23:22:05.459535 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:05.425881  4984 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.459535  4944 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.544580  5024 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:05.559881 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 300966us
I20260809 23:22:05.520201  4359 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:05.494742  5082 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:05.572832  5036 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.010s	sys 0.013s
I20260809 23:22:05.547031  3979 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:05.595958 24311 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260809 23:22:05.426657  5018 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:05.577505  5004 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.827s	user 0.007s	sys 0.006s
W20260809 23:22:05.586612 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 258406us
W20260809 23:22:05.593226 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 133769us
I20260809 23:22:05.572724  4996 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:05.571353  4968 raft_consensus.cc:3037] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:05.584968  4945 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.976s	user 0.010s	sys 0.004s
I20260809 23:22:05.597376 24435 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260809 23:22:05.444480  4985 consensus_queue.cc:237] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.603989 24321 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260809 23:22:05.598109  5036 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.010s	sys 0.013s
I20260809 23:22:05.580454  4639 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:05.444113  5079 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.613137 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:05.594867  5038 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.706s	user 0.010s	sys 0.002s
I20260809 23:22:05.591833  5024 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:05.606050 24435 pending_rounds.cc:76] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:22:05.586099  4659 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260809 23:22:05.592308  5077 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.615646 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:05.615509 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:05.615774 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:05.608004  5030 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.684s	user 0.010s	sys 0.000s
W20260809 23:22:05.664371  4976 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.770s	user 0.013s	sys 0.000s
I20260809 23:22:05.664578  5030 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.666569  5030 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:05.606019  4774 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.001s
I20260809 23:22:05.667196  4774 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.669353  4774 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:05.670002 24431 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:05.610911  4973 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:05.597002  4812 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:05.559268  4974 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.650817  3979 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.652697  4984 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:05.547518  5067 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.655413  4993 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:05.656663  4359 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.659896  4659 consensus_queue.cc:1225] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:05.661389 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:05.663853 24314 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:05.663868  4589 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:05.745059 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:05.674153 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.671445  5038 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:05.675814 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.745965 24430 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:05.676245 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.636802 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:05.644124 24435 pending_rounds.cc:97] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260809 23:22:05.747107 24435 raft_consensus.cc:2866] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:05.675577 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:05.747406 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52020.7us
I20260809 23:22:05.679219 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:05.615875  4945 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21.
I20260809 23:22:05.681299 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.748132 25370 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:05.559867  5062 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.639039  5088 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.611070  4996 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:05.689363 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76319.4us
I20260809 23:22:05.690133 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:05.770368 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:05.693648 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.694015 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.695533 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.696154 24546 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260809 23:22:05.745246 24540 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.21-23.21]
W20260809 23:22:05.745620 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61735.8us
I20260809 23:22:05.676608 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.773509 24539 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:05.774446 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:05.775801 24539 raft_consensus.cc:2354] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:05.682729 24545 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:05.680366 24431 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:05.747798 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3407) took 1604 ms. Trace:
I20260809 23:22:05.774576 25370 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:05.777298 24435 rpcz_store.cc:276] 0809 23:22:04.143229 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.143373 (+   144us) service_pool.cc:227] Handling call
0809 23:22:04.143735 (+   362us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:05.270962 (+1127227us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54004a1120. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:22:05.747584 (+476622us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:05.747750 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881582976}
I20260809 23:22:05.753297 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.756558 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.778528 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.677496  5036 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:05.767680 24320 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:05.642823  4745 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.065s
I20260809 23:22:05.608951  5004 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.778486 24423 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:05.787388 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:05.789156 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99121.7us
I20260809 23:22:05.693552  4812 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.780956 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.789724 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:05.787955 24423 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:05.780941 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:05.792055 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93834.6us
I20260809 23:22:05.694769  5079 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:05.793690 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:05.678872  4862 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.090s
W20260809 23:22:05.781798  4611 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260809 23:22:05.739761  5087 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.806759 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.835142 24425 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 21: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:05.799028 24314 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:05.799185 24535 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:05.799940 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:05.811085 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.811403 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.838229 24435 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:05.804082 24427 raft_consensus.cc:2354] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26.
I20260809 23:22:05.816011 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:05.816857 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.840199 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:05.818099 24539 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:05.844316 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.767831  4841 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:05.825426 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:05.857486 24544 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:05.826573  5102 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260809 23:22:05.828949 24545 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:05.860544 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.793900  4678 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:22:05.823587 24430 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:05.689242  4973 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:05.851589 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.824652 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:05.839464 24423 raft_consensus.cc:2362] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26. Re-sending same reply.
I20260809 23:22:05.853521  5038 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:05.855751 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:05.879089 24428 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:05.863662 24435 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:05.794467  5057 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:05.828969 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:05.831648  5060 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260809 23:22:05.868049 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:05.694621  5105 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.175s
I20260809 23:22:05.869535 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
W20260809 23:22:05.860708  5055 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260809 23:22:05.873762 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:05.894532 24314 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:05.914039 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:05.922036 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111450us
I20260809 23:22:05.814119  4812 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.929243 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:05.930788 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3775) took 1389 ms (client timeout 1643 ms). Trace:
I20260809 23:22:05.884864  4846 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:05.925499 24430 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:22:05.926712 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:05.927263 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53631.4us
I20260809 23:22:05.936902 24380 leader_election.cc:398] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:05.937292 24380 leader_election.cc:401] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
W20260809 23:22:05.931219 24320 rpcz_store.cc:269] 0809 23:22:04.541437 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.541589 (+   152us) service_pool.cc:227] Handling call
0809 23:22:05.930638 (+1389049us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003c37d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:05.930717 (+    79us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2759667456}
I20260809 23:22:05.937983 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:05.938866 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:05.847558  4568 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.065s
I20260809 23:22:05.792539  5088 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.948869 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:05.985015 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:05.864771  5010 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:05.797422  5036 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:05.993148 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54850.9us
I20260809 23:22:05.876314  4999 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:05.978971 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:05.929572  5102 log.cc:924] Time spent T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260809 23:22:05.866006  4841 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.985908 24314 raft_consensus.cc:2362] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23. Re-sending same reply.
I20260809 23:22:05.980705  4588 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:05.998173 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:05.984741  4945 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:06.017361 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.035950 24323 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:22:05.936669  4948 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260809 23:22:06.017463 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 201550us
I20260809 23:22:06.032951 24428 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:06.005967 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.040014 24320 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:05.991588 24544 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
W20260809 23:22:06.017355  5093 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.387s	user 0.001s	sys 0.001s
W20260809 23:22:06.044955 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105830us
I20260809 23:22:06.098912 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.051987 24425 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:05.857971  5087 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.106739 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.108537 24320 raft_consensus.cc:1226] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:05.992615  5030 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:05.985994  5105 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:05.999430  5018 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:05.912693  5057 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:05.995998  4974 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.132035 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.146865 24425 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 28: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:06.021435  4823 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:06.050580  4948 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:06.149652 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:06.144675 24320 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:06.163064 24323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
W20260809 23:22:06.121269  5094 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.478s	user 0.000s	sys 0.001s
I20260809 23:22:06.194176 24321 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:06.199049 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 260233us
I20260809 23:22:06.200953 25321 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:06.201298 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 206342us
I20260809 23:22:06.198413 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.203629 24425 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:06.232513  4757 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.233316  4757 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:06.235229  5063 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.011s	sys 0.030s
W20260809 23:22:06.235698  5063 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.011s	sys 0.030s
I20260809 23:22:06.237640  4757 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.237833 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.023183  4999 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.120627  4945 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:06.244241 24320 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 27. Current term is 28. Ops: []
I20260809 23:22:06.246214 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.247244 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.163426  5105 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:06.247854 24545 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:06.163249  4948 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.264891 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:06.269164 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 161378us
I20260809 23:22:06.198329  5067 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:06.251842  5108 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.483s	user 0.001s	sys 0.001s
I20260809 23:22:06.277916 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.279155 24320 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260809 23:22:06.283015 24320 pending_rounds.cc:76] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 14
I20260809 23:22:06.283269 24320 pending_rounds.cc:97] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260809 23:22:06.283525 24320 raft_consensus.cc:2866] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:06.297518 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.264873  5105 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:06.274549  4996 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:06.301926 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 977768us
I20260809 23:22:06.319799 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:06.320223 24428 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:06.203219  4823 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.302809 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.322341 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.275900  4589 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.301842 24320 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:06.327728 24320 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:06.323266 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:06.284192  4588 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:06.345700 25371 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260809 23:22:06.303385  5122 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:06.354306 25370 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260809 23:22:06.354899 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52173.8us
I20260809 23:22:06.329977 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:06.296119  4973 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.289250  5067 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260809 23:22:06.322723  5114 raft_consensus.cc:2741] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:22:06.328830  5105 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:06.349948 24320 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.295100  5106 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
W20260809 23:22:06.376310 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.013s	sys 0.060s
I20260809 23:22:06.379132 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:06.381562 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.013s	sys 0.060s
I20260809 23:22:06.346288  5004 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:06.382272 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 582441us
I20260809 23:22:06.347163  4588 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:06.379817 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:06.327467  5139 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:06.386178 25370 raft_consensus.cc:2354] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:06.389562 25372 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:06.366580  4757 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.394754 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:06.396512 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.397317 25368 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260809 23:22:06.397351 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:06.398921 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:06.398943 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.359725  5018 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:06.364974  4812 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.367786  5140 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:06.366405  4996 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:06.346909  4589 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.379245  4984 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.375090  5130 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:06.362528  5137 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:06.410264  5114 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:06.425287 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:06.415401  4639 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.773s	user 0.010s	sys 0.002s
I20260809 23:22:06.396832  4841 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.387995  4588 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:06.410513  5106 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:06.409806  5088 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.414309  5077 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:06.443858  5141 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
W20260809 23:22:06.445885  5138 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260809 23:22:06.446885  5144 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260809 23:22:06.417613  5018 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:06.448112  5134 log.cc:924] Time spent T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260809 23:22:06.449240  5129 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:22:06.451494  5030 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:06.422109  4812 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.455179 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.348503  4945 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.589295  5004 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.589814  5004 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.590977  4375 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:06.591888  5004 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:06.484772  5057 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.593297 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.594681 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:06.595268  5112 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.601s	user 0.000s	sys 0.004s
I20260809 23:22:06.568599  4999 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.596359  4812 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:06.598354  5148 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260809 23:22:06.416548  5125 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.065s
W20260809 23:22:06.599284  5147 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260809 23:22:06.483235  5113 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.396s	user 0.000s	sys 0.002s
W20260809 23:22:06.485280  5115 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.308s	user 0.000s	sys 0.001s
I20260809 23:22:06.600898 24320 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:06.601994 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:06.431730  5110 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.300s	user 0.001s	sys 0.000s
I20260809 23:22:06.602689  5018 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:06.499272 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3829) took 2540 ms (client timeout 2070 ms). Trace:
W20260809 23:22:06.603739 24534 rpcz_store.cc:269] 0809 23:22:03.958644 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.960523 (+  1879us) service_pool.cc:227] Handling call
0809 23:22:06.488713 (+2528190us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b5400012554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:06.499175 (+ 10462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1578653952}
I20260809 23:22:06.604187 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2011) took 1018 ms. Trace:
I20260809 23:22:06.604398 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3905) took 1477 ms. Trace:
I20260809 23:22:06.583984  5030 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:06.604627 25363 rpcz_store.cc:276] 0809 23:22:05.586243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.591227 (+  4984us) service_pool.cc:227] Handling call
0809 23:22:05.591665 (+   438us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:05.592065 (+   400us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:05.592103 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:05.592605 (+   502us) log.cc:842] Serialized 26 byte log entry
0809 23:22:05.783392 (+190787us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:22:05.783433 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:05.783493 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:06.603957 (+820464us) raft_consensus.cc:1671] finished
0809 23:22:06.603979 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:06.604130 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3322,"raft.queue_time_us":3653,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":3441,"thread_start_us":166921,"threads_started":1,"wal-append.queue_time_us":338286}
W20260809 23:22:06.504036  5150 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.080s	user 0.000s	sys 0.002s
I20260809 23:22:06.605990  5030 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 23.21, Last appended by leader: 21, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:06.504518  4820 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.143s	user 0.006s	sys 0.008s
I20260809 23:22:06.461259  4639 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.505548  4589 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.505602 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:06.508498  4968 consensus_meta.cc:287] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.897s	user 0.001s	sys 0.010s
I20260809 23:22:06.430100  4984 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.520232 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3783) took 1616 ms. Trace:
I20260809 23:22:06.458817  5087 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.522687  4611 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.619539 24317 raft_consensus.cc:2396] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260809 23:22:06.523559  5042 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.432384  5130 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.524533  4823 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.424311  5140 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.404600  4757 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.525204  4841 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.462354  5036 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.528213 24427 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:06.639120 24427 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:06.529289  5062 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.530033  4678 raft_consensus.cc:3032] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:06.534638 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:06.536345  5024 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.000s
W20260809 23:22:06.540016 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 769707us
I20260809 23:22:06.649195 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.650321 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:06.418824  4996 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:06.651196  4996 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:06.651767 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 266530us
I20260809 23:22:06.653525 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.654249  4823 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:06.577031 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 546656us
W20260809 23:22:06.559283  5025 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.187s	user 0.013s	sys 0.000s
I20260809 23:22:06.585129 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.568202  5106 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:06.588271 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.658170 24321 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260809 23:22:06.433583  5065 consensus_meta.cc:287] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.934s	user 0.012s	sys 0.000s
I20260809 23:22:06.659198  5065 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.660333 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.660642 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.660986 24541 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:06.661337  5065 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.604789 24431 rpcz_store.cc:276] 0809 23:22:05.126631 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.126825 (+   194us) service_pool.cc:227] Handling call
0809 23:22:06.604211 (+1477386us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400061b54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:06.604312 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":621537536}
I20260809 23:22:06.661896 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.662380 24313 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:06.662686  5065 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.663367 24540 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260809 23:22:06.663169  5065 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.663642  5106 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.665344  5065 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.667811 24270 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:06.668649  5065 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:06.609035 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:06.598577  4812 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.615593 24311 rpcz_store.cc:276] 0809 23:22:04.903521 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.903688 (+   167us) service_pool.cc:227] Handling call
0809 23:22:04.921516 (+ 17828us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:05.598349 (+676833us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:06.488311 (+889962us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b540019cdd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:06.520151 (+ 31840us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1550454016}
I20260809 23:22:06.669217  5065 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:06.669545 24321 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:06.669770  5065 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:06.617285 24428 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:06.600950  5082 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.993s	user 0.011s	sys 0.001s
W20260809 23:22:06.628616 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.006s	sys 0.014s
I20260809 23:22:06.607570  5141 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:06.670689  5082 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.671180 24321 raft_consensus.cc:1261] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260809 23:22:06.631402 24312 raft_consensus.cc:2354] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:06.671283 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.006s	sys 0.014s
I20260809 23:22:06.613327  4976 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:06.671918  4976 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:06.672228 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1963) took 1069 ms. Trace:
I20260809 23:22:06.672683 24321 raft_consensus.cc:1203] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Deduplicated request from leader. Original: 22.13->[24.14-24.14]   Dedup: 24.14->[]
W20260809 23:22:06.608137 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.007s	sys 0.001s
W20260809 23:22:06.643537 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3834) took 2677 ms (client timeout 1700 ms). Trace:
I20260809 23:22:06.672766  5082 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:06.646240 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 445408us
I20260809 23:22:06.671373  5141 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:06.674515 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.007s	sys 0.002s
I20260809 23:22:06.668571 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.675210  5141 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.676290 25321 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:06.676473  5141 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.677430 25321 leader_election.cc:398] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26.
I20260809 23:22:06.677767 25321 leader_election.cc:401] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:06.672593 24309 rpcz_store.cc:276] 0809 23:22:05.602686 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.602877 (+   191us) service_pool.cc:227] Handling call
0809 23:22:05.616089 (+ 13212us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:05.616346 (+   257us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:05.828563 (+212217us) log.cc:842] Serialized 14 byte log entry
0809 23:22:06.671911 (+843348us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:06.671965 (+    54us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:06.672010 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:06.672083 (+    73us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:06.672179 (+    96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":178270,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":6679,"raft.queue_time_us":1303,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"thread_start_us":1026891,"threads_started":2}
I20260809 23:22:06.672763  4976 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:06.673684 24543 rpcz_store.cc:269] 0809 23:22:03.966452 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:03.967868 (+  1416us) service_pool.cc:227] Handling call
0809 23:22:06.643351 (+2675483us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:06.643473 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1980490112}
I20260809 23:22:06.612432  5030 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:06.643081 24427 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.687831 24427 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.673455 24323 raft_consensus.cc:595] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:06.688688 24323 raft_consensus.cc:603] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:06.689343 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:06.689872 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80941.4us
I20260809 23:22:06.690675 24543 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.21-23.21]
I20260809 23:22:06.691017 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:06.673491 24321 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.14-24.14]
W20260809 23:22:06.685423 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 286591us
I20260809 23:22:06.701571 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:06.703639 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.680303 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.682561 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:06.685482 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:06.709020 24427 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260809 23:22:06.709537 24427 pending_rounds.cc:76] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 14
I20260809 23:22:06.644625  4678 raft_consensus.cc:729] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:06.689426  3889 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:06.689615  4375 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:06.709745 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:06.709847 24427 pending_rounds.cc:97] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260809 23:22:06.708307 24539 raft_consensus.cc:2362] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26. Re-sending same reply.
I20260809 23:22:06.785108 24309 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: [28.16-28.16]
W20260809 23:22:06.735378 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55139.4us
I20260809 23:22:06.793215 24427 raft_consensus.cc:2866] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:06.797792 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:06.799386  5152 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260809 23:22:06.799892  4757 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.800071  5118 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:06.805819  5071 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
W20260809 23:22:06.810698 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 101062us
W20260809 23:22:06.768786 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44523us
W20260809 23:22:06.743409 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60967.7us
I20260809 23:22:06.711288 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:06.779495 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 70438.7us
I20260809 23:22:06.704319  4976 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:06.809507 24316 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:06.633522  4611 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.812589 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:06.703687  5139 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:06.648103  4841 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.636581  5042 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.810407 24431 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:06.814164 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:06.814464 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.844340 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:06.845177 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:06.845543 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.845760 24431 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.820964 24430 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:06.795622  4589 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.824447  4745 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:06.827688  5154 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.826263  4639 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.826071  4588 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.730508  4968 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
I20260809 23:22:06.683565  5024 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.793354  3889 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:06.827536  4948 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.000s
I20260809 23:22:06.829062  5163 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26.
I20260809 23:22:06.832373  4812 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:06.781527  4820 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:06.836714  5079 consensus_meta.cc:287] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.022s	user 0.003s	sys 0.008s
I20260809 23:22:06.838910  4611 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.795193  4375 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:06.941846 24429 raft_consensus.cc:2396] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260809 23:22:06.841244  5155 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260809 23:22:06.731748  5132 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:06.841759  5172 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:06.779090  4985 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:06.689525  4493 consensus_queue.cc:1035] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.107s
I20260809 23:22:06.843472 24309 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:06.891433  5182 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:06.846580 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.848090 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:06.957922 24427 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:06.935926  5087 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:06.963528 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 115533us
I20260809 23:22:06.965286 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:06.948799  4375 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.691167  4989 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.125s
I20260809 23:22:06.852659 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:06.853315 24539 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:06.853440 24320 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:06.970710 24541 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:06.824877 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.971005 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3196) took 1186 ms. Trace:
I20260809 23:22:06.872148 24431 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:06.952884  5132 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:06.877179 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 172139us
I20260809 23:22:06.820019  4251 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.831329  5160 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:06.756212  4993 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260809 23:22:06.845187 24540 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:06.886917  4996 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.887375  4588 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.755519  4359 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.105s
I20260809 23:22:06.851392  4757 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:06.974884  5158 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
W20260809 23:22:06.912467 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 210993us
I20260809 23:22:06.925109 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:06.948415 24424 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:07.011881 24543 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 29
I20260809 23:22:07.011982 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:22:06.949013 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 205396us
I20260809 23:22:06.898502  3889 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:06.871884  4589 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.848404 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:06.968366 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:06.872483 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.971472 24320 rpcz_store.cc:276] 0809 23:22:05.784489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.301388 (+516899us) service_pool.cc:227] Handling call
0809 23:22:06.301721 (+   333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:06.600662 (+298941us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5800884fb0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:22:06.970723 (+370061us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:06.970950 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":502143360}
I20260809 23:22:07.020386 24427 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:07.020571 24430 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:07.020943 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:06.973120 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:06.968487  5139 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:06.971112  5088 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:22:06.971232  5163 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:06.975360 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.888104  5154 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.884702  4678 consensus_queue.cc:260] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:06.984686 24309 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:06.987329 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:06.940871  4820 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:06.975455  3810 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260809 23:22:06.964534  5130 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:06.854722  5025 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:07.097373  5025 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:07.098507  5025 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.923610  4948 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:06.981123  4968 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:06.971419  5182 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:07.100739  4968 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.100997 24321 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:07.101233  4968 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:07.101394  5182 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.102068  4948 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.103318  5042 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:06.984692  5132 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:06.975064  5053 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.105757  5053 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.041433  5193 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.079s
W20260809 23:22:07.022547 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49517.1us
I20260809 23:22:07.115833 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.116791 24541 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:07.117254 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:06.975361  5140 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:07.024389 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 176053us
I20260809 23:22:07.029682 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.030488 24540 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:07.003574  5172 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.039176 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:06.971622  5029 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.047s
I20260809 23:22:07.131330 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.132198  5172 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.134482  5172 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.014521  4949 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.047158  5186 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:07.135461  4493 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:07.135211  4949 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.135970  4493 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:07.135654  5186 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.137190  4949 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.037807 24427 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:07.138245 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.138465  5186 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.046442 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.015501  5160 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.018419  4757 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.219256 24539 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.004118  4251 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.026707  4588 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:07.218106 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:07.060186  5179 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260809 23:22:07.051882  4984 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.051019  5187 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:07.095824  5139 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.021667 24320 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:07.118759 24430 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:07.234658 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.049443  4974 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.090415  4996 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.124501 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:07.223062  5029 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:06.972163  5087 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.141608  5130 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:07.065011  4820 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.142064 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:07.194574 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 77390.6us
I20260809 23:22:07.196332 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.261138 24541 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:07.059697 24543 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:07.264760 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:07.084245 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3190) took 1389 ms (client timeout 1735 ms). Trace:
W20260809 23:22:07.265983 24317 rpcz_store.cc:269] 0809 23:22:05.694703 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.747400 (+ 52697us) service_pool.cc:227] Handling call
0809 23:22:07.083942 (+1336542us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b540007d354. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:07.084165 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1866786048}
I20260809 23:22:07.087596 24311 raft_consensus.cc:1261] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260809 23:22:07.051309  5163 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.026837  5079 raft_consensus.cc:2741] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260809 23:22:07.240662  5156 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:07.228159 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:07.277379 24544 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:07.065171  5183 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.232726 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.278807 24431 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:07.279841 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.280860 24431 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260809 23:22:07.157698  5004 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.119410 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:07.278661 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:07.083918  4841 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.240149 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:07.235477 24545 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:22:07.124491  4973 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:07.085951  4589 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.047215 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.242434 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.063699  5024 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.136911  4493 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.224593  5116 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.066263  4862 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:07.335246  4862 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.223044  5172 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:07.267887 24311 pending_rounds.cc:76] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 14
I20260809 23:22:07.337477 24311 pending_rounds.cc:97] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260809 23:22:07.337919 24311 raft_consensus.cc:2866] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:07.234161  5187 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:07.258752 24542 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:07.338989  4973 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.226735  4588 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:07.339545  5187 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.340610  4588 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.341500  4973 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.262264 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
W20260809 23:22:07.089051  5105 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.698s	user 0.008s	sys 0.003s
I20260809 23:22:07.270104 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:07.343384  5105 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:07.343847 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74070.5us
I20260809 23:22:07.344197  5163 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.345369  5105 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.080190  5173 consensus_queue.cc:1046] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.217s
I20260809 23:22:07.069450  3889 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:07.283949 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 164621us
I20260809 23:22:07.346055  4757 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.346712  4757 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:07.347299  4757 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:07.347564  3889 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.348333  4757 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.286190 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.256404  5029 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.349970 24317 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260809 23:22:07.350124  5029 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:07.290484 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.337865  4862 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.352051 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.354842  5106 consensus_queue.cc:1035] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:07.355827  5024 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:07.357049  5106 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:07.306209 24544 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
W20260809 23:22:07.307500 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67437.9us
I20260809 23:22:07.307551 24309 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:07.310920 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.360263 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.309432 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:07.361805 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52500.7us
I20260809 23:22:07.363114 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:07.363466  4862 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.363952  4862 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.315106  4841 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:07.365193  4841 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:07.365593  4841 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:07.365976  4862 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.366513  4862 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:07.366839  4862 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.367194  4862 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:07.282534  5156 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.368672  5156 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:07.299198  5079 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:07.371366  5186 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:07.371891  5186 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.372268  5087 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.276764  4948 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:07.373099  4948 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.373512  4948 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:07.373832  5186 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.375157  4984 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:07.300323  5004 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:07.317677  5065 consensus_meta.cc:287] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.647s	user 0.010s	sys 0.003s
I20260809 23:22:07.323359  5195 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.327685  5132 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.331147  5116 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.337515  5172 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:07.282702 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 430077us
I20260809 23:22:07.349071 24534 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
W20260809 23:22:07.298108 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 286184us
I20260809 23:22:07.308584  5154 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.316735 24320 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260809 23:22:07.403383 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3795) took 1016 ms. Trace:
I20260809 23:22:07.403788 24320 rpcz_store.cc:276] 0809 23:22:06.387347 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.315932 (+928585us) service_pool.cc:227] Handling call
0809 23:22:07.316323 (+   391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.403071 (+ 86748us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:07.403315 (+   244us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:07.378161 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.378917 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.405310 25363 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:07.382457 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:07.404731 24498 connection.cc:232] RPC call timeout handler was delayed by -2.086157051846385s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23950000000000002s for a response
I20260809 23:22:07.406389 25372 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:07.380959 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.383611 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.384186 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:07.409152 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:07.409277 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.409662 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.375674  4984 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:07.246220  4974 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.388819 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.389132 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:07.415602 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.390584 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.387827  5195 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.374405  4862 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.361603 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.400636 25321 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24.
I20260809 23:22:07.436997 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:07.427459 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.438622 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:07.439742 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.440222  5105 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:07.440389 25372 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:07.440800  5105 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:07.441787  5105 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.438299 24320 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:07.411870  4984 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:07.456588 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.300422  5183 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.415634  5163 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:07.457242 25363 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:22:07.458700 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:07.466971 24320 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:07.388161  5065 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.412007  5141 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:07.461591 25365 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:07.442433  5186 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:07.453564 25372 raft_consensus.cc:2396] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 24 because replica has last-logged OpId of term: 23 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 20.
W20260809 23:22:07.465184 25321 connection.cc:232] RPC call timeout handler was delayed by -1.4486678577959538s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
I20260809 23:22:07.424573  5154 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:07.478664  5202 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.479243  5183 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.479678  5202 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.483434  5202 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.485134  5202 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.486099  5183 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.486703  5183 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.489843  5183 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.491466  5132 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:07.490621 25321 connection.cc:232] RPC call timeout handler was delayed by -1.1237678579054773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
I20260809 23:22:07.493772 25372 raft_consensus.cc:1261] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:07.495033  4974 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.485703  5202 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.496923 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.497720 24320 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:07.497887  5202 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.498886 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.499626 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.497061  5116 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.501072 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.501770 25372 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:07.503204 24320 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:07.504261  5172 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.506867 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.507539 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:07.508414 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.508646 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:07.509013 24320 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:22:07.509178  4375 consensus_meta.cc:287] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.524s	user 0.001s	sys 0.008s
I20260809 23:22:07.509718  4375 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.510179  5154 consensus_queue.cc:1035] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:07.510541 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:07.511811 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:07.512158  4375 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.512522  5154 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:07.512888 24546 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260809 23:22:07.513342  5154 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:07.513825  5154 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.515609 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.514832  5154 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:07.517304  5116 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:07.517795  5116 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.518126  5116 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:07.502782 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.524863 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:07.525480 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 708545us
I20260809 23:22:07.525972 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.526705 25372 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.526242  5116 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.527446 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.528115 25372 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:07.528838 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.529793 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.448794  4862 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.535065 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.535802 24430 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.540269  4968 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:07.544536  5184 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.659s	user 0.004s	sys 0.004s
I20260809 23:22:07.546569 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:07.547175 24534 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:07.548432 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.549959 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3913) took 1521 ms. Trace:
I20260809 23:22:07.550377 24425 rpcz_store.cc:276] 0809 23:22:06.028679 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.204351 (+175672us) service_pool.cc:227] Handling call
0809 23:22:06.204670 (+   319us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:06.279653 (+ 74983us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:06.279683 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:06.279731 (+    48us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:06.279773 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:07.549720 (+1269947us) spinlock_profiling.cc:243] Waited 74.8 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:07.549770 (+    50us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:07.549902 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172012800}
I20260809 23:22:07.552006 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.559377 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.560150 24425 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:07.563063 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.564146 24544 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:07.569249 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1919) took 1216 ms. Trace:
I20260809 23:22:07.592446 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.598519 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:07.599889 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:07.601297 24320 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:07.569844 25366 rpcz_store.cc:276] 0809 23:22:06.352396 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.352564 (+   168us) service_pool.cc:227] Handling call
0809 23:22:06.352982 (+   418us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:06.756793 (+403811us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:06.756838 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:06.759822 (+  2984us) log.cc:842] Serialized 26 byte log entry
0809 23:22:06.777924 (+ 18102us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:06.778074 (+   150us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:06.778383 (+   309us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:07.543903 (+765520us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:07.546051 (+  2148us) raft_consensus.cc:1671] finished
0809 23:22:07.546072 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:07.569179 (+ 23107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14570,"raft.queue_time_us":34388,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":19667,"spinlock_wait_cycles":8129664,"thread_start_us":406286,"threads_started":2,"wal-append.queue_time_us":122305}
I20260809 23:22:07.609916 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:07.610566 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:07.609112 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.023s	sys 0.017s
W20260809 23:22:07.614495 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.023s	sys 0.017s
W20260809 23:22:07.627157 25321 leader_election.cc:334] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.834s (SENT)
I20260809 23:22:07.627679 25321 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:07.627882 24544 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:07.629144  4862 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:07.629420 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.630198 24542 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.633033 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:07.635041 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.635756 24542 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:22:07.641481 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.655229 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:07.549999  5030 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.661258 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.662585 24425 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:07.664258 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.664857 25366 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:07.664183 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:07.665695 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.667644 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.553534  5195 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:07.592175  4507 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.023s	sys 0.014s
I20260809 23:22:07.673626 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.671064 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.569522  5183 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.675414 24428 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 19.
I20260809 23:22:07.680511 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.674674 24425 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:07.686302 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.687043 24425 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:07.676633 24534 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:07.688515 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:07.548573  4948 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:07.693356 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:07.657068  4976 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:07.698108 25366 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:22:07.695680  5183 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.712798  5183 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.715261 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3790) took 1373 ms. Trace:
I20260809 23:22:07.715708 24312 rpcz_store.cc:276] 0809 23:22:06.341510 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.601733 (+260223us) service_pool.cc:227] Handling call
0809 23:22:07.715067 (+1113334us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b54000dbbd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:07.715201 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65727744}
I20260809 23:22:07.716838 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.717459 24312 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:07.718291 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.720476  3979 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:07.721692  5154 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.669317  4862 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:07.723416  4862 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:07.723896  4862 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:07.726997  5163 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:07.727819 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1713) took 1841 ms. Trace:
I20260809 23:22:07.728174 24435 rpcz_store.cc:276] 0809 23:22:05.885997 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.886135 (+   138us) service_pool.cc:227] Handling call
0809 23:22:05.886477 (+   342us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:05.998593 (+112116us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:05.998874 (+   281us) log.cc:842] Serialized 14 byte log entry
0809 23:22:06.675008 (+676134us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:06.675059 (+    51us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:06.675120 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:07.727603 (+1052483us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002f9ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:07.727648 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:07.727764 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":256,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":958,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":257254784,"thread_start_us":675766,"threads_started":1,"wal-append.queue_time_us":826561}
I20260809 23:22:07.730032 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.731186 25375 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260809 23:22:07.730896  5029 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.732383 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.733076  5029 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.709118  5195 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:22:07.734097  5195 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:07.735109  5195 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.699887 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.741655 24535 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:22:07.701205  4841 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.669661  5030 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.750053 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:07.664500  5116 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:07.696925 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.755882 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:07.707650 24317 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: []
W20260809 23:22:07.678630  4507 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.024s	sys 0.015s
I20260809 23:22:07.743885 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:07.689500  5172 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260809 23:22:07.690526  5186 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:07.753566  5004 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.746021 24312 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:07.707269  4958 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:07.767045 25372 raft_consensus.cc:2379] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:07.768196 24428 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:07.784953 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.785106 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.785665 24435 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:07.785696 24427 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:07.707237  5160 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.765659 24320 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:07.759109  5029 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.796710 24425 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:07.764813  5186 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:07.807675 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 400465us
I20260809 23:22:07.794744 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:07.773890  4958 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:07.808511 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.894s (SENT)
I20260809 23:22:07.786475  4968 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.793473  4945 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:07.807824  5160 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:07.760232  5025 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260809 23:22:07.795956 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:07.832844 24320 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:07.810912  5186 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:07.818825 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.108s (SENT)
I20260809 23:22:07.806160 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.793246  5004 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:07.852043 24312 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:07.783342  4862 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.842978 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.859553 24320 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:07.813306  5139 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:07.817963  5065 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.817167  5154 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:07.847379  4984 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.814611  4958 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:07.839834 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 62) took 2187 ms. Trace:
I20260809 23:22:07.804277 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:07.828962  5221 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.058s	user 0.001s	sys 0.001s
W20260809 23:22:07.870442 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51641.6us
W20260809 23:22:07.883141 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.137s (SENT)
I20260809 23:22:07.883756 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:07.885131 24498 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 2.682s (SENT) [suppressed 24 similar messages]
I20260809 23:22:07.886767  5205 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:22:07.805328 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:07.888233 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.873705 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:07.811092  4948 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:07.872452 24539 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:07.822345  5141 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:07.877216 24425 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:07.840866  5013 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:07.876895 24323 rpcz_store.cc:276] 0809 23:22:05.643373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.098130 (+454757us) service_pool.cc:227] Handling call
0809 23:22:07.830921 (+1732791us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b5400243bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385abc3 00007fac287a481a 00007fac21e97d04 00007fac21e97c92 00007fac21e97bc1 00007fac21e97b3c 00007fac21e96df2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:07.831089 (+   168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1171899264}
I20260809 23:22:07.890244 24312 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.891273 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:07.927138 24498 connection.cc:232] RPC call timeout handler was delayed by -1.47011962486431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20260809 23:22:07.927738 24498 connection.cc:232] RPC call timeout handler was delayed by -1.3630196247249842s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
I20260809 23:22:07.929049 25365 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:07.929925 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:07.933733 24266 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:07.935807 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.823247  5160 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.809873  5029 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.968981 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:07.891319  5139 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:07.972927 24435 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:07.984206 24266 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:07.987337 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45493.9us
I20260809 23:22:07.988034 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:07.984727 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:07.995766 25368 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:07.932061  5079 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:07.881290  5004 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.001708 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:08.001876 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.000116 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.004711 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:08.000835 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 337654us
I20260809 23:22:08.005676 24427 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:07.934594  5053 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.007346 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.009218 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.005614 24425 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:08.010229 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:08.017794 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.018896 24427 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:08.023435 24266 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:08.027263 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.029251 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.068706 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:08.068995 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:07.998107  5079 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:07.929279  5013 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:07.982818  5206 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:07.987396  4948 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:08.076195 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:07.999466  4968 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:08.075910  4974 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:08.076021  4976 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.083715 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:08.166682 24323 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:08.084272 24314 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260809 23:22:08.087550 24435 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
W20260809 23:22:08.071388 24378 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.016s	sys 0.012s
I20260809 23:22:08.088052 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:08.171895 24378 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.016s	sys 0.012s
I20260809 23:22:08.077729  5202 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:08.088999 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3788) took 1748 ms. Trace:
I20260809 23:22:08.090638 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 22) took 1638 ms. Trace:
I20260809 23:22:08.079524  4841 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:08.097770 24429 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1699) took 2423 ms (client timeout 1834 ms). Trace:
W20260809 23:22:08.098172 24316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3772) took 3603 ms (client timeout 1896 ms). Trace:
W20260809 23:22:08.180193 24429 rpcz_store.cc:269] 0809 23:22:05.674666 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.674793 (+   127us) service_pool.cc:227] Handling call
0809 23:22:08.097292 (+2422499us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000af854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.097679 (+   387us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2689112448}
W20260809 23:22:08.099043 24313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 1968) took 2493 ms (client timeout 2070 ms). Trace:
I20260809 23:22:08.102684 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:08.105568  5122 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:08.107412 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 831399us
I20260809 23:22:08.108696 24320 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:08.112517 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:08.114212  5160 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.197691 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:08.198419 25363 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:08.116696  5227 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260809 23:22:08.115605 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:08.199664 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84157.3us
I20260809 23:22:08.118906  5004 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.200538 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.025182  5154 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:08.025821  5183 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:08.201843  5183 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:08.025427  5030 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.202356  5116 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:08.033294  4588 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:08.203933  4588 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:08.069373  5038 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.102s
I20260809 23:22:08.204434  5122 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.204916  4588 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.168354  4968 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:08.207672  4968 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.168710  4997 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.169190  4974 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:08.172258 24312 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:08.172250 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 781951us
I20260809 23:22:08.172992 24321 rpcz_store.cc:276] 0809 23:22:06.340507 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.693327 (+352820us) service_pool.cc:227] Handling call
0809 23:22:06.693664 (+   337us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.101482 (+407818us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.088767 (+987285us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b540043c1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:08.088942 (+   175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":936301824}
I20260809 23:22:08.173131 24422 rpcz_store.cc:276] 0809 23:22:06.452430 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.591936 (+139506us) service_pool.cc:227] Handling call
0809 23:22:08.090188 (+1498252us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.090555 (+   367us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":575607680}
I20260809 23:22:08.096520 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:08.124936  5079 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:08.180397 24316 rpcz_store.cc:269] 0809 23:22:04.494115 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:04.494638 (+   523us) service_pool.cc:227] Handling call
0809 23:22:08.097732 (+3603094us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b54002530d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.097841 (+   109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4797236608}
I20260809 23:22:08.226164  5030 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:08.227002 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:08.227828  5030 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:08.228258 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:08.228816 24316 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:08.230383 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:08.233891  4958 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.235200 24378 leader_election.cc:398] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
W20260809 23:22:08.181499 24313 rpcz_store.cc:269] 0809 23:22:05.605726 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.605833 (+   107us) service_pool.cc:227] Handling call
0809 23:22:08.098793 (+2492960us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x7b54003c2dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.098954 (+   161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2299787648}
I20260809 23:22:08.190493 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.175196  4862 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.191051  4948 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:08.191998 25372 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:08.194308  4823 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.125761  4375 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260809 23:22:08.192040  4945 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.003s
I20260809 23:22:08.247716 24316 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:08.195199 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.252411 24429 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:08.252486 25323 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:08.195943  4251 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.205098  5038 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
W20260809 23:22:08.120450  5156 consensus_meta.cc:287] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.751s	user 0.013s	sys 0.000s
I20260809 23:22:08.202373 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:08.218600 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:08.211205  4997 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.223722  4974 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.225121 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:08.238920 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.241559 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.244796 24541 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:08.246635 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.254616 25375 raft_consensus.cc:1261] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:08.255317 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.235917  5187 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:08.256990 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54689.9us
I20260809 23:22:08.286559 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.286806 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:08.262499 24312 raft_consensus.cc:2396] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:08.262673 24422 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:08.288094 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.288609 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.288738 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.260715 24435 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:08.261266 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:08.249346 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:08.268054 25375 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:08.273427  4974 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:08.294480 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.265339  5156 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.265082  4862 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:08.283322 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:08.244266  5065 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.250726  4823 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.256735 24321 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:08.244479  4841 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:08.172989  4976 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.294307 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:08.297358 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.256270  5038 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:08.265223  5053 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:08.293511 25368 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:08.244416  5063 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260809 23:22:08.298995 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.414784 24541 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:08.247879  5122 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.300688 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:08.251323  3810 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:08.243430  4948 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.298178 24422 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260809 23:22:08.448447 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.013s	sys 0.034s
W20260809 23:22:08.448788 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.013s	sys 0.034s
W20260809 23:22:08.441639 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 141053us
I20260809 23:22:08.303022 24429 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:08.285450  4678 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:08.471385 24321 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
W20260809 23:22:08.296804  5195 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.005s	sys 0.035s
I20260809 23:22:08.318135 24316 raft_consensus.cc:1261] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:08.268580  5116 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:08.314400 24323 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:22:08.370679 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.030s	sys 0.033s
I20260809 23:22:08.295594 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:08.485407 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.030s	sys 0.034s
I20260809 23:22:08.376675 25368 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
W20260809 23:22:08.291198 25371 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.017s	sys 0.019s
W20260809 23:22:08.486287 25371 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.018s	sys 0.019s
I20260809 23:22:08.377202 25372 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
W20260809 23:22:08.422622 25373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.754s	user 0.008s	sys 0.036s
W20260809 23:22:08.487347 25373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.009s	sys 0.036s
W20260809 23:22:08.423025 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.004s	sys 0.036s
W20260809 23:22:08.488268 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.004s	sys 0.036s
W20260809 23:22:08.367700 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 73491.2us
I20260809 23:22:08.297937  4945 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:08.296716  5154 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:08.297245  4985 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:08.178041  5139 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.450563 24422 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260809 23:22:08.475723  5084 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:08.490406  5025 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.504455  5025 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.505061 24422 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:08.506462  5025 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:08.453838 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.021s	sys 0.030s
I20260809 23:22:08.497103  4958 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:08.507165 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.021s	sys 0.030s
I20260809 23:22:08.508867  5172 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:08.509362  5172 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:08.455265 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:08.370464  4507 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.033s	sys 0.065s
I20260809 23:22:08.474624 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.514022 24541 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:08.296942  5187 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.305145  5030 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:08.478955 24316 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260809 23:22:08.358865  4984 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.377909  5202 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:08.487221 24323 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:22:08.376538  4862 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:08.256083  5105 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.036s	sys 0.051s
I20260809 23:22:08.329558  4973 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.144s
W20260809 23:22:08.493219 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 197692us
W20260809 23:22:08.517901  4507 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.033s	sys 0.066s
I20260809 23:22:08.329718  5026 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.161s
I20260809 23:22:08.496990  4841 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:08.469830  5125 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:22:08.535604  5224 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.415s	user 0.000s	sys 0.001s
W20260809 23:22:08.477752  5195 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.005s	sys 0.035s
I20260809 23:22:08.519407 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.475363  5200 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:08.548851 24541 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:08.529838 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:08.531108 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:08.484159  5029 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.489887  5116 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:08.535241 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:08.495543 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:08.412487  5065 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.476577  3810 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:08.568811 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 73461.1us
I20260809 23:22:08.570392 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:08.539033  5105 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.036s	sys 0.051s
I20260809 23:22:08.291409  5024 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:08.527047  5202 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:08.563747 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:08.514637  5166 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260809 23:22:08.537916  5013 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.598101 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:08.573132 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:08.524235  4745 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260809 23:22:08.530547  5183 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260809 23:22:08.624718  5183 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:08.625422  5183 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.546360  5125 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:08.627326  5125 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:08.628044  5187 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:08.628643  5187 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:08.628955  5187 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:08.629793  5187 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.580730 25369 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:08.567895  5116 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:08.589535  5079 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:08.632340  5239 log.cc:924] Time spent T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260809 23:22:08.622368  5166 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:08.633572  5079 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.633647  5166 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:08.586660  4823 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.634671  5166 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.635356  4823 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:08.595568  5202 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:08.587430  5156 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.605434 24541 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:08.578791  5024 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:08.513710  5038 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.639411 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.656006 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3857) took 1968 ms. Trace:
I20260809 23:22:08.604717  5105 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:08.643424  3810 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.645051 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3204) took 2854 ms. Trace:
I20260809 23:22:08.659865 24311 rpcz_store.cc:276] 0809 23:22:05.790341 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.669955 (+879614us) service_pool.cc:227] Handling call
0809 23:22:07.087078 (+417123us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.339148 (+252070us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.644824 (+1305676us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b540007d350. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.644979 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":958417152}
I20260809 23:22:08.623484  5160 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:08.656335 24546 rpcz_store.cc:276] 0809 23:22:06.687970 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.688458 (+   488us) service_pool.cc:227] Handling call
0809 23:22:06.703629 (+ 15171us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.513348 (+809719us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.655740 (+1142392us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b5400012cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:08.655943 (+   203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1860513536}
I20260809 23:22:08.661677 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.664590 24541 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:08.646286  5187 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:08.655956  5024 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:08.672040 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:08.669692  5186 consensus_meta.cc:287] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.794s	user 0.011s	sys 0.000s
W20260809 23:22:08.674991 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.036s	sys 0.031s
W20260809 23:22:08.676234 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.036s	sys 0.031s
I20260809 23:22:08.663614  5105 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:08.678226  5186 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.655823  5156 consensus_queue.cc:1035] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260809 23:22:08.698391 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3866) took 1056 ms. Trace:
W20260809 23:22:08.680390  5235 log.cc:924] Time spent T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: Append to log took a long time: real 0.599s	user 0.004s	sys 0.001s
I20260809 23:22:08.683517  5065 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:08.706064 24424 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1698) took 3031 ms (client timeout 2195 ms). Trace:
W20260809 23:22:08.711347 24424 rpcz_store.cc:269] 0809 23:22:05.674384 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:05.675601 (+  1217us) service_pool.cc:227] Handling call
0809 23:22:08.705845 (+3030244us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:08.705999 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2737021312}
I20260809 23:22:08.712409 24430 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:08.683320 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.684465  5183 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.685153  5202 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.665690  4976 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.698791  5105 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:08.689328 25366 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:08.670969  3810 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:08.674975  5160 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.030s
I20260809 23:22:08.735849  5160 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:08.736207  5160 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:08.724277  5173 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:08.736863 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2016) took 1242 ms. Trace:
I20260809 23:22:08.705380 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2014) took 1212 ms. Trace:
I20260809 23:22:08.751467 25369 rpcz_store.cc:276] 0809 23:22:07.494941 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.498482 (+  3541us) service_pool.cc:227] Handling call
0809 23:22:07.498792 (+   310us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.631218 (+1132426us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.736614 (+105396us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:08.736793 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2487655680}
I20260809 23:22:08.751724 25365 rpcz_store.cc:276] 0809 23:22:07.493324 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.533099 (+ 39775us) service_pool.cc:227] Handling call
0809 23:22:07.533445 (+   346us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.931902 (+398457us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.705109 (+773207us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54007d05d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:08.705311 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":841968896}
I20260809 23:22:08.752895 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:08.746577 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1911) took 1014 ms. Trace:
I20260809 23:22:08.751111 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.739868 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1905) took 1189 ms. Trace:
I20260809 23:22:08.702381 24542 rpcz_store.cc:276] 0809 23:22:07.642790 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.643515 (+   725us) service_pool.cc:227] Handling call
0809 23:22:07.643867 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.644117 (+   250us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:07.960326 (+316209us) log.cc:842] Serialized 14 byte log entry
0809 23:22:08.697807 (+737481us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:08.697867 (+    60us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:08.697913 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.698152 (+   239us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:08.698309 (+   157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":56706,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":20775,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":2831,"thread_start_us":1031667,"threads_started":2}
I20260809 23:22:08.713560 24546 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:08.688979  5206 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.519s	user 0.011s	sys 0.000s
I20260809 23:22:08.673426  3979 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.043s
I20260809 23:22:08.652781  4862 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:08.733330  4996 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.889s	user 0.015s	sys 0.044s
I20260809 23:22:08.754594 24543 rpcz_store.cc:276] 0809 23:22:07.731948 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.733436 (+  1488us) service_pool.cc:227] Handling call
0809 23:22:07.733791 (+   355us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.734154 (+   363us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.734181 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:07.734585 (+   404us) log.cc:842] Serialized 26 byte log entry
0809 23:22:08.449112 (+714527us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:08.449153 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.449229 (+    76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:08.746216 (+296987us) raft_consensus.cc:1671] finished
0809 23:22:08.746257 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:08.746484 (+   227us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1285,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":714161,"threads_started":1,"wal-append.queue_time_us":1009839}
I20260809 23:22:08.756825 24540 rpcz_store.cc:276] 0809 23:22:07.550749 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.551075 (+   326us) service_pool.cc:227] Handling call
0809 23:22:07.553231 (+  2156us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.591186 (+ 37955us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.591234 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:07.611052 (+ 19818us) log.cc:842] Serialized 26 byte log entry
0809 23:22:08.485838 (+874786us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:08.485894 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.485967 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:08.739657 (+253690us) raft_consensus.cc:1671] finished
0809 23:22:08.739681 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:08.739807 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8976,"raft.queue_time_us":9194,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":24096,"thread_start_us":803249,"threads_started":1,"wal-append.queue_time_us":1059453}
W20260809 23:22:08.744926  5229 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.658s	user 0.005s	sys 0.001s
I20260809 23:22:08.769344 24541 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:08.819752 24542 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:08.707935  5024 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:08.779446 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.013s	sys 0.009s
W20260809 23:22:08.846262 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.013s	sys 0.009s
I20260809 23:22:08.847532 24535 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:08.786386 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:08.743980  5232 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.662s	user 0.007s	sys 0.005s
W20260809 23:22:08.823470 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.026s	sys 0.036s
I20260809 23:22:08.820428 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:22:08.853451 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.026s	sys 0.036s
I20260809 23:22:08.779282  5206 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:08.761148  3810 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:08.768445 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.014s	sys 0.009s
W20260809 23:22:08.899930 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.015s	sys 0.010s
W20260809 23:22:08.900478 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 869869us
W20260809 23:22:08.850435 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.005s	sys 0.014s
I20260809 23:22:08.889817 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3867) took 1244 ms. Trace:
W20260809 23:22:08.902266 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.005s	sys 0.014s
W20260809 23:22:08.896018 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75733.9us
I20260809 23:22:08.896749 24546 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
W20260809 23:22:08.781644  4996 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.017s	sys 0.046s
W20260809 23:22:08.906746 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51664.3us
I20260809 23:22:08.907229 24541 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: []
I20260809 23:22:08.902570 24544 rpcz_store.cc:276] 0809 23:22:07.646581 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.647190 (+   609us) service_pool.cc:227] Handling call
0809 23:22:07.647520 (+   330us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.647750 (+   230us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:22:07.984930 (+337180us) log.cc:842] Serialized 14 byte log entry
0809 23:22:08.889415 (+904485us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:08.889480 (+    65us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:08.889529 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.889603 (+    74us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:08.889748 (+   145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35429,"prepare.queue_time_us":53266,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":36437,"thread_start_us":1205398,"threads_started":2}
I20260809 23:22:08.913712 24539 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260809 23:22:08.915655 24543 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:08.915871 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.743191  5116 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.750763  5173 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.925446 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.750206  5125 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.931602 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.938452 24543 raft_consensus.cc:2396] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260809 23:22:08.939044  5202 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.944573 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.899271  4984 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:08.946182  4984 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.948347  4984 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.949394  3810 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.952997  5186 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:08.736977  5160 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:08.954190  5160 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:08.956312  4745 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:08.960330  5125 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:08.911290  5205 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260809 23:22:08.961609  5205 consensus_queue.cc:1225] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 26.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:08.971269 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.972007 24543 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:08.912616  5141 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:08.977983 25368 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:08.985031 24543 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:08.986413 25325 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:08.987942 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:08.988704 24543 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:08.992206 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1926) took 1615 ms. Trace:
I20260809 23:22:08.995453 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:08.997047 24542 raft_consensus.cc:2379] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:08.992666 25371 rpcz_store.cc:276] 0809 23:22:07.377240 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.377418 (+   178us) service_pool.cc:227] Handling call
0809 23:22:07.377766 (+   348us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:07.378144 (+   378us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.378171 (+    27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:07.387708 (+  9537us) log.cc:842] Serialized 52 byte log entry
0809 23:22:07.738420 (+350712us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:08.486720 (+748300us) log.cc:842] Serialized 14 byte log entry
0809 23:22:08.486986 (+   266us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.487075 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:08.744289 (+257214us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:08.968248 (+223959us) raft_consensus.cc:1671] finished
0809 23:22:08.991939 (+ 23691us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:08.992133 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1537,"prepare.queue_time_us":976360,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":51,"raft.queue_time_us":5903,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":8602,"spinlock_wait_cycles":69271808,"thread_start_us":1081768,"threads_started":2,"wal-append.queue_time_us":699160}
I20260809 23:22:08.898113  5079 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.081s
I20260809 23:22:09.009003 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:08.998876  4989 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.995s	user 0.011s	sys 0.030s
I20260809 23:22:08.967669  5010 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260809 23:22:08.998875  5186 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:22:09.025424 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.012s	sys 0.024s
W20260809 23:22:09.027281 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.012s	sys 0.024s
W20260809 23:22:09.026758 24312 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.016s	sys 0.020s
W20260809 23:22:09.029868 24312 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.016s	sys 0.020s
I20260809 23:22:08.991698  5030 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:08.999845  5141 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:09.040390 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 505178us
I20260809 23:22:09.041693  4678 consensus_queue.cc:1035] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:09.024858  5024 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:09.025545  4989 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.037s	user 0.011s	sys 0.031s
I20260809 23:22:09.043874 25323 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:09.052291  4745 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:09.052552  5141 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:09.047457 25375 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.010s	sys 0.037s
W20260809 23:22:09.055797 24316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.004s	sys 0.014s
W20260809 23:22:09.056703 24316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.004s	sys 0.014s
I20260809 23:22:09.036876  5186 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:09.047767 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.015s	sys 0.031s
W20260809 23:22:09.062466 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.015s	sys 0.032s
W20260809 23:22:09.056267 25375 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.010s	sys 0.037s
I20260809 23:22:09.060047 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3188) took 3365 ms. Trace:
I20260809 23:22:09.079672 24314 rpcz_store.cc:276] 0809 23:22:05.694169 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.107713 (+413544us) service_pool.cc:227] Handling call
0809 23:22:06.150792 (+ 43079us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.167851 (+2017059us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.059805 (+891954us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54000db6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:09.059990 (+   185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4445756160}
I20260809 23:22:09.037364  5030 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:09.057463  5139 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.002s	sys 0.016s
I20260809 23:22:09.084124  5116 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:09.085049  4985 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.085492  4678 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:22:09.088125  5163 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:09.035787  5025 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260809 23:22:09.079218  5187 raft_consensus.cc:2781] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
W20260809 23:22:09.109443 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.011s	sys 0.048s
W20260809 23:22:09.109750 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.011s	sys 0.048s
W20260809 23:22:09.110324 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 812981us
I20260809 23:22:09.100723  5264 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Stepping down as leader of term 27
W20260809 23:22:09.114059 25373 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.007s	sys 0.008s
I20260809 23:22:09.095247 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:09.114696 25373 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.007s	sys 0.009s
I20260809 23:22:09.096300 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:09.101408 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.001s	sys 0.017s
W20260809 23:22:09.115702 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.002s	sys 0.018s
W20260809 23:22:09.115837 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.016s	sys 0.016s
W20260809 23:22:09.095965  5154 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.008s	sys 0.010s
W20260809 23:22:09.095338  5139 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.002s	sys 0.016s
I20260809 23:22:09.100560  5262 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:09.103029  4588 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.011s	sys 0.035s
I20260809 23:22:09.113093 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:09.167524 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56577.6us
W20260809 23:22:09.102068  4846 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.008s	sys 0.033s
W20260809 23:22:09.169626  4846 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.008s	sys 0.033s
W20260809 23:22:09.058022  4968 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.004s	sys 0.016s
W20260809 23:22:09.116166 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.016s	sys 0.016s
W20260809 23:22:09.107518  5231 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.020s	user 0.001s	sys 0.003s
W20260809 23:22:09.117235 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.009s	sys 0.045s
W20260809 23:22:09.113844  4945 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.001s	sys 0.018s
I20260809 23:22:09.161409 24311 raft_consensus.cc:2379] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:09.116029 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 660843us
I20260809 23:22:09.180958 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:09.115149  5187 raft_consensus.cc:686] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:09.173942 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.009s	sys 0.045s
W20260809 23:22:09.183743 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.012s	sys 0.024s
W20260809 23:22:09.191995 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.012s	sys 0.024s
W20260809 23:22:09.176384  4968 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.004s	sys 0.016s
I20260809 23:22:09.175139 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:09.114508  5264 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:09.185659 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:09.167019  4588 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.011s	sys 0.035s
I20260809 23:22:09.187378 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:09.162115  5154 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.009s	sys 0.010s
I20260809 23:22:09.176795  5025 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:09.161370  5010 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:09.105969  4985 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.195405 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.133754 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:22:09.168722  4375 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:09.219945 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.220587 24422 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:09.184542  4945 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.002s	sys 0.018s
I20260809 23:22:09.183486 25369 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:09.211946 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.211994 24321 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:09.217604 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:09.222880 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.012s	sys 0.019s
W20260809 23:22:09.227928 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.012s	sys 0.019s
W20260809 23:22:09.223688 24429 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.002s	sys 0.029s
W20260809 23:22:09.215417  4948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.009s	sys 0.020s
W20260809 23:22:09.229022 24429 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.003s	sys 0.029s
I20260809 23:22:09.199719  5118 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:09.234488  5118 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:09.235423 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.011s	sys 0.008s
W20260809 23:22:09.235764 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.011s	sys 0.008s
I20260809 23:22:09.237095  5118 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:09.237335 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3443) took 1975 ms. Trace:
I20260809 23:22:09.237931 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3874) took 1272 ms. Trace:
I20260809 23:22:09.218025  5010 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.238372 24534 rpcz_store.cc:276] 0809 23:22:07.965536 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.965663 (+   127us) service_pool.cc:227] Handling call
0809 23:22:07.998442 (+ 32779us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.998839 (+   397us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.998868 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:08.002704 (+  3836us) log.cc:842] Serialized 26 byte log entry
0809 23:22:08.903396 (+900692us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:08.903456 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.903551 (+    95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:09.237698 (+334147us) raft_consensus.cc:1671] finished
0809 23:22:09.237723 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.237871 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2402,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":8610,"thread_start_us":792149,"threads_started":1,"wal-append.queue_time_us":1043148}
W20260809 23:22:09.234982  5261 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260809 23:22:09.240109  5010 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:09.230437  4948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.009s	sys 0.020s
I20260809 23:22:09.222486  5139 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.241214  5105 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.241540  5139 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.243942  5139 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:09.237772 24431 rpcz_store.cc:276] 0809 23:22:07.262225 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.353140 (+ 90915us) service_pool.cc:227] Handling call
0809 23:22:07.353553 (+   413us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:07.354005 (+   452us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.354046 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:07.380212 (+ 26166us) log.cc:842] Serialized 52 byte log entry
0809 23:22:07.759642 (+379430us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:07.784632 (+ 24990us) log.cc:842] Serialized 14 byte log entry
0809 23:22:07.787758 (+  3126us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:07.787855 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:09.091668 (+1303813us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:09.237010 (+145342us) raft_consensus.cc:1671] finished
0809 23:22:09.237116 (+   106us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.237274 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":24694,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":3960,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":230,"spinlock_wait_cycles":78967552,"thread_start_us":333895,"threads_started":1,"wal-append.queue_time_us":689038}
I20260809 23:22:09.247043 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:09.247679 25369 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:09.247577  5264 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.248605 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:09.249131  5264 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:09.249565 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.250600 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.251128 24428 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:09.251542  4985 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:09.249172 25369 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 28
I20260809 23:22:09.246690 25365 raft_consensus.cc:2379] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:09.168656  5186 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 23.21, Last appended by leader: 21, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.257766 24542 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260809 23:22:09.258790 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.259406 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:09.253763  4958 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:09.204191  5154 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:09.246698 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.263868 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.266059 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:09.274744 24543 raft_consensus.cc:2396] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 25 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260809 23:22:09.275612 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:09.224541  5166 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.018s	sys 0.010s
I20260809 23:22:09.266404 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.279884 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:09.268426 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:09.168995  5262 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.278723 25323 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.282406 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.282424 24546 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:09.189397  5160 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260809 23:22:09.286595 24323 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [21.16-23.17]
I20260809 23:22:09.294452 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.296645 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:09.299419 25365 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:09.226270  4945 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:09.308830 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:09.306334 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:09.243935  5105 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:09.278378 25369 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:22:09.269060  3979 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.314105 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:09.281145  5166 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.019s	sys 0.010s
W20260809 23:22:09.316351 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 47855.7us
I20260809 23:22:09.317652 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.317889 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:09.322563 24323 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
W20260809 23:22:09.286303  4973 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.011s	sys 0.019s
W20260809 23:22:09.316838 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50866.1us
I20260809 23:22:09.321372 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.327226 25369 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260809 23:22:09.329059 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:09.273772 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 354846us
I20260809 23:22:09.296641  4985 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:09.285876  4958 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:09.227576  4375 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.396148 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.350546 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:09.372126 25365 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:09.364908 25321 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.419664 24314 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260809 23:22:09.358158 24546 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:09.421077 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.307753  5290 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:09.331925  4973 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.012s	sys 0.019s
I20260809 23:22:09.316879  5030 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:09.426487 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:09.434593 24323 raft_consensus.cc:2354] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
W20260809 23:22:09.402786 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52343.1us
I20260809 23:22:09.361675 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.437037 24431 raft_consensus.cc:2396] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260809 23:22:09.436844 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.331842  5105 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:09.443336 24534 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:09.444332 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2000) took 1184 ms. Trace:
I20260809 23:22:09.444958 24316 rpcz_store.cc:276] 0809 23:22:08.259559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.524361 (+264802us) service_pool.cc:227] Handling call
0809 23:22:08.524644 (+   283us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.057271 (+532627us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:09.098345 (+ 41074us) log.cc:842] Serialized 14 byte log entry
0809 23:22:09.443734 (+345389us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:09.443831 (+    97us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:09.443909 (+    78us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.444022 (+   113us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.444236 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":41667,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":1581,"raft.queue_time_us":43905,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6459,"thread_start_us":876877,"threads_started":2}
I20260809 23:22:09.319469  5010 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.470552 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:09.474326 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53015.2us
I20260809 23:22:09.426151  4757 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:09.476646 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:09.485219 24323 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:09.485487 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:09.483820 24316 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:09.294653  5187 consensus_queue.cc:237] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.327278  3979 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.490734 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:09.489534 24270 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:09.505281 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68519us
I20260809 23:22:09.506275 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:09.506430 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.509220 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.520853  5269 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.476733  5029 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:09.537606 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.542703 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.542855 24323 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:09.543159 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:09.489482  4862 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.018s	sys 0.028s
I20260809 23:22:09.539760 24428 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:22:09.553344 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63899.3us
I20260809 23:22:09.554158 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.555778 24270 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:09.488023  4757 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:09.553367 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:09.550033 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:09.429857  5301 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.149s
I20260809 23:22:09.443442  4823 consensus_queue.cc:1046] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.102s
I20260809 23:22:09.566262 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:22:09.573529 24316 raft_consensus.cc:595] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:09.573891 24316 raft_consensus.cc:603] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:09.574069  5280 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:09.576309 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1986) took 2162 ms. Trace:
I20260809 23:22:09.576766 24312 rpcz_store.cc:276] 0809 23:22:07.413868 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.295258 (+881390us) service_pool.cc:227] Handling call
0809 23:22:08.368547 (+ 73289us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.368759 (+   212us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:09.032085 (+663326us) log.cc:842] Serialized 14 byte log entry
0809 23:22:09.575913 (+543828us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:09.575987 (+    74us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:09.576044 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.576125 (+    81us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.576249 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108094,"prepare.queue_time_us":569130,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":129982,"thread_start_us":1018319,"threads_started":3}
I20260809 23:22:09.535303  4678 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:09.563649  4757 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
W20260809 23:22:09.585301  5284 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260809 23:22:09.586625  5030 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260809 23:22:09.591511  5289 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.283s	user 0.000s	sys 0.001s
I20260809 23:22:09.552217  5269 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.591665  5338 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:09.593796  5308 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:09.580024  3810 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:09.551858  4862 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.019s	sys 0.028s
W20260809 23:22:09.594880  5272 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260809 23:22:09.487852  5010 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:09.639720 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102180us
W20260809 23:22:09.601910 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46218.9us
W20260809 23:22:09.603734 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 180903us
I20260809 23:22:09.607402 24323 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:09.614562  5325 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.618871 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3462) took 1327 ms. Trace:
I20260809 23:22:09.619889  5262 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:09.483726  4945 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.621405  5301 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:09.622401 24540 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:09.658439 24540 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:09.443359  5154 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.552647  5278 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:09.627061 24424 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:09.659940 24424 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:09.624624 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3457) took 1607 ms. Trace:
I20260809 23:22:09.661934 24435 rpcz_store.cc:276] 0809 23:22:08.017076 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.289757 (+272681us) service_pool.cc:227] Handling call
0809 23:22:09.623938 (+1334181us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b5800004598. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900bce 000000000034aea9 00007fac2872ca30 00007fac287f27b4 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:22:09.624125 (+   187us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.624334 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:09.624360 (+    26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:09.624403 (+    43us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:09.624433 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.624486 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.624574 (+    88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9006720}
I20260809 23:22:09.573208  5160 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:09.566883 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:09.637231  3979 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:09.643522 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.673148 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.630686  5308 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:09.645669 24544 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260809 23:22:09.646059 24429 rpcz_store.cc:276] 0809 23:22:08.291611 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.482533 (+190922us) service_pool.cc:227] Handling call
0809 23:22:08.518216 (+ 35683us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:08.535409 (+ 17193us) raft_consensus.cc:1520] Early marking committed up to index 9
0809 23:22:08.535440 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:08.541582 (+  6142us) log.cc:842] Serialized 52 byte log entry
0809 23:22:09.230900 (+689318us) raft_consensus.cc:1631] Marking committed up to 9
0809 23:22:09.230957 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.231019 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:09.618547 (+387528us) raft_consensus.cc:1671] finished
0809 23:22:09.618579 (+    32us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:09.618788 (+   209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9007,"raft.queue_time_us":12594,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":4196,"thread_start_us":654009,"threads_started":1,"wal-append.queue_time_us":951642}
I20260809 23:22:09.669538 24422 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:09.563405  5287 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.098s
I20260809 23:22:09.682958 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.683568 24422 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:09.644783 25371 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [21.16-23.17]
W20260809 23:22:09.685603 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 118810us
I20260809 23:22:09.687431 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:09.661039 24424 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.688534 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.688951 24424 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:09.689435  5010 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:09.689579  5347 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:09.692312  5304 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260809 23:22:09.693563  5172 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:09.557703  4958 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.671815 24540 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.698910 24540 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:09.697094 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.645258  5338 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:09.687812 24378 leader_election.cc:398] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:09.674688 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:09.671434  5262 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:09.739928 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:09.738433 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63855.4us
I20260809 23:22:09.742094 24378 leader_election.cc:401] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:09.685199  5308 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:09.745139 24540 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260809 23:22:09.746726 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45039.7us
I20260809 23:22:09.748430 24543 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260809 23:22:09.747366 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1917) took 1098 ms. Trace:
I20260809 23:22:09.749377 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.750174 24540 rpcz_store.cc:276] 0809 23:22:08.649059 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.783309 (+134250us) service_pool.cc:227] Handling call
0809 23:22:08.784767 (+  1458us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.622294 (+837527us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54002e1e20. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237f0730 00007fac23862300 00007fac23863371 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc
0809 23:22:09.747074 (+124780us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.747288 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":609358720}
I20260809 23:22:09.751145 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.760819 24270 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.762383 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:09.695990  5331 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260809 23:22:09.776093  5278 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:09.776453  5275 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:09.776721  5278 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:09.777618  5278 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.778410  5262 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:09.778813  5262 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:09.674741  5030 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
I20260809 23:22:09.701263  5276 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:09.791162  5275 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260809 23:22:09.794194 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.794790 24540 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:09.739617  5287 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:09.739921  5329 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:09.748174  5160 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:09.744696  4375 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:09.737401  5172 consensus_queue.cc:1046] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.100s
I20260809 23:22:09.779225  5262 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:09.669278  5325 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.779418  5347 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:09.802184 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:09.841845  5359 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:09.843705  5268 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.616s	user 0.001s	sys 0.004s
I20260809 23:22:09.846549  5024 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:09.780136  5156 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.057s
I20260809 23:22:09.847846  5338 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.850227  4375 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.850742  4375 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.852731  4375 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:09.779575  5337 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
W20260809 23:22:09.815949  5288 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260809 23:22:09.808823  5276 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:09.701253  5302 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.132s
I20260809 23:22:09.847023  5160 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:09.889251  5160 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:09.847007  5024 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:09.891389 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.892140 24542 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:09.894852 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:09.895931 24540 raft_consensus.cc:2396] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 30 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260809 23:22:09.896721 24429 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260809 23:22:09.897826 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:09.870708 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 68644.7us
I20260809 23:22:09.810554  5053 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:09.810527  5349 consensus_queue.cc:1035] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260809 23:22:09.809803  5329 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:09.878804  5223 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.767s	user 0.001s	sys 0.002s
I20260809 23:22:09.880033  5325 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:09.880674  5269 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:09.881085 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3832) took 1668 ms (client timeout 1584 ms). Trace:
I20260809 23:22:09.880962 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:22:09.917687 24316 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:09.918203 24316 raft_consensus.cc:603] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:09.881523  5276 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:09.899320 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:09.917361 24539 rpcz_store.cc:269] 0809 23:22:08.212938 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.213230 (+   292us) service_pool.cc:227] Handling call
0809 23:22:09.880835 (+1667605us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:09.881015 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1496099072}
I20260809 23:22:09.957087 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:09.912997  5275 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:09.914824  5301 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:09.891745  5024 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:10.004688 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 287649us
I20260809 23:22:09.916762  5329 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:09.916045  5053 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:10.033502 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 390160us
I20260809 23:22:10.048976 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1980) took 2982 ms. Trace:
I20260809 23:22:10.050279 24309 rpcz_store.cc:276] 0809 23:22:07.066846 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.361268 (+294422us) service_pool.cc:227] Handling call
0809 23:22:07.361563 (+   295us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.361875 (+   312us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.361902 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:07.362270 (+   368us) log.cc:842] Serialized 26 byte log entry
0809 23:22:07.798884 (+436614us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:07.799161 (+   277us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:07.799279 (+   118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:09.878159 (+2078880us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:10.045130 (+166971us) raft_consensus.cc:1671] finished
0809 23:22:10.045152 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.048918 (+  3766us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1234,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"spinlock_wait_cycles":10487680,"thread_start_us":436224,"threads_started":1,"wal-append.queue_time_us":749215}
I20260809 23:22:09.920046  5262 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.051720 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.034520  5331 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:22:10.025398 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 126185us
I20260809 23:22:09.999168  5352 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.051489  5287 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:10.030496 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 344046us
I20260809 23:22:10.052345 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3796) took 3658 ms. Trace:
I20260809 23:22:10.074215 24311 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:22:10.075799 24317 rpcz_store.cc:276] 0809 23:22:06.394131 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.350798 (+956667us) service_pool.cc:227] Handling call
0809 23:22:07.381461 (+ 30663us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:07.759660 (+378199us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.052088 (+2292428us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b540007dd54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.052285 (+   197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":749719424}
I20260809 23:22:10.077275 24311 raft_consensus.cc:595] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.077605  5378 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:10.083201 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.080314 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:10.086797  5308 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.088517 24546 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:10.091202 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.092549 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:10.090201 24498 connection.cc:232] RPC call timeout handler was delayed by -1.2110447371378541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
I20260809 23:22:10.137249 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:10.139148 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44696.3us
I20260809 23:22:10.138118 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:10.141987 24498 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:10.142475 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.143555 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:10.135182  5269 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:10.146337  5269 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:10.146728  5269 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:10.149303  3889 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:10.150395 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.151247 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.081977  5331 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:10.152841 24428 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:10.144331 24542 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:22:10.079686  5024 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:10.006979  5275 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.135361  5030 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:10.176826  5264 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:10.155750  4375 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:10.151021  3889 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.043s
I20260809 23:22:10.198838  5173 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:10.199407  5366 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20260809 23:22:10.078235  5378 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.200747  5275 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.101074  4757 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.202900  4757 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.161018  5331 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:10.153955  5351 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:10.154738  5368 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:10.203933 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2031) took 2590 ms. Trace:
I20260809 23:22:10.204936  5375 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260809 23:22:10.206861  5375 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:10.207209  5375 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.208010  5262 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.073760  5352 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.139892  5301 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.095661  5287 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:10.198884  5367 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260809 23:22:10.222182  5375 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.223766  5375 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:10.224617  5308 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:10.232942  5260 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.182s	user 0.001s	sys 0.002s
I20260809 23:22:10.238477  5287 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:10.240484  5236 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 2.125s	user 0.004s	sys 0.000s
W20260809 23:22:10.243274  5237 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.166s	user 0.009s	sys 0.014s
I20260809 23:22:10.245817  5352 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:10.250329  5256 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.335s	user 0.003s	sys 0.000s
I20260809 23:22:10.250479  5378 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:10.255880  5365 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260809 23:22:10.256220  5331 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:10.259011  5246 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.590s	user 0.001s	sys 0.002s
I20260809 23:22:10.251776  5329 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.262583  5269 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.265527  5269 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.265619  5359 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:10.266275  5359 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.266733  5329 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.267302  5359 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.268567  5359 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:10.143268 24309 raft_consensus.cc:2362] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25. Re-sending same reply.
I20260809 23:22:10.320827 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:10.328120 24320 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3207) took 2909 ms (client timeout 1961 ms). Trace:
I20260809 23:22:10.328506 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 1995) took 2238 ms. Trace:
I20260809 23:22:10.327831 24317 raft_consensus.cc:2454] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:10.320487 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.328948 24321 rpcz_store.cc:276] 0809 23:22:08.089832 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.472341 (+382509us) service_pool.cc:227] Handling call
0809 23:22:08.472761 (+   420us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.230636 (+757875us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.327774 (+1097138us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b540007e254. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.328258 (+   484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1684933376}
W20260809 23:22:10.328750 24320 rpcz_store.cc:269] 0809 23:22:07.418323 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.873329 (+455006us) service_pool.cc:227] Handling call
0809 23:22:10.327527 (+2454198us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003c28d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:10.327790 (+   263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":293174528}
I20260809 23:22:10.332439 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.333678 24320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:22:10.334563 24320 raft_consensus.cc:595] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.330853 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.336570 24320 raft_consensus.cc:603] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:10.339748 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3807) took 1083 ms. Trace:
I20260809 23:22:10.351039 24314 rpcz_store.cc:276] 0809 23:22:09.256022 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.256133 (+   111us) service_pool.cc:227] Handling call
0809 23:22:09.256444 (+   311us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:09.423701 (+167257us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.326214 (+902513us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400253ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.339481 (+ 13267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":374600448}
I20260809 23:22:10.340713 24320 raft_consensus.cc:1226] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260809 23:22:10.345834 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:22:10.351765 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.353441 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3428) took 3555 ms. Trace:
I20260809 23:22:10.204316 25375 rpcz_store.cc:276] 0809 23:22:07.613757 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.301921 (+688164us) service_pool.cc:227] Handling call
0809 23:22:08.302241 (+   320us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:09.068078 (+765837us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:09.068123 (+    45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:09.095789 (+ 27666us) log.cc:842] Serialized 26 byte log entry
0809 23:22:09.514118 (+418329us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:09.514189 (+    71us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.203515 (+689326us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b540065d6d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.203568 (+    53us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.203617 (+    49us) raft_consensus.cc:1671] finished
0809 23:22:10.203630 (+    13us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.203872 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15168,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":1959,"spinlock_wait_cycles":550850816,"thread_start_us":880769,"threads_started":2,"wal-append.queue_time_us":501518}
I20260809 23:22:10.338680 24321 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:10.377637 25375 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260809 23:22:10.378396 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1936) took 1165 ms. Trace:
W20260809 23:22:10.379772 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.961s (SENT)
I20260809 23:22:10.272152  5331 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.355675 24423 rpcz_store.cc:276] 0809 23:22:06.797949 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:06.798120 (+   171us) service_pool.cc:227] Handling call
0809 23:22:06.798688 (+   568us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:06.815666 (+ 16978us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:06.815709 (+    43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:06.816142 (+   433us) log.cc:842] Serialized 26 byte log entry
0809 23:22:07.832269 (+1016127us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b1800392bc0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:08.507560 (+675291us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:08.507852 (+   292us) log.cc:842] Serialized 14 byte log entry
0809 23:22:08.508053 (+   201us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.508130 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.249813 (+1741683us) spinlock_profiling.cc:243] Waited 4.82 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:10.352764 (+102951us) raft_consensus.cc:1671] finished
0809 23:22:10.352811 (+    47us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.353174 (+   363us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50976,"prepare.queue_time_us":648,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":27220,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":16048,"spinlock_wait_cycles":19192064,"thread_start_us":675094,"threads_started":1,"wal-append.queue_time_us":1969248}
I20260809 23:22:10.372452  3889 raft_consensus.cc:3032] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:10.390959  3889 raft_consensus.cc:729] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:10.373306  4375 raft_consensus.cc:686] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:10.378731 25375 rpcz_store.cc:276] 0809 23:22:09.212986 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.376274 (+1163288us) service_pool.cc:227] Handling call
0809 23:22:10.377035 (+   761us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.378059 (+  1024us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.378282 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:10.406883 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3472) took 1119 ms. Trace:
W20260809 23:22:10.408260 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2027) took 2901 ms (client timeout 3110 ms). Trace:
W20260809 23:22:10.424721 25366 rpcz_store.cc:269] 0809 23:22:07.506373 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.755562 (+249189us) service_pool.cc:227] Handling call
0809 23:22:10.407739 (+2652177us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b54003dfed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:10.407986 (+   247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2144177408}
I20260809 23:22:10.428725  5270 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:10.429742  5270 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.418722 24429 rpcz_store.cc:276] 0809 23:22:09.287778 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.689511 (+401733us) service_pool.cc:227] Handling call
0809 23:22:09.689792 (+   281us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.897182 (+207390us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.406240 (+509058us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.406602 (+   362us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465729408}
I20260809 23:22:10.431970 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:10.432775 24429 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:10.435626 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1744) took 1177 ms. Trace:
I20260809 23:22:10.436189 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.436064 24435 rpcz_store.cc:276] 0809 23:22:09.258331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.663368 (+405037us) service_pool.cc:227] Handling call
0809 23:22:09.854472 (+191104us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b5800004598. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:22:09.854686 (+   214us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:09.854895 (+   209us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:09.854921 (+    26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:09.855397 (+   476us) log.cc:842] Serialized 26 byte log entry
0809 23:22:09.946726 (+ 91329us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:09.946767 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.946839 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.434977 (+488138us) raft_consensus.cc:1671] finished
0809 23:22:10.435026 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.435467 (+   441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12355840,"thread_start_us":90969,"threads_started":1,"wal-append.queue_time_us":193211}
I20260809 23:22:10.438292 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.439323  5378 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.440047 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.440222  5378 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.441571  5368 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:10.442649  5368 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:10.443347  5368 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
W20260809 23:22:10.440400 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 89306us
I20260809 23:22:10.447703 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1904) took 2898 ms. Trace:
I20260809 23:22:10.448019 24378 leader_election.cc:398] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:10.448515 24378 leader_election.cc:401] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260809 23:22:10.449391  5337 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:10.449642  5331 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24.
I20260809 23:22:10.448342 24545 rpcz_store.cc:276] 0809 23:22:07.549213 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.549347 (+   134us) service_pool.cc:227] Handling call
0809 23:22:07.589113 (+ 39766us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:07.589550 (+   437us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.589591 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:07.660190 (+ 70599us) log.cc:842] Serialized 26 byte log entry
0809 23:22:08.161068 (+500878us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:08.161129 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.161222 (+    93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.258254 (+2097032us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:10.446985 (+188731us) raft_consensus.cc:1671] finished
0809 23:22:10.447047 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.447423 (+   376us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":69701,"spinlock_wait_cycles":4363520,"thread_start_us":470544,"threads_started":1,"wal-append.queue_time_us":993383}
I20260809 23:22:10.409567  5084 raft_consensus.cc:3032] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
W20260809 23:22:10.461249 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.09825e+06us
I20260809 23:22:10.406822  4251 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.470031 25369 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:10.470753 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1885) took 3040 ms. Trace:
I20260809 23:22:10.471382 25370 rpcz_store.cc:276] 0809 23:22:07.429845 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.429985 (+   140us) service_pool.cc:227] Handling call
0809 23:22:07.430288 (+   303us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:07.524431 (+ 94143us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.524463 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:07.524969 (+   506us) log.cc:842] Serialized 52 byte log entry
0809 23:22:07.824435 (+299466us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:07.824480 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:08.089971 (+265491us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:08.090017 (+    46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.239302 (+2149285us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:10.454413 (+215111us) raft_consensus.cc:1671] finished
0809 23:22:10.465409 (+ 10996us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.470492 (+  5083us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10044,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":3433,"spinlock_wait_cycles":219785216,"thread_start_us":299101,"threads_started":1,"wal-append.queue_time_us":551654}
I20260809 23:22:10.470049 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.484745  5280 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:10.485643 24540 raft_consensus.cc:2396] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260809 23:22:10.488996 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.495550  5053 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.495939  5378 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.496775 24435 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:10.497269 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:10.497550 25368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2039) took 1981 ms (client timeout 1520 ms). Trace:
I20260809 23:22:10.498055 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:10.499194  5344 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.499364 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.499961  5344 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.502794  5337 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.503485  5319 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:10.503806  5337 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.455449  5084 raft_consensus.cc:729] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.506330  5084 consensus_queue.cc:260] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.490657 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.498574 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:10.438793 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:10.498124 25368 rpcz_store.cc:269] 0809 23:22:08.515897 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.577960 (+ 62063us) service_pool.cc:227] Handling call
0809 23:22:10.496960 (+1919000us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b540069df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:10.497256 (+   296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":871746944}
I20260809 23:22:10.515686 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2046) took 1305 ms. Trace:
I20260809 23:22:10.406157  3889 consensus_queue.cc:260] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.517758 24316 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:10.516237 25365 rpcz_store.cc:276] 0809 23:22:09.209886 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.427067 (+217181us) service_pool.cc:227] Handling call
0809 23:22:09.427400 (+   333us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:09.474916 (+ 47516us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:09.474965 (+    49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:09.475488 (+   523us) log.cc:842] Serialized 52 byte log entry
0809 23:22:09.658259 (+182771us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:09.676086 (+ 17827us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.514875 (+838789us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.515132 (+   257us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.515207 (+    75us) raft_consensus.cc:1671] finished
0809 23:22:10.515227 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.515485 (+   258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":877879,"prepare.queue_time_us":19320,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":948258,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":6389,"thread_start_us":182002,"threads_started":1,"wal-append.queue_time_us":328315}
I20260809 23:22:10.525135 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1934) took 1314 ms. Trace:
I20260809 23:22:10.506893  5337 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.530277 24423 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:10.530105  4251 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.531611  5084 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:10.533102 24435 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:10.525682 25371 rpcz_store.cc:276] 0809 23:22:09.206243 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.206453 (+   210us) service_pool.cc:227] Handling call
0809 23:22:09.206888 (+   435us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:09.696878 (+489990us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.520310 (+823432us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54002ab7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.520919 (+   609us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1000442624}
I20260809 23:22:10.404534  4375 consensus_queue.cc:237] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.532318  5344 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.554250 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2022) took 3039 ms. Trace:
I20260809 23:22:10.554867 25372 rpcz_store.cc:276] 0809 23:22:07.501394 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.243564 (+742170us) service_pool.cc:227] Handling call
0809 23:22:08.243849 (+   285us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.488183 (+244334us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.525422 (+2037239us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540001ac54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.540371 (+ 14949us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":306205952}
I20260809 23:22:10.565044 24539 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:22:10.574975 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.582662  4968 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.583171  5163 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.584662  5163 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.586447  5173 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:10.587249  5173 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.587751  5270 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.589661  5173 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.590193  5173 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.590342  4251 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.592585  5301 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:10.596165  4993 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260809 23:22:10.598114 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3464) took 1410 ms. Trace:
I20260809 23:22:10.598771 24431 rpcz_store.cc:276] 0809 23:22:09.187368 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.580012 (+392644us) service_pool.cc:227] Handling call
0809 23:22:10.507391 (+927379us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5800004598. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:22:10.543137 (+ 35746us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.597305 (+ 54168us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.597360 (+    55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:10.597444 (+    84us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:10.597525 (+    81us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.597632 (+   107us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.597844 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33193856}
I20260809 23:22:10.601277 24431 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
W20260809 23:22:10.605598 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 157951us
I20260809 23:22:10.606732 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.608705 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1889) took 3137 ms. Trace:
I20260809 23:22:10.608915 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.583353  5141 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:10.610312  5329 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:10.611030  5329 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.610813  5141 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.611948  5013 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:10.613245  5187 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.613538  5141 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.615355  4984 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:10.612725  5329 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:10.616371 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 177756us
I20260809 23:22:10.615959 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.646180  5026 raft_consensus.cc:3032] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:22:10.649457  5029 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:10.649987  5029 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:10.584044  4968 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.600560  4997 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.651808 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.590684  5173 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.661655  4968 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.662882  4993 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:10.664719  5347 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:10.658264  5004 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260809 23:22:10.661468  5010 consensus_queue.cc:1046] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260809 23:22:10.682510  5118 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:22:10.683010  5010 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:10.683538  5010 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:10.684347 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.685050  5301 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:10.685479 25371 raft_consensus.cc:2379] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 28: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:10.684944  5010 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.667277 25366 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:10.685875  5301 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.687038  5172 raft_consensus.cc:3032] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:10.667431  5116 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:10.688603 25368 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:10.688277  5116 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.691632 25370 raft_consensus.cc:1226] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:10.692864 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.692504  5116 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:10.693614 25375 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:10.667711  4984 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.698753 24423 raft_consensus.cc:2379] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:10.699195 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2006) took 1227 ms. Trace:
I20260809 23:22:10.614288  5187 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.702798 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1743) took 1454 ms. Trace:
I20260809 23:22:10.704540  5200 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:10.705109  5200 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:10.703791  5415 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:10.701063 24431 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:10.706259 24431 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.699606 24312 rpcz_store.cc:276] 0809 23:22:09.471323 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.578401 (+107078us) service_pool.cc:227] Handling call
0809 23:22:09.578771 (+   370us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:09.579940 (+  1169us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:09.580061 (+   121us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:09.581386 (+  1325us) log.cc:842] Serialized 26 byte log entry
0809 23:22:10.333294 (+751908us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b1800080260. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:10.556978 (+223684us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:10.571787 (+ 14809us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.575926 (+  4139us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.616635 (+ 40709us) raft_consensus.cc:1671] finished
0809 23:22:10.616655 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.698996 (+ 82341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11475,"raft.queue_time_us":434,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":27030,"spinlock_wait_cycles":29238656,"thread_start_us":142894,"threads_started":1,"wal-append.queue_time_us":994767}
W20260809 23:22:10.703791 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 88203.1us
I20260809 23:22:10.646772  5026 raft_consensus.cc:729] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:10.683403 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:10.687501  5172 raft_consensus.cc:729] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:10.710763 24429 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:10.724390 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.725135 25370 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260809 23:22:10.726069  5359 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:10.726276 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.683051  5118 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:10.726881  5359 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.728101  5118 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.729395  5118 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:10.729709  5359 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.730530  5301 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:10.732242  5403 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260809 23:22:10.732632 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:10.733981  5337 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:10.734503 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2007) took 1262 ms. Trace:
I20260809 23:22:10.735430 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.735711  5337 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:10.736017 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.736315  5337 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:10.737671  5337 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.738888  5026 consensus_queue.cc:260] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.739367  5116 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:10.738986 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.609144 25373 rpcz_store.cc:276] 0809 23:22:07.471479 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.471617 (+   138us) service_pool.cc:227] Handling call
0809 23:22:07.505459 (+ 33842us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:07.537252 (+ 31793us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:07.537282 (+    30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:07.547967 (+ 10685us) log.cc:842] Serialized 52 byte log entry
0809 23:22:08.487998 (+940031us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:09.115077 (+627079us) log.cc:842] Serialized 14 byte log entry
0809 23:22:09.115479 (+   402us) log.cc:842] Serialized 14 byte log entry
0809 23:22:09.617977 (+502498us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b18003afc00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac234fde10 00007fac234f1663 00007fac234e702a 00007fac234e757b 00007fac2386a88a 00007fac23879f0f 00007fac23879e88
0809 23:22:10.606993 (+989016us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.608128 (+  1135us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:10.608194 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.608293 (+    99us) raft_consensus.cc:1671] finished
0809 23:22:10.608310 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.608518 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1441190,"prepare.queue_time_us":294883,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":1102591,"raft.run_cpu_time_us":496,"raft.run_wall_time_us":22732,"spinlock_wait_cycles":62739200,"thread_start_us":1411164,"threads_started":2,"wal-append.queue_time_us":1147694}
I20260809 23:22:10.750902 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3474) took 1098 ms. Trace:
I20260809 23:22:10.704825 24424 rpcz_store.cc:276] 0809 23:22:09.203238 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.624780 (+421542us) service_pool.cc:227] Handling call
0809 23:22:09.626892 (+  2112us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:09.911256 (+284364us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:22:10.433648 (+522392us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.569473 (+135825us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.574156 (+  4683us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:10.575842 (+  1686us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:10.578749 (+  2907us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.617019 (+ 38270us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.657679 (+ 40660us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":495682,"prepare.run_cpu_time_us":143,"prepare.run_wall_time_us":918,"raft.queue_time_us":90817,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":42253,"spinlock_wait_cycles":5997184,"thread_start_us":733239,"threads_started":3,"wal-append.queue_time_us":118733}
I20260809 23:22:10.751413 24428 rpcz_store.cc:276] 0809 23:22:09.582622 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.157204 (+574582us) service_pool.cc:227] Handling call
0809 23:22:10.157443 (+   239us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.157845 (+   402us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:10.509660 (+351815us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.579677 (+ 70017us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:10.623723 (+ 44046us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:10.641976 (+ 18253us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.654713 (+ 12737us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.681053 (+ 26340us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":237677,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":2139,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":9366,"spinlock_wait_cycles":45636864,"thread_start_us":336585,"threads_started":2,"wal-append.queue_time_us":161336}
I20260809 23:22:10.739421  5376 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:10.746281  5163 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.779291 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.791796 24544 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:10.795315 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.784758 24498 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:10.735183 24323 rpcz_store.cc:276] 0809 23:22:09.471730 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.660494 (+188764us) service_pool.cc:227] Handling call
0809 23:22:09.682454 (+ 21960us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:09.684445 (+  1991us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:09.736138 (+ 51693us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.332128 (+595990us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b180047b9c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:10.542986 (+210858us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.583638 (+ 40652us) log.cc:842] Serialized 26 byte log entry
0809 23:22:10.583832 (+   194us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:10.583872 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.584090 (+   218us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.733812 (+149722us) raft_consensus.cc:1671] finished
0809 23:22:10.733934 (+   122us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.734088 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99223,"prepare.queue_time_us":103957,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":2081,"raft.queue_time_us":166902,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":3077,"spinlock_wait_cycles":9857920,"thread_start_us":59908,"threads_started":1,"wal-append.queue_time_us":653722}
I20260809 23:22:10.800525 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:10.780570  5376 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:10.804536 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3939) took 1012 ms. Trace:
I20260809 23:22:10.785853  5325 consensus_queue.cc:1046] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.101s
I20260809 23:22:10.805181 24431 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.813503 24431 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:10.815542 24317 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:10.820237  5278 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:10.823320  4251 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:10.825218  4251 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:10.810264 24435 rpcz_store.cc:276] 0809 23:22:09.788127 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.571234 (+783107us) service_pool.cc:227] Handling call
0809 23:22:10.615188 (+ 43954us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.615500 (+   312us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:10.615918 (+   418us) log.cc:842] Serialized 14 byte log entry
0809 23:22:10.800352 (+184434us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:10.800414 (+    62us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:10.800466 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.800533 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.800646 (+   113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":8357,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":4801,"raft.queue_time_us":2179,"thread_start_us":183906,"threads_started":1}
I20260809 23:22:10.837754 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:10.838812 24323 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:10.840866 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 93577.8us
I20260809 23:22:10.843385  4993 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:10.834859 24424 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:10.843861 24435 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:10.848632 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 23) took 1332 ms. Trace:
I20260809 23:22:10.848763  5319 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:10.849064 24424 rpcz_store.cc:276] 0809 23:22:09.516434 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.845391 (+1328957us) service_pool.cc:227] Handling call
0809 23:22:10.848541 (+  3150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:10.849906 25372 raft_consensus.cc:2396] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260809 23:22:10.850260  5376 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.851292  5029 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.852077  5376 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:10.866555  5173 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.856709  5302 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:10.867631  5187 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:10.867796  5302 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.869022 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:10.871876 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 162941us
I20260809 23:22:10.873145 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.873208 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.874809 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:10.875041 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68615.5us
W20260809 23:22:10.876295 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122602us
I20260809 23:22:10.877417 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:10.878312  5026 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:10.879613 24431 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260809 23:22:10.874722 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.855921  5084 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:10.879974 24431 pending_rounds.cc:76] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 15
I20260809 23:22:10.889329 25373 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:10.888501 24431 pending_rounds.cc:97] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260809 23:22:10.893579 24311 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:10.893683 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.894709 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.896315 25375 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:10.896895 25368 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:10.902348  5347 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:10.902959  5347 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:10.903256 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:10.903431  5029 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.907516 24431 raft_consensus.cc:2866] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:10.907866 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49625.6us
I20260809 23:22:10.846602  5368 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:10.884068  5187 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:10.891865  5173 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:10.947867 24428 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:10.950028 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 75429.7us
I20260809 23:22:10.957379 24429 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:10.962174 25368 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260809 23:22:10.963045 24424 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:22:10.908275  5084 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.966617 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.966903  5200 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:10.977404 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.994961 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:10.978202 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:10.979104 24311 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 21.
I20260809 23:22:10.997710 25375 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:10.979380 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.980234 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.980620 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:11.003890 25372 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:11.004518  5423 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:11.005100  5423 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:11.005482  5423 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:10.981007 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.006253 25371 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:10.983156 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3803) took 3595 ms. Trace:
I20260809 23:22:10.987888 24540 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 20. Preceding OpId from leader: term: 25 index: 22. (index mismatch)
I20260809 23:22:10.966996  5302 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:10.988626 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:10.992012 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:10.968452 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:11.010919 24424 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:11.017753 24427 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:22:11.018589 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3468) took 1827 ms. Trace:
I20260809 23:22:11.018903 24427 rpcz_store.cc:276] 0809 23:22:09.190965 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.587909 (+396944us) service_pool.cc:227] Handling call
0809 23:22:11.000199 (+1412290us) spinlock_profiling.cc:243] Waited 9.51 ms on lock 0x7b5800004598. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287b62b0 00007fac287a17f9 00007fac287a12a2 00007fac21e87684
0809 23:22:11.000614 (+   415us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.018218 (+ 17604us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.018501 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21872512}
I20260809 23:22:11.021173 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260809 23:22:11.021579 24427 pending_rounds.cc:76] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 15
I20260809 23:22:11.021829 24427 pending_rounds.cc:97] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260809 23:22:11.022096 24427 raft_consensus.cc:2866] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:11.023260 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.024935  5187 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260809 23:22:11.028061 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:11.029542 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 61174.5us
I20260809 23:22:11.042814 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.043051 24544 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:22:11.036234 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:11.060273 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:10.959761  5173 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260809 23:22:11.085075 25373 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:11.042647  5337 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:11.086117 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.089921 25368 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260809 23:22:11.077337 24431 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:11.091516 25368 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 17
I20260809 23:22:11.093111 25371 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:11.059247  5415 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:11.100986 25368 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260809 23:22:11.101600 25368 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:11.006996 24313 rpcz_store.cc:276] 0809 23:22:07.387217 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.245166 (+857949us) service_pool.cc:227] Handling call
0809 23:22:08.264555 (+ 19389us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:08.266408 (+  1853us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:09.029449 (+763041us) log.cc:842] Serialized 14 byte log entry
0809 23:22:09.470902 (+441453us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.741507 (+1270605us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b2c00056b88. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:10.742175 (+   668us) log.cc:842] Serialized 26 byte log entry
0809 23:22:10.834462 (+ 92287us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:10.835541 (+  1079us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.855034 (+ 19493us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.982691 (+127657us) raft_consensus.cc:1671] finished
0809 23:22:10.982815 (+   124us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:10.982968 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63252,"prepare.queue_time_us":660987,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":9217,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":25480,"spinlock_wait_cycles":37139200,"thread_start_us":1253611,"threads_started":3,"wal-append.queue_time_us":796609,"wal-append.run_cpu_time_us":1924,"wal-append.run_wall_time_us":1006035}
W20260809 23:22:11.114187 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78089.4us
I20260809 23:22:11.115952 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.113776  5160 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.080s
I20260809 23:22:11.090765  5337 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.118896 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.125327 25370 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:11.117909  5084 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.131580 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.143139 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.143561 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:11.123435  5337 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:11.144202 24313 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:11.133112  4251 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:11.165786  5368 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:11.166226  5200 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.169088 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:11.170689 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1715) took 3620 ms. Trace:
W20260809 23:22:11.163734  5437 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260809 23:22:11.176402 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 184542us
I20260809 23:22:11.142505  5026 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:11.137602  5187 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:11.187004 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.179057 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:11.184069 24430 rpcz_store.cc:276] 0809 23:22:07.550522 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:07.550650 (+   128us) service_pool.cc:227] Handling call
0809 23:22:08.706438 (+1155788us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.712880 (+  6442us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.170033 (+2457153us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002f8ad0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:11.170503 (+   470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2656724736}
I20260809 23:22:11.137878  5376 consensus_queue.cc:1046] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.053s
I20260809 23:22:11.146759  5416 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:11.209758 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3228) took 1102 ms. Trace:
I20260809 23:22:11.163602  5311 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:11.225304 25375 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:11.199693 24323 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:11.233969 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3839) took 2786 ms. Trace:
I20260809 23:22:11.234578 24535 rpcz_store.cc:276] 0809 23:22:08.447231 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.783184 (+335953us) service_pool.cc:227] Handling call
0809 23:22:08.847348 (+ 64164us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:08.854691 (+  7343us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.196588 (+2341897us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b5400148550. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:11.233791 (+ 37203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82354560}
I20260809 23:22:11.236034 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.237542 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.163656  5205 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.110s
I20260809 23:22:11.239133 24435 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:11.239414  5026 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:11.239207  5122 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.242293  5122 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:11.243552  5428 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260809 23:22:11.245242 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 197183us
I20260809 23:22:11.245889  5122 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.246124 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:11.247778 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:11.176661  5347 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.249989 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:11.250825 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.251485 24323 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:11.251981  5347 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.252398 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:11.255568 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:11.276605 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.292639 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:11.301496  5378 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:11.303429  4507 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260809 23:22:11.242229  5026 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:11.224885  5311 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:11.306684  5311 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:11.307078  5024 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 1.060s	user 0.008s	sys 0.012s
W20260809 23:22:11.307929  5435 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260809 23:22:11.307971  5194 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:11.308879  5084 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:11.309465  5084 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.310011 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.311280 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.311662  5084 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.311985 24311 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:11.312752 24422 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260809 23:22:11.312943 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.314111 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:11.315140  4993 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:11.315513  4993 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:11.316747  4993 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:11.317482 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 105385us
I20260809 23:22:11.318225  4993 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:11.319097  4976 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.222836 24320 rpcz_store.cc:276] 0809 23:22:10.091939 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.425566 (+333627us) service_pool.cc:227] Handling call
0809 23:22:10.428243 (+  2677us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.579841 (+151598us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.579892 (+    51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.654312 (+ 74420us) log.cc:842] Serialized 26 byte log entry
0809 23:22:11.013454 (+359142us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:11.015613 (+  2159us) log.cc:842] Serialized 14 byte log entry
0809 23:22:11.132669 (+117056us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.132741 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:11.194070 (+ 61329us) raft_consensus.cc:1671] finished
0809 23:22:11.194099 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:11.194265 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":226882,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":284702,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":12886,"thread_start_us":345026,"threads_started":2,"wal-append.queue_time_us":404577}
I20260809 23:22:11.320214 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.320874 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.323109 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:11.318156 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:11.319818  4976 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.321597 24311 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:11.332267  4375 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.007s
I20260809 23:22:11.340940  4976 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.343106  5347 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:11.353906 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:11.357990 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 75253.2us
I20260809 23:22:11.283947  5368 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:11.332279  5416 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:11.356909  5118 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:11.381294 24323 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:11.365536  5424 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260809 23:22:11.393359 24313 raft_consensus.cc:1226] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260809 23:22:11.379570  5118 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:11.397775  5205 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:11.380376  5421 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.689s	user 0.005s	sys 0.000s
I20260809 23:22:11.404210  5118 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:11.401309 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 47508.8us
I20260809 23:22:11.379910  4507 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260809 23:22:11.418279  5183 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.136s
I20260809 23:22:11.379235  5122 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.419935  5122 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.377903  5116 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:11.422291 24312 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260809 23:22:11.427480  4993 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:11.428063  4993 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:11.429292  4993 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.430286 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:11.438576  5433 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
I20260809 23:22:11.433779 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1891) took 3897 ms. Trace:
I20260809 23:22:11.435596 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:11.438650  5038 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:11.442484  5200 raft_consensus.cc:3032] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:11.410264  5205 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:11.445963 24312 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:11.446094 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:11.377825  5423 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.446501  5116 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.447381  5290 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:11.449061 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:11.416742  5368 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.376027  5416 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:11.454236  5347 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:11.454972  5026 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:11.455219  5173 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:11.462759  4973 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:11.464097  5122 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.471390 25363 rpcz_store.cc:276] 0809 23:22:07.535795 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.295775 (+759980us) service_pool.cc:227] Handling call
0809 23:22:10.545449 (+2249674us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.567754 (+ 22305us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.567802 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.568474 (+   672us) log.cc:842] Serialized 26 byte log entry
0809 23:22:10.758573 (+190099us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:10.758635 (+    62us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.769141 (+ 10506us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:10.831162 (+ 62021us) raft_consensus.cc:1671] finished
0809 23:22:10.831186 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:11.433328 (+602142us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b540001ac50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:11.433588 (+   260us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23618,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":162,"spinlock_wait_cycles":5120680192,"thread_start_us":174508,"threads_started":1,"wal-append.queue_time_us":231604}
I20260809 23:22:11.482532 24323 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:11.477619  5200 raft_consensus.cc:729] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:11.501935 24313 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260809 23:22:11.502550 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.431231  4993 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:11.496686  5378 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:11.506161 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 254403us
I20260809 23:22:11.497694  5173 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:11.494810  5278 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:11.552148 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2047) took 2342 ms. Trace:
I20260809 23:22:11.616925 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3860) took 1565 ms. Trace:
W20260809 23:22:11.664736 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 405941us
I20260809 23:22:11.617450 24542 rpcz_store.cc:276] 0809 23:22:09.909039 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.950276 (+ 41237us) service_pool.cc:227] Handling call
0809 23:22:10.003606 (+ 53330us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.159415 (+155809us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.426762 (+1267347us) spinlock_profiling.cc:243] Waited 88.8 ms on lock 0x7b54001fc554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:11.474040 (+ 47278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":204136320}
I20260809 23:22:11.616669 25369 rpcz_store.cc:276] 0809 23:22:09.210179 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.803704 (+593525us) service_pool.cc:227] Handling call
0809 23:22:10.469478 (+665774us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.785148 (+315670us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.785180 (+    32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:10.788274 (+  3094us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:10.788332 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.794027 (+  5695us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:11.551964 (+757937us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54003dfed0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:11.552088 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46256,"raft.queue_time_us":60223,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":13909,"spinlock_wait_cycles":1396709248}
I20260809 23:22:11.691169  5200 consensus_queue.cc:260] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.526059  5376 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:11.527035  5368 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.696350  5368 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.696846  5375 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:11.697206  5029 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.697364  5375 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:11.697887  5029 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.699682  5375 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:11.551746  5173 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:11.701987  5029 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.698787  5118 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.707513  4841 consensus_queue.cc:1046] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.233s
I20260809 23:22:11.696866  5160 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:11.711465  5160 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.712337  5026 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:11.716270  5105 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.140s
I20260809 23:22:11.717343 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3942) took 1306 ms. Trace:
I20260809 23:22:11.717979 24429 rpcz_store.cc:276] 0809 23:22:10.410743 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.744041 (+333298us) service_pool.cc:227] Handling call
0809 23:22:10.760604 (+ 16563us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.962752 (+202148us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.717028 (+754276us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54003b9554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:11.717262 (+   234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443879680}
I20260809 23:22:11.716524  5118 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:11.807395 25372 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.009s	sys 0.025s
W20260809 23:22:11.810971 25372 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.009s	sys 0.025s
I20260809 23:22:11.720259 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:11.829061 24429 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
W20260809 23:22:11.844327 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 458889us
I20260809 23:22:11.528770  5024 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:11.740442 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.872889 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:11.881294 24541 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:11.681888  5013 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:11.899466  4862 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:11.899924  5415 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.795s	user 0.007s	sys 0.004s
I20260809 23:22:11.900256  4862 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.901897  5013 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:11.846298  5433 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260809 23:22:11.891280 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.915997 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:11.924619 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:11.937716 24429 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:11.919796 24269 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.520s (SENT)
W20260809 23:22:11.902655  5311 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.595s	user 0.007s	sys 0.003s
W20260809 23:22:11.907790  5402 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.118s	user 0.007s	sys 0.007s
W20260809 23:22:11.928066 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55319.8us
I20260809 23:22:11.897979  5024 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.941380 24542 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:11.965296 24542 raft_consensus.cc:1226] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:11.965936  5024 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:11.966840  5280 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:11.967085 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:11.967435 24542 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:11.967625 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3231) took 1108 ms. Trace:
I20260809 23:22:11.906059  5347 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:11.951484  5415 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:11.905642  5423 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:11.984220 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:11.971998 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:11.972007  5311 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:11.974741 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:12.019672 24429 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 24
W20260809 23:22:11.987349 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 103803us
I20260809 23:22:11.993630 24316 rpcz_store.cc:276] 0809 23:22:10.858951 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.859103 (+   152us) service_pool.cc:227] Handling call
0809 23:22:10.882014 (+ 22911us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.950098 (+ 68084us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.950136 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.007917 (+ 57781us) log.cc:842] Serialized 26 byte log entry
0809 23:22:11.148210 (+140293us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:11.175350 (+ 27140us) log.cc:842] Serialized 14 byte log entry
0809 23:22:11.231859 (+ 56509us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.231938 (+    79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:11.967407 (+735469us) raft_consensus.cc:1671] finished
0809 23:22:11.967447 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:11.967568 (+   121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56175,"prepare.queue_time_us":17333,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":71549,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":5946,"thread_start_us":81950,"threads_started":1,"wal-append.queue_time_us":770328}
I20260809 23:22:11.620157  5285 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.330s
I20260809 23:22:11.943660  4973 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:11.904807  5013 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:12.064798 24435 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.825s	user 0.012s	sys 0.002s
I20260809 23:22:12.072626 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:12.018216  5026 raft_consensus.cc:2741] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260809 23:22:12.067516 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:12.005539  5319 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.003s	sys 0.033s
W20260809 23:22:12.088493  5457 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260809 23:22:12.090410  5440 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260809 23:22:12.091249 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:12.094267  5280 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.097517  5280 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.100019 24435 raft_consensus.cc:2396] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260809 23:22:12.035388  4973 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:12.062215  4968 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
W20260809 23:22:12.189019 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.023s	sys 0.038s
W20260809 23:22:12.189538 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.023s	sys 0.038s
W20260809 23:22:12.099385  5319 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.004s	sys 0.034s
W20260809 23:22:12.194465 24535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.797s	user 0.025s	sys 0.037s
W20260809 23:22:12.196476 24535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.025s	sys 0.037s
I20260809 23:22:12.027555  5347 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:12.226622 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 264301us
I20260809 23:22:12.086479  5024 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:12.231585 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 88726.8us
W20260809 23:22:12.150204 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 321459us
I20260809 23:22:12.263520 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:12.222177 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.243221 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3873) took 1139 ms. Trace:
I20260809 23:22:12.165530  5160 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.073390  5285 consensus_queue.cc:1225] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 21.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260809 23:22:12.197716  5194 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.018s	sys 0.044s
I20260809 23:22:12.084906  5301 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:12.266719 24535 rpcz_store.cc:276] 0809 23:22:11.103399 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.276877 (+173478us) service_pool.cc:227] Handling call
0809 23:22:11.311341 (+ 34464us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:11.311654 (+   313us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:22:11.312006 (+   352us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.231684 (+919678us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.231746 (+    62us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:12.242757 (+ 11011us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.242890 (+   133us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.243104 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69707,"prepare.queue_time_us":7614,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":114889,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"thread_start_us":833184,"threads_started":1,"wal-append.queue_time_us":3825}
I20260809 23:22:12.193501  5205 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.231135  5368 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.317469 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:12.140383  5378 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:12.263090  4973 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:12.286901  5194 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.018s	sys 0.046s
I20260809 23:22:12.383767 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:12.285395  5116 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.395309 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:12.397643  4862 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.398001 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3923) took 4672 ms. Trace:
W20260809 23:22:12.398365  5460 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
W20260809 23:22:12.399938  5459 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.301s	user 0.000s	sys 0.001s
I20260809 23:22:12.398566 24425 rpcz_store.cc:276] 0809 23:22:07.654226 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:08.010671 (+356445us) service_pool.cc:227] Handling call
0809 23:22:08.528175 (+517504us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:08.535839 (+  7664us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:08.535891 (+    52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:08.536346 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:22:09.062923 (+526577us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:09.062999 (+    76us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:09.063090 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:09.737207 (+674117us) raft_consensus.cc:1671] finished
0809 23:22:09.737232 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.200571 (+2463339us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54002f85d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.326218 (+125647us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9186,"raft.queue_time_us":12938,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3554,"spinlock_wait_cycles":1188895104,"thread_start_us":518860,"threads_started":1,"wal-append.queue_time_us":674767,"wal-append.run_cpu_time_us":3773,"wal-append.run_wall_time_us":1563486}
I20260809 23:22:12.401352 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:12.402235  5451 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260809 23:22:12.403606 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3491) took 1328 ms. Trace:
I20260809 23:22:12.404167 24431 rpcz_store.cc:276] 0809 23:22:11.075333 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.103105 (+ 27772us) service_pool.cc:227] Handling call
0809 23:22:11.115300 (+ 12195us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:11.162284 (+ 46984us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.162323 (+    39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:11.183202 (+ 20879us) log.cc:842] Serialized 52 byte log entry
0809 23:22:11.504989 (+321787us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:11.505038 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.505140 (+   102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:12.403012 (+897872us) raft_consensus.cc:1671] finished
0809 23:22:12.403053 (+    41us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.403491 (+   438us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3654,"raft.queue_time_us":21589,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":22002,"thread_start_us":319305,"threads_started":1,"wal-append.queue_time_us":829599}
I20260809 23:22:12.406131 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:12.330888  5205 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.341409  5368 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.410073  5368 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:12.411464 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 126217us
I20260809 23:22:12.413071 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.414353 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.415103 24541 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 19.
I20260809 23:22:12.415953 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.389675  4974 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:12.426214 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:12.378480  5160 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.431881 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:12.273240  5024 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.407631  5205 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.434962 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
W20260809 23:22:12.430843  5163 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.086s	user 0.049s	sys 0.086s
I20260809 23:22:12.461378 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:12.392625  5194 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Stepping down as leader of term 22
W20260809 23:22:12.471833 24427 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.032s	sys 0.034s
W20260809 23:22:12.500794 24427 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.978s	user 0.032s	sys 0.034s
W20260809 23:22:12.448913  4375 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.067s	user 0.050s	sys 0.138s
W20260809 23:22:12.451637  4993 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.926s	user 0.009s	sys 0.003s
W20260809 23:22:12.532943 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.004s	sys 0.024s
I20260809 23:22:12.528452 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3490) took 1482 ms. Trace:
W20260809 23:22:12.535625 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.005s	sys 0.025s
I20260809 23:22:12.536659 24535 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:12.505707  5163 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.119s	user 0.049s	sys 0.086s
W20260809 23:22:12.526961 25320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.014s	sys 0.054s
W20260809 23:22:12.539464 24321 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.026s	sys 0.040s
W20260809 23:22:12.541059 24321 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.119s	user 0.026s	sys 0.040s
W20260809 23:22:12.525068  4375 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.050s	sys 0.138s
I20260809 23:22:12.539971 24541 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
W20260809 23:22:12.538607 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.000s	sys 0.030s
W20260809 23:22:12.539529 25320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.014s	sys 0.056s
W20260809 23:22:12.543020 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.001s	sys 0.030s
W20260809 23:22:12.539880 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.013s	user 0.006s	sys 0.010s
I20260809 23:22:12.500506  5194 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:12.544286 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.006s	sys 0.010s
I20260809 23:22:12.524174  5311 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.535610 24427 rpcz_store.cc:276] 0809 23:22:11.046099 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.066753 (+ 20654us) service_pool.cc:227] Handling call
0809 23:22:11.093455 (+ 26702us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:11.148645 (+ 55190us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.148683 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:11.189493 (+ 40810us) log.cc:842] Serialized 52 byte log entry
0809 23:22:11.484510 (+295017us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:12.501422 (+1016912us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.501886 (+   464us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.528025 (+ 26139us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.528122 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:12.528180 (+    58us) raft_consensus.cc:1671] finished
0809 23:22:12.528197 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.528383 (+   186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19228,"raft.queue_time_us":14485,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":6402,"thread_start_us":1223589,"threads_started":2,"wal-append.queue_time_us":667401}
I20260809 23:22:12.449553  5160 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.551519 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3874) took 1444 ms. Trace:
W20260809 23:22:12.547187 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 1.02043e+06us
I20260809 23:22:12.547721 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3828) took 1522 ms. Trace:
W20260809 23:22:12.567737 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.013s	sys 0.021s
W20260809 23:22:12.568261 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.014s	sys 0.022s
I20260809 23:22:12.438469  4974 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.548813 24429 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:12.579775  5465 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.580386  5465 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.582106  4984 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.582547  5465 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:12.584867 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3851) took 3311 ms (client timeout 1722 ms). Trace:
W20260809 23:22:12.585268 24543 rpcz_store.cc:269] 0809 23:22:09.273046 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.275395 (+  2349us) service_pool.cc:227] Handling call
0809 23:22:12.584331 (+3308936us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b540029de54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.584550 (+   219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1084971392}
I20260809 23:22:12.587869  5408 raft_consensus.cc:979] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:22:12.588738  5408 raft_consensus.cc:1067] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:22:12.593201  4974 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.596099  5471 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:12.596772  5471 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.599666  4678 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:12.600116  5024 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.600272  4678 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.601284  5024 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:12.601986  5024 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:12.594808  5410 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.815s	user 0.000s	sys 0.012s
I20260809 23:22:12.605751 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3225) took 2514 ms. Trace:
I20260809 23:22:12.609784 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.610683 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:12.611756 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.611433  5194 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.612972  5194 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:12.613693  5160 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.615250 24543 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:12.615435  5290 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:12.615990  5290 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:12.602914  4678 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.617219 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.618557 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3496) took 1108 ms. Trace:
I20260809 23:22:12.619005 24427 rpcz_store.cc:276] 0809 23:22:11.509864 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.566605 (+1056741us) service_pool.cc:227] Handling call
0809 23:22:12.617938 (+ 51333us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.618256 (+   318us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.618284 (+    28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.618322 (+    38us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:12.618362 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.618413 (+    51us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.618502 (+    89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.620239  5172 consensus_queue.cc:260] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.627426 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3499) took 1100 ms. Trace:
I20260809 23:22:12.630311 24427 rpcz_store.cc:276] 0809 23:22:11.520788 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.619964 (+1099176us) service_pool.cc:227] Handling call
0809 23:22:12.620316 (+   352us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.620498 (+   182us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.620520 (+    22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.620557 (+    37us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.620587 (+    30us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.620635 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.620733 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:22:12.632095 25372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.008s	sys 0.014s
W20260809 23:22:12.632563 25372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.008s	sys 0.014s
I20260809 23:22:12.552052 24539 rpcz_store.cc:276] 0809 23:22:11.107375 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.490986 (+383611us) service_pool.cc:227] Handling call
0809 23:22:11.508676 (+ 17690us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:11.695222 (+186546us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:11.699503 (+  4281us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.551183 (+851680us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.551241 (+    58us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.551284 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.551345 (+    61us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.551460 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3890,"prepare.queue_time_us":4123,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"spinlock_wait_cycles":1920,"thread_start_us":844173,"threads_started":1}
I20260809 23:22:12.575779  5183 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.626328  5172 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:12.639605 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3498) took 1119 ms. Trace:
I20260809 23:22:12.606160 24314 rpcz_store.cc:276] 0809 23:22:10.091089 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.367242 (+276153us) service_pool.cc:227] Handling call
0809 23:22:10.367850 (+   608us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.512198 (+144348us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.512228 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.525280 (+ 13052us) log.cc:842] Serialized 26 byte log entry
0809 23:22:10.573521 (+ 48241us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:10.573574 (+    53us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:10.573696 (+   122us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:12.594042 (+2020346us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:12.605424 (+ 11382us) raft_consensus.cc:1671] finished
0809 23:22:12.605450 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.605678 (+   228us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":215381,"raft.queue_time_us":154849,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":5507,"spinlock_wait_cycles":5784832,"thread_start_us":47768,"threads_started":1,"wal-append.queue_time_us":135131}
I20260809 23:22:12.606815 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3483) took 1663 ms. Trace:
I20260809 23:22:12.576333  5163 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:12.654170 24429 rpcz_store.cc:276] 0809 23:22:11.520453 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.579175 (+1058722us) service_pool.cc:227] Handling call
0809 23:22:12.638903 (+ 59728us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.639188 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.639222 (+    34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.639278 (+    56us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:12.639328 (+    50us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.639399 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.639533 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.655218 24428 rpcz_store.cc:276] 0809 23:22:10.943220 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.995761 (+ 52541us) service_pool.cc:227] Handling call
0809 23:22:10.996231 (+   470us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:11.085812 (+ 89581us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.085850 (+    38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:11.112127 (+ 26277us) log.cc:842] Serialized 52 byte log entry
0809 23:22:11.264757 (+152630us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:11.264797 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.606265 (+1341468us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b5400332cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:12.606335 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:12.606437 (+   102us) raft_consensus.cc:1671] finished
0809 23:22:12.606456 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.606739 (+   283us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":151,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":20996,"spinlock_wait_cycles":89323904,"thread_start_us":102083,"threads_started":1,"wal-append.queue_time_us":951913}
I20260809 23:22:12.577282  4375 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
I20260809 23:22:12.694413 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:12.695199  5024 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:12.699000  5079 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.069s	user 0.012s	sys 0.018s
W20260809 23:22:12.699386  5079 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.229s	user 0.012s	sys 0.018s
I20260809 23:22:12.693298 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2059) took 1714 ms. Trace:
W20260809 23:22:12.700922  5038 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.219s	user 0.009s	sys 0.006s
I20260809 23:22:12.692073  5163 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.706956  4507 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:12.692404  5160 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:12.707554  4507 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:12.708214  5160 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.708930  4507 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.711285 24424 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260809 23:22:12.711474  5205 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.711732 24424 pending_rounds.cc:76] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 16
I20260809 23:22:12.653097  4678 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.712149 24424 pending_rounds.cc:97] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260809 23:22:12.712551 24424 raft_consensus.cc:2866] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:12.712378  4678 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.714501  4678 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.715583  5166 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.716144  5166 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.718084  3889 raft_consensus.cc:3037] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:12.719987  5195 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.720624  5195 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.722602  5195 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.711742 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.723354 25371 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:12.728845 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3485) took 1775 ms. Trace:
I20260809 23:22:12.710311  5160 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.729712 24424 rpcz_store.cc:276] 0809 23:22:10.954123 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.016232 (+ 62109us) service_pool.cc:227] Handling call
0809 23:22:11.043749 (+ 27517us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.713116 (+1669367us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.713848 (+   732us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54003908d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:12.728766 (+ 14918us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3833818880}
I20260809 23:22:12.653609  5347 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.689947  3979 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:12.710491  4507 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:12.733330 24314 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 19.
I20260809 23:22:12.701691  5471 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.723462  5013 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.740011  3979 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:12.707582  5163 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:12.753116 24423 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260809 23:22:12.756774 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.753690 24423 pending_rounds.cc:76] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 15
I20260809 23:22:12.759007 24423 pending_rounds.cc:97] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260809 23:22:12.701042  5183 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.759325 24423 raft_consensus.cc:2866] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:12.770684 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.011s	sys 0.011s
I20260809 23:22:12.779690 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3890) took 3135 ms. Trace:
W20260809 23:22:12.786504 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.012s	sys 0.012s
W20260809 23:22:12.750893 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.199s	user 0.012s	sys 0.017s
I20260809 23:22:12.704815 25372 rpcz_store.cc:276] 0809 23:22:10.978877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.004612 (+ 25735us) service_pool.cc:227] Handling call
0809 23:22:11.005144 (+   532us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:11.005535 (+   391us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:11.811468 (+805933us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.688690 (+877222us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.688752 (+    62us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.692972 (+  4220us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.693064 (+    92us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.693229 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42724,"prepare.queue_time_us":160816,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":58642,"thread_start_us":1473309,"threads_started":2}
I20260809 23:22:12.750793 24429 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
W20260809 23:22:12.787050 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.256s	user 0.012s	sys 0.017s
I20260809 23:22:12.761683 24424 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:22:12.773557 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3222) took 2683 ms. Trace:
I20260809 23:22:12.782420 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:22:12.786716 24546 rpcz_store.cc:276] 0809 23:22:09.644413 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.644606 (+   193us) service_pool.cc:227] Handling call
0809 23:22:09.644934 (+   328us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:10.088889 (+443955us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.779219 (+2690330us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b54000ed8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:12.779499 (+   280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1019623680}
I20260809 23:22:12.788491 24429 pending_rounds.cc:76] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 15
I20260809 23:22:12.801391 24429 pending_rounds.cc:97] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260809 23:22:12.801750 24429 raft_consensus.cc:2866] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:12.803720 24541 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:12.753078  3979 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
W20260809 23:22:12.760046  5105 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.015s	sys 0.068s
W20260809 23:22:12.760242 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3938) took 3018 ms (client timeout 2365 ms). Trace:
W20260809 23:22:12.779796  5411 log.cc:924] Time spent T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 2.091s	user 0.023s	sys 0.005s
I20260809 23:22:12.760880  4973 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.788787 24309 rpcz_store.cc:276] 0809 23:22:10.090290 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.374992 (+284702us) service_pool.cc:227] Handling call
0809 23:22:10.858019 (+483027us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:10.862624 (+  4605us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:10.862658 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:10.863177 (+   519us) log.cc:842] Serialized 26 byte log entry
0809 23:22:11.012083 (+148906us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:11.012518 (+   435us) log.cc:842] Serialized 14 byte log entry
0809 23:22:11.043142 (+ 30624us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:11.043219 (+    77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:11.437647 (+394428us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:11.976209 (+538562us) raft_consensus.cc:1671] finished
0809 23:22:11.976352 (+   143us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.773387 (+797035us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54003a48d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.773499 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":115581,"prepare.queue_time_us":236,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":39888,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":22309,"spinlock_wait_cycles":1109781632,"thread_start_us":148414,"threads_started":1,"wal-append.queue_time_us":264685}
I20260809 23:22:12.772661  5166 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.809731 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1938) took 1359 ms. Trace:
I20260809 23:22:12.811985 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:12.812242 24546 rpcz_store.cc:276] 0809 23:22:11.450775 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.791851 (+1341076us) service_pool.cc:227] Handling call
0809 23:22:12.808985 (+ 17134us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.809470 (+   485us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.809494 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.809529 (+    35us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:12.809555 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.809599 (+    44us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.809682 (+    83us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.812955  4973 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:12.813544 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:12.810434 24429 pending_rounds.cc:97] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260809 23:22:12.815268 24429 raft_consensus.cc:2866] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:12.808979  5105 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.925s	user 0.015s	sys 0.069s
I20260809 23:22:12.816934  5105 raft_consensus.cc:3032] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:22:12.817342  5105 raft_consensus.cc:729] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:12.809232 24422 rpcz_store.cc:269] 0809 23:22:09.740604 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.897529 (+156925us) service_pool.cc:227] Handling call
0809 23:22:12.749169 (+2851640us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b5400287154. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.758794 (+  9625us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3251015552}
I20260809 23:22:12.819847  5166 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:12.820187  4568 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.000s	sys 0.025s
W20260809 23:22:12.820626  4568 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.950s	user 0.000s	sys 0.025s
I20260809 23:22:12.820547  5166 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.753839  5013 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:12.818351  5105 consensus_queue.cc:260] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.824193  5105 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:12.822721  5166 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:12.826596 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1939) took 1356 ms. Trace:
I20260809 23:22:12.828497 24546 rpcz_store.cc:276] 0809 23:22:11.470427 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.814610 (+1344183us) service_pool.cc:227] Handling call
0809 23:22:12.814906 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.826271 (+ 11365us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.826301 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.826343 (+    42us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:12.826380 (+    37us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.826440 (+    60us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.826545 (+   105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":498,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87}
I20260809 23:22:12.839105 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3883) took 1320 ms. Trace:
I20260809 23:22:12.839622 24546 rpcz_store.cc:276] 0809 23:22:11.518831 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.837836 (+1319005us) service_pool.cc:227] Handling call
0809 23:22:12.838280 (+   444us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.838499 (+   219us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.838534 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.838764 (+   230us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:22:12.838809 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.838885 (+    76us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.839037 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.842491 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:12.844228 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:12.554603 24321 rpcz_store.cc:276] 0809 23:22:11.025001 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.420258 (+395257us) service_pool.cc:227] Handling call
0809 23:22:11.420731 (+   473us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:11.421107 (+   376us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:11.421525 (+   418us) log.cc:842] Serialized 14 byte log entry
0809 23:22:12.543694 (+1122169us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.543762 (+    68us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:12.543821 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.543896 (+    75us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.547664 (+  3768us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1737,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":105,"raft.queue_time_us":921,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4399,"thread_start_us":1120825,"threads_started":1}
I20260809 23:22:12.852010 24321 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:12.853655 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2026) took 1686 ms. Trace:
I20260809 23:22:12.854244 24321 rpcz_store.cc:276] 0809 23:22:11.167394 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.851324 (+1683930us) service_pool.cc:227] Handling call
0809 23:22:12.851727 (+   403us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.853372 (+  1645us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.853568 (+   196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.856817 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2027) took 1689 ms. Trace:
I20260809 23:22:12.857304 24321 rpcz_store.cc:276] 0809 23:22:11.167736 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.855340 (+1687604us) service_pool.cc:227] Handling call
0809 23:22:12.856159 (+   819us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.856410 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.856446 (+    36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.856501 (+    55us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.856555 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.856619 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.856743 (+   124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.859548 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3886) took 1281 ms. Trace:
I20260809 23:22:12.860140 24546 rpcz_store.cc:276] 0809 23:22:11.577964 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.841089 (+1263125us) service_pool.cc:227] Handling call
0809 23:22:12.841460 (+   371us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.859223 (+ 17763us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.859253 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:12.859290 (+    37us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:12.859319 (+    29us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.859368 (+    49us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:12.859484 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.862080 24543 raft_consensus.cc:2396] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:12.808590 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3879) took 1505 ms. Trace:
I20260809 23:22:12.865408 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:12.863217 24541 rpcz_store.cc:276] 0809 23:22:11.303029 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.770261 (+1467232us) service_pool.cc:227] Handling call
0809 23:22:12.803274 (+ 33013us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.804344 (+  1070us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.808533 (+  4189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.889381 24546 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:12.867290 24321 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
W20260809 23:22:12.859166 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.012s	sys 0.019s
I20260809 23:22:12.891300 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:12.892292 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:12.892620 24535 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:12.893739 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:12.894418 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:12.894918 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.013s	sys 0.019s
W20260809 23:22:12.895905 24540 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.053s	sys 0.061s
I20260809 23:22:12.805145  5163 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:12.905913 24540 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.053s	sys 0.062s
I20260809 23:22:12.758796  5290 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.908690 25369 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:22:12.900065 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2028) took 1573 ms. Trace:
I20260809 23:22:12.867807  5301 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.911085 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.910032 24321 rpcz_store.cc:276] 0809 23:22:11.326270 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.858545 (+1532275us) service_pool.cc:227] Handling call
0809 23:22:12.858946 (+   401us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.899755 (+ 40809us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.899998 (+   243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.890182  5194 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:12.913898  5301 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.891023  5166 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:12.918098  5194 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.917932  5166 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:12.918437 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:12.921494 24317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40856 (request call id 3230) took 2086 ms (client timeout 2356 ms). Trace:
W20260809 23:22:12.921646 24323 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40860 (request call id 3824) took 1897 ms (client timeout 1913 ms). Trace:
W20260809 23:22:12.921973 24317 rpcz_store.cc:269] 0809 23:22:10.834838 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.918167 (+2083329us) service_pool.cc:227] Handling call
0809 23:22:12.921424 (+  3257us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:22:12.922093 24323 rpcz_store.cc:269] 0809 23:22:11.023748 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.252136 (+228388us) service_pool.cc:227] Handling call
0809 23:22:12.921449 (+1669313us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400252bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.921566 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294275968}
I20260809 23:22:12.924219  5166 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.924597  5200 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:12.925881  5194 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:12.926632 24535 raft_consensus.cc:1226] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260809 23:22:12.927855 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.928642 24535 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:12.929591 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.930776 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:12.945401 25371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1945) took 1899 ms (client timeout 2132 ms). Trace:
W20260809 23:22:12.946708 25371 rpcz_store.cc:269] 0809 23:22:11.046313 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.109823 (+ 63510us) service_pool.cc:227] Handling call
0809 23:22:12.945201 (+1835378us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400652cd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:12.945331 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3632783104}
I20260809 23:22:12.943274 24535 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:12.949158 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.952848  5010 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:12.927433 24311 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:12.950217 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:12.906967  5173 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.205s	user 0.012s	sys 0.002s
I20260809 23:22:12.972123 24535 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260809 23:22:12.978884 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:12.985059 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:12.913893  5122 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:12.921680  5105 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:13.017902 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.019100 24543 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:13.023901 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.027624 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3234) took 2115 ms. Trace:
I20260809 23:22:13.030092 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:13.028446 24311 rpcz_store.cc:276] 0809 23:22:10.912432 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.351367 (+438935us) service_pool.cc:227] Handling call
0809 23:22:12.927262 (+1575895us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.967498 (+ 40236us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.027283 (+ 59785us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400252bd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:13.027525 (+   242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3567802240}
I20260809 23:22:13.031476 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3908) took 1033 ms. Trace:
I20260809 23:22:13.036643 24535 rpcz_store.cc:276] 0809 23:22:11.998054 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.992714 (+994660us) service_pool.cc:227] Handling call
0809 23:22:12.993108 (+   394us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.030948 (+ 37840us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.030991 (+    43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:13.031059 (+    68us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:13.031113 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.031201 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:13.031403 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:22:13.044844 24534 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.018s	user 0.012s	sys 0.000s
I20260809 23:22:13.045531 24534 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:12.988427  5122 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:13.046172 24534 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3900) took 2639 ms (client timeout 1913 ms). Trace:
I20260809 23:22:12.977032  5173 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.015651  5278 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:13.048537 24534 rpcz_store.cc:269] 0809 23:22:10.414330 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.983755 (+1569425us) service_pool.cc:227] Handling call
0809 23:22:13.046057 (+1062302us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:12.980738  5302 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.067s
I20260809 23:22:13.052731 24541 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:22:13.051918 24546 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
W20260809 23:22:13.052058 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63434.4us
I20260809 23:22:12.983852  5163 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.061257 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3479) took 2711 ms. Trace:
I20260809 23:22:13.062539 24423 rpcz_store.cc:276] 0809 23:22:10.349530 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.699469 (+349939us) service_pool.cc:227] Handling call
0809 23:22:12.750627 (+2051158us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.786257 (+ 35630us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.059636 (+273379us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b5400287150. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:13.061189 (+  1553us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4716256000}
I20260809 23:22:13.060292 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:13.061199 25365 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:13.066390 24546 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:13.068185 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3909) took 1001 ms. Trace:
I20260809 23:22:13.035224  5105 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.068621 24543 rpcz_store.cc:276] 0809 23:22:12.076033 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.036571 (+960538us) service_pool.cc:227] Handling call
0809 23:22:13.048006 (+ 11435us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.067820 (+ 19814us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.067850 (+    30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:13.067895 (+    45us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:13.067956 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.068018 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:13.068126 (+   108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:13.075610  4984 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:13.076049  4984 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:13.076414  4984 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:13.076682  5173 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.076756  5202 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:13.077620  5013 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.077459  5202 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.083470 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2022) took 1940 ms. Trace:
I20260809 23:22:13.083979 24312 rpcz_store.cc:276] 0809 23:22:11.147628 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.480053 (+332425us) service_pool.cc:227] Handling call
0809 23:22:11.502059 (+ 22006us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.509998 (+  7939us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.510025 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.524148 (+ 14123us) log.cc:842] Serialized 26 byte log entry
0809 23:22:12.536685 (+1012537us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:12.536732 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.536799 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:13.083197 (+546398us) raft_consensus.cc:1671] finished
0809 23:22:13.083226 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:13.083399 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":827947,"threads_started":1,"wal-append.queue_time_us":1320918}
W20260809 23:22:13.086246 25321 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.2:42099 timed out after 1.552s (SENT) [suppressed 5 similar messages]
I20260809 23:22:13.087690 24535 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:38102
I20260809 23:22:13.088445 24535 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:22:13.088851 24535 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:13.091310  5183 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.094233  5105 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:13.097805  5474 log.cc:924] Time spent T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.196s	user 0.004s	sys 0.000s
I20260809 23:22:13.100519  5202 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.106894  5122 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.108053  5122 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:13.055682  5166 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:13.109330  5163 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.110689  5166 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:13.111160  5166 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:13.100994 24543 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:12.819811 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:22:13.122385  5337 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.933s	user 0.011s	sys 0.000s
I20260809 23:22:13.122160  4984 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.122545  5302 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:13.123426  5302 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.123914  5290 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.125639  5302 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.125982  5301 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.127655  4984 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.129457 24545 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:13.065397  5416 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:13.063439  5415 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260809 23:22:13.124964 24168 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 19. New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:13.114420 24543 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:13.139279 24535 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.147113 25371 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 20.
I20260809 23:22:13.152122 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:13.194082 24535 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.149575  5302 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.067313  5278 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.122869  5337 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:13.180470 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.938s	user 0.044s	sys 0.080s
W20260809 23:22:13.182735 24380 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.000s	sys 0.034s
W20260809 23:22:13.198995 24380 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.000s	sys 0.035s
W20260809 23:22:13.199313 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.19837e+06us
W20260809 23:22:13.199947 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.044s	sys 0.080s
W20260809 23:22:13.188726 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.910s	user 0.022s	sys 0.018s
I20260809 23:22:13.196627 24543 raft_consensus.cc:1226] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 19. Current term is 20. Ops: []
I20260809 23:22:13.188862  5416 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:13.199127 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47116.5us
W20260809 23:22:13.203372 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.924s	user 0.022s	sys 0.018s
I20260809 23:22:13.197034 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1930) took 2458 ms. Trace:
I20260809 23:22:13.194316  5415 consensus_queue.cc:1225] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260809 23:22:13.204459 24545 rpcz_store.cc:276] 0809 23:22:10.738165 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:10.739301 (+  1136us) service_pool.cc:227] Handling call
0809 23:22:10.739657 (+   356us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.186304 (+2446647us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.196805 (+ 10501us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:13.196979 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5495112064}
I20260809 23:22:13.208588 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:13.203953 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 934632us
W20260809 23:22:13.202606 24380 connection.cc:232] RPC call timeout handler was delayed by -1.1308651377912611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19130000000000003s for a response
I20260809 23:22:13.205567 24168 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 9 to 20, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 20 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:13.242185 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.050s	sys 0.057s
W20260809 23:22:13.251699 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.051s	sys 0.058s
W20260809 23:22:13.254446 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.926s	user 0.011s	sys 0.027s
W20260809 23:22:13.256702 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.943s	user 0.011s	sys 0.027s
I20260809 23:22:13.261878 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:13.276970 25371 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 24 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:13.197510  3979 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.203477  5485 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260809 23:22:13.204530  5378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.012s	sys 0.020s
I20260809 23:22:13.242265  4984 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:13.290266 24380 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:13.291642 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.202510  5302 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.296155 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.304466 24323 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 29 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260809 23:22:13.285316 24168 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 12 to 25. New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:13.307017 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:13.307766 24311 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:13.309967 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.307263 24542 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:13.314111 24314 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:22:13.314791 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.010s	sys 0.030s
W20260809 23:22:13.315297 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.010s	sys 0.030s
I20260809 23:22:13.315235 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:13.278595  5010 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.277825  5172 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.317557 24545 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: []
W20260809 23:22:13.289361  5378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.880s	user 0.013s	sys 0.020s
I20260809 23:22:13.314661 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 23 index: 21 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:13.284678  5485 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:13.325219 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:13.319286 24431 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.022s	sys 0.017s
I20260809 23:22:13.323525 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:13.326520 24545 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 23
W20260809 23:22:13.327052 24431 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.022s	sys 0.018s
I20260809 23:22:13.327855 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.292280  4984 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:13.250825  5482 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260809 23:22:13.335464 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:13.334256 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.609s	user 0.021s	sys 0.034s
W20260809 23:22:13.337327 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.612s	user 0.021s	sys 0.034s
I20260809 23:22:13.336726 25363 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:13.363071 24535 raft_consensus.cc:2396] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 30 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:13.364689 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:13.316967  5194 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:13.365639 25369 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:13.370591 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3875) took 2255 ms. Trace:
I20260809 23:22:13.372876 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.374106  3889 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:13.373062 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.377568 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:13.372702  4993 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:13.377704 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.378829 24423 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260809 23:22:13.372961 24544 rpcz_store.cc:276] 0809 23:22:11.115305 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.490914 (+375609us) service_pool.cc:227] Handling call
0809 23:22:11.508998 (+ 18084us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:11.724243 (+215245us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:11.724601 (+   358us) log.cc:842] Serialized 14 byte log entry
0809 23:22:13.370165 (+1645564us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:13.370224 (+    59us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:13.370270 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.370416 (+   146us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:13.370532 (+   116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30645,"prepare.queue_time_us":87057,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"thread_start_us":1614475,"threads_started":1}
I20260809 23:22:13.379341 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.381158 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.381794 24317 raft_consensus.cc:1226] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260809 23:22:13.304510  4507 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260809 23:22:13.324587  5378 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:13.388228 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:13.388439 24542 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 3907) took 1390 ms (client timeout 1514 ms). Trace:
W20260809 23:22:13.388808 24542 rpcz_store.cc:269] 0809 23:22:11.997711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.013874 (+ 16163us) service_pool.cc:227] Handling call
0809 23:22:13.388284 (+1374410us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000eddd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:13.388386 (+   102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2813036928}
I20260809 23:22:13.336681  4984 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:13.388877 25371 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:13.314194  5290 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260809 23:22:13.405512 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.392426 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.330711  5485 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:13.400689 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.401769 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:13.390661  3889 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.405512 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.404770 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.407859 24423 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:13.422609 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.427093  5299 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260809 23:22:13.436543 24544 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:13.436543 24422 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:13.443454 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.323269  5172 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.410187 24168 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 13 to 21, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 21 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:13.456054 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:13.458222  3889 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.323489  5010 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.462266 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.469609  5202 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:13.470039  5202 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:13.469938 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:13.470561  5202 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:13.470846 24423 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:13.457432  5302 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.472252 24168 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 17 to 20, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:13.475020 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.014s	sys 0.027s
W20260809 23:22:13.475373 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.014s	sys 0.027s
I20260809 23:22:13.475504 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.476748 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.405731  5416 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.477923  5122 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:22:13.478971 24167 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 243) took 21989 ms (client timeout 15000 ms). Trace:
I20260809 23:22:13.479017  5416 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:13.483400  5487 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260809 23:22:13.478767  5202 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.455950  5013 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.485415  5202 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.485644  5013 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.492000  5013 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.492882  5013 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:13.493335  5013 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:13.493873  5013 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:13.477804 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.496532 24380 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:13.503136 24435 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260809 23:22:13.504690 24535 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:13.437126  5368 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:13.384158  5194 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:13.479569 24167 rpcz_store.cc:269] 0809 23:21:51.212998 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:51.221447 (+  8449us) service_pool.cc:227] Handling call
0809 23:22:13.199578 (+21978131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:04.337946 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:04.338731 (+   785us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:04.338812 (+    81us) write_op.cc:435] Acquired schema lock
0809 23:22:04.338831 (+    19us) tablet.cc:660] Decoding operations
0809 23:22:04.340899 (+  2068us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:04.341027 (+   128us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:04.341082 (+    55us) tablet.cc:683] Acquiring locks for 18 operations
0809 23:22:04.342877 (+  1795us) tablet.cc:699] Row locks acquired
0809 23:22:04.342912 (+    35us) write_op.cc:260] PREPARE: finished
0809 23:22:04.447300 (+104388us) write_op.cc:270] Start()
0809 23:22:04.482222 (+ 34922us) write_op.cc:276] Timestamp: P: 1786317724388077 usec, L: 0
0809 23:22:04.492001 (+  9779us) op_driver.cc:331] REPLICATION: starting
0809 23:22:04.588351 (+ 96350us) log.cc:842] Serialized 6990 byte log entry
0809 23:22:05.747480 (+1159129us) op_driver.cc:444] REPLICATION: finished
0809 23:22:06.294308 (+546828us) write_op.cc:301] APPLY: starting
0809 23:22:06.294491 (+   183us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:06.528650 (+234159us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:06.528779 (+   129us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:07.454513 (+925734us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:07.492489 (+ 37976us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=0,mrs_lookups=0
0809 23:22:07.492557 (+    68us) write_op.cc:312] APPLY: finished
0809 23:22:07.493127 (+   570us) log.cc:842] Serialized 164 byte log entry
0809 23:22:07.792943 (+299816us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:07.813505 (+ 20562us) write_op.cc:454] Released schema lock
0809 23:22:07.839380 (+ 25875us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:07.891892 (+ 52512us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:07.892972 (+  1080us) write_op.cc:454] Released schema lock
0809 23:22:08.679820 (+786848us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
Metrics: {"mutex_wait_us":12910,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":453907,"apply.run_cpu_time_us":46735,"apply.run_wall_time_us":1688636,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79295,"num_ops":18,"prepare.queue_time_us":558610,"prepare.run_cpu_time_us":24804,"prepare.run_wall_time_us":449338,"raft.queue_time_us":398169,"raft.run_cpu_time_us":30583,"raft.run_wall_time_us":528910,"replication_time_us":1194422,"spinlock_wait_cycles":82625792,"thread_start_us":1397241,"threads_started":5,"wal-append.queue_time_us":870533,"wal-append.run_cpu_time_us":47503,"wal-append.run_wall_time_us":3434008}]]}
I20260809 23:22:13.513893 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:13.515262 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.461305  5200 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:13.519469 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.521637 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:13.467149  5278 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.499396  5337 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.528913 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:13.533196 24544 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:13.538559 24168 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 10 to 20, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 20 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:13.542968 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.545542 24422 raft_consensus.cc:2396] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260809 23:22:13.545500 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.545868 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:13.541522  5010 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.548754 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.551394 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:13.553531 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.554292 24535 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:13.534858  4993 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.559365 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.549722 24422 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:13.517650  5368 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.564818 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.571571 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:13.573311 24422 raft_consensus.cc:2396] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260809 23:22:13.572685 24168 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 11 to 17, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 17 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
W20260809 23:22:13.574203 24546 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.507s	user 0.013s	sys 0.000s
I20260809 23:22:13.580420 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.550860  5280 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:13.584671 24546 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260809 23:22:13.588234 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3913) took 1173 ms. Trace:
I20260809 23:22:13.585620 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.594322 24422 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260809 23:22:13.588862 24546 rpcz_store.cc:276] 0809 23:22:12.415170 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.056241 (+641071us) service_pool.cc:227] Handling call
0809 23:22:13.066242 (+ 10001us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.587919 (+521677us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.588167 (+   248us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:22:13.604724 24425 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.054s	user 0.032s	sys 0.080s
I20260809 23:22:13.597891 24168 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 14 to 21, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 21 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:13.547535  5200 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.589932  5376 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:13.605147 24425 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.057s	user 0.032s	sys 0.080s
I20260809 23:22:13.588315  5337 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:13.587936  5280 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:13.626631 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.575490  4993 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.610496 24168 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 18 to 24, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:13.634920  5172 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.622339 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.539523  5378 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:13.639513 24316 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.351s	user 0.022s	sys 0.063s
I20260809 23:22:13.639446 24435 raft_consensus.cc:2396] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260809 23:22:13.645957 24422 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:22:13.647962 24316 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.023s	sys 0.063s
I20260809 23:22:13.653788 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:13.660645  5491 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260809 23:22:13.641243  5280 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:13.666477 24168 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:13.672066 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:13.673399 24422 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:13.694248 24430 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.002s	user 0.016s	sys 0.028s
W20260809 23:22:13.694641 24430 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.006s	user 0.016s	sys 0.028s
W20260809 23:22:13.700169 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.012s	sys 0.033s
W20260809 23:22:13.700865 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.000s	user 0.012s	sys 0.033s
I20260809 23:22:13.702098 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.641739  5376 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.710367 24423 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:13.719426 24422 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:13.731328 24423 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 21. Preceding OpId from leader: term: 25 index: 22. (index mismatch)
I20260809 23:22:13.643249  5337 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:13.727038 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2023) took 2560 ms. Trace:
I20260809 23:22:13.746608 24313 rpcz_store.cc:276] 0809 23:22:11.166242 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.521719 (+355477us) service_pool.cc:227] Handling call
0809 23:22:11.621057 (+ 99338us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.621343 (+   286us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.621376 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.689841 (+ 68465us) log.cc:842] Serialized 26 byte log entry
0809 23:22:12.895429 (+1205588us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.895475 (+    46us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.896474 (+   999us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:13.726722 (+830248us) raft_consensus.cc:1671] finished
0809 23:22:13.726758 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:13.726966 (+   208us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":905646,"threads_started":1,"wal-append.queue_time_us":1645792}
I20260809 23:22:13.729846 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.752285 24422 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:13.758631 24422 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:13.760264 24422 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 22. Current term is 23. Ops: []
W20260809 23:22:13.719913 24539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.011s	sys 0.035s
W20260809 23:22:13.692654  5166 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.581s	user 0.008s	sys 0.004s
I20260809 23:22:13.745519 24311 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
W20260809 23:22:13.727101  5079 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.026s	user 0.018s	sys 0.030s
I20260809 23:22:13.765194 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:13.700657  5194 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.772683 24423 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 18
I20260809 23:22:13.773077 24423 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260809 23:22:13.773389 24423 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:13.751480  5278 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:13.778990 24539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s	user 0.012s	sys 0.035s
I20260809 23:22:13.803023 24545 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:13.804574 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:13.805581 24309 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.015s	sys 0.011s
I20260809 23:22:13.763516  5503 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:13.768630  5079 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.069s	user 0.018s	sys 0.030s
I20260809 23:22:13.766759  5166 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:13.809587 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.057s	user 0.036s	sys 0.051s
W20260809 23:22:13.809284 24309 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.015s	sys 0.011s
W20260809 23:22:13.923754 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.341s	user 0.036s	sys 0.066s
W20260809 23:22:13.826133 24320 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.999s	user 0.013s	sys 0.023s
W20260809 23:22:13.924930 24320 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.017s	sys 0.029s
I20260809 23:22:13.838254 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:13.826829 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.046s	user 0.009s	sys 0.037s
W20260809 23:22:13.927217 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s	user 0.014s	sys 0.042s
I20260809 23:22:13.861402 24168 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 17 to 20, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 20 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:13.813793  5311 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.198s	user 0.011s	sys 0.011s
I20260809 23:22:13.807487  4993 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:13.838601  5482 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:13.939874  5311 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.324s	user 0.014s	sys 0.021s
W20260809 23:22:13.951133 24541 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.002s	sys 0.030s
W20260809 23:22:13.970263 24541 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.002s	sys 0.030s
I20260809 23:22:13.806656  5368 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.976329 24544 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:13.807606  5278 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:13.978844 25325 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:13.887715  4984 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:13.955570 25321 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.868s	user 0.012s	sys 0.008s
W20260809 23:22:13.986239 25321 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.013s	sys 0.009s
W20260809 23:22:13.986624 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 900350us
I20260809 23:22:13.873634  5163 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:13.838851  5194 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:13.802181  3979 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:13.855062  5200 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.969312 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:13.955248 24266 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.005s	sys 0.013s
W20260809 23:22:14.000033 24266 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.005s	sys 0.013s
W20260809 23:22:14.000478 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 760763us
W20260809 23:22:13.975409  5502 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
W20260809 23:22:13.979200  4568 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.016s	sys 0.018s
W20260809 23:22:13.975889 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.006s	sys 0.028s
W20260809 23:22:14.009498 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.006s	sys 0.028s
W20260809 23:22:14.010658  5490 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260809 23:22:13.988848  5163 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:14.012061 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1937) took 2847 ms. Trace:
I20260809 23:22:13.982790  5337 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:14.009277  4568 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.016s	sys 0.018s
I20260809 23:22:13.975057  5482 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.003690  5512 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:14.018883  5484 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:14.019392  5512 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:14.019748  5512 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.020316  5482 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:13.995731  5488 raft_consensus.cc:2781] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:13.934908  5503 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.028435 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:14.012435 24540 rpcz_store.cc:276] 0809 23:22:11.175782 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.972416 (+796634us) service_pool.cc:227] Handling call
0809 23:22:11.972794 (+   378us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.973244 (+   450us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.973282 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.973926 (+   644us) log.cc:842] Serialized 26 byte log entry
0809 23:22:12.908070 (+934144us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:12.915245 (+  7175us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:12.915350 (+   105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.011844 (+1096494us) raft_consensus.cc:1671] finished
0809 23:22:14.011877 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.012002 (+   125us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":352,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":933752,"threads_started":1,"wal-append.queue_time_us":1597531}
I20260809 23:22:14.012908  3979 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:14.006970  5194 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:14.041889  5488 raft_consensus.cc:686] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:14.042161  5496 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260809 23:22:14.047636  5416 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.568s	user 0.008s	sys 0.005s
I20260809 23:22:14.042438  3979 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:14.076251 25375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.014s	sys 0.022s
I20260809 23:22:14.012775  5163 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:14.066915  5520 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260809 23:22:14.078044 25375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.014s	sys 0.022s
I20260809 23:22:14.060416  5376 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:14.083588  5278 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:14.071796  5166 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:14.090886 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.000s	sys 0.039s
W20260809 23:22:14.091436 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.000s	sys 0.039s
I20260809 23:22:14.096978 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3503) took 2393 ms. Trace:
I20260809 23:22:14.102206 24428 rpcz_store.cc:276] 0809 23:22:11.703386 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.689596 (+986210us) service_pool.cc:227] Handling call
0809 23:22:12.703491 (+ 13895us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:12.729644 (+ 26153us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.729691 (+    47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:12.740580 (+ 10889us) log.cc:842] Serialized 26 byte log entry
0809 23:22:13.927754 (+1187174us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:13.927797 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.927877 (+    80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.096734 (+168857us) raft_consensus.cc:1671] finished
0809 23:22:14.096758 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.096906 (+   148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3108,"raft.queue_time_us":22989,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":7940,"thread_start_us":1158665,"threads_started":1,"wal-append.queue_time_us":1341760}
I20260809 23:22:14.112344 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:14.112962 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3493) took 2990 ms. Trace:
I20260809 23:22:14.113379 24431 rpcz_store.cc:276] 0809 23:22:11.119173 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.417921 (+1298748us) service_pool.cc:227] Handling call
0809 23:22:12.418226 (+   305us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.418498 (+   272us) raft_consensus.cc:1520] Early marking committed up to index 12
0809 23:22:13.327521 (+909023us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.092238 (+764717us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.108723 (+ 16485us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.108873 (+   150us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.108922 (+    49us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:22:14.108963 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.109034 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.109146 (+   112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":526516,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":324,"raft.run_wall_time_us":7044,"spinlock_wait_cycles":16086016,"thread_start_us":1614904,"threads_started":2}
I20260809 23:22:14.111457 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3501) took 2559 ms. Trace:
I20260809 23:22:14.115391 24427 rpcz_store.cc:276] 0809 23:22:11.551881 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.631491 (+1079610us) service_pool.cc:227] Handling call
0809 23:22:12.651243 (+ 19752us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:12.652683 (+  1440us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.652720 (+    37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:12.653246 (+   526us) log.cc:842] Serialized 26 byte log entry
0809 23:22:13.725986 (+1072740us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:13.726099 (+   113us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.726309 (+   210us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.111251 (+384942us) raft_consensus.cc:1671] finished
0809 23:22:14.111282 (+    31us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.111399 (+   117us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":43219,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":112,"thread_start_us":1000804,"threads_started":1,"wal-append.queue_time_us":1273411}
I20260809 23:22:14.068488  5194 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:14.140339 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.013s	sys 0.022s
W20260809 23:22:14.140775 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.013s	sys 0.022s
W20260809 23:22:14.102227 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.020s	sys 0.021s
W20260809 23:22:14.141557 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.020s	sys 0.021s
I20260809 23:22:14.109320  5416 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:14.113919  5483 log.cc:924] Time spent T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
W20260809 23:22:14.123365 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.015s	sys 0.019s
W20260809 23:22:14.139797 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.013s	sys 0.036s
I20260809 23:22:14.109401  5368 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:14.151826 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.015s	sys 0.019s
W20260809 23:22:14.152220 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.013s	sys 0.036s
W20260809 23:22:14.152287 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 737950us
W20260809 23:22:14.152931 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 775358us
I20260809 23:22:14.153255 24269 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:14.142091 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 628249us
I20260809 23:22:14.087109  5488 consensus_queue.cc:237] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:14.150504  4507 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.019s	sys 0.015s
W20260809 23:22:14.158281 24543 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.015s	sys 0.037s
W20260809 23:22:14.164340 24543 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.015s	sys 0.037s
I20260809 23:22:14.140712  4984 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:14.154919 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:14.163318 24496 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.019s	sys 0.017s
W20260809 23:22:14.175606 24496 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.020s	sys 0.018s
I20260809 23:22:14.179147  5496 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:14.179651  5496 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:14.179837 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:14.181788  5280 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:14.184304  5013 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.690s	user 0.008s	sys 0.006s
W20260809 23:22:14.176441 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 802018us
I20260809 23:22:14.184804  5013 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.186251 24429 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:14.190061  5512 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.191110  5503 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.195147  5512 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:14.196187  5278 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:14.191121  5013 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.197779  5376 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:14.198619 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:14.204517 24498 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:14.158567  5368 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.200039  5280 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:14.156075  4251 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.016s	sys 0.019s
W20260809 23:22:14.165287  4507 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.020s	sys 0.015s
I20260809 23:22:14.155735  5416 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.219847  5163 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:14.173741  3979 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:14.209350  4251 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.016s	sys 0.020s
W20260809 23:22:14.225039 24542 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.020s	sys 0.032s
W20260809 23:22:14.232837 24542 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.020s	sys 0.033s
I20260809 23:22:14.234344 24427 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260809 23:22:14.240887 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:14.224366  5518 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260809 23:22:14.244678 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:14.225925  5521 log.cc:924] Time spent T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260809 23:22:14.246614 24427 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:14.223443  5376 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:14.225867  5506 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:14.247925 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:14.224900  5280 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:14.248172 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:14.258955  3889 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:14.264937 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:14.231352  5511 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:14.236696  3979 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:14.271057  5500 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:14.271365  5301 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:14.210067  5079 consensus_queue.cc:1046] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260809 23:22:14.275226  5494 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:14.276324  5416 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.276714 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3492) took 3172 ms. Trace:
I20260809 23:22:14.277650 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3878) took 2975 ms. Trace:
I20260809 23:22:14.278604  5376 raft_consensus.cc:686] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:14.279847  5368 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:14.264493  5280 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:14.265095 24431 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
W20260809 23:22:14.265908 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 61466us
W20260809 23:22:14.309446 24535 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.013s	sys 0.034s
I20260809 23:22:14.326910 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:14.334218 24316 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.011s	sys 0.038s
W20260809 23:22:14.390635 24535 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.014s	sys 0.034s
W20260809 23:22:14.390947 24316 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.011s	sys 0.038s
I20260809 23:22:14.282435  5496 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.394003  5496 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:14.250819  5506 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:14.394598  5506 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:14.338111 24539 rpcz_store.cc:276] 0809 23:22:11.302717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.640639 (+1337922us) service_pool.cc:227] Handling call
0809 23:22:12.641112 (+   473us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:12.714292 (+ 73180us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:12.714326 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:12.714829 (+   503us) log.cc:842] Serialized 26 byte log entry
0809 23:22:13.789734 (+1074905us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:13.789777 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.789849 (+    72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.277349 (+487500us) raft_consensus.cc:1671] finished
0809 23:22:14.277385 (+    36us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.277569 (+   184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49696,"raft.queue_time_us":75307,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":3352,"thread_start_us":1074484,"threads_started":1,"wal-append.queue_time_us":1273481}
I20260809 23:22:14.391676 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2024) took 3225 ms. Trace:
I20260809 23:22:14.399744 24316 rpcz_store.cc:276] 0809 23:22:11.166694 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.031815 (+865121us) service_pool.cc:227] Handling call
0809 23:22:12.101867 (+ 70052us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.197582 (+ 95715us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:13.648548 (+1450966us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.391319 (+742771us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.391378 (+    59us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:14.391423 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.391511 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.391625 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1606019,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":3749,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":2056263,"threads_started":2}
I20260809 23:22:14.407281  5506 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.415028  5013 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:14.415992  5013 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.418545  5013 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.301180  5511 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.421777  5511 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.302522  5301 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:14.423550  5301 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.425294  5301 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:14.415246  5506 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.336364  5500 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.429948  5500 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.434682 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3891) took 2040 ms. Trace:
I20260809 23:22:14.435189 24534 rpcz_store.cc:276] 0809 23:22:12.394719 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.060879 (+666160us) service_pool.cc:227] Handling call
0809 23:22:13.062234 (+  1355us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.074313 (+ 12079us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.074361 (+    48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:13.150100 (+ 75739us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.009973 (+859873us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:14.010027 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.010170 (+   143us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.434417 (+424247us) raft_consensus.cc:1671] finished
0809 23:22:14.434450 (+    33us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.434613 (+   163us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9542,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9426,"thread_start_us":844650,"threads_started":1,"wal-append.queue_time_us":1272666}
I20260809 23:22:14.442675  5280 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.443456  5512 raft_consensus.cc:3032] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:14.443953  5512 raft_consensus.cc:729] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:14.444986  5512 consensus_queue.cc:260] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.447773  5376 consensus_queue.cc:237] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:14.333039  5485 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.871s	user 0.014s	sys 0.000s
I20260809 23:22:14.451896  5485 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.454493  5485 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:14.456516 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66133.1us
W20260809 23:22:14.459688 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 132920us
I20260809 23:22:14.460322  5506 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:14.460729  5506 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:14.460834 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.461129  5506 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.336752 24430 rpcz_store.cc:276] 0809 23:22:11.103931 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.384883 (+280952us) service_pool.cc:227] Handling call
0809 23:22:11.416487 (+ 31604us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.509732 (+ 93245us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.509762 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.715110 (+205348us) log.cc:842] Serialized 26 byte log entry
0809 23:22:12.568739 (+853629us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:13.694956 (+1126217us) log.cc:842] Serialized 14 byte log entry
0809 23:22:13.695147 (+   191us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:13.695215 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.276256 (+581041us) raft_consensus.cc:1671] finished
0809 23:22:14.276296 (+    40us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.276603 (+   307us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2433,"prepare.queue_time_us":375657,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":263253,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":24526,"thread_start_us":1859933,"threads_started":2,"wal-append.queue_time_us":2028039}
W20260809 23:22:14.475054 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 446673us
I20260809 23:22:14.481102  5506 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.492916 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:14.493798  3889 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:14.494146  3889 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:14.494465  3889 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:14.523468 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:14.524230 24431 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.531275  3889 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.532191 24431 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:14.533519 24496 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.534513  4507 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:14.535075  4507 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:14.535982  4507 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.539989  3889 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:14.541117 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:14.544662  5200 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:14.546106  5506 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:14.554435 24546 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.034s	sys 0.028s
W20260809 23:22:14.612388 24546 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.034s	sys 0.028s
I20260809 23:22:14.589417  4993 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:14.659025  4993 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.661859  4993 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:14.665494 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:14.580530  5484 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.005s	sys 0.055s
W20260809 23:22:14.681217 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.039s	sys 0.025s
W20260809 23:22:14.670749  5484 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.005s	sys 0.056s
W20260809 23:22:14.685770 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.023s	sys 0.050s
W20260809 23:22:14.687206 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.024s	sys 0.050s
W20260809 23:22:14.684939 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.040s	sys 0.025s
W20260809 23:22:14.686156 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.025s	sys 0.029s
W20260809 23:22:14.685447 24435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.033s	sys 0.035s
W20260809 23:22:14.689293 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.025s	sys 0.029s
W20260809 23:22:14.689510 24435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.033s	sys 0.035s
W20260809 23:22:14.687901 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.025s	sys 0.061s
I20260809 23:22:14.690066 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3826) took 3665 ms. Trace:
I20260809 23:22:14.615628  5200 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.690255 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2025) took 3523 ms. Trace:
W20260809 23:22:14.692189 24541 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.013s	sys 0.054s
W20260809 23:22:14.694802 24541 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.013s	sys 0.054s
W20260809 23:22:14.691473 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.964s	user 0.025s	sys 0.062s
I20260809 23:22:14.692734 24309 rpcz_store.cc:276] 0809 23:22:11.167060 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.813169 (+1646109us) service_pool.cc:227] Handling call
0809 23:22:12.813453 (+   284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:12.813700 (+   247us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:22:13.924178 (+1110478us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.689708 (+765530us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.689814 (+   106us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:14.689897 (+    83us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.690008 (+   111us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.690162 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13289,"prepare.queue_time_us":1126246,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":1603,"raft.queue_time_us":98,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":116,"thread_start_us":1830803,"threads_started":2}
I20260809 23:22:14.695849 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1940) took 1904 ms. Trace:
I20260809 23:22:14.690440 24320 rpcz_store.cc:276] 0809 23:22:11.024393 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.366658 (+342265us) service_pool.cc:227] Handling call
0809 23:22:11.417029 (+ 50371us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:11.448488 (+ 31459us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:11.448528 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:11.494509 (+ 45981us) log.cc:842] Serialized 26 byte log entry
0809 23:22:12.787630 (+1293121us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:13.925267 (+1137637us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.689699 (+764432us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.689799 (+   100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.689845 (+    46us) raft_consensus.cc:1671] finished
0809 23:22:14.689864 (+    19us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.690015 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1146,"prepare.queue_time_us":1097087,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":85989,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":3111131,"threads_started":3,"wal-append.queue_time_us":1869812}
I20260809 23:22:14.702912 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1763) took 1284 ms. Trace:
I20260809 23:22:14.709918 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1959) took 1412 ms. Trace:
I20260809 23:22:14.702265 24541 rpcz_store.cc:276] 0809 23:22:12.791244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.112067 (+320823us) service_pool.cc:227] Handling call
0809 23:22:13.112373 (+   306us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.113634 (+  1261us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:13.970787 (+857153us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.695474 (+724687us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.695540 (+    66us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:14.695598 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.695683 (+    85us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.695794 (+   111us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":999640,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":6670,"thread_start_us":1578685,"threads_started":2}
I20260809 23:22:14.728525 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:14.729173 24429 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:14.734400 24544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.003s	sys 0.010s
W20260809 23:22:14.734953 24544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.003s	sys 0.010s
I20260809 23:22:14.707209 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3898) took 1636 ms. Trace:
W20260809 23:22:14.735992 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.009s	sys 0.005s
W20260809 23:22:14.736445 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.009s	sys 0.005s
I20260809 23:22:14.741451 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.726454 24535 rpcz_store.cc:276] 0809 23:22:13.297571 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.569615 (+272044us) service_pool.cc:227] Handling call
0809 23:22:13.570217 (+   602us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.570500 (+   283us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.570535 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:13.581061 (+ 10526us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.391095 (+810034us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:14.464173 (+ 73078us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.464380 (+   207us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.464455 (+    75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.709650 (+245195us) raft_consensus.cc:1671] finished
0809 23:22:14.709680 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.709849 (+   169us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":73353,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":627,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":781889,"threads_started":1,"wal-append.queue_time_us":1120558}
I20260809 23:22:14.722198 24435 rpcz_store.cc:276] 0809 23:22:13.418296 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.798979 (+380683us) service_pool.cc:227] Handling call
0809 23:22:13.810090 (+ 11111us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.850653 (+ 40563us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:13.866893 (+ 16240us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.702540 (+835647us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.702590 (+    50us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:14.702639 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.702717 (+    78us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.702851 (+   134us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":16461,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":12509,"thread_start_us":812545,"threads_started":1}
I20260809 23:22:14.745512 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:14.746990 24431 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:14.752377 24168 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.000s	sys 0.013s
I20260809 23:22:14.695725  5506 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:14.761502 24168 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.000s	sys 0.014s
W20260809 23:22:14.750715 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67664.4us
I20260809 23:22:14.702142  5172 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:14.737129  5378 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.011s	sys 0.002s
I20260809 23:22:14.735612 24543 rpcz_store.cc:276] 0809 23:22:13.070952 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.203724 (+132772us) service_pool.cc:227] Handling call
0809 23:22:13.204121 (+   397us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.214406 (+ 10285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.214450 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:13.292503 (+ 78053us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.165020 (+872517us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:14.165484 (+   464us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.165594 (+   110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.706991 (+541397us) raft_consensus.cc:1671] finished
0809 23:22:14.707017 (+    26us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.707150 (+   133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35161,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6455,"thread_start_us":798498,"threads_started":1,"wal-append.queue_time_us":1355895}
I20260809 23:22:14.765969  3889 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:14.755007  5311 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.006s	sys 0.007s
I20260809 23:22:14.708688  5200 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:14.759457 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:14.756115  4568 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.004s	sys 0.009s
I20260809 23:22:14.772132  3889 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:14.766824 24498 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:14.764083  4993 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:14.787918  3889 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:14.787451 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 25) took 3479 ms. Trace:
W20260809 23:22:14.772677  5378 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.012s	sys 0.002s
W20260809 23:22:14.777935  4568 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.005s	sys 0.010s
I20260809 23:22:14.792397 24424 rpcz_store.cc:276] 0809 23:22:11.305923 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.814842 (+1508919us) service_pool.cc:227] Handling call
0809 23:22:14.784232 (+1969390us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:14.784722 (+   490us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4324330368}
I20260809 23:22:14.795037  3889 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader election won for term 20
I20260809 23:22:14.797044  3889 raft_consensus.cc:686] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:14.775338  5311 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.007s	sys 0.007s
I20260809 23:22:14.769191  5506 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.784135  5200 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.777499  5172 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.806430  5547 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260809 23:22:14.799084  5540 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.115s	user 0.001s	sys 0.001s
I20260809 23:22:14.790439  4993 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:14.760254  5541 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260809 23:22:14.818359 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1760) took 1401 ms. Trace:
I20260809 23:22:14.822319 24423 rpcz_store.cc:276] 0809 23:22:13.417297 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.760954 (+343657us) service_pool.cc:227] Handling call
0809 23:22:13.772406 (+ 11452us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.832347 (+ 59941us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.832404 (+    57us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:13.832935 (+   531us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.694772 (+861837us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:14.695173 (+   401us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.695353 (+   180us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.818109 (+122756us) raft_consensus.cc:1671] finished
0809 23:22:14.818137 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.818290 (+   153us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":855926,"threads_started":1,"wal-append.queue_time_us":982931}
I20260809 23:22:14.807713  5416 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
W20260809 23:22:14.794322 24542 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.006s	sys 0.009s
W20260809 23:22:14.821959 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.009s	sys 0.057s
W20260809 23:22:14.828768 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.010s	sys 0.057s
W20260809 23:22:14.827317 24542 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.006s	sys 0.010s
I20260809 23:22:14.821641  4993 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:14.833703 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.001s	sys 0.011s
I20260809 23:22:14.811443  3889 consensus_queue.cc:237] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:14.847318 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.002s	sys 0.011s
W20260809 23:22:14.847784 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 689402us
W20260809 23:22:14.832497 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.007s	sys 0.006s
W20260809 23:22:14.848806 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.007s	sys 0.006s
W20260809 23:22:14.849609 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 669791us
I20260809 23:22:14.850584 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:14.832892  5194 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.001s	sys 0.013s
W20260809 23:22:14.843087 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.019s	sys 0.029s
I20260809 23:22:14.852442 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1960) took 1554 ms. Trace:
W20260809 23:22:14.855027 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.020s	sys 0.038s
W20260809 23:22:14.834545 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.000s	sys 0.013s
W20260809 23:22:14.857810 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.019s	sys 0.030s
W20260809 23:22:14.842859  5166 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.023s	sys 0.025s
W20260809 23:22:14.862980 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.021s	sys 0.038s
W20260809 23:22:14.852041 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.005s	sys 0.008s
W20260809 23:22:14.863780 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.005s	sys 0.009s
W20260809 23:22:14.860765 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.001s	sys 0.013s
I20260809 23:22:14.836257  5562 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:14.852469  5488 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.002s	sys 0.012s
W20260809 23:22:14.864266 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 825341us
W20260809 23:22:14.864532 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 815381us
I20260809 23:22:14.867944 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:14.857688  5194 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.001s	sys 0.013s
I20260809 23:22:14.873268 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:14.862600  5166 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.023s	sys 0.025s
I20260809 23:22:14.876619 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:14.859440  5337 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:22:14.862812  5122 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.015s	sys 0.015s
I20260809 23:22:14.877286 24543 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:14.899909 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:14.872117  5488 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.002s	sys 0.013s
I20260809 23:22:14.906018 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:14.911394 24535 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:14.873778  5494 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:14.908232 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:14.837863  5280 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:14.911830 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:14.901393  5122 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.016s	sys 0.015s
W20260809 23:22:14.912643 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43759.4us
I20260809 23:22:14.914080 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:14.954167 24543 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:14.916810 24541 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:14.874014  5550 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
W20260809 23:22:14.917693  5565 log.cc:924] Time spent T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260809 23:22:14.917937 25363 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260809 23:22:14.918342 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 3952) took 3588 ms. Trace:
I20260809 23:22:14.966032 24425 rpcz_store.cc:276] 0809 23:22:11.329934 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.468547 (+1138613us) service_pool.cc:227] Handling call
0809 23:22:12.537505 (+ 68958us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:13.637912 (+1100407us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.637944 (+    32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:13.656845 (+ 18901us) log.cc:842] Serialized 52 byte log entry
0809 23:22:14.696094 (+1039249us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:14.696151 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.696252 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.918125 (+221873us) raft_consensus.cc:1671] finished
0809 23:22:14.918146 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.918283 (+   137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":55620,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":52,"thread_start_us":2058956,"threads_started":2,"wal-append.queue_time_us":1166148}
I20260809 23:22:14.920066  5013 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:14.920563  5302 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
W20260809 23:22:14.921295 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53447.7us
I20260809 23:22:14.970537 24380 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.972609 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:14.923408 24534 raft_consensus.cc:1226] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:14.923950  5547 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:14.925475 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1963) took 1399 ms. Trace:
I20260809 23:22:14.928977  5566 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:14.931118  5555 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260809 23:22:14.912424 25369 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:14.961158 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1905) took 2063 ms. Trace:
I20260809 23:22:14.964089 24539 raft_consensus.cc:1261] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260809 23:22:14.913300  5558 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260809 23:22:14.967572 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:14.978808 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.992218 25375 rpcz_store.cc:276] 0809 23:22:12.897864 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:12.900190 (+  2326us) service_pool.cc:227] Handling call
0809 23:22:12.902053 (+  1863us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.107700 (+205647us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:14.080136 (+972436us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.859970 (+779834us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.860129 (+   159us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.860172 (+    43us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:14.860229 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.960919 (+100690us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:14.960967 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.961097 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4151,"prepare.queue_time_us":1324158,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":9100,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":10028,"spinlock_wait_cycles":482536192,"thread_start_us":1722994,"threads_started":2,"wal-append.queue_time_us":890085}
I20260809 23:22:14.978368 24544 rpcz_store.cc:276] 0809 23:22:13.526265 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.984768 (+458503us) service_pool.cc:227] Handling call
0809 23:22:13.985160 (+   392us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.985568 (+   408us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.985609 (+    41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:14.003397 (+ 17788us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.735516 (+732119us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:14.735575 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.735679 (+   104us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:14.925237 (+189558us) raft_consensus.cc:1671] finished
0809 23:22:14.925264 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.925414 (+   150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3165,"raft.queue_time_us":512,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":10615,"thread_start_us":718307,"threads_started":1,"wal-append.queue_time_us":826779}
I20260809 23:22:15.011420 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:14.991016  5482 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:15.040787  5280 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.040594  5482 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.045014 25371 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260809 23:22:15.009518  5566 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.049569 25371 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:15.049614  5566 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:15.050472  5566 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:15.051100 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.052278  5280 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.052791 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.053483 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.055086 25371 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:15.055307 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.052767  5280 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.056340  5280 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:15.057636  5278 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:15.058174  5278 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:15.050885  5566 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:15.059443  3979 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:15.059216  5278 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.060322  3979 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:15.061537  3979 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.062952 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1962) took 1562 ms. Trace:
I20260809 23:22:15.064352  5581 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:15.063455 24546 rpcz_store.cc:276] 0809 23:22:13.500678 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.647016 (+146338us) service_pool.cc:227] Handling call
0809 23:22:13.662867 (+ 15851us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:13.663589 (+   722us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:13.663633 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:13.690740 (+ 27107us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.612863 (+922123us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:14.612921 (+    58us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.613018 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:15.062659 (+449641us) raft_consensus.cc:1671] finished
0809 23:22:15.062689 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:15.062862 (+   173us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":264,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":23463,"thread_start_us":846216,"threads_started":1,"wal-append.queue_time_us":1050974}
I20260809 23:22:15.064766  5581 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.065713  5581 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.068008  5506 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:14.978123  5547 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:15.030650  5593 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.054s	user 0.001s	sys 0.001s
I20260809 23:22:15.014657  5378 raft_consensus.cc:2741] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260809 23:22:15.087330 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:14.897063  5562 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:15.111718 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 181214us
I20260809 23:22:15.084017 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:15.138171 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54254.5us
I20260809 23:22:15.138835 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.139899 24544 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:15.086333 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.134303 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.140228 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:15.143213 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 383887us
I20260809 23:22:15.143350  5581 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:15.145939 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:15.146567  5376 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.147167  5512 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:15.147138  5376 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.149322  5376 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.149983 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.150202 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1764) took 1731 ms. Trace:
I20260809 23:22:15.090595  5566 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.151301 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.152431 25372 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:14.857851 24542 rpcz_store.cc:276] 0809 23:22:13.297838 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.574992 (+277154us) service_pool.cc:227] Handling call
0809 23:22:13.591603 (+ 16611us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:13.592020 (+   417us) raft_consensus.cc:1520] Early marking committed up to index 22
0809 23:22:14.233690 (+641670us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.844774 (+611084us) log.cc:842] Serialized 14 byte log entry
0809 23:22:14.852064 (+  7290us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:14.852118 (+    54us) raft_consensus.cc:1631] Marking committed up to 22
0809 23:22:14.852179 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:14.852250 (+    71us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:14.852380 (+   130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11630,"prepare.queue_time_us":522426,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":14801,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":10145,"spinlock_wait_cycles":11469184,"thread_start_us":1230806,"threads_started":2}
I20260809 23:22:15.141178 24545 raft_consensus.cc:2362] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24. Re-sending same reply.
I20260809 23:22:15.153419 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.154615 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.161155 25365 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 29: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:15.161247 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.162707 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.163484 24535 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:15.164363 24534 raft_consensus.cc:2362] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24. Re-sending same reply.
I20260809 23:22:15.162616 25371 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:15.175875 25369 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:15.184065 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:15.176450 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.185384 24535 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:15.128707  5547 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:15.187420 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.195899 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.196724 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.198836 24429 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:15.199699 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.197307 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.207573 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.208387 24542 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:15.209229 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 345422us
I20260809 23:22:15.210879 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.216394 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:15.178215  5577 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260809 23:22:15.214501 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:15.217216 24430 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:15.231106 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.232832 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.233122 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.233664 24425 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:15.238265 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.241919 24427 raft_consensus.cc:2379] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:15.200327 24429 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:15.223438 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.150640 24422 rpcz_store.cc:276] 0809 23:22:13.418625 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:13.801734 (+383109us) service_pool.cc:227] Handling call
0809 23:22:13.802159 (+   425us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:14.043519 (+241360us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:14.043561 (+    42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:14.044087 (+   526us) log.cc:842] Serialized 26 byte log entry
0809 23:22:14.736903 (+692816us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:15.016800 (+279897us) log.cc:842] Serialized 14 byte log entry
0809 23:22:15.017275 (+   475us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:15.149767 (+132492us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54002f8ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:15.149824 (+    57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:15.149896 (+    72us) raft_consensus.cc:1671] finished
0809 23:22:15.149999 (+   103us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:15.150146 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6159,"raft.queue_time_us":12697,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":12182,"spinlock_wait_cycles":544944512,"thread_start_us":947917,"threads_started":2,"wal-append.queue_time_us":771383}
I20260809 23:22:15.225752 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:15.142156  5482 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:15.201355 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:15.228423 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.248167 24430 raft_consensus.cc:2396] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260809 23:22:15.244032 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:15.127291  5550 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.080s
I20260809 23:22:15.192540  5194 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.201761  5378 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:15.261543 24431 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260809 23:22:15.274833 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 73770.5us
I20260809 23:22:15.188285  5582 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:15.288153 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.289343  3889 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:15.249725 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.244882 24540 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:15.265604 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.266669 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.267055  5537 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.268443  5337 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.271378 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.312167 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.183987  4568 consensus_queue.cc:1046] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.101s
I20260809 23:22:15.262993 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.284499 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.285934  5194 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:15.305248 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:15.326408 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 358957us
I20260809 23:22:15.202270  5166 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.084s
I20260809 23:22:15.303036 25366 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.320314 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:15.306775 24424 raft_consensus.cc:3037] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.335667 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:15.312988  5592 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.303s	user 0.003s	sys 0.000s
W20260809 23:22:15.330919 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 274517us
W20260809 23:22:15.334785 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 85178.1us
I20260809 23:22:15.256172 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.307386  5537 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:15.340090 24427 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:15.317891  5337 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.311843  5013 raft_consensus.cc:2741] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260809 23:22:15.358539 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.323809  5194 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:15.351212 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.354804 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.348743 24266 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:15.370204 24435 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:15.371640  5595 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260809 23:22:15.371747  5611 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.372282  5611 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:15.372638  5611 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:15.375021  5596 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.103s	user 0.001s	sys 0.001s
I20260809 23:22:15.342950  4993 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.376508 24546 raft_consensus.cc:2354] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:15.354737  5537 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:15.391814 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 214551us
I20260809 23:22:15.342527  5311 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:15.397457 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:15.361295  4568 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:15.317545  5378 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.407433  5378 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.407812  5302 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:22:15.408506 24428 raft_consensus.cc:2396] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260809 23:22:15.408824  5484 consensus_queue.cc:1035] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.007s
I20260809 23:22:15.416501 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:15.405089  5562 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.405189  5311 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:15.419265 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80557.8us
I20260809 23:22:15.420665 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.421371  5311 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.398504  5545 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.423586  5378 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:15.423861  5545 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:15.424153  5378 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.424328  5545 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:15.425223  5378 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.425936 24541 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:15.427116  5550 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:15.427645  5550 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.398320  5013 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:15.428478  5550 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.430090 24540 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:15.423022 24380 leader_election.cc:398] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:15.444717 24380 leader_election.cc:401] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260809 23:22:15.443074 24543 raft_consensus.cc:2454] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 29.
I20260809 23:22:15.451423 24545 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: []
W20260809 23:22:15.430866  5200 consensus_meta.cc:287] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.619s	user 0.003s	sys 0.005s
I20260809 23:22:15.462235 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:22:15.463892 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.461565 24424 raft_consensus.cc:2454] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
W20260809 23:22:15.475442 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 296923us
I20260809 23:22:15.441888  5494 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:15.400897  5172 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:15.520717 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58426.5us
I20260809 23:22:15.443698  5511 raft_consensus.cc:2741] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260809 23:22:15.403439  4568 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:15.529902  5200 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:15.534873 24429 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
W20260809 23:22:15.533953 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 185263us
I20260809 23:22:15.539100 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:15.543133  5378 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:15.545236 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122377us
W20260809 23:22:15.560509 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 76036us
I20260809 23:22:15.561124  5579 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:15.531026  5582 raft_consensus.cc:3032] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:15.536172  5494 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:15.569020 24423 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260809 23:22:15.605386  3889 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.648124  3889 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:15.605473 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 221514us
I20260809 23:22:15.451030  5566 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:15.609582 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 231936us
W20260809 23:22:15.562028 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 98231.9us
I20260809 23:22:15.657935 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.561002 24544 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260809 23:22:15.625523 24424 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e"
dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
mode: GRACEFUL
new_leader_uuid: "60d5c69728134c8bb918a900eb6a8c21"
 from {username='slave'} at 127.0.0.1:40954
I20260809 23:22:15.627507 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.628588 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.630370 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.632122 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.660808 24311 raft_consensus.cc:2454] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:15.634012 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.661892 24313 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:15.604216  5548 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.662227 24313 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:15.635730 24314 raft_consensus.cc:1261] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260809 23:22:15.641943 24543 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:22:15.649804 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.664438 24269 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.659411 24424 raft_consensus.cc:595] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Received request to transfer leadership to TS 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:15.639933 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:15.670454  5280 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.003s
I20260809 23:22:15.651046  5582 raft_consensus.cc:729] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:15.676591 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:15.676913 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.678076 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.656446  5494 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:15.689076 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.666051 24313 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.682375 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.690321  3889 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.692526 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.693672 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:15.677734  5541 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:15.665431  5622 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.031s
I20260809 23:22:15.696755  5079 consensus_queue.cc:1046] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:15.697705  5079 raft_consensus.cc:3032] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:15.698035  5079 raft_consensus.cc:729] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.693369 24429 raft_consensus.cc:2362] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24. Re-sending same reply.
I20260809 23:22:15.666069  5378 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:15.701917 24423 raft_consensus.cc:2396] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 28 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
W20260809 23:22:15.705614 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53005.1us
I20260809 23:22:15.678587  4993 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:15.723453 24545 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:15.655995  5337 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.724972 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.726241 25321 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:15.661737  5537 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.727495 24313 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:15.722180 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 183247us
I20260809 23:22:15.729620 24309 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:15.701241  5302 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:15.708971  5541 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260809 23:22:15.729187 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:15.700484  5629 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260809 23:22:15.698850  5079 consensus_queue.cc:260] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.693334  5643 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:15.709076  5416 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:15.726419  5547 consensus_meta.cc:287] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.000s
I20260809 23:22:15.681299  5512 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260809 23:22:15.704507  5604 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:15.673187  5280 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.743199 24314 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
W20260809 23:22:15.755775 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97966us
I20260809 23:22:15.765657 24545 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:15.766129 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:15.759317 25325 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:15.769354 24424 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260809 23:22:15.770373 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:15.771155  5637 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260809 23:22:15.771872 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.772188 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.772434 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.773576 24312 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:15.773613 24314 raft_consensus.cc:1261] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:15.745141  5648 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:15.779224 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.779973 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.781386 24321 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.743098  5566 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:15.784251 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.747939  5585 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:15.746976  5541 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:15.788244  5512 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Stepping down as leader of term 25
W20260809 23:22:15.789083  5633 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260809 23:22:15.790313  5619 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260809 23:22:15.809574 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:15.809778 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 84712.6us
I20260809 23:22:15.791118 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.791684 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.795938 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.797713 24425 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:15.680612  4568 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.729599  5378 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.049s
I20260809 23:22:15.814957  5378 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:15.744659  5646 consensus_queue.cc:1046] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.055s
I20260809 23:22:15.815140 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.816006 24312 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:15.816551 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.816818 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.817461 24312 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 21.
I20260809 23:22:15.817955 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.818282 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.756547  5582 consensus_queue.cc:260] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.820374  5582 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:15.821666 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.822410 24314 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:15.823132  5301 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:15.823386 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.824167 24314 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:15.774984  5631 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:15.811605 24309 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:15.842231 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.783064  5648 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:15.812777  5639 log.cc:924] Time spent T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260809 23:22:15.813102  5337 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:15.813956  4568 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:15.815348  5378 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.800056  5541 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:15.825112  5537 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:15.800566  5512 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.832971 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.783300  5079 raft_consensus.cc:3037] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:15.672978  5548 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.893570 25363 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:22:15.894263 25363 raft_consensus.cc:595] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.803597 24542 raft_consensus.cc:1226] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:15.768564  5643 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:15.837788 24313 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:15.803109 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.838809 24545 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260809 23:22:15.847421  5631 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.905118 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.848122 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.849066  5511 raft_consensus.cc:3032] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Stepping down as leader of term 19
I20260809 23:22:15.851608 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:15.861344 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50834.8us
I20260809 23:22:15.913409 24314 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
I20260809 23:22:15.810386  5163 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:22:15.872769 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 146627us
W20260809 23:22:15.877175 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 84731.2us
W20260809 23:22:15.878664 25366 consensus_meta.cc:287] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.541s	user 0.011s	sys 0.000s
I20260809 23:22:15.914460 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:15.826674 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:15.774289  5547 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.885648 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.914812 25366 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:15.916508 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:15.865649  5488 raft_consensus.cc:2781] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:15.908141 24425 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.851769  5648 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:15.910409 24309 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:15.919265 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:15.804157  5585 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.924914 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:15.926468 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.929636 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:15.930428 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:15.935015 24320 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:15.936517 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:15.935884 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 176628us
I20260809 23:22:15.894840  5629 consensus_queue.cc:1046] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260809 23:22:15.913044  5511 raft_consensus.cc:729] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.924997  5172 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:15.926222  5581 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:15.956489  3889 consensus_queue.cc:1046] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:15.936014  5013 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:15.965761 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:15.970546 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:15.974478  5653 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260809 23:22:15.974687  5301 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:15.979717  3889 raft_consensus.cc:3032] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:15.980209  3889 raft_consensus.cc:729] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:15.961587  5646 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:15.992245  5646 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:15.992101  5643 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.992911  5646 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
W20260809 23:22:15.975426  5611 consensus_meta.cc:287] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.602s	user 0.011s	sys 0.001s
I20260809 23:22:15.986749  3889 consensus_queue.cc:260] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.990275  5585 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:15.923584  5378 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.968519  5581 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:15.988724 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:15.998816 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.028561 24323 raft_consensus.cc:2379] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 20: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:15.967458  5172 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:15.957324  5337 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260809 23:22:15.999837  5280 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:16.024672  5163 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:16.026342 24314 raft_consensus.cc:1226] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:16.027354 24546 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260809 23:22:15.972496  5571 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:16.030473 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.003247  5611 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:15.946099  5666 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.058s
I20260809 23:22:16.032094  3889 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:15.932147  5512 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.050340 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.122700 24313 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:16.002851  5574 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:16.155298  5545 consensus_meta.cc:287] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.730s	user 0.009s	sys 0.005s
I20260809 23:22:16.156294  5545 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:15.981179  5494 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:16.095306  5636 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260809 23:22:16.058538  5378 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:16.096849  5625 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.256s	user 0.001s	sys 0.001s
W20260809 23:22:16.114796 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84375.6us
W20260809 23:22:15.986660  5649 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
W20260809 23:22:16.103036  5654 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260809 23:22:16.184531 24321 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
W20260809 23:22:16.118331 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 187990us
W20260809 23:22:16.121325 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60277us
W20260809 23:22:15.863812  5624 consensus_queue.cc:1159] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "60d5c69728134c8bb918a900eb6a8c21" responder_term: 22 status { last_received { term: 21 index: 14 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 21. Current term is 22. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:16.104063  5488 raft_consensus.cc:686] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:16.120932  5301 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:16.131326  5658 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:22:16.132472  5660 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260809 23:22:16.133440  5547 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:16.135607  5194 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.762s	user 0.007s	sys 0.006s
I20260809 23:22:16.136438  5548 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:16.137424  5163 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:16.139912  5659 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260809 23:22:16.119810  5571 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.159762  5631 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:16.122448  5666 consensus_queue.cc:1225] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer 808fed3ed3f04882ad757a3db9abfdd9 log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:16.164963  5550 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:16.048817  5376 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
W20260809 23:22:16.117022 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.704s (SENT)
I20260809 23:22:16.189697 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.190798 24317 raft_consensus.cc:1226] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
W20260809 23:22:16.190323 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 71345.5us
I20260809 23:22:16.215502 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:16.193917 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 264345us
I20260809 23:22:16.216590 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.214465 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:16.217744 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 100778us
I20260809 23:22:16.218096 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.197090 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:16.218710 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.727s (SENT)
I20260809 23:22:16.219259 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.219504 24543 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:16.201746  5658 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:16.228236  5658 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:16.171628  5581 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.222909 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.207882  5301 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:16.211290  5194 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.224927  5670 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:16.226754  5676 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:16.221046  5677 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:16.222986 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:22:16.220366  5663 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.226s	user 0.000s	sys 0.002s
I20260809 23:22:16.232462 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.232764 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.236362  5512 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:16.236639 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.237466 25371 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260809 23:22:16.237929  5622 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:16.223403  5574 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:16.267908  5574 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:16.268287  5574 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:16.268128 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.271215 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.272054 24313 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:16.275787 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.276403 24309 raft_consensus.cc:2362] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27. Re-sending same reply.
I20260809 23:22:16.277041  5482 raft_consensus.cc:2741] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260809 23:22:16.278632  3889 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260809 23:22:16.279822 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 56927.8us
I20260809 23:22:16.280818 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.277320  5574 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.282389 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:16.284760  5676 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:16.284943  5629 raft_consensus.cc:2741] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260809 23:22:16.285300  5676 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:16.222986  5376 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:16.240173 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.300441  5376 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.300897 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.241258 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 22 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.243230 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.249404  5506 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:16.249526  5194 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.220085  5643 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:16.337014 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.258397 24321 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:16.340046 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.266253 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:16.239264 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:16.287971 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.705s (SENT)
I20260809 23:22:16.291061 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.291312 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.343680 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.289448  5488 consensus_queue.cc:237] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.292207 24316 raft_consensus.cc:1226] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:16.298861  5648 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.296698  5571 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:16.297618  5545 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:16.309073 24425 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:16.322139 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.336045 24309 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260809 23:22:16.373176 24309 pending_rounds.cc:76] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 14
I20260809 23:22:16.373502 24309 pending_rounds.cc:97] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260809 23:22:16.373793 24309 raft_consensus.cc:2866] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:16.342525 25369 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:16.295555  5611 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:16.354696 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.357117 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.364148 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:16.365510 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:16.385340 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 97388.2us
I20260809 23:22:16.368496 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.320590  5013 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:16.368788 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.369907 25370 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:16.370841 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.284899  5574 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.285845  5676 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:16.375370 24429 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: [25.22-25.22]
I20260809 23:22:16.293675  5494 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:16.376744 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.396912 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.399310 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:16.387848 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65850us
I20260809 23:22:16.388108  3889 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:16.391335 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.389590 24534 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:16.394656 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.395155 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:16.404088 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.404651 24321 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:16.405462 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.405973 25323 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.394783 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.411369 24323 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:16.411954 24316 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:16.416771 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.417572 24314 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:16.415649 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:16.420138 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:16.421327 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.423032 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:16.424932 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.425751 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.427500 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.396389  5166 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:16.336419  5506 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.390250  5013 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:16.439081 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:16.436903 24321 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:16.441399 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.443001 24321 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:16.444352 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.445895 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.431329 24543 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:16.447943 24380 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:16.448660  5681 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:16.449205  5681 raft_consensus.cc:686] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:16.449623 24543 pending_rounds.cc:76] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 17
I20260809 23:22:16.450024 24543 pending_rounds.cc:97] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 27.18
I20260809 23:22:16.450408 24543 raft_consensus.cc:2866] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:16.450111  5681 consensus_queue.cc:237] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.452509 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.453079 24534 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:16.453229 24316 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:16.453990 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.457178 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.458361 24309 raft_consensus.cc:2454] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:16.463917 25365 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:16.464840 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.465519 25365 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:16.466240 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.467312 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.468091 25365 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:16.411386  5079 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:16.420825  5548 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:16.473786 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.476164  3889 consensus_queue.cc:1035] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:16.512176 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.516866 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.454491 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:16.520321 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.386626  5547 consensus_queue.cc:1046] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.043s
I20260809 23:22:16.518124 24321 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:16.432155  5311 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:16.438728  5574 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:16.434873  5337 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:16.527139 24321 raft_consensus.cc:1261] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260809 23:22:16.453961  5579 consensus_meta.cc:287] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.790s	user 0.007s	sys 0.006s
I20260809 23:22:16.527635 24316 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:16.524314 24323 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 24: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:16.524482 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.534251 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.536646 24423 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:16.537092 24498 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.537846 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:16.538156 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.453585  5013 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:16.543812 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.412448  5643 consensus_queue.cc:1035] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.107s
I20260809 23:22:16.468583  5652 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:16.468137  5582 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:16.552258 24309 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260809 23:22:16.521476 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:16.593410 24323 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
W20260809 23:22:16.589596 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 68206.5us
I20260809 23:22:16.596199 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.562605 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:16.568059 24321 raft_consensus.cc:1226] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:16.573474 25369 raft_consensus.cc:2396] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260809 23:22:16.573926 24313 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:16.575249 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.579425 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.600840 25365 raft_consensus.cc:2379] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:16.527673  5545 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:16.586107 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.534879  5311 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:16.420305  5301 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.639123 25372 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:16.640022 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.516863  5200 consensus_queue.cc:1046] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.060s
I20260809 23:22:16.604483 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:16.651165 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46808.6us
I20260809 23:22:16.653338 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.648938 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.640619 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.511637  5656 consensus_queue.cc:1046] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.128s
I20260809 23:22:16.662477 24309 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:16.663313 25365 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:16.541146  5550 consensus_queue.cc:1046] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260809 23:22:16.666405 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.672829 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.674501 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.674798 25369 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:16.679900 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:16.641552  5629 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:16.647398  5311 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:16.693606 25365 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:16.691148  3889 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:16.582228  5579 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:16.646356  5631 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:16.695505 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:16.646462  5652 raft_consensus.cc:686] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:16.702086 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.660815  5581 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:16.711328 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.665119  5635 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:16.583670  5337 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.701462  5484 raft_consensus.cc:3032] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:16.718005  5484 raft_consensus.cc:729] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:16.719352  5693 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260809 23:22:16.719249  5484 consensus_queue.cc:260] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.720940  5484 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:16.722211  5689 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260809 23:22:16.724725 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.725476 24431 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:16.717746  5661 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 25.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.209s
I20260809 23:22:16.727792  5661 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:16.728159  5661 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:16.729023  5661 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 25.22, Last appended by leader: 22, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.730378  5661 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:16.732569 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.735280 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.736099  5482 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:16.736284 24427 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:16.739207 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.740064  5494 raft_consensus.cc:3032] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:16.740638  5494 raft_consensus.cc:729] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:16.740978  5482 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:16.746604  5651 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.814s	user 0.003s	sys 0.000s
W20260809 23:22:16.748456 25366 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1921) took 1603 ms (client timeout 1542 ms). Trace:
W20260809 23:22:16.751065  5668 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.654s	user 0.000s	sys 0.002s
I20260809 23:22:16.706815 25369 raft_consensus.cc:1226] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:16.758677 25373 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260809 23:22:16.693385  5629 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:16.764465 25366 rpcz_store.cc:269] 0809 23:22:15.144835 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.176653 (+ 31818us) service_pool.cc:227] Handling call
0809 23:22:16.748224 (+1571571us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x7b5400708fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:16.748373 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":167864064}
I20260809 23:22:16.772686 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:16.709613  5585 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:16.780337 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2030) took 1016 ms. Trace:
I20260809 23:22:16.781458 24320 rpcz_store.cc:276] 0809 23:22:15.763805 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.763979 (+   174us) service_pool.cc:227] Handling call
0809 23:22:15.783107 (+ 19128us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:15.949126 (+166019us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:16.780036 (+830910us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54003854d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:16.780261 (+   225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":342574976}
I20260809 23:22:16.752454  5648 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.751946  5631 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:16.815553  5646 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.822s	user 0.008s	sys 0.004s
I20260809 23:22:16.816045  5646 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.818064 25365 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:16.819420  5337 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.741813  5494 consensus_queue.cc:260] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.821003  5494 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:16.821144  5337 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:16.822799 24540 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3937) took 1182 ms. Trace:
I20260809 23:22:16.823223 24540 rpcz_store.cc:276] 0809 23:22:15.640709 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.641303 (+   594us) service_pool.cc:227] Handling call
0809 23:22:15.665051 (+ 23748us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:15.665412 (+   361us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:15.665439 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:15.670357 (+  4918us) log.cc:842] Serialized 26 byte log entry
0809 23:22:15.732725 (+ 62368us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:15.732763 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:15.732827 (+    64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:16.745903 (+1013076us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:16.822377 (+ 76474us) raft_consensus.cc:1671] finished
0809 23:22:16.822401 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:16.822720 (+   319us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3198,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":3473,"spinlock_wait_cycles":20923136,"thread_start_us":61981,"threads_started":1,"wal-append.queue_time_us":221397}
I20260809 23:22:16.824236  5511 consensus_queue.cc:260] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:16.825636  5511 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 19 FOLLOWER]: Advancing to term 20
I20260809 23:22:16.805555  5585 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:16.822153  5541 consensus_meta.cc:287] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.928s	user 0.009s	sys 0.000s
I20260809 23:22:16.828682 24544 raft_consensus.cc:1226] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:16.828473  5541 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.829761 24312 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 20.
I20260809 23:22:16.828732 24320 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f"
dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:44596
I20260809 23:22:16.829284 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2029) took 1080 ms. Trace:
I20260809 23:22:16.781548  5548 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:16.832860 24311 rpcz_store.cc:276] 0809 23:22:15.754063 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.773853 (+ 19790us) service_pool.cc:227] Handling call
0809 23:22:15.788948 (+ 15095us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:15.864014 (+ 75066us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:15.864060 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:15.864555 (+   495us) log.cc:842] Serialized 26 byte log entry
0809 23:22:15.997072 (+132517us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:15.997126 (+    54us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:15.997213 (+    87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:16.750373 (+753160us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:16.828882 (+ 78509us) raft_consensus.cc:1671] finished
0809 23:22:16.828905 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:16.829156 (+   251us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74630,"raft.queue_time_us":87847,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":11710,"spinlock_wait_cycles":5500288,"thread_start_us":98318,"threads_started":1,"wal-append.queue_time_us":150182}
I20260809 23:22:16.775899  5629 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:16.818794 24314 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:16.767776 24431 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:16.838142 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:16.839201 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 302196us
I20260809 23:22:16.843544 24320 raft_consensus.cc:595] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.844034 24320 raft_consensus.cc:603] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260809 23:22:16.837551  5378 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.658s	user 0.008s	sys 0.004s
I20260809 23:22:16.845255 24498 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:16.915275  5548 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 25.22, Last appended by leader: 22, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:16.870654  5506 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.938170  5711 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:16.850986  5643 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:16.951663 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:16.961581 25368 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:16.947278  5711 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:16.930601  5506 consensus_queue.cc:1046] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
I20260809 23:22:16.953907  5646 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:16.997105 24423 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:16.957150  5643 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:17.047086 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2049) took 1126 ms. Trace:
I20260809 23:22:17.047544 24312 rpcz_store.cc:276] 0809 23:22:15.920490 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.920758 (+   268us) service_pool.cc:227] Handling call
0809 23:22:17.046813 (+1126055us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b54003a3ed4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.047007 (+   194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2052351360}
I20260809 23:22:17.049256 24316 raft_consensus.cc:1261] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 17. (index mismatch)
I20260809 23:22:17.061884 24539 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260809 23:22:17.064136 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:17.007093  5676 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.590s	user 0.008s	sys 0.004s
W20260809 23:22:17.073694 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 477579us
W20260809 23:22:17.103812 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 438337us
I20260809 23:22:17.104686 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:17.105240 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:17.106112  5547 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:17.107372  5547 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:16.907970  5652 consensus_queue.cc:237] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.006637  5673 raft_consensus.cc:3032] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:17.113605 24546 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:17.121884 24543 raft_consensus.cc:1226] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 19. Current term is 20. Ops: []
I20260809 23:22:17.120877 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:17.132804 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:16.998107  5711 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:17.134248 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:17.044341  5579 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:16.992244  5631 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.132179  5166 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.071290  5635 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.133421  5506 raft_consensus.cc:3032] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:17.161214 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:17.162163 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:17.163659 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 99612.5us
I20260809 23:22:17.139521  5673 raft_consensus.cc:729] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:17.069865  5200 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.097s
I20260809 23:22:17.139955  5541 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.169334 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:17.170295 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:17.056058  5172 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.106s
I20260809 23:22:17.053637  5646 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.108s
W20260809 23:22:17.171906  5702 log.cc:924] Time spent T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260809 23:22:17.172019  5661 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:17.153079  5545 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:17.160616  5579 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.168279  5166 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.178668 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:17.110641  5676 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.109754  5311 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.166500  5506 raft_consensus.cc:729] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:17.207695 24430 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:17.183400  5571 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:17.203192  3979 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:17.177754  5585 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:17.164712  5643 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.212164  5676 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.209568  5571 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:17.213873  5716 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:17.214998  5673 consensus_queue.cc:260] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.197664  5574 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.238461  5574 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:17.239562  5687 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.919s	user 0.002s	sys 0.000s
I20260809 23:22:17.241075  5574 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:17.187724  5666 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:17.222296 25372 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260809 23:22:17.203393  5545 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:17.227572 25371 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:17.231436  5311 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.232805  5166 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:17.235251  5698 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.408s	user 0.003s	sys 0.000s
I20260809 23:22:17.230862  5571 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.168617  5635 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:17.237720  5013 consensus_meta.cc:287] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.000s
I20260809 23:22:17.226464  5579 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.236874  5707 raft_consensus.cc:3032] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
W20260809 23:22:17.244807  5550 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.018s	sys 0.043s
W20260809 23:22:17.257094 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 378081us
W20260809 23:22:17.263006 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.013s	sys 0.025s
W20260809 23:22:17.263486 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.014s	sys 0.025s
W20260809 23:22:17.263849 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 836952us
I20260809 23:22:17.264688 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:17.245465  5378 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:17.256588  5550 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.018s	sys 0.044s
I20260809 23:22:17.256455  5681 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:17.298149 25375 raft_consensus.cc:1226] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:17.314162 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:17.255398  5537 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.008s	sys 0.016s
I20260809 23:22:17.258356  5013 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.335067  5378 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:17.338963 24427 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 3988) took 1513 ms (client timeout 1584 ms). Trace:
I20260809 23:22:17.339282 24431 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 22: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:17.339399 24427 rpcz_store.cc:269] 0809 23:22:15.825454 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:15.826281 (+   827us) service_pool.cc:227] Handling call
0809 23:22:17.338738 (+1512457us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54003bae54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.338864 (+   126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1823928064}
I20260809 23:22:17.342682 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1799) took 1011 ms. Trace:
I20260809 23:22:17.343087 24429 rpcz_store.cc:276] 0809 23:22:16.331559 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.331706 (+   147us) service_pool.cc:227] Handling call
0809 23:22:16.375067 (+ 43361us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:16.406487 (+ 31420us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.342493 (+936006us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b54003bb350. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.342624 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98874752}
W20260809 23:22:17.346771  5511 consensus_meta.cc:287] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.002s
I20260809 23:22:17.339329  5378 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.347412 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.348182 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3922) took 1345 ms. Trace:
W20260809 23:22:17.348412  5658 consensus_meta.cc:287] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.093s	user 0.012s	sys 0.000s
I20260809 23:22:17.349036  5378 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:17.348941  5658 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.251797  5506 consensus_queue.cc:260] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.351109  5506 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:17.250229  5676 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260809 23:22:17.356293  5635 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:17.358502  5629 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.283327  5166 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:17.359655  5013 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.333761  5571 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:17.360874  5629 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.348537  5079 raft_consensus.cc:3032] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:17.363554  5079 raft_consensus.cc:729] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:17.348529 24541 rpcz_store.cc:276] 0809 23:22:16.002443 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.002603 (+   160us) service_pool.cc:227] Handling call
0809 23:22:17.347254 (+1344651us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:22:17.348122 (+   868us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5137664}
I20260809 23:22:17.258028  5707 raft_consensus.cc:729] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:17.367743 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 53704.5us
I20260809 23:22:17.368780 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:17.333833  5537 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.008s	sys 0.017s
I20260809 23:22:17.380182 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1797) took 1059 ms. Trace:
I20260809 23:22:17.380683 24428 rpcz_store.cc:276] 0809 23:22:16.321109 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.337950 (+ 16841us) service_pool.cc:227] Handling call
0809 23:22:16.338334 (+   384us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:16.338585 (+   251us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:16.758594 (+420009us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.242580 (+483986us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:17.242652 (+    72us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:17.242722 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.242816 (+    94us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.380115 (+137299us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37787,"prepare.queue_time_us":386755,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":92794,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":43734,"thread_start_us":854827,"threads_started":2,"wal-append.queue_time_us":597256}
I20260809 23:22:17.388484  5707 consensus_queue.cc:260] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.390168  5707 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:17.375602  5661 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:17.392386  5681 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:17.364415  5079 consensus_queue.cc:260] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.373777  5166 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:17.393584  5079 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:17.394659 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1798) took 1063 ms. Trace:
I20260809 23:22:17.395076 24424 rpcz_store.cc:276] 0809 23:22:16.331151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.351637 (+ 20486us) service_pool.cc:227] Handling call
0809 23:22:16.556001 (+204364us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:16.556223 (+   222us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:16.556258 (+    35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:16.556311 (+    53us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:16.556351 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:16.556418 (+    67us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.394482 (+838064us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54002f9ed0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.394600 (+   118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":469029632}
I20260809 23:22:17.396559 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2077) took 1070 ms. Trace:
I20260809 23:22:17.396926 25365 rpcz_store.cc:276] 0809 23:22:16.325793 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.723341 (+397548us) service_pool.cc:227] Handling call
0809 23:22:16.723671 (+   330us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:16.818412 (+ 94741us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.396333 (+577921us) spinlock_profiling.cc:243] Waited 94.2 ms on lock 0x7b540071dc54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:17.396503 (+   170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":216622976}
I20260809 23:22:17.364775  5550 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.398262  5550 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.400367  5550 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.401912 25368 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:17.390697  5716 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:17.436829 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 274787us
I20260809 23:22:17.441658 25373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1924) took 1206 ms. Trace:
I20260809 23:22:17.442183 25373 rpcz_store.cc:276] 0809 23:22:16.234827 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.235059 (+   232us) service_pool.cc:227] Handling call
0809 23:22:16.752716 (+517657us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:16.769202 (+ 16486us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.439123 (+669921us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b5400708fd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.441570 (+  2447us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1189768448}
W20260809 23:22:17.442414 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 286634us
W20260809 23:22:17.442749 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71908.1us
I20260809 23:22:17.462059 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.415198  5723 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.481945 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:17.504637 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48182.5us
I20260809 23:22:17.493229 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:17.528438 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.531625 24427 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:17.527812 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.533684 24428 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:17.535705 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 121757us
I20260809 23:22:17.538537 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:17.532869 24542 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:17.438702  5537 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:17.550556 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 371930us
I20260809 23:22:17.551612 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:17.553743 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37950 (request call id 1959) took 1206 ms. Trace:
W20260809 23:22:17.551934 25323 connection.cc:232] RPC call timeout handler was delayed by -1.0574614312499762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
I20260809 23:22:17.554235 25375 rpcz_store.cc:276] 0809 23:22:16.347848 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.785687 (+437839us) service_pool.cc:227] Handling call
0809 23:22:16.786074 (+   387us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:17.333714 (+547640us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.553441 (+219727us) spinlock_profiling.cc:243] Waited 510 ms on lock 0x7b540052dfd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:17.553670 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1172562816}
I20260809 23:22:17.539372 25373 raft_consensus.cc:2362] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26. Re-sending same reply.
I20260809 23:22:17.481431  5723 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.557070  5611 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:17.564640  5723 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.565918 25371 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:17.562695 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:17.568001  5611 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:17.575736  5611 consensus_queue.cc:1046] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:17.575709  5716 consensus_queue.cc:1046] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:17.578949  5611 raft_consensus.cc:3032] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:17.468598  5729 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.615028 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:17.529762  5550 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:17.608191 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 476574us
W20260809 23:22:17.608992 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 71917.9us
W20260809 23:22:17.606560 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64210.6us
I20260809 23:22:17.468102  5652 consensus_queue.cc:1046] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.119s
I20260809 23:22:17.613785 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:17.589310  5611 raft_consensus.cc:729] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:17.561698  5537 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.585937  5488 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.056s
I20260809 23:22:17.585467  5635 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.060s
I20260809 23:22:17.680044 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:17.679463 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:17.685851 25371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:22:17.679479 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.656805 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.682159 24543 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:17.740734 24423 raft_consensus.cc:2362] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28. Re-sending same reply.
I20260809 23:22:17.658298 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3262) took 1261 ms. Trace:
I20260809 23:22:17.590340  5711 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:17.701870  5721 log.cc:924] Time spent T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
W20260809 23:22:17.676122  5730 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260809 23:22:17.704777  5541 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.742398 24316 rpcz_store.cc:276] 0809 23:22:16.396877 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.559561 (+162684us) service_pool.cc:227] Handling call
0809 23:22:17.048824 (+489263us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:17.049717 (+   893us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.658058 (+608341us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54003a3ed0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:17.658224 (+   166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1043316992}
I20260809 23:22:17.740645 25371 raft_consensus.cc:595] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:17.680848  5714 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.751986 25371 raft_consensus.cc:603] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260809 23:22:17.687296  5673 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:17.756157 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:17.748392 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 96608.8us
W20260809 23:22:17.682776  5732 log.cc:924] Time spent T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260809 23:22:17.751194 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.055s	sys 0.121s
I20260809 23:22:17.743686 24541 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
I20260809 23:22:17.753835 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:17.685555  5658 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:17.763641 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.055s	sys 0.121s
I20260809 23:22:17.763613 24534 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:17.764253 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 657577us
I20260809 23:22:17.764828 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:17.747172  5511 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.653543  5716 consensus_queue.cc:1046] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.099s
I20260809 23:22:17.766311 24427 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 21. Preceding OpId from leader: term: 25 index: 22. (index mismatch)
I20260809 23:22:17.636981  5550 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.660310  5494 consensus_queue.cc:1046] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.102s
I20260809 23:22:17.635061  5729 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.659889  5739 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.103s
W20260809 23:22:17.763011  5280 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.000s	sys 0.024s
W20260809 23:22:17.772141 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.015s	sys 0.006s
W20260809 23:22:17.763111  5624 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.006s	sys 0.019s
I20260809 23:22:17.647567  5722 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
W20260809 23:22:17.755883  5548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.000s	sys 0.023s
I20260809 23:22:17.786885 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:17.815871 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.015s	sys 0.006s
W20260809 23:22:17.763236  5643 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.008s	sys 0.013s
I20260809 23:22:17.818157 24496 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:17.819662  5280 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.000s	sys 0.024s
W20260809 23:22:17.788348  5512 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.021s	sys 0.050s
W20260809 23:22:17.857714  5512 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.022s	sys 0.050s
I20260809 23:22:17.859959  5571 raft_consensus.cc:3032] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:17.860538  5571 raft_consensus.cc:729] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:17.839643 25371 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
mode: GRACEFUL
new_leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:48046
I20260809 23:22:17.861742 24431 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 25 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
W20260809 23:22:17.834834  5643 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.009s	sys 0.013s
I20260809 23:22:17.844393  5166 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:17.844864  5652 raft_consensus.cc:3032] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
W20260809 23:22:17.846735  5741 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260809 23:22:17.847632  5278 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:17.761044  5714 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.850550  5762 raft_consensus.cc:3032] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:17.780733  5718 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.076s
I20260809 23:22:17.851543  5488 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:17.855065 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2082) took 1471 ms. Trace:
I20260809 23:22:17.856437  5280 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:17.856542  5711 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:17.868527  5166 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:17.889873 25366 rpcz_store.cc:276] 0809 23:22:16.383749 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.806996 (+423247us) service_pool.cc:227] Handling call
0809 23:22:16.819310 (+ 12314us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:16.819699 (+   389us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:17.205023 (+385324us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.548827 (+343804us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:17.556155 (+  7328us) log.cc:842] Serialized 52 byte log entry
0809 23:22:17.556351 (+   196us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:17.556639 (+   288us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.578639 (+ 22000us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.578853 (+   214us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.578920 (+    67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:17.702756 (+123836us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b1800359a40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11
0809 23:22:17.703416 (+   660us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b1800359a40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:17.841938 (+138522us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b1800359a40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac208af4ca 00007fac29f90f99 00007fac27708bcf 00007fac209387b6 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d
0809 23:22:17.854819 (+ 12881us) raft_consensus.cc:1671] finished
0809 23:22:17.854842 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.855007 (+   165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":287671,"prepare.queue_time_us":661070,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":4668,"raft.queue_time_us":67551,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":37584,"spinlock_wait_cycles":34275456,"thread_start_us":672511,"threads_started":2,"wal-append.queue_time_us":643003}
W20260809 23:22:17.786847  5667 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.797s	user 0.014s	sys 0.014s
I20260809 23:22:17.862028 25371 raft_consensus.cc:595] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Received request to transfer leadership to TS 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:17.840154  5585 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:17.928535  5585 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:17.929013  5585 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:17.930429 24428 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260809 23:22:17.820251  5624 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.006s	sys 0.019s
I20260809 23:22:17.868460  5079 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260809 23:22:17.932299  5079 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:17.932752  5079 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:17.934038  5729 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.933777  5079 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:17.900017  5280 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.936084 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2079) took 1609 ms. Trace:
I20260809 23:22:17.937024  5280 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.937654 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260809 23:22:17.939352 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.939746 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:22:17.939951 24546 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:22:17.831398  5548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.001s	sys 0.023s
I20260809 23:22:17.941532 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:17.941977  5714 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:17.942973  5512 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:17.929983  5585 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.943398  5550 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:17.944070  5585 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:17.887302  5762 raft_consensus.cc:729] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:17.945676  5762 consensus_queue.cc:260] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.943400  5512 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:17.947989  5512 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:17.947997  5079 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:17.824725  5581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.005s	sys 0.019s
W20260809 23:22:17.844254 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 229330us
I20260809 23:22:17.952795 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.865947 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2085) took 1331 ms. Trace:
I20260809 23:22:17.889581  5488 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:17.953564 25363 rpcz_store.cc:276] 0809 23:22:16.534188 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.824487 (+290299us) service_pool.cc:227] Handling call
0809 23:22:16.824793 (+   306us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:16.825009 (+   216us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:16.825041 (+    32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:16.825496 (+   455us) log.cc:842] Serialized 26 byte log entry
0809 23:22:17.250765 (+425269us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:17.768704 (+517939us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.865570 (+ 96866us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.865667 (+    97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:17.865723 (+    56us) raft_consensus.cc:1671] finished
0809 23:22:17.865740 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.865880 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98322,"prepare.queue_time_us":104936,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":96662,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":1409,"thread_start_us":909375,"threads_started":2,"wal-append.queue_time_us":615152}
I20260809 23:22:17.869161 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2080) took 1540 ms. Trace:
I20260809 23:22:17.955570 25369 rpcz_store.cc:276] 0809 23:22:16.328670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.787542 (+458872us) service_pool.cc:227] Handling call
0809 23:22:16.787908 (+   366us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:16.807949 (+ 20041us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:17.221529 (+413580us) log.cc:842] Serialized 14 byte log entry
0809 23:22:17.868836 (+647307us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:17.868889 (+    53us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:17.868946 (+    57us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.869010 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.869111 (+   101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44036,"prepare.queue_time_us":547270,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":47588,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":14454,"thread_start_us":973803,"threads_started":2}
I20260809 23:22:17.956934  5643 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.936455 25370 rpcz_store.cc:276] 0809 23:22:16.326376 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:16.753305 (+426929us) service_pool.cc:227] Handling call
0809 23:22:16.780024 (+ 26719us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:16.803529 (+ 23505us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:16.803562 (+    33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:16.809261 (+  5699us) log.cc:842] Serialized 26 byte log entry
0809 23:22:16.938591 (+129330us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b18005326c0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:17.371161 (+432570us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:17.371210 (+    49us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:17.371301 (+    91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:17.935766 (+564465us) raft_consensus.cc:1671] finished
0809 23:22:17.935788 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:17.936017 (+   229us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9346,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":1458,"spinlock_wait_cycles":50703616,"thread_start_us":432347,"threads_started":1,"wal-append.queue_time_us":831478}
W20260809 23:22:17.865057 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.014s	sys 0.012s
I20260809 23:22:17.762599  5541 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.779938  5511 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.970721 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:17.971743 25371 raft_consensus.cc:2379] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 20: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:17.860143  5506 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.000s
I20260809 23:22:17.768199  5611 consensus_queue.cc:260] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:17.872593  5652 raft_consensus.cc:729] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:17.874428  5278 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:17.899207  5631 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.028s	sys 0.045s
W20260809 23:22:17.920799  5667 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.931s	user 0.015s	sys 0.015s
W20260809 23:22:17.990056  5631 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.028s	sys 0.045s
I20260809 23:22:17.919406  5166 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:17.966471 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:17.992602 24427 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:17.967099 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.014s	sys 0.012s
I20260809 23:22:17.963169  5762 raft_consensus.cc:3037] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:17.993568 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 832829us
I20260809 23:22:17.886955  5717 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:17.968569 24424 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260809 23:22:17.897926  5545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.023s	sys 0.046s
I20260809 23:22:17.974522  5752 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:17.964733  5581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.005s	sys 0.019s
I20260809 23:22:17.977703 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:17.978193 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:17.980697 25366 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:17.863041 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45027us
I20260809 23:22:17.958213  5537 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:17.967761 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.035125 24320 raft_consensus.cc:1226] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
W20260809 23:22:17.978394  5194 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.018s	sys 0.008s
W20260809 23:22:18.052816 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74702.7us
I20260809 23:22:17.992393  5278 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:18.032426  5545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.023s	sys 0.047s
I20260809 23:22:17.958120  5643 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.070693 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:18.075326 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 468272us
I20260809 23:22:18.032109  5611 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:18.039448  5729 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:18.051244  5748 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:17.963111  5571 consensus_queue.cc:260] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.093071 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.093775  5676 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:17.955579  5488 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.095311 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.055485  5635 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260809 23:22:18.095723 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.021s	sys 0.007s
I20260809 23:22:18.069926  5717 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:18.085201  5194 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.019s	sys 0.008s
I20260809 23:22:18.039510  5550 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.048s
W20260809 23:22:18.114849 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.021s	sys 0.007s
I20260809 23:22:18.129115 24543 raft_consensus.cc:1261] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260809 23:22:18.129122 24424 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260809 23:22:18.130430 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:18.131157 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 357552us
I20260809 23:22:18.131645 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.132098 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.133213 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.133394 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.134413 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.136916 24427 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:18.134197 24428 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:18.098440  5548 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.048s
I20260809 23:22:18.151805  5511 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.154213  5545 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:18.158355  5765 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:18.159204  5765 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:18.103667  5748 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.161463 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:22:18.162343  5748 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.163725 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:18.081346  5719 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260809 23:22:18.167261 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:18.167557 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.168408 24543 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:22:18.171684 24543 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:18.127815  5571 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.188314  5759 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:18.188120  5748 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:18.189092  5759 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:18.189761  5748 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:18.192884  5548 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:18.193337  5548 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.193783  5548 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:18.194154  5773 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260809 23:22:18.196007  5718 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260809 23:22:18.200124  5771 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260809 23:22:18.201853  5742 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260809 23:22:18.051617  5752 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.127072  5646 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:18.102397  5729 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:18.219638 25323 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.607s (SENT) [suppressed 42 similar messages]
I20260809 23:22:18.211189 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:18.248731 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.250203 25321 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.086884  5652 consensus_queue.cc:260] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.257385 24427 pending_rounds.cc:76] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 18
I20260809 23:22:18.257761 24427 pending_rounds.cc:97] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260809 23:22:18.258128 24427 raft_consensus.cc:2866] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:18.208406  5488 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:18.125160  5506 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.119s
I20260809 23:22:18.171695  5581 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:18.262262 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51192.8us
I20260809 23:22:18.275507 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.276247 24543 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:18.237900  5646 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.286414 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:18.230327  5658 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:18.291872 25371 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260809 23:22:18.263896  5482 raft_consensus.cc:2781] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:18.271518  5635 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260809 23:22:18.279284  5122 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.020s	sys 0.034s
I20260809 23:22:18.288942  5646 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:18.333735 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.333766 24431 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:18.286739  5643 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.341154 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:18.291201  5541 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.297508  5547 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:18.362727  5482 raft_consensus.cc:686] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:18.363780  5482 consensus_queue.cc:237] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.367749  5790 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.368290  5790 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.370321  5790 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.358834  5547 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.372592  5547 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:18.372812  5782 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260809 23:22:18.349622  5484 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:18.375607  5484 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.378033  5484 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:18.324471  5635 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.387342  5635 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:18.387624 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.388267 24542 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:18.389122 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.390178  5537 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:18.390575 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.390892  5537 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:18.392077  5729 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.393858  5751 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.425983  5751 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.428242  5751 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.430109  5548 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.394418  5791 consensus_queue.cc:1035] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:18.396078  5661 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:18.398569  5582 raft_consensus.cc:979] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: : Instructing follower 808fed3ed3f04882ad757a3db9abfdd9 to start an election
W20260809 23:22:18.400409  5673 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.642s	user 0.012s	sys 0.006s
I20260809 23:22:18.288568  5717 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.285653  5581 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:18.325990  5122 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.020s	sys 0.034s
I20260809 23:22:18.403517  5166 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.413720 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.462423 24534 raft_consensus.cc:2379] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:18.462702 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.296471  5658 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.394678  5752 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.434609 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.439759 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.458529 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 1986) took 1005 ms. Trace:
I20260809 23:22:18.436029 24312 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:18.438522  5666 raft_consensus.cc:3032] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Stepping down as leader of term 21
I20260809 23:22:18.435154  5506 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:18.487278 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:18.467193 25368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.020s	sys 0.042s
I20260809 23:22:18.484210 24429 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:18.488520 25368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.020s	sys 0.043s
I20260809 23:22:18.483687 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:18.473153  5172 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.263s	user 0.005s	sys 0.047s
I20260809 23:22:18.496078 25373 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260809 23:22:18.496172 25325 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:18.496555 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 408319us
I20260809 23:22:18.478713 24545 rpcz_store.cc:276] 0809 23:22:17.453308 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:17.453490 (+   182us) service_pool.cc:227] Handling call
0809 23:22:17.466707 (+ 13217us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:17.466985 (+   278us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:17.995848 (+528863us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.457784 (+461936us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.458065 (+   281us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:18.458132 (+    67us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:18.458196 (+    64us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.458296 (+   100us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:18.458454 (+   158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17570,"prepare.queue_time_us":142594,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":102,"raft.queue_time_us":4272,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":38519,"spinlock_wait_cycles":37515264,"thread_start_us":898907,"threads_started":2}
I20260809 23:22:18.467356  5582 raft_consensus.cc:1067] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Signalling peer 808fed3ed3f04882ad757a3db9abfdd9 to start an election
I20260809 23:22:18.492877  5790 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:18.522266 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 387936us
I20260809 23:22:18.528520 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.510259 25365 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:22:18.527983  5790 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.530788 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.484573  5666 raft_consensus.cc:729] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:18.491319  5506 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.543327 24266 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.548030 24545 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994"
dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
 from {username='slave'} at 127.0.0.1:46304
I20260809 23:22:18.548756 24545 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:22:18.489666  5719 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:18.553175 24314 raft_consensus.cc:1261] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:18.553504 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:18.542445  5512 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.003s
I20260809 23:22:18.554838  5512 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.555143 24317 raft_consensus.cc:1226] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:18.549830 24545 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:18.556834 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:18.497534  5172 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.295s	user 0.006s	sys 0.047s
I20260809 23:22:18.557979 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.562192 24543 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.495146  5711 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:18.502147  5643 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:18.561472 24422 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:18.489893  5670 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:18.523825  5611 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:18.579674 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.561751  5719 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.550127  5506 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:18.576978  5585 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.006s
I20260809 23:22:18.583146 25372 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:18.558547  5752 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:18.599529 24266 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:18.603909 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 384270us
I20260809 23:22:18.561425  5631 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.600396 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.583724 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.612538 25323 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.614116  5751 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:18.578294  5716 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:18.615880  5166 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.616446  5494 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:18.607041  5796 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:18.601639  5749 consensus_queue.cc:1046] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260809 23:22:18.603541  5611 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.625404  5611 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.627138  5631 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.629184  5631 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.629644 24543 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:18.631542 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.631393  5759 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:18.632625  5759 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:18.632787  5658 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.633028  5759 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:18.633947  5759 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.584152  5748 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:18.637714  5805 log.cc:924] Time spent T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:22:18.637917  5581 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.592533  5711 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:18.638695  5548 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:18.639355  5723 consensus_queue.cc:1035] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:18.643285  5512 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.639374  5711 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.645437  5652 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.647243 24429 raft_consensus.cc:1226] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []
I20260809 23:22:18.648694 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.651304 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.651475  5723 consensus_queue.cc:1035] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:18.652818  5723 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.653246  5723 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.635604 24546 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
W20260809 23:22:18.655613 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49773.8us
I20260809 23:22:18.525674  5278 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.635164 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.579353  5722 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:18.638471  5798 log.cc:924] Time spent T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260809 23:22:18.586822  5670 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:18.666777 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54328.7us
I20260809 23:22:18.668023 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.676008 24314 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:18.676923 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.662384 24544 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:18.672567 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.678167 24321 raft_consensus.cc:2362] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25. Re-sending same reply.
I20260809 23:22:18.673245  5748 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.673830 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.687916 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.687989  5748 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.655009  5646 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.689137 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.690155  5579 raft_consensus.cc:3032] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:18.690431  5646 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.690586  5579 raft_consensus.cc:729] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:18.691133 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.680277 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.691959 24311 raft_consensus.cc:2396] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260809 23:22:18.692044 24321 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:18.692835 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.692965 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.693612 24321 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
I20260809 23:22:18.687165 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.694386 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:18.668605  5545 consensus_meta.cc:287] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.514s	user 0.005s	sys 0.011s
I20260809 23:22:18.679587 24270 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:18.691569  5579 consensus_queue.cc:260] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.674027 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.708876 24422 raft_consensus.cc:1226] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:18.711018 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.709182 24317 raft_consensus.cc:2362] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25. Re-sending same reply.
I20260809 23:22:18.717684 24313 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 31: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:18.634505  5635 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.687279 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:18.694298 24424 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.007s	sys 0.018s
W20260809 23:22:18.729982 24424 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.008s	sys 0.018s
I20260809 23:22:18.635782  5759 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:18.686453 24320 raft_consensus.cc:2454] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:18.690315  5748 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.733970 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.734647 24422 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
I20260809 23:22:18.693392  5670 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.664934 24545 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.726542 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:18.726640 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47135.4us
I20260809 23:22:18.726686 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:18.727305 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.008s	sys 0.020s
I20260809 23:22:18.727253 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.728040 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.743929 24540 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:18.744683 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.008s	sys 0.021s
I20260809 23:22:18.704362 25323 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.720531  5545 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:18.711896 24429 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:18.754794 24545 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.642689  5547 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.745625 24317 raft_consensus.cc:1226] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
W20260809 23:22:18.749301 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45072.4us
I20260809 23:22:18.722690 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.752017 24314 raft_consensus.cc:2354] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 20. Current term is 21.
I20260809 23:22:18.666080  5648 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260809 23:22:18.754196 24323 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:22:18.731134  5722 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:18.763536 24431 raft_consensus.cc:1226] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:18.765798 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.767021 24422 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:18.768184 24422 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 27. Current term is 28. Ops: []
I20260809 23:22:18.769347 24422 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260809 23:22:18.771160 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: true dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:18.771739 25372 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:18.772028 25372 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:18.773144 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.773712 24320 raft_consensus.cc:2362] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28. Re-sending same reply.
I20260809 23:22:18.774698 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.688017  5801 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.172s
I20260809 23:22:18.775794 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.758214  5166 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:18.776430 24320 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:18.777431 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.754940  5622 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:18.760347  5579 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.779621  5801 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:18.779857  5783 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.596s	user 0.001s	sys 0.002s
I20260809 23:22:18.780139  5801 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.782263  5801 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.783921  5635 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.771160 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.785197 24311 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.787761 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.788389 24544 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:18.790825 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.791507 24313 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:18.792379 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.793066 24313 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:18.793805 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.708775  5548 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.046s
I20260809 23:22:18.795045 24544 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:18.795563  5311 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:18.796092  5311 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:18.797408  5311 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.662909  5666 consensus_queue.cc:260] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.798884  5311 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
W20260809 23:22:18.806173 24498 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.542s (SENT)
I20260809 23:22:18.806882 24498 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.807238  5278 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:18.807931  5800 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260809 23:22:18.808344  5718 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:18.809950 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3955) took 1067 ms. Trace:
I20260809 23:22:18.801780 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.811513 24314 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:18.713003  5643 consensus_queue.cc:1035] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260809 23:22:18.813300 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.772773 25372 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.778481 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:18.816483 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Timed out: RequestConsensusVote RPC to 127.21.255.1:36681 timed out after 1.557s (SENT)
I20260809 23:22:18.824282 24270 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:18.816543 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.810317 24535 rpcz_store.cc:276] 0809 23:22:17.743028 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:17.743216 (+   188us) service_pool.cc:227] Handling call
0809 23:22:17.764126 (+ 20910us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:17.767194 (+  3068us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:17.767228 (+    34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:17.770976 (+  3748us) log.cc:842] Serialized 26 byte log entry
0809 23:22:18.292861 (+521885us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:18.292921 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.292982 (+    61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:18.807244 (+514262us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:18.809001 (+  1757us) raft_consensus.cc:1671] finished
0809 23:22:18.809021 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:18.809781 (+   760us) spinlock_profiling.cc:243] Waited 9.87 ms on lock 0x7b5400081e50. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:18.809895 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2675,"raft.queue_time_us":3418,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":10007,"spinlock_wait_cycles":128308352,"thread_start_us":427412,"threads_started":1,"wal-append.queue_time_us":796172}
I20260809 23:22:18.794894 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: true dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.826352 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.829367 24313 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:18.830549 24540 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:18.831894 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.832662 24545 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:18.834367 24323 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:18.814132 24431 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:18.835923 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.836169 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:18.841583 24544 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:18.843796 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.844786 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.845690 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.847605 24317 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:18.852588 25325 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.852020 25372 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:18.844111 24314 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:18.855492 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:18.856249 24314 raft_consensus.cc:3037] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:18.840510  5801 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election won for term 21
I20260809 23:22:18.802596  5666 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:18.870492  5658 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:18.845649  5541 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:18.866248  5801 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:18.873581  5801 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.874236  5541 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.874661  5541 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:18.870864  5484 raft_consensus.cc:2781] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:18.879141  5166 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.879245  5484 raft_consensus.cc:686] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:18.833647  5729 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.879684  5166 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.880388  5729 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.881745  5166 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.882474  5729 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:18.882727 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.851079  5548 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:18.883623  5548 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:18.884498  5548 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:18.840857  5718 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260809 23:22:18.851200  5585 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:18.846518  5714 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.869216 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.887194  5585 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.888294  5547 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.834137  5648 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer ab5aefaeb3d14775b30a74a2a6d3357f log is divergent from this leader: its last log entry 23.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:18.833562  5622 raft_consensus.cc:686] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:18.889174  5585 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:18.889855  5622 consensus_queue.cc:237] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.866700 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.867045 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.868194 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.891516 24431 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:18.878499  5661 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:18.823212  5752 consensus_queue.cc:1159] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b9b54436d0e94a5595cdded73fe33da9" responder_term: 22 status { last_received { term: 21 index: 15 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 21. Current term is 22. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:18.801982  5719 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:18.764809  5722 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:18.806092  5790 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:18.868718  5723 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:18.892222 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.892967 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:18.893260 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:18.940899 24540 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:22:18.920722 25372 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:22:18.942358 25366 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.010s	sys 0.037s
W20260809 23:22:18.942799 25366 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.010s	sys 0.037s
I20260809 23:22:18.880395 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.893888 24313 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:18.858664 24311 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:18.899360 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:18.923776 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.944568 25368 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
W20260809 23:22:18.946250 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.013s	sys 0.020s
W20260809 23:22:18.946614 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.013s	sys 0.020s
I20260809 23:22:18.919394 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.910756 24266 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.904611 24321 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:18.945652 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:18.955590 24427 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.020s	sys 0.020s
W20260809 23:22:18.955929 24427 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.020s	sys 0.020s
W20260809 23:22:18.927045 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.021s	sys 0.030s
W20260809 23:22:18.959805 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.021s	sys 0.030s
W20260809 23:22:18.960642 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 828994us
I20260809 23:22:18.961439 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.968287 24498 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.969638 24498 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:18.971140 24498 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:18.977761 24311 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:19.013705 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:19.015519  5541 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:19.012275 24269 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:18.990460 24540 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
W20260809 23:22:19.003038 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110882us
I20260809 23:22:19.032648 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:19.010178 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:18.924300  5658 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:19.026633  5652 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:19.027267 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:18.973892  5488 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.024s	sys 0.009s
I20260809 23:22:18.945974  5722 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:18.953323  5537 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:19.040994 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:19.055172  5816 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:19.062842 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:19.042222 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51288.2us
I20260809 23:22:19.001801  5681 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:19.047284 24535 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 30
I20260809 23:22:19.070719 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:19.020511  5541 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:19.051721 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.055902 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:19.005638  5723 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:18.880219  5484 consensus_queue.cc:237] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:18.902817  5646 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.127s
W20260809 23:22:19.057031  5488 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.025s	sys 0.009s
I20260809 23:22:19.091689 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:19.076197 25363 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.004s	sys 0.025s
W20260809 23:22:19.092492 25363 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.004s	sys 0.025s
I20260809 23:22:19.015559 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:19.044950 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:19.088760 24311 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 30. Current term is 31. Ops: []
W20260809 23:22:19.093550 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 78045.3us
I20260809 23:22:19.093655 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:18.917917  5661 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:19.091130 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 78968.1us
I20260809 23:22:19.047081  5719 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:19.047101  5652 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:19.062476  5673 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:19.118462 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:19.112421 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.123629 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:19.124464 25373 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 22: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:19.114099 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69312.3us
W20260809 23:22:19.118877 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53962.1us
I20260809 23:22:19.125200 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.129033 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.129937  5488 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:19.130381 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.130743  5488 consensus_queue.cc:1225] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 21.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260809 23:22:19.133718  5643 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:19.109354 24314 raft_consensus.cc:2454] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:19.136615 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.137482 24314 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 29: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:19.139003 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:19.139870 24314 raft_consensus.cc:2379] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:19.133555 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:19.146576 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:19.167474 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:19.175952 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:19.091380  5537 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:19.179364 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:19.190986 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:19.184917 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:19.095058  5677 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:19.090862  5723 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:19.112689  5571 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:19.121719  5652 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:19.192761 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:19.198757 24323 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:19.188737 24270 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:19.222570 25366 raft_consensus.cc:1261] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:19.057451  5658 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:19.246878 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58263.5us
I20260809 23:22:19.247056 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:19.249377 25371 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.012s	sys 0.032s
W20260809 23:22:19.249922 25371 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.012s	sys 0.032s
I20260809 23:22:19.193387 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:19.181869 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:19.316802 24534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.020s	sys 0.035s
W20260809 23:22:19.317257 24534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.020s	sys 0.036s
W20260809 23:22:19.195953  5122 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.016s	sys 0.022s
I20260809 23:22:19.320606 24314 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 30: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:19.176427  5729 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:19.324862 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91875.4us
W20260809 23:22:19.328526 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 135231us
I20260809 23:22:19.329536 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:19.337005 24320 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:19.337761 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:19.205302  5719 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:19.217000  5648 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:19.350229 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:19.357254 25373 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:19.218698  5571 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:19.438597  5571 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:19.218004  5723 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:19.449370  5506 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.786s	user 0.003s	sys 0.005s
I20260809 23:22:19.450062  5506 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:19.371789  5830 log.cc:924] Time spent T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260809 23:22:19.373273  5826 log.cc:924] Time spent T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.259s	user 0.000s	sys 0.000s
I20260809 23:22:19.387127  5666 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:19.390914 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4020) took 1205 ms. Trace:
I20260809 23:22:19.488554 24424 rpcz_store.cc:276] 0809 23:22:18.185773 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.185932 (+   159us) service_pool.cc:227] Handling call
0809 23:22:18.186224 (+   292us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:18.186631 (+   407us) raft_consensus.cc:1520] Early marking committed up to index 21
0809 23:22:18.187263 (+   632us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.730301 (+543038us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:18.736890 (+  6589us) log.cc:842] Serialized 52 byte log entry
0809 23:22:18.737093 (+   203us) raft_consensus.cc:1631] Marking committed up to 22
0809 23:22:18.737601 (+   508us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.737833 (+   232us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.737919 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:19.390626 (+652707us) raft_consensus.cc:1671] finished
0809 23:22:19.390655 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.390837 (+   182us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":305,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":67,"raft.queue_time_us":200,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":180,"thread_start_us":542548,"threads_started":1,"wal-append.queue_time_us":1029747}
I20260809 23:22:19.491595 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:19.186390  5681 consensus_queue.cc:1046] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.150s
I20260809 23:22:19.413872  5658 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:19.247795  5537 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:19.314493  5796 raft_consensus.cc:2741] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
W20260809 23:22:19.502192 24316 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.036s	sys 0.034s
W20260809 23:22:19.502604 24316 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.895s	user 0.036s	sys 0.034s
I20260809 23:22:19.416702  5661 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:19.419380 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4026) took 1016 ms. Trace:
I20260809 23:22:19.319620  5752 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:19.320016  5585 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:19.324900  5122 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.017s	sys 0.023s
I20260809 23:22:19.431653 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:19.437963  5714 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.551s	user 0.005s	sys 0.001s
I20260809 23:22:19.068219  5816 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.526746 24539 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:19.354916  5729 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:19.363811  5759 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:19.462828 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:19.464293 24321 raft_consensus.cc:1226] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:19.503539 24430 rpcz_store.cc:276] 0809 23:22:18.403368 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.403532 (+   164us) service_pool.cc:227] Handling call
0809 23:22:18.403973 (+   441us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.404301 (+   328us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.404330 (+    29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.404742 (+   412us) log.cc:842] Serialized 26 byte log entry
0809 23:22:18.961377 (+556635us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:18.961417 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.961485 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:19.419122 (+457637us) raft_consensus.cc:1671] finished
0809 23:22:19.419150 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.419314 (+   164us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1252,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":541970,"threads_started":1,"wal-append.queue_time_us":1012482}
W20260809 23:22:19.466647 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 137218us
I20260809 23:22:19.467648  5579 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:22:19.468173 24422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.020s	sys 0.060s
I20260809 23:22:19.467751  5545 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:19.472016 24542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.935s	user 0.016s	sys 0.036s
W20260809 23:22:19.474643 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.029s	sys 0.038s
I20260809 23:22:19.527350  5752 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:19.498944 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.921s	user 0.011s	sys 0.048s
W20260809 23:22:19.572651 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.011s	sys 0.048s
I20260809 23:22:19.535467  5714 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:19.507032 24320 pending_rounds.cc:76] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting all ops after (but not including) 16
I20260809 23:22:19.577215 24320 pending_rounds.cc:97] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260809 23:22:19.577554 24320 raft_consensus.cc:2866] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:19.501307 24424 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:19.579121 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:19.579910 24424 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:19.581086 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: true dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:19.581698 24424 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:19.521847 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 90287.5us
I20260809 23:22:19.578671 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1951) took 1093 ms. Trace:
I20260809 23:22:19.528963 24540 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:19.551097 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:19.585358 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:19.553867 25375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.015s	sys 0.035s
W20260809 23:22:19.586889 25375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.015s	sys 0.035s
W20260809 23:22:19.554390 24425 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.023s	sys 0.031s
W20260809 23:22:19.587782 24425 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.023s	sys 0.031s
W20260809 23:22:19.554669 24309 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.066s	sys 0.138s
I20260809 23:22:19.544523 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:19.588382 24309 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.066s	sys 0.139s
W20260809 23:22:19.588893 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44508.7us
W20260809 23:22:19.589537 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.935s	user 0.019s	sys 0.020s
I20260809 23:22:19.563437 24535 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
W20260809 23:22:19.565990 24542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.032s	user 0.016s	sys 0.036s
W20260809 23:22:19.591087 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.020s	sys 0.020s
W20260809 23:22:19.566150 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.936s	user 0.029s	sys 0.039s
W20260809 23:22:19.479203  5482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.016s	sys 0.031s
I20260809 23:22:19.567180 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1820) took 1243 ms. Trace:
I20260809 23:22:19.594384 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:19.334802 24311 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:19.590149 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:19.561993 24422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.022s	sys 0.061s
I20260809 23:22:19.591691 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2000) took 1100 ms. Trace:
I20260809 23:22:19.593350 24380 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:19.584522 24430 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:19.594648 24435 rpcz_store.cc:276] 0809 23:22:18.323774 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.323967 (+   193us) service_pool.cc:227] Handling call
0809 23:22:18.324413 (+   446us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.324843 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.324883 (+    40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.325423 (+   540us) log.cc:842] Serialized 26 byte log entry
0809 23:22:18.734960 (+409537us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:18.735001 (+    41us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.735061 (+    60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:19.566871 (+831810us) raft_consensus.cc:1671] finished
0809 23:22:19.566901 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.567099 (+   198us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4697,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":3789,"thread_start_us":338667,"threads_started":1,"wal-append.queue_time_us":786093}
I20260809 23:22:19.597981 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:19.598582 24378 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:19.599287 24541 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:19.583694 25371 rpcz_store.cc:276] 0809 23:22:18.485514 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.485667 (+   153us) service_pool.cc:227] Handling call
0809 23:22:18.504225 (+ 18558us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.504510 (+   285us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.504546 (+    36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.505127 (+   581us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.250329 (+745202us) raft_consensus.cc:1631] Marking committed up to 22
0809 23:22:19.250381 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.250471 (+    90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:19.578421 (+327950us) raft_consensus.cc:1671] finished
0809 23:22:19.578448 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.578602 (+   154us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":744838,"threads_started":1,"wal-append.queue_time_us":1071428}
I20260809 23:22:19.603533 24435 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:19.595804 24542 rpcz_store.cc:276] 0809 23:22:18.491256 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.491754 (+   498us) service_pool.cc:227] Handling call
0809 23:22:18.492085 (+   331us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:18.492334 (+   249us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:18.533498 (+ 41164us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.591385 (+1057887us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:19.591439 (+    54us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:19.591484 (+    45us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.591543 (+    59us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.591643 (+   100us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33090,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":9491,"raft.queue_time_us":29732,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8663,"thread_start_us":1057621,"threads_started":1}
I20260809 23:22:19.520618  5585 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:19.610325 24428 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.002s	sys 0.038s
W20260809 23:22:19.639354 24428 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.006s	sys 0.038s
W20260809 23:22:19.610512 24313 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.005s	sys 0.035s
W20260809 23:22:19.641320 24313 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.039s
I20260809 23:22:19.642355 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:19.643236 24430 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:19.599995  5670 consensus_meta.cc:287] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.844s	user 0.007s	sys 0.005s
I20260809 23:22:19.601698  5791 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:19.646175 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:19.653318 24423 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:19.621356 24424 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
W20260809 23:22:19.594468  5482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.016s	sys 0.031s
W20260809 23:22:19.604619  5311 consensus_meta.cc:287] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.799s	user 0.011s	sys 0.002s
W20260809 23:22:19.599399  5278 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.008s	sys 0.009s
W20260809 23:22:19.608623  5548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.016s	sys 0.031s
W20260809 23:22:19.647225  5801 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.018s	sys 0.028s
I20260809 23:22:19.655797 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2064) took 1900 ms. Trace:
W20260809 23:22:19.647223  5792 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260809 23:22:19.685702 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:19.656958 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:19.686488 24429 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:19.594259  5790 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:19.688015  5790 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.689462 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1942) took 1435 ms. Trace:
I20260809 23:22:19.602476  5545 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:19.691574  5545 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:19.692178  5545 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:19.693239  5545 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.694407  5748 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:19.607331  5681 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:22:19.695842  5681 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:19.696985  5681 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.610499  5537 raft_consensus.cc:2741] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election decision vote started in defunct term 22: won
I20260809 23:22:19.697543  5635 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:19.704129  5635 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.690655  5790 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:19.680073 24316 rpcz_store.cc:276] 0809 23:22:17.755429 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:17.755585 (+   156us) service_pool.cc:227] Handling call
0809 23:22:17.774269 (+ 18684us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:18.133839 (+359570us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.133880 (+    41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:18.167008 (+ 33128us) log.cc:842] Serialized 52 byte log entry
0809 23:22:18.586898 (+419890us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:19.503063 (+916165us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.503458 (+   395us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.503830 (+   372us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.504023 (+   193us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:19.655579 (+151556us) raft_consensus.cc:1671] finished
0809 23:22:19.655599 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.655740 (+   141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59674,"prepare.queue_time_us":1059739,"raft.queue_time_us":79754,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":7285,"spinlock_wait_cycles":27184640,"thread_start_us":1493547,"threads_started":3,"wal-append.queue_time_us":787420}
W20260809 23:22:19.650346 25369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.013s	sys 0.037s
W20260809 23:22:19.711141 25369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.838s	user 0.013s	sys 0.037s
W20260809 23:22:19.661541  5622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.014s	sys 0.017s
W20260809 23:22:19.663244  5717 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.007s	sys 0.025s
I20260809 23:22:19.721005 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46304 (request call id 1996) took 1334 ms. Trace:
I20260809 23:22:19.727772 24539 rpcz_store.cc:276] 0809 23:22:18.386485 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.387360 (+   875us) service_pool.cc:227] Handling call
0809 23:22:18.600258 (+212898us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800253b98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900bce 000000000034aea9 00007fac2872ca30 00007fac287f27b4 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:22:19.720695 (+1120437us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b54002706d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:19.720918 (+   223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2077404032}
W20260809 23:22:19.663386  5278 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.009s	sys 0.009s
I20260809 23:22:19.659941  5791 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:19.715246 24423 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:19.659428  5579 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:19.689960 25363 rpcz_store.cc:276] 0809 23:22:18.254432 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.254571 (+   139us) service_pool.cc:227] Handling call
0809 23:22:18.254842 (+   271us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:18.494232 (+239390us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:18.494547 (+   315us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.094255 (+599708us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:19.094311 (+    56us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:19.094355 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.689177 (+594822us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540055f0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:19.689239 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:19.689391 (+   152us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":131,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":1289,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":20767,"spinlock_wait_cycles":551451776,"thread_start_us":598088,"threads_started":1,"wal-append.queue_time_us":1048457}
W20260809 23:22:19.679921  5801 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.018s	sys 0.028s
W20260809 23:22:19.663123  5548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.017s	sys 0.032s
W20260809 23:22:19.722944  5622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.014s	sys 0.018s
W20260809 23:22:19.718918  5711 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.011s	sys 0.052s
I20260809 23:22:19.679793  5631 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:19.706797  5635 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:19.727625  5717 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.008s	sys 0.025s
I20260809 23:22:19.726095  5748 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:19.655751  5670 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.739688  5791 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:19.750510  5506 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:19.766412 25365 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:19.757555  5718 raft_consensus.cc:2741] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Leader pre-election decision vote started in defunct term 24: won
W20260809 23:22:19.774437  5673 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.001s
W20260809 23:22:19.746505  5711 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.012s	sys 0.052s
W20260809 23:22:19.780851 24317 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.014s	sys 0.026s
W20260809 23:22:19.781181 24317 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.014s	sys 0.026s
W20260809 23:22:19.814097 24431 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.018s	sys 0.027s
I20260809 23:22:19.738269  5752 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:19.815012 24431 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.019s	sys 0.027s
W20260809 23:22:19.780175  5827 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
I20260809 23:22:19.777768  5729 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:19.778402  5673 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:19.877172  5843 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260809 23:22:19.847330  5833 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.291s	user 0.000s	sys 0.000s
I20260809 23:22:19.894057 25372 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:22:19.855890 24429 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
W20260809 23:22:19.932403 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.020s	sys 0.041s
W20260809 23:22:19.940488 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.020s	sys 0.041s
W20260809 23:22:19.943220 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 826940us
I20260809 23:22:19.946405 24430 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:19.883639  5729 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:19.887799  5835 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.286s	user 0.001s	sys 0.001s
W20260809 23:22:19.951395 24312 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.193s	user 0.028s	sys 0.066s
W20260809 23:22:19.953956 24312 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.196s	user 0.028s	sys 0.066s
I20260809 23:22:19.901564  5585 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:19.899845  5673 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:19.962884 24543 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3953) took 1332 ms (client timeout 1705 ms). Trace:
W20260809 23:22:19.963841 24543 rpcz_store.cc:269] 0809 23:22:18.630717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.631271 (+   554us) service_pool.cc:227] Handling call
0809 23:22:19.962567 (+1331296us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001fc054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:19.962789 (+   222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2693501312}
W20260809 23:22:19.964915 24534 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.009s	sys 0.009s
W20260809 23:22:19.965473 24534 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.009s	sys 0.009s
I20260809 23:22:19.948686  5729 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:19.999657  5670 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:20.000589  5858 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260809 23:22:19.960548  5816 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:20.039060  5673 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:20.238454 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.008s	sys 0.012s
W20260809 23:22:20.505149 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.008s	sys 0.012s
W20260809 23:22:20.505916 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 907514us
I20260809 23:22:20.506862 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:20.239346 24425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.009s	sys 0.011s
W20260809 23:22:20.507695 24425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.009s	sys 0.011s
W20260809 23:22:20.242152 24380 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.004s	sys 0.016s
W20260809 23:22:20.309676 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.022s	sys 0.027s
W20260809 23:22:20.508191 24380 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.004s	sys 0.016s
W20260809 23:22:20.508316 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.023s	sys 0.028s
W20260809 23:22:20.508646 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 915306us
W20260809 23:22:20.237973 24499 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.005s	sys 0.014s
W20260809 23:22:20.510316 24499 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.005s	sys 0.014s
W20260809 23:22:20.511950  5828 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.991s	user 0.011s	sys 0.002s
W20260809 23:22:20.496479  5722 consensus_meta.cc:287] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.407s	user 0.025s	sys 0.004s
I20260809 23:22:20.513155  5722 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:20.489699  5855 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.464s	user 0.000s	sys 0.011s
I20260809 23:22:20.515007 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:20.515367  5722 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:20.515834 24542 raft_consensus.cc:2379] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 30: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:20.517293  5729 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:20.518038  5849 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.633s	user 0.002s	sys 0.010s
W20260809 23:22:20.488811  5571 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.050s	user 0.017s	sys 0.009s
W20260809 23:22:20.515852  5869 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.737s	user 0.012s	sys 0.000s
I20260809 23:22:20.519346  5571 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.519809 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1816) took 2268 ms. Trace:
I20260809 23:22:20.520310 25375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1952) took 2034 ms. Trace:
I20260809 23:22:20.521286  5571 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:20.521641 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3557) took 1842 ms. Trace:
I20260809 23:22:20.522023 24422 rpcz_store.cc:276] 0809 23:22:18.679518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.825284 (+145766us) service_pool.cc:227] Handling call
0809 23:22:18.853601 (+ 28317us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.853944 (+   343us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:22:18.853974 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.891746 (+ 37772us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.595739 (+703993us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:19.595794 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.595877 (+    83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.521429 (+925552us) raft_consensus.cc:1671] finished
0809 23:22:20.521453 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.521585 (+   132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":291,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":26227,"thread_start_us":649290,"threads_started":1,"wal-append.queue_time_us":853701}
I20260809 23:22:20.524227  5670 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:20.524796  5670 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.525290  5484 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:20.525785  5484 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:20.526588  5868 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.526s	user 0.009s	sys 0.008s
I20260809 23:22:20.526911 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:20.527171  5670 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:20.533032 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:20.528759 24425 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3553) took 1882 ms. Trace:
W20260809 23:22:20.504632  5851 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.605s	user 0.000s	sys 0.014s
I20260809 23:22:20.545559 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:20.531821  5829 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 1.011s	user 0.002s	sys 0.000s
I20260809 23:22:20.546273 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3284) took 1834 ms. Trace:
I20260809 23:22:20.533504 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2103) took 1662 ms. Trace:
I20260809 23:22:20.528098 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:20.546813 24309 rpcz_store.cc:276] 0809 23:22:18.767285 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.767529 (+   244us) service_pool.cc:227] Handling call
0809 23:22:18.767868 (+   339us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.768212 (+   344us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.768243 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.768675 (+   432us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.588721 (+820046us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:19.663889 (+ 75168us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.664127 (+   238us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.664228 (+   101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.546039 (+881811us) raft_consensus.cc:1671] finished
0809 23:22:20.546069 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.546200 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96395,"prepare.queue_time_us":176871,"prepare.run_cpu_time_us":1178,"prepare.run_wall_time_us":3352,"raft.queue_time_us":39318,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":216,"thread_start_us":819690,"threads_started":1,"wal-append.queue_time_us":1072353}
I20260809 23:22:20.520764 25375 rpcz_store.cc:276] 0809 23:22:18.485872 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.486012 (+   140us) service_pool.cc:227] Handling call
0809 23:22:18.506118 (+ 20106us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.590369 (+ 84251us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.590400 (+    31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.606809 (+ 16409us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.587354 (+980545us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:19.587409 (+    55us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.695652 (+108243us) spinlock_profiling.cc:243] Waited 80.3 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:19.695717 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.520019 (+824302us) raft_consensus.cc:1671] finished
0809 23:22:20.520039 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.520251 (+   212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3538,"raft.queue_time_us":3761,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":12552,"spinlock_wait_cycles":184691072,"thread_start_us":879111,"threads_started":1,"wal-append.queue_time_us":1116034}
I20260809 23:22:20.536733 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:20.341049 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.003s	sys 0.015s
I20260809 23:22:20.540014 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:20.552299 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.200s	user 0.003s	sys 0.016s
I20260809 23:22:20.552580 24422 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
W20260809 23:22:20.341625 24377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.008s	sys 0.004s
W20260809 23:22:20.553947 24377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.008s	sys 0.004s
W20260809 23:22:20.342242 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.004s	sys 0.015s
W20260809 23:22:20.555840 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.138s	user 0.004s	sys 0.015s
W20260809 23:22:20.342689 24270 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.003s	sys 0.011s
W20260809 23:22:20.351045 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 387328us
W20260809 23:22:20.556644 24270 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.922s	user 0.003s	sys 0.011s
W20260809 23:22:20.253328  5122 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.008s	sys 0.011s
W20260809 23:22:20.238356  5278 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.003s	sys 0.009s
W20260809 23:22:20.368803 24323 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.010s	sys 0.003s
W20260809 23:22:20.387224  5545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.004s	sys 0.006s
W20260809 23:22:20.309875 24423 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.007s	sys 0.003s
W20260809 23:22:20.357998  5681 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.002s	sys 0.008s
W20260809 23:22:20.358903 24313 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.000s	sys 0.011s
I20260809 23:22:20.568104 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:20.574986 24313 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.002s	sys 0.011s
I20260809 23:22:20.575291 24542 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:20.343032 24314 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.002s	sys 0.012s
W20260809 23:22:20.575863 24314 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.002s	sys 0.014s
W20260809 23:22:20.575289  5278 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.003s	sys 0.009s
I20260809 23:22:20.539933  5751 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:20.576710  5751 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:20.462769 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3962) took 1570 ms. Trace:
I20260809 23:22:20.577430 24541 rpcz_store.cc:276] 0809 23:22:18.892495 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.892975 (+   480us) service_pool.cc:227] Handling call
0809 23:22:20.032114 (+1139139us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400271ad4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:20.462706 (+430592us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1505443200}
I20260809 23:22:20.577701  5751 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.579035 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:20.579309  5751 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:19.940968  5194 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:20.580883  5194 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:20.238737 24320 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.013s	sys 0.006s
W20260809 23:22:20.583163 24320 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.016s	sys 0.008s
I20260809 23:22:19.887419  5748 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:20.341953 24435 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.009s	sys 0.009s
W20260809 23:22:20.586369 24435 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.952s	user 0.011s	sys 0.010s
W20260809 23:22:20.237505 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.013s	sys 0.007s
W20260809 23:22:20.494494  5723 consensus_meta.cc:287] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.051s	user 0.014s	sys 0.013s
I20260809 23:22:20.551719 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:20.547447 25369 rpcz_store.cc:276] 0809 23:22:18.870628 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.871947 (+  1319us) service_pool.cc:227] Handling call
0809 23:22:18.872336 (+   389us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.872776 (+   440us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:22:18.873051 (+   275us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.711667 (+838616us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:19.712192 (+   525us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.712345 (+   153us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:19.712385 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.712471 (+    86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.533178 (+820707us) raft_consensus.cc:1671] finished
0809 23:22:20.533208 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.533414 (+   206us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":690,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":9660,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":6215,"thread_start_us":838173,"threads_started":1,"wal-append.queue_time_us":976655}
I20260809 23:22:20.559957 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:20.561058 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:20.546286 24535 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:20.520270 24427 rpcz_store.cc:276] 0809 23:22:18.251358 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.256782 (+  5424us) service_pool.cc:227] Handling call
0809 23:22:18.257176 (+   394us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.291354 (+ 34178us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.291421 (+    67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.292201 (+   780us) log.cc:842] Serialized 26 byte log entry
0809 23:22:18.956225 (+664024us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:18.956264 (+    39us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.956326 (+    62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.035345 (+1079019us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:20.519460 (+484115us) raft_consensus.cc:1671] finished
0809 23:22:20.519485 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.519741 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27581184,"thread_start_us":663698,"threads_started":1,"wal-append.queue_time_us":1187889}
W20260809 23:22:20.343771 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.007s	sys 0.011s
W20260809 23:22:20.615871 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.007s	sys 0.012s
W20260809 23:22:20.574839 24323 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.011s	sys 0.004s
W20260809 23:22:20.429946  5648 consensus_meta.cc:287] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.978s	user 0.019s	sys 0.007s
W20260809 23:22:20.316103 24428 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.005s	sys 0.005s
W20260809 23:22:20.620348 24428 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.007s	sys 0.007s
I20260809 23:22:20.620256 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:20.620946 24544 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:20.527711  5670 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:20.588168 24423 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s	user 0.020s	sys 0.030s
I20260809 23:22:20.593276 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:20.527755  5484 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:20.599766 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1837) took 1139 ms. Trace:
I20260809 23:22:20.605046 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:20.606350 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3296) took 1833 ms. Trace:
W20260809 23:22:20.494437  5652 consensus_meta.cc:287] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 1.172s	user 0.012s	sys 0.016s
I20260809 23:22:20.461793  5813 raft_consensus.cc:3032] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Stepping down as leader of term 30
I20260809 23:22:20.544696 24425 rpcz_store.cc:276] 0809 23:22:18.646471 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.653212 (+  6741us) service_pool.cc:227] Handling call
0809 23:22:18.653492 (+   280us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:18.653688 (+   196us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:19.594295 (+940607us) log.cc:842] Serialized 14 byte log entry
0809 23:22:20.528439 (+934144us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:20.528489 (+    50us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:20.528540 (+    51us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:20.528602 (+    62us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.528706 (+   104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20286,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"thread_start_us":1847680,"threads_started":2}
I20260809 23:22:20.664685 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:20.611557 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.015s	user 0.020s	sys 0.009s
I20260809 23:22:20.612264 24311 raft_consensus.cc:1226] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260809 23:22:20.529021  5816 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:20.539561  5494 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Stepping down as leader of term 27
W20260809 23:22:20.621791 25325 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.019s	user 0.012s	sys 0.041s
W20260809 23:22:20.705897 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.110s	user 0.012s	sys 0.041s
W20260809 23:22:20.706238 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 1.11314e+06us
W20260809 23:22:20.566114  5122 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.009s	sys 0.012s
I20260809 23:22:20.616613 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:20.529026  5673 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:20.528687  5646 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:20.602047  5571 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:20.655138 24309 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:20.655504 24313 rpcz_store.cc:276] 0809 23:22:18.772972 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.943838 (+170866us) service_pool.cc:227] Handling call
0809 23:22:18.944160 (+   322us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.944778 (+   618us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.944824 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.945252 (+   428us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.642680 (+697428us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:20.575457 (+932777us) log.cc:842] Serialized 14 byte log entry
0809 23:22:20.606040 (+ 30583us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:20.606114 (+    74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:20.606153 (+    39us) raft_consensus.cc:1671] finished
0809 23:22:20.606163 (+    10us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.606294 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155500,"prepare.queue_time_us":127348,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":31,"raft.queue_time_us":312,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":25451,"thread_start_us":1605170,"threads_started":2,"wal-append.queue_time_us":1001583}
I20260809 23:22:20.791702 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:20.638827 25375 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
W20260809 23:22:20.593796  5681 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.002s	sys 0.009s
W20260809 23:22:20.593775  5545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.004s	sys 0.007s
I20260809 23:22:20.700680 24316 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:20.614382  5818 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:20.538794  5729 consensus_queue.cc:1035] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.107s
W20260809 23:22:20.752784  5791 consensus_meta.cc:287] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.987s	user 0.012s	sys 0.005s
I20260809 23:22:20.748982 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:20.622036  5718 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:20.891516 24424 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
W20260809 23:22:20.893857 25372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37950 (request call id 1971) took 1809 ms (client timeout 2168 ms). Trace:
W20260809 23:22:20.894397 25372 rpcz_store.cc:269] 0809 23:22:19.084957 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.090918 (+  5961us) service_pool.cc:227] Handling call
0809 23:22:20.893658 (+1802740us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400717854. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:20.893794 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1821051264}
I20260809 23:22:20.622566  5874 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:20.652871 24435 rpcz_store.cc:276] 0809 23:22:19.460211 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.604259 (+144048us) service_pool.cc:227] Handling call
0809 23:22:19.604541 (+   282us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:20.587029 (+982488us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:20.598879 (+ 11850us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:20.598956 (+    77us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:20.599136 (+   180us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:20.599220 (+    84us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:20.599649 (+   429us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29546,"raft.queue_time_us":145546,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":5258,"thread_start_us":952372,"threads_started":1}
I20260809 23:22:20.647572  5648 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:20.787906 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
W20260809 23:22:20.755604 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 139078us
I20260809 23:22:20.817601 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:20.691484  5666 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:20.706405  5868 log.cc:924] Time spent T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260809 23:22:20.664002  5891 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260809 23:22:20.839670  5829 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260809 23:22:20.691617  5813 raft_consensus.cc:729] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:20.747442 25325 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:20.744771  5622 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:20.754840  5122 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:20.968302 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 416107us
I20260809 23:22:20.635892  5723 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:20.908452 25366 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 27. Current term is 28. Ops: []
I20260809 23:22:20.705751  5494 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:20.839196  5866 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260809 23:22:20.691468  5670 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.691385  5652 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:20.905771  5776 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.009s	sys 0.012s
W20260809 23:22:20.950395 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 203117us
I20260809 23:22:20.905978  5545 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:20.978289  5622 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.908226  5311 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:20.949291  5889 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:20.816270  5646 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.006986  5752 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.046037 24539 raft_consensus.cc:1226] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 30. Current term is 31. Ops: []
I20260809 23:22:20.830780  5571 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:21.023811  5873 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260809 23:22:20.957329  5891 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:20.879940  5818 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.081118  5818 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:21.049489  5776 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.010s	sys 0.012s
I20260809 23:22:21.035848  5882 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:20.894588  5791 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.042547  5673 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.007332  5122 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:20.922796  5718 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.056320  5890 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.076251  5923 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260809 23:22:21.088686  5656 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.065s	sys 0.086s
W20260809 23:22:21.076773  5875 log.cc:924] Time spent T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260809 23:22:21.085457  5280 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.052407  5311 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.056257 25363 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 27: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:21.056861  5829 log.cc:924] Time spent T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:22:20.978468  5648 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:22:21.094434 25365 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2100) took 2569 ms (client timeout 1866 ms). Trace:
I20260809 23:22:21.047029  5722 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:20.957175  5666 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:21.076493  5862 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.518s	user 0.001s	sys 0.001s
I20260809 23:22:21.073047  5714 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.077999  5571 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:20.957828  5194 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.075526  5891 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:21.078989  5816 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:21.056061  5898 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.095027  5885 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:21.115327  5656 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.066s	sys 0.086s
I20260809 23:22:21.056560  5889 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.070901  5752 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.069731  5494 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.264137  5923 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:21.337123  5311 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.337507  5729 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:21.338977  5931 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:21.088591  5882 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.049760  5545 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.105546  5673 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.325623  5900 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:21.342370  5878 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.466s	user 0.000s	sys 0.001s
I20260809 23:22:20.948077  5874 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.111747  5890 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.286372  5280 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:21.358485  5751 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.779s	user 0.010s	sys 0.005s
I20260809 23:22:21.345404  5670 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:20.921986 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:20.920676 24309 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:21.085397 24428 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1829) took 2209 ms. Trace:
W20260809 23:22:21.111469 25321 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.018s	sys 0.022s
W20260809 23:22:21.794237 24545 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.181s	user 0.021s	sys 0.040s
W20260809 23:22:21.372648 25365 rpcz_store.cc:269] 0809 23:22:18.525141 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.530287 (+  5146us) service_pool.cc:227] Handling call
0809 23:22:21.094224 (+2563937us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:21.094363 (+   139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2838344576}
W20260809 23:22:21.815237 24545 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.203s	user 0.022s	sys 0.046s
I20260809 23:22:21.816095 25363 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 1955) took 1069 ms. Trace:
I20260809 23:22:21.817260 24545 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3973) took 1250 ms. Trace:
I20260809 23:22:21.816808 25363 rpcz_store.cc:276] 0809 23:22:20.746322 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.748701 (+  2379us) service_pool.cc:227] Handling call
0809 23:22:21.815276 (+1066575us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54006027d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:21.815652 (+   376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381726080}
I20260809 23:22:21.360514 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2107) took 2225 ms. Trace:
W20260809 23:22:21.819257 25321 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.021s	sys 0.025s
I20260809 23:22:21.818852 25366 rpcz_store.cc:276] 0809 23:22:19.135626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.234498 (+ 98872us) service_pool.cc:227] Handling call
0809 23:22:20.898621 (+1664123us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:20.979224 (+ 80603us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.360295 (+381071us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400717850. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:21.360446 (+   151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3826252032}
I20260809 23:22:21.793781 24539 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3964) took 1282 ms. Trace:
W20260809 23:22:21.746027 24423 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s	user 0.002s	sys 0.010s
W20260809 23:22:21.822208 24423 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.003s	sys 0.019s
W20260809 23:22:21.784371 24496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.000s	sys 0.003s
W20260809 23:22:21.823894 24496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.002s	sys 0.015s
I20260809 23:22:21.821496 24539 rpcz_store.cc:276] 0809 23:22:20.511674 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.514289 (+  2615us) service_pool.cc:227] Handling call
0809 23:22:20.514556 (+   267us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:21.062767 (+548211us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.792932 (+730165us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b54000ee7d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:21.793468 (+   536us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1211360000}
I20260809 23:22:21.764284 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1836) took 2622 ms. Trace:
I20260809 23:22:21.678524 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:21.826124 24431 rpcz_store.cc:276] 0809 23:22:19.141456 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.143187 (+  1731us) service_pool.cc:227] Handling call
0809 23:22:19.143562 (+   375us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:19.143786 (+   224us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:19.143821 (+    35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:19.144377 (+   556us) log.cc:842] Serialized 26 byte log entry
0809 23:22:19.815588 (+671211us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:19.815648 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.815746 (+    98us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.763623 (+1947877us) raft_consensus.cc:1671] finished
0809 23:22:21.763669 (+    46us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.764035 (+   366us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":670765,"threads_started":1,"wal-append.queue_time_us":1599456}
I20260809 23:22:21.828881 24311 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:21.829592 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2087) took 1220 ms. Trace:
I20260809 23:22:21.699013 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2084) took 1090 ms. Trace:
I20260809 23:22:21.843112 24321 rpcz_store.cc:276] 0809 23:22:20.608107 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.609825 (+  1718us) service_pool.cc:227] Handling call
0809 23:22:20.648795 (+ 38970us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:21.678822 (+1030027us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:21.678954 (+   132us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:21.679019 (+    65us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:21.679080 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.698486 (+ 19406us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003ca0d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:21.698556 (+    70us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.698801 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2366307072}
W20260809 23:22:21.747185 24376 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.215s	user 0.009s	sys 0.006s
W20260809 23:22:21.844877 24376 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.313s	user 0.009s	sys 0.017s
I20260809 23:22:21.679883 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3295) took 2907 ms. Trace:
I20260809 23:22:21.847187 24317 rpcz_store.cc:276] 0809 23:22:18.772690 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.903600 (+130910us) service_pool.cc:227] Handling call
0809 23:22:18.955094 (+ 51494us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:18.990864 (+ 35770us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.990910 (+    46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:19.009317 (+ 18407us) log.cc:842] Serialized 52 byte log entry
0809 23:22:19.781750 (+772433us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:19.860126 (+ 78376us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.880795 (+ 20669us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.881077 (+   282us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.881176 (+    99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.435394 (+1554218us) spinlock_profiling.cc:243] Waited 7.68 ms on lock 0x7b180018ac00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:21.679085 (+243691us) raft_consensus.cc:1671] finished
0809 23:22:21.679149 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.679630 (+   481us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":67034,"prepare.queue_time_us":189481,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":10886,"raft.queue_time_us":94915,"raft.run_cpu_time_us":432,"raft.run_wall_time_us":10440,"spinlock_wait_cycles":17653632,"thread_start_us":719990,"threads_started":1,"wal-append.queue_time_us":2029438}
I20260809 23:22:21.851123 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
W20260809 23:22:21.671454 24266 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.001s	sys 0.010s
I20260809 23:22:21.785854 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2002) took 3234 ms. Trace:
W20260809 23:22:21.678007 24320 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.030s	user 0.006s	sys 0.008s
W20260809 23:22:21.672870 24314 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.000s	sys 0.015s
W20260809 23:22:21.679626 24323 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.063s	user 0.011s	sys 0.004s
I20260809 23:22:21.672468 24313 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:22:21.747255 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.196s	user 0.015s	sys 0.059s
I20260809 23:22:21.824568 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3569) took 2095 ms. Trace:
I20260809 23:22:21.755169 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1841) took 1224 ms. Trace:
I20260809 23:22:21.818732 24428 rpcz_store.cc:276] 0809 23:22:18.876320 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.889423 (+ 13103us) service_pool.cc:227] Handling call
0809 23:22:18.889694 (+   271us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.891091 (+  1397us) raft_consensus.cc:1520] Early marking committed up to index 17
0809 23:22:18.915376 (+ 24285us) log.cc:842] Serialized 14 byte log entry
0809 23:22:20.620706 (+1705330us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:20.621135 (+   429us) log.cc:842] Serialized 26 byte log entry
0809 23:22:20.621283 (+   148us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:20.621315 (+    32us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:20.621388 (+    73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.085134 (+463746us) raft_consensus.cc:1671] finished
0809 23:22:21.085163 (+    29us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.085325 (+   162us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32950,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":124737,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":2326,"thread_start_us":1704841,"threads_started":2,"wal-append.queue_time_us":934210}
W20260809 23:22:21.793962 25323 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.010s	sys 0.011s
W20260809 23:22:21.820051 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 1.21958e+06us
I20260809 23:22:21.835403 24311 rpcz_store.cc:276] 0809 23:22:20.609070 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.920186 (+311116us) service_pool.cc:227] Handling call
0809 23:22:21.829506 (+909320us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:21.690853 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:22:21.823185 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2093) took 1187 ms. Trace:
I20260809 23:22:21.746613 24380 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:21.763413 24430 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.003s	sys 0.011s
I20260809 23:22:21.837224 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1945) took 3551 ms. Trace:
I20260809 23:22:21.837663 25369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1954) took 1091 ms. Trace:
I20260809 23:22:21.760705 24167 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:21.817909 24545 rpcz_store.cc:276] 0809 23:22:20.567113 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.567611 (+   498us) service_pool.cc:227] Handling call
0809 23:22:20.569462 (+  1851us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:20.569693 (+   231us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:20.569723 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:20.612313 (+ 42590us) log.cc:842] Serialized 26 byte log entry
0809 23:22:21.816453 (+1204140us) raft_consensus.cc:1631] Marking committed up to 12
0809 23:22:21.816594 (+   141us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.816782 (+   188us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.816871 (+    89us) raft_consensus.cc:1671] finished
0809 23:22:21.816894 (+    23us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.817188 (+   294us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1203163,"threads_started":1,"wal-append.queue_time_us":856289}
W20260809 23:22:21.803994 25325 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.009s	sys 0.013s
W20260809 23:22:21.675861 24269 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.010s	user 0.003s	sys 0.011s
I20260809 23:22:21.446751  5122 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:20.889848  5166 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:21.370334  5278 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.009s	sys 0.014s
I20260809 23:22:21.855211  5311 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.855930  5545 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.400724  5622 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.856724  5494 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:21.407289  5643 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.447099  5646 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:21.372577  5648 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:21.900449  5729 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.901595  5648 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:21.364673  5652 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.903343  5311 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.903642  5166 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.905272  5673 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:21.676710 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 756036us
I20260809 23:22:21.906164  5643 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.817757 24316 raft_consensus.cc:2379] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 23: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:21.874996 24266 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.228s	user 0.007s	sys 0.019s
I20260809 23:22:21.424650  5813 consensus_queue.cc:260] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:21.444168  5852 log.cc:924] Time spent T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 1.492s	user 0.007s	sys 0.024s
W20260809 23:22:21.432636  5904 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.410s	user 0.004s	sys 0.008s
W20260809 23:22:21.426075  5850 log.cc:924] Time spent T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.403s	user 0.002s	sys 0.009s
I20260809 23:22:21.364611  5930 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:21.437039  5857 log.cc:924] Time spent T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.430s	user 0.001s	sys 0.000s
W20260809 23:22:21.439988  5909 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260809 23:22:21.868583 24264 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.069s	user 0.013s	sys 0.023s
I20260809 23:22:21.866654 24424 rpcz_store.cc:276] 0809 23:22:20.530683 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.531012 (+   329us) service_pool.cc:227] Handling call
0809 23:22:20.531387 (+   375us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:20.906957 (+375570us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.754219 (+847262us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b540012f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:21.754842 (+   623us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":793125632}
W20260809 23:22:21.882511 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.331s	user 0.020s	sys 0.065s
W20260809 23:22:21.927706 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 181835us
W20260809 23:22:21.877578 24314 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.301s	user 0.000s	sys 0.029s
I20260809 23:22:21.860044  5280 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:21.881186 24323 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.264s	user 0.014s	sys 0.015s
W20260809 23:22:21.885644 24430 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.006s	sys 0.017s
I20260809 23:22:21.911054 24321 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2085) took 1302 ms. Trace:
W20260809 23:22:21.882193 24269 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.013s	sys 0.013s
W20260809 23:22:21.884208 24320 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.248s	user 0.006s	sys 0.022s
I20260809 23:22:21.957880 24320 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2083) took 2795 ms. Trace:
I20260809 23:22:21.868995 25369 rpcz_store.cc:276] 0809 23:22:20.745973 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.748086 (+  2113us) service_pool.cc:227] Handling call
0809 23:22:20.748494 (+   408us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:21.065053 (+316559us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:21.065097 (+    44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:21.065151 (+    54us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:21.065189 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.837061 (+771872us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b540065c554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:21.837150 (+    89us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.837427 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":727372672}
I20260809 23:22:21.890393  5903 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:21.896335  5906 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:21.883118  5816 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.897840  5898 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.864024 24534 rpcz_store.cc:276] 0809 23:22:18.551442 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.551632 (+   190us) service_pool.cc:227] Handling call
0809 23:22:18.552022 (+   390us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:19.323983 (+771961us) raft_consensus.cc:1520] Early marking committed up to index 18
0809 23:22:19.363939 (+ 39956us) log.cc:842] Serialized 14 byte log entry
0809 23:22:19.966070 (+602131us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:20.351909 (+385839us) log.cc:842] Serialized 26 byte log entry
0809 23:22:20.475577 (+123668us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:20.475629 (+    52us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:20.475725 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.784976 (+1309251us) raft_consensus.cc:1671] finished
0809 23:22:21.785044 (+    68us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.785576 (+   532us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14457,"prepare.queue_time_us":40503,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":38875,"raft.run_cpu_time_us":11367,"raft.run_wall_time_us":517056,"thread_start_us":1366732,"threads_started":2,"wal-append.queue_time_us":1454189}
W20260809 23:22:21.898664 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 855964us
I20260809 23:22:21.868813 25370 rpcz_store.cc:276] 0809 23:22:18.285836 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.285963 (+   127us) service_pool.cc:227] Handling call
0809 23:22:18.303140 (+ 17177us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:18.326229 (+ 23089us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.326260 (+    31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:18.326945 (+   685us) log.cc:842] Serialized 52 byte log entry
0809 23:22:18.735258 (+408313us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:18.755447 (+ 20189us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.758478 (+  3031us) log.cc:842] Serialized 14 byte log entry
0809 23:22:18.761968 (+  3490us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:18.762033 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:18.867805 (+105772us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b18003d1d40. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:20.531161 (+1663356us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:21.836568 (+1305407us) raft_consensus.cc:1671] finished
0809 23:22:21.836622 (+    54us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.836989 (+   367us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88269,"prepare.queue_time_us":64782,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":42113,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":23704,"spinlock_wait_cycles":55906432,"thread_start_us":375743,"threads_started":1,"wal-append.queue_time_us":970399}
I20260809 23:22:21.896317 24313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2086) took 1287 ms. Trace:
I20260809 23:22:21.896251 24317 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:21.869081 24309 rpcz_store.cc:276] 0809 23:22:20.635406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.817267 (+181861us) service_pool.cc:227] Handling call
0809 23:22:21.822897 (+1005630us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:21.834697 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:21.865312 24423 rpcz_store.cc:276] 0809 23:22:19.728695 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.737504 (+  8809us) service_pool.cc:227] Handling call
0809 23:22:19.739540 (+  2036us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:20.658433 (+918893us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:20.658479 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:20.705482 (+ 47003us) log.cc:842] Serialized 26 byte log entry
0809 23:22:21.823979 (+1118497us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:21.824071 (+    92us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.824220 (+   149us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.824285 (+    65us) raft_consensus.cc:1671] finished
0809 23:22:21.824297 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.824499 (+   202us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":198392,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":11753,"thread_start_us":1849734,"threads_started":2,"wal-append.queue_time_us":709116}
W20260809 23:22:21.877694 25323 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.011s	sys 0.020s
I20260809 23:22:21.445853  5718 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.439329  5884 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:21.947031  5813 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:21.931500  5900 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.437419  5791 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:21.444857  5928 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:21.947866  5670 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.400501  5723 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:21.948684  5643 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:21.943189 24316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2088) took 1333 ms. Trace:
W20260809 23:22:21.909118  5278 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.011s	sys 0.016s
W20260809 23:22:21.953248 24264 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.154s	user 0.013s	sys 0.024s
W20260809 23:22:21.957978 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 1.32221e+06us
I20260809 23:22:21.906818  5166 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.921847  5666 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.906736  5311 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:21.946015 24321 rpcz_store.cc:276] 0809 23:22:20.608467 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:21.860121 (+1251654us) service_pool.cc:227] Handling call
0809 23:22:21.860775 (+   654us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:21.910622 (+ 49847us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:21.910659 (+    37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:21.910701 (+    42us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:21.910739 (+    38us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.910803 (+    64us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.910959 (+   156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2837,"raft.queue_time_us":3516,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":184}
I20260809 23:22:21.936451  5883 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:21.993419  5918 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:21.908119 24167 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 22, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:21.958417 24320 rpcz_store.cc:276] 0809 23:22:19.162609 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.483068 (+320459us) service_pool.cc:227] Handling call
0809 23:22:19.506760 (+ 23692us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:19.586166 (+ 79406us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:19.586210 (+    44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:19.604246 (+ 18036us) log.cc:842] Serialized 52 byte log entry
0809 23:22:20.583657 (+979411us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:20.584039 (+   382us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.957211 (+1373172us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.957376 (+   165us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.957469 (+    93us) raft_consensus.cc:1671] finished
0809 23:22:21.957491 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.957782 (+   291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49630,"prepare.queue_time_us":109989,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":400126,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":46966,"thread_start_us":2248126,"threads_started":2,"wal-append.queue_time_us":1728688}
I20260809 23:22:21.994458  5611 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:21.994284  5918 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.995702 24309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2090) took 1361 ms. Trace:
I20260809 23:22:21.995280  5611 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.995234 24167 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 23, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 23 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:21.996048 24309 rpcz_store.cc:276] 0809 23:22:20.634360 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:21.994522 (+1360162us) service_pool.cc:227] Handling call
0809 23:22:21.995067 (+   545us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:21.995385 (+   318us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:21.995409 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:21.995462 (+    53us) raft_consensus.cc:1631] Marking committed up to 18
0809 23:22:21.995488 (+    26us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.995533 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.995632 (+    99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:21.996773  5918 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.998912  5874 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.999415  5611 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.994047  5816 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.002118  5928 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.003646 24314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40860 (request call id 3864) took 2902 ms. Trace:
I20260809 23:22:22.004846  5928 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.004115 24314 rpcz_store.cc:276] 0809 23:22:19.100926 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.321606 (+220680us) service_pool.cc:227] Handling call
0809 23:22:19.562667 (+241061us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:19.569615 (+  6948us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:19.702064 (+132449us) spinlock_profiling.cc:243] Waited 7.18 ms on lock 0x7b480015c258. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:22:20.576417 (+874353us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.946871 (+1370454us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:21.948196 (+  1325us) log.cc:842] Serialized 52 byte log entry
0809 23:22:21.948442 (+   246us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:21.976690 (+ 28248us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.982733 (+  6043us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.983236 (+   503us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:22.002816 (+ 19580us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:22.003036 (+   220us) raft_consensus.cc:1671] finished
0809 23:22:22.003053 (+    17us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:22.003584 (+   531us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8317,"prepare.queue_time_us":186125,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":18145,"raft.run_cpu_time_us":3537,"raft.run_wall_time_us":129109,"spinlock_wait_cycles":17772032,"thread_start_us":2332058,"threads_started":3,"wal-append.queue_time_us":830697}
W20260809 23:22:22.008359 24542 consensus_meta.cc:287] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.432s	user 0.015s	sys 0.015s
I20260809 23:22:22.009071 24542 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:22.009827 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 3972) took 1443 ms. Trace:
I20260809 23:22:22.010169 24542 rpcz_store.cc:276] 0809 23:22:20.566750 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.567796 (+  1046us) service_pool.cc:227] Handling call
0809 23:22:22.009692 (+1441896us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:22.011557 24311 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:21.966545 24313 rpcz_store.cc:276] 0809 23:22:20.608769 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.791406 (+182637us) service_pool.cc:227] Handling call
0809 23:22:21.895984 (+1104578us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400225e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:21.896219 (+   235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2022304128}
I20260809 23:22:21.969532 24317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51376 (request call id 2092) took 1332 ms. Trace:
I20260809 23:22:21.958691  5930 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.907128  5923 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.925625  5885 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.932260  5931 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.983999  5681 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:21.985373  5796 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:22.017306  5681 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.018554  5931 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.018647  5885 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.911073  5714 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:21.989013  5882 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.989255  5890 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:21.978996 24316 rpcz_store.cc:276] 0809 23:22:20.609360 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.830830 (+221470us) service_pool.cc:227] Handling call
0809 23:22:21.943063 (+1112233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:21.986624 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1847) took 1434 ms. Trace:
I20260809 23:22:21.931835  5729 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.012212  5166 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.012328  5889 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:21.924144  5752 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.013386 24317 rpcz_store.cc:276] 0809 23:22:20.635118 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:21.850586 (+1215468us) service_pool.cc:227] Handling call
0809 23:22:21.967215 (+116629us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:21.961171  5903 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:21.998202 24167 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 18 to 21, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.004329  5311 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:21.984851  5906 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.015482 24311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2091) took 1380 ms. Trace:
W20260809 23:22:22.016410 24498 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.003s	sys 0.016s
W20260809 23:22:22.023286 24495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.006s	sys 0.015s
W20260809 23:22:22.034138 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53667.9us
I20260809 23:22:22.034690 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3267) took 3367 ms. Trace:
I20260809 23:22:22.034614  5816 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.026135  5643 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.043867 24311 rpcz_store.cc:276] 0809 23:22:20.634740 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.010449 (+1375709us) service_pool.cc:227] Handling call
0809 23:22:22.011030 (+   581us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:22.014291 (+  3261us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:22.015404 (+  1113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:22:22.041247 24323 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:51376 (request call id 2081) took 2942 ms. Trace:
W20260809 23:22:22.046881 24498 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.941s	user 0.004s	sys 0.016s
I20260809 23:22:22.056336  5930 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:22.059259  5816 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.062250  5816 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.063136  5884 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.066241  5884 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.068384  5898 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.091392  5898 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.092182  5898 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.093852  5681 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.093158 24167 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 24. New cstate: current_term: 24 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:22.070539 24495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.007s	sys 0.016s
I20260809 23:22:22.095626 24167 catalog_manager.cc:5495] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 22 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:22.044015 24312 rpcz_store.cc:276] 0809 23:22:18.665649 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.665785 (+   136us) service_pool.cc:227] Handling call
0809 23:22:18.666093 (+   308us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:18.674884 (+  8791us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:18.674945 (+    61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:18.687179 (+ 12234us) log.cc:842] Serialized 26 byte log entry
0809 23:22:18.757395 (+ 70216us) spinlock_profiling.cc:243] Waited 6.4 ms on lock 0x7b180050f560. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:19.957665 (+1200270us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:19.957730 (+    65us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:19.957826 (+    96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.425417 (+1467591us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:22.024768 (+599351us) raft_consensus.cc:1671] finished
0809 23:22:22.024856 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:22.033238 (+  8382us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8369,"raft.queue_time_us":9096,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":20457,"spinlock_wait_cycles":54509184,"thread_start_us":1200001,"threads_started":1,"wal-append.queue_time_us":2335633}
I20260809 23:22:22.041875  5311 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.099951 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.100409  5311 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:22.101140 24545 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:22.103868 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.104890 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:22.105477 24428 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.107478 24435 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 27. Current term is 28. Ops: []
I20260809 23:22:22.109984 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.110177 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.111640 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.031873 24430 rpcz_store.cc:276] 0809 23:22:20.551910 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:20.552490 (+   580us) service_pool.cc:227] Handling call
0809 23:22:20.552774 (+   284us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:20.552975 (+   201us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:21.046534 (+493559us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.986001 (+939467us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:21.986126 (+   125us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:21.986248 (+   122us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.986370 (+   122us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:21.986558 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35973,"prepare.queue_time_us":786490,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":30562,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":7584,"thread_start_us":1343024,"threads_started":2,"wal-append.queue_time_us":427712}
I20260809 23:22:22.070657  5923 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.071249  5951 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.114329  5946 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.115097  5946 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.075187  5166 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.117372  5946 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:22.076189 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 970887us
I20260809 23:22:22.031450  5670 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.080251  5900 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.118966 24545 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260809 23:22:22.082697  5903 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.120640  5670 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.083280  5643 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.121510 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.122525  5903 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.122884  5903 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.123271  5643 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.086149  5729 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.124431  5729 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.124891  5900 consensus_queue.cc:1046] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:22.126155  5681 raft_consensus.cc:3032] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Stepping down as leader of term 30
I20260809 23:22:22.126569  5681 raft_consensus.cc:729] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:22.087711  5918 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.127344  5918 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.127779  5681 consensus_queue.cc:260] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.129660  5918 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.048840 24323 rpcz_store.cc:276] 0809 23:22:19.098574 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:19.262025 (+163451us) service_pool.cc:227] Handling call
0809 23:22:19.262582 (+   557us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:19.314626 (+ 52044us) raft_consensus.cc:1520] Early marking committed up to index 15
0809 23:22:19.705290 (+390664us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x7b1800513400. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f 00007fac23879e88
0809 23:22:20.616689 (+911399us) log.cc:842] Serialized 14 byte log entry
0809 23:22:21.981188 (+1364499us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:21.984579 (+  3391us) log.cc:842] Serialized 26 byte log entry
0809 23:22:21.984877 (+   298us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:21.984924 (+    47us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:21.986657 (+  1733us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:21.990612 (+  3955us) raft_consensus.cc:1671] finished
0809 23:22:22.016123 (+ 25511us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:22.041155 (+ 25032us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52041,"prepare.queue_time_us":428511,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":65467,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":36419,"spinlock_wait_cycles":168215296,"thread_start_us":2240630,"threads_started":2,"wal-append.queue_time_us":789712}
I20260809 23:22:22.089108  5545 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.132040  5545 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.133785 24540 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:22.134308  5545 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.089742  5890 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.134805  5681 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:22.135366  5890 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.136665 24424 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.137665  5890 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.090269  5883 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.138412  5883 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.140558  5883 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.141999  5891 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.146194 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.123273  5903 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.088641  5906 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.152228 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.152942 24424 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.114652  5951 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.155359 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.156737  5951 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.079521  5714 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.157099 24544 raft_consensus.cc:1261] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:22.126672  5729 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.117789  5166 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.162940 24270 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:22.163278  5729 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.163990  5729 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.165041  5889 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:22.165609  5889 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:22.165788 24168 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 345) took 41714 ms (client timeout 15000 ms). Trace:
I20260809 23:22:22.166297  5729 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.150751 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.151296  5752 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.094769  5898 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.097741 24167 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.186512 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.185154  5666 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.180477  5796 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.217257 24167 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 24, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 24 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:22.179358  5946 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:22.218291  5796 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.220036  5752 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.187078  5882 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.223659  5752 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.181563 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:22.229727 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 395388us
I20260809 23:22:22.218319 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
W20260809 23:22:22.234134 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 52701.1us
I20260809 23:22:22.191402  5611 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.166827  5889 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.223081  5882 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.203676 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.186865 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.192601 24312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:22.215560 24425 raft_consensus.cc:2454] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:22.227275 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 44943.4us
I20260809 23:22:22.225914 24424 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:22.191075 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.198963 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:22.252460 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 170100us
I20260809 23:22:22.225567  5891 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.260093 24430 raft_consensus.cc:2396] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260809 23:22:22.191789  5887 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.269917 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.229885 24167 catalog_manager.cc:5495] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.267241 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.231148  5796 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.272975 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:22.276098  5656 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.276800 24431 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.279150 24312 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:22.217262  5666 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.283392  5670 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.284492 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.214396  5648 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.285868 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.287056 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:22.287698 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.288822  5646 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.289399  5311 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.298420 25366 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:22.299256 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.300071 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:22.300323 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.300627  5882 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.302623 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:22.166378 24168 rpcz_store.cc:269] 0809 23:21:40.501670 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:40.595384 (+ 93714us) service_pool.cc:227] Handling call
0809 23:22:22.165490 (+41570106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:14.918678 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:14.919374 (+   696us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:14.919438 (+    64us) write_op.cc:435] Acquired schema lock
0809 23:22:14.919448 (+    10us) tablet.cc:660] Decoding operations
0809 23:22:14.920571 (+  1123us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:14.920665 (+    94us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:14.920708 (+    43us) tablet.cc:683] Acquiring locks for 11 operations
0809 23:22:14.921755 (+  1047us) tablet.cc:699] Row locks acquired
0809 23:22:14.921785 (+    30us) write_op.cc:260] PREPARE: finished
0809 23:22:14.921949 (+   164us) write_op.cc:270] Start()
0809 23:22:14.922093 (+   144us) write_op.cc:276] Timestamp: P: 1786317734921886 usec, L: 0
0809 23:22:14.922128 (+    35us) op_driver.cc:331] REPLICATION: starting
0809 23:22:14.922794 (+   666us) log.cc:842] Serialized 4365 byte log entry
0809 23:22:15.989470 (+1066676us) op_driver.cc:444] REPLICATION: finished
0809 23:22:17.238305 (+1248835us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac2091a5fe 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac27270410
0809 23:22:18.204737 (+966432us) write_op.cc:301] APPLY: starting
0809 23:22:18.204888 (+   151us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:18.626152 (+421264us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:18.626198 (+    46us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:19.580032 (+953834us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:19.580512 (+   480us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0809 23:22:19.580595 (+    83us) write_op.cc:312] APPLY: finished
0809 23:22:19.581084 (+   489us) log.cc:842] Serialized 105 byte log entry
0809 23:22:21.081861 (+1500777us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:21.082026 (+   165us) write_op.cc:454] Released schema lock
0809 23:22:21.082096 (+    70us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:21.082470 (+   374us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:21.082507 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1450550,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":2148886,"apply.run_cpu_time_us":43023,"apply.run_wall_time_us":2878278,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50063,"num_ops":11,"prepare.queue_time_us":882718,"prepare.run_cpu_time_us":15332,"prepare.run_wall_time_us":540559,"raft.queue_time_us":138811,"raft.run_cpu_time_us":29936,"raft.run_wall_time_us":2000987,"replication_time_us":1067227,"spinlock_wait_cycles":12867328,"thread_start_us":4759044,"threads_started":5,"wal-append.queue_time_us":2522660,"wal-append.run_cpu_time_us":31042,"wal-append.run_wall_time_us":1326652}]]}
I20260809 23:22:22.310770 24545 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:22.305680 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.257647  5951 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:22.305759 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.319109 24540 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:22.321615 24544 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.273406  5643 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.311239  5813 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:22.330240 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 45934.4us
I20260809 23:22:22.331458 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.328963 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:22.333367 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.333945 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.334079 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:22.335503 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.336143 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.336789  5643 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.337294 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.338402 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.333359 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.338833  5903 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.339107  5646 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.339787 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.340271 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.340402 24314 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:22.341043  5903 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.341346 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.341742 24544 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:22.341579  5670 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.343101 24435 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:22.343472 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.343919  5670 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:22.344408 24309 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:22.344671  5887 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.345715 24309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.328886  5796 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260809 23:22:22.346416 24309 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:22.347237 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.347368 24320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.347514 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.347801 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.347949 24540 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:22.348438 25373 raft_consensus.cc:2354] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
W20260809 23:22:22.349299 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 49379us
I20260809 23:22:22.349354 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.350073  5311 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:22.350282 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.350765 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.351038  5311 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:22.351737  5311 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:22.351795 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.352689 24545 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:22.353083  5194 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:22.316051  5611 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.354285 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.351046  5885 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:22.335538 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 91551.7us
I20260809 23:22:22.355489  5813 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.348455 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.359629 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.360169 24321 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:22.355144 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.360888 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.361104 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.361946 24321 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:22.361994 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.362085 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.322551  5891 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.362977 24316 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:22.363524  5891 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.364004 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.364406 24313 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:22.365932  5891 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.346720 24167 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 24, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 24 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.351624 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.367081  5656 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.368817 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.369448  5656 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:22.369693 24323 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:22.370064 24496 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.372095  5885 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:22.372529 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.372648  5885 raft_consensus.cc:686] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:22.373235 24423 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 22.
I20260809 23:22:22.373821  5885 consensus_queue.cc:237] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.370200 25372 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:22.334726 25371 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.334323 24427 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260809 23:22:22.376075 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.377243 24428 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 29.
I20260809 23:22:22.341445  5646 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.369710 25375 raft_consensus.cc:2379] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 29: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:22.359521 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.378340 24321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.367857 24167 catalog_manager.cc:5495] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 26. New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:22.345741 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 58841.5us
I20260809 23:22:22.383438 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.378815 24546 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 20.
I20260809 23:22:22.387336 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.387691 24539 raft_consensus.cc:2396] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260809 23:22:22.390283 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:22.391029 24535 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:22.347998 24320 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 28.
I20260809 23:22:22.388305 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.393321 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.378790 24543 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:22.396201 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.396683 24424 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260809 23:22:22.397080 24546 raft_consensus.cc:2379] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:22.397420 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:22.397902 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.398063 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.399067 24317 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:22.403571 24311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:22.404551 24311 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.407446 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.403601 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.412302 24323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.351058  5648 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.411958 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:22.421206 24435 raft_consensus.cc:3037] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.419840 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.422338 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.423894 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:22.424871 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 74712.6us
I20260809 23:22:22.425244 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.426430 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.427202  5887 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.428717  5670 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:22.419605 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.432827 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.413470 24167 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 23, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.408329 24313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21" is_pre_election: true
I20260809 23:22:22.436007 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.420256 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.433192 25365 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.439101 24314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "60d5c69728134c8bb918a900eb6a8c21"
I20260809 23:22:22.412338  5903 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:22.446687 24541 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:22.408991  5643 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.444959 24427 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:22.421231  5951 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:22.445560 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.444548 24323 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:22.445468 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.458902 24545 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.450680 24316 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 29.
W20260809 23:22:22.453154 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94641.2us
I20260809 23:22:22.454092 24167 catalog_manager.cc:5495] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.455106  5611 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.394236  5891 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.044s
I20260809 23:22:22.437139  5656 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:22.477311 24543 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260809 23:22:22.477842 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.478510 24431 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:22.479599 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:22.483291 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.486019 25373 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:22.487243 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.488071 25366 raft_consensus.cc:2379] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:22.488941 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.489606 25363 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:22.489782 24312 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:22.490219 25371 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:22.463899 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.465529 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.465768  5311 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.458402 24534 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:22.465363 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.473210  5813 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.487188 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.471019 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.509585 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.493019 25373 raft_consensus.cc:2454] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 20.
W20260809 23:22:22.511082 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47475.2us
I20260809 23:22:22.516796 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.447950  5903 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.518440 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.519040 24380 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.526129 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.514283 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:22.527663 24380 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.477418  5656 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.524868 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:22.530097 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 200961us
I20260809 23:22:22.536132 24540 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:22.529784 25375 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:22.528132 24167 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 28. New cstate: current_term: 28 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.541483 25369 raft_consensus.cc:2379] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:22.525436  5903 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:22.547230 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:22.529263  5813 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260809 23:22:22.555622 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 156760us
I20260809 23:22:22.541232  5656 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:22.539197  5643 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.559094  5801 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.559962  5801 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.560442  5801 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:22.561275 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 90445.5us
I20260809 23:22:22.553563 24380 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.564416 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:22.573520 24546 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:22.559201  5643 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.566259 24435 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:22.579998 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.552987  5891 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.559104  5952 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:22.550062 24167 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:22.551568  5813 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.598806 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.595268 24311 raft_consensus.cc:2454] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:22.592468  5166 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:22.612342 24430 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:22:22.607470 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.608273 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.608453  5571 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:22.588792  5891 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.626603  5891 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.627121 25368 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:22.627580 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.628432  5801 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.629096 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:22.630523  5801 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.631469  5280 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:22.631805  5801 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:22.632208  5801 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.632583  5801 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:22.633580  5903 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.609488 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.610333  5965 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:22.581233  5643 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.636574  5965 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.611217 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.601686 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.640105 24424 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
I20260809 23:22:22.641440 24266 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.587589  5681 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:22.643204  5681 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.643617  5681 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:22.642818  5278 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.645684  5278 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.584682  5796 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:22.565711 24378 connection.cc:232] RPC call timeout handler was delayed by -1.2955258809961379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
I20260809 23:22:22.646658  5278 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.581496  5666 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:22.613723  5813 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:22.622684  5952 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:22.623277  5887 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:22.625635 24270 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:22.642653 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.653704 24167 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 24, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:22.659600 24431 raft_consensus.cc:2454] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
I20260809 23:22:22.647787 24378 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.666838 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.667622 24430 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:22.638767  5965 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:22.673520 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 47984.6us
I20260809 23:22:22.674595 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:22.677774 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
W20260809 23:22:22.679097 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 143980us
I20260809 23:22:22.655800  5918 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.681494 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:22.684267 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 125357us
I20260809 23:22:22.698643  5889 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:22.699153  5889 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:22.699594  5889 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:22.701609 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.705299 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.686784 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:22.711239 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63545.1us
I20260809 23:22:22.712307 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.713411 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:22.714087  5751 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:22.714625  5751 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.714996  5751 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:22.676371  5898 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:22.715123  5965 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.715639  5965 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:22.715807  5898 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:22.716106  5965 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.716446  5898 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:22.717473  5898 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.676733  5666 raft_consensus.cc:2741] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260809 23:22:22.724541  5946 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:22.725005  5718 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:22.725695  5718 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.719447 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.727885  5718 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:22.727491  5952 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.728477  5718 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:22.728857  5718 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.729231  5718 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:22.731299  5903 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:22.733093 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46426.7us
I20260809 23:22:22.734018 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.734196  5957 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.734632  5957 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.734989  5957 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.676956  5796 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.735852 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.737573  5796 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.738610 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.739511  5918 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.740002 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.740202  5961 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:22.741848  5918 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:22.682916  5882 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:22.658625  5571 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:22.755295 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.768447 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.769145 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.769827 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.771595 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:22.767023 24167 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:22.796226 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53330us
I20260809 23:22:22.803673 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.768186 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.806308 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.807512 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.807898 24435 raft_consensus.cc:2396] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260809 23:22:22.726500  5751 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.751055  5882 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:22.808151 25375 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
W20260809 23:22:22.814882 24498 connection.cc:232] RPC call timeout handler was delayed by -1.2417621079366654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
I20260809 23:22:22.816435 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.756232  5666 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:22.820676 24378 leader_election.cc:398] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:22.821040 24378 leader_election.cc:401] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:22.820825 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:22.821708 24544 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.817277 24540 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
W20260809 23:22:22.823261 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 55074.6us
I20260809 23:22:22.825058 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.825702 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.824993 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:22.825363 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:22.829015 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64807.4us
I20260809 23:22:22.830287 24378 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.826877 24546 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:22.833274 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:22.833972 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.835539 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.836812 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:22.837493 24435 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:22.835254 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:22.841536 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 166998us
I20260809 23:22:22.850766 25373 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:22.817739  5882 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:22.842286 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:22.848845 24167 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 25, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:22.793404  5903 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.043s
W20260809 23:22:22.889302 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 138879us
I20260809 23:22:22.764400  5801 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:22.848592  5666 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:22.897437 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52337.1us
I20260809 23:22:22.811168  5887 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260809 23:22:22.903740 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:22.909016 24424 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:22.910441 24167 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 26, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:22.913300 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 71099.4us
I20260809 23:22:22.914700 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.913004  5939 raft_consensus.cc:2781] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:22.924204 24167 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 23, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 23 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:22.926306 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:22.899475  5903 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:22.841197  5545 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:22.917038  5939 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:22.999827  5484 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:22:23.000316  5484 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.000710  5484 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:23.000833  5939 raft_consensus.cc:3037] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:23.002066 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.003123  5885 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:23.004215  5801 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.004832  5801 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:23.005167  5801 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:23.005494  5801 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:22.836192  5643 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.011621 24167 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 17 to 20, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 20 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:22.930439  5975 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260809 23:22:22.929397  5918 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:23.015590  5545 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:23.027258 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:22.967290  5752 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260809 23:22:23.030349 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 204348us
W20260809 23:22:23.039561 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 209318us
I20260809 23:22:23.043253 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:23.044529 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 160361us
I20260809 23:22:23.046607 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:23.048313 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46433.3us
I20260809 23:22:23.049810 25365 raft_consensus.cc:2379] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:23.055882 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:23.058430 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:23.066615 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 173437us
I20260809 23:22:23.025409  5545 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.041487  5752 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:23.069197 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:22.913179  5666 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.040658  5918 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:23.071308 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 237407us
I20260809 23:22:23.097069 24435 raft_consensus.cc:2454] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:23.014953  5957 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.088743 24496 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.064253  5946 raft_consensus.cc:465] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Not starting pre-election -- already a leader
I20260809 23:22:23.081864  5652 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:23.087731  5752 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:23.083397  5545 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:23.130633 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:23.155720 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46652.3us
W20260809 23:22:23.157629 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 99322.6us
I20260809 23:22:23.159580 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.148097 24534 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:23.158365 24496 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:23.134160  5951 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:23.130192  5652 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:23.122059  5571 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.127381  5903 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:23.179744 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 253511us
I20260809 23:22:23.092546  5918 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.193190 24496 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:23.149330  5670 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:23.182505 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:23.127844  5194 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:23.147863  5751 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:23.193212 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.197197 25321 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:23.182560 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.118618  5791 consensus_queue.cc:1035] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.277s
I20260809 23:22:23.194139  5652 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:23.187983  5951 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:23.231535  5656 consensus_meta.cc:287] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.674s	user 0.013s	sys 0.000s
I20260809 23:22:23.232081  5656 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.224080  5891 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.002s
I20260809 23:22:23.235363  5891 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.237444  5891 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.226540  5751 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:23.239207 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 46117.8us
I20260809 23:22:23.239667 24429 raft_consensus.cc:2379] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:23.241036  5311 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:23.241413 24496 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:23.241632  6007 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260809 23:22:23.242473  5194 raft_consensus.cc:2781] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:23.241429  5311 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:23.243295  5311 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:23.244496  5951 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:23.247754  5957 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.249588  5571 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:23.251667  5946 raft_consensus.cc:2781] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:23.252223  5946 raft_consensus.cc:686] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:23.253118  5946 consensus_queue.cc:237] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:23.257417  5989 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260809 23:22:23.258165 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.259052 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:23.259471  5990 log.cc:924] Time spent T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
W20260809 23:22:23.223323  5978 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.309s	user 0.001s	sys 0.000s
I20260809 23:22:23.267153  5656 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.261466 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.221990  5903 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:23.270165  5903 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.270651  5903 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:23.250000  5918 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:23.277382  5997 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260809 23:22:23.243301 25368 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:23.265139  5939 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.261749  6003 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:23.240566  5891 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:23.250666  5571 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:23.288136 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 224414us
I20260809 23:22:23.226784  5839 raft_consensus.cc:2781] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:23.312219 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:23.313706 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 143512us
I20260809 23:22:23.292919 24544 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
W20260809 23:22:23.298699  5889 consensus_meta.cc:287] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.591s	user 0.004s	sys 0.009s
I20260809 23:22:23.320541 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.326448 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.291253  5643 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.331256 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.349427 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.320312  5891 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:23.360071  5166 consensus_meta.cc:287] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.744s	user 0.005s	sys 0.007s
I20260809 23:22:23.359961  5891 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.361567  5681 consensus_meta.cc:287] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.717s	user 0.013s	sys 0.000s
I20260809 23:22:23.361945  5681 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.363917  5681 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.364583 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.309309  6005 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260809 23:22:23.368038 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 285423us
I20260809 23:22:23.360483  5166 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:23.297979  5194 raft_consensus.cc:686] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:23.310334  5571 raft_consensus.cc:3037] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:23.350723 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.375653 24311 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:23.372167 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58474.9us
I20260809 23:22:23.230211  5670 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.375712 24429 raft_consensus.cc:2454] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:23.392514 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.393306 24430 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:23.331943  5839 raft_consensus.cc:686] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:23.394609 24270 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:23.358547  5882 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.413918 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:23.392002  5813 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.000s
I20260809 23:22:23.368141  6005 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:23.432286 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 105951us
I20260809 23:22:23.433218 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.434336 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.390708  5791 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:23.438176 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.420585 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:23.418860 24429 raft_consensus.cc:2379] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 31: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:23.429683 24311 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:23.430068 24544 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:23.330227  5889 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:23.481247 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48140us
W20260809 23:22:23.419697 24269 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 2.851s (SENT) [suppressed 6 similar messages]
I20260809 23:22:23.358865  5311 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:435: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
I20260809 23:22:23.478675 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.482823 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.436054  6005 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:23.486609 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.486948 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
I20260809 23:22:23.487257  5959 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260809 23:22:23.488219  5993 log.cc:924] Time spent T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
I20260809 23:22:23.487830  5801 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.492897 24430 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:23.383841  5903 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.515775 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 274437us
I20260809 23:22:23.507028 24429 raft_consensus.cc:2379] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:23.510653 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 313493us
I20260809 23:22:23.508610  5646 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:23.510320  5887 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:23.583627  5801 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:23.514801  5965 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.798s	user 0.013s	sys 0.000s
I20260809 23:22:23.571179  6000 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.572680 25321 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.573217  6004 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:23.574075  6009 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260809 23:22:23.576088  5311 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.519025  5611 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:23.431536  5813 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.580227  5939 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.539031  5646 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:23.587134 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 75648.9us
I20260809 23:22:23.628423 24312 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:23.595412 24430 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:23.596429 24378 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.004s	sys 0.015s
W20260809 23:22:23.629412 24378 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.005s	sys 0.015s
W20260809 23:22:23.588063  5484 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.002s
W20260809 23:22:23.629705 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 573863us
I20260809 23:22:23.633074  5752 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.639685 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:23.645903 24431 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:23.620985  5796 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.019s
I20260809 23:22:23.602186  5887 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:23.635450  5646 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:23.614874  5643 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:23.533638  5951 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:23.660742  5796 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.006s	sys 0.019s
I20260809 23:22:23.686021 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.586902  5652 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.690122 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:23.690333 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.672762  5681 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:23.634611  5801 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260809 23:22:23.714970 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.626202  6000 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.639520  5611 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:23.719354 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 299669us
I20260809 23:22:23.603413  5965 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.681766  5280 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:23.726415 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 305890us
I20260809 23:22:23.639371  5484 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.736714 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.738590 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38092 (request call id 3992) took 1001 ms. Trace:
I20260809 23:22:23.736860 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.602639  5839 consensus_queue.cc:237] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.766969 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:23.716234  5889 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.603101  5194 consensus_queue.cc:237] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.801338 24428 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:23.729562  6007 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:23.745415 24535 rpcz_store.cc:276] 0809 23:22:22.737526 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.743251 (+  5725us) service_pool.cc:227] Handling call
0809 23:22:22.794523 (+ 51272us) raft_consensus.cc:1457] Updating replica for 2 ops
0809 23:22:22.816318 (+ 21795us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:22.816377 (+    59us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0809 23:22:22.882668 (+ 66291us) log.cc:842] Serialized 52 byte log entry
0809 23:22:23.062226 (+179558us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:23.062266 (+    40us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:23.062332 (+    66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:23.738352 (+676020us) raft_consensus.cc:1671] finished
0809 23:22:23.738376 (+    24us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:23.738531 (+   155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15460,"raft.queue_time_us":9100,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11727,"thread_start_us":152000,"threads_started":1,"wal-append.queue_time_us":313274}
I20260809 23:22:23.804179 24423 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:22:23.792273 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 55505.8us
W20260809 23:22:23.726063  5622 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.017s	sys 0.006s
I20260809 23:22:23.816903 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:23.813858  6007 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.735868  5681 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:23.820524  6015 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260809 23:22:23.822569  6033 consensus_queue.cc:1046] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
W20260809 23:22:23.824086  6023 log.cc:924] Time spent T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.164s	user 0.001s	sys 0.001s
W20260809 23:22:23.824636  6024 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260809 23:22:23.747213  5670 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:23.768784  5751 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.767810  6012 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:23.855382 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 338492us
I20260809 23:22:23.789916  5882 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.858345 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:23.828424  5622 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.857s	user 0.017s	sys 0.013s
I20260809 23:22:23.812127  5891 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:23.833596  6007 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
W20260809 23:22:23.825008  6014 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260809 23:22:23.878301 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:23.745046  5280 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.853546  5903 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.893253 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:23.894836 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:23.901643 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 231682us
I20260809 23:22:23.902303 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:23.739203  5887 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:23.913410 24540 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:23.875607  6033 consensus_queue.cc:1225] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer 60d5c69728134c8bb918a900eb6a8c21 log is divergent from this leader: its last log entry 23.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260809 23:22:23.915169 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 62108.1us
I20260809 23:22:23.913244 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.930428 25363 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260809 23:22:23.906440  5718 consensus_meta.cc:287] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 1.171s	user 0.012s	sys 0.006s
I20260809 23:22:23.891026  5813 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.934631 25371 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:23.898769  5643 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:23.932808 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.936451 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.939640  5652 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:23.939791  6025 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260809 23:22:23.883687  5751 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.897285  5752 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:23.946283  6034 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:22:23.946826  5965 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.947094  6044 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:23.902015  6012 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.951933 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:23.952694 24540 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:23.954459 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.951231 24546 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 3997) took 1141 ms. Trace:
I20260809 23:22:23.963209 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.956023 24546 rpcz_store.cc:276] 0809 23:22:22.809733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.832328 (+ 22595us) service_pool.cc:227] Handling call
0809 23:22:22.847762 (+ 15434us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:22.896787 (+ 49025us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:22.896833 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:22.915868 (+ 19035us) log.cc:842] Serialized 26 byte log entry
0809 23:22:23.261559 (+345691us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b48000c0558. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac287077f1 00007fac286fe8e7 00007fac286fe77a 00007fac287f295f 00007fac287f2627 00007fac28806b2c 00007fac28806ad8 00007fac28806a58 00007fac28806a13 00007fac28805cbc 00007fac27787be6
0809 23:22:23.328543 (+ 66984us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:23.328602 (+    59us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:23.328694 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:23.937213 (+608519us) raft_consensus.cc:1671] finished
0809 23:22:23.937243 (+    30us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:23.951144 (+ 13901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48582,"raft.queue_time_us":60618,"raft.run_cpu_time_us":4531,"raft.run_wall_time_us":351357,"spinlock_wait_cycles":18720128,"thread_start_us":306418,"threads_started":1,"wal-append.queue_time_us":595422}
I20260809 23:22:23.933082  5801 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:23.948787  5951 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.973982 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.965590  5652 consensus_queue.cc:1046] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260809 23:22:23.975108 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:23.901669  5891 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:23.967406  5484 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:23.998751 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.999787 24266 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:24.001497  6037 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260809 23:22:24.004925  5278 consensus_meta.cc:287] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 1.335s	user 0.004s	sys 0.008s
I20260809 23:22:24.005460  5278 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.007010  5611 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.007524  5801 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.009810  5801 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.013893  5891 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:24.014170 24580 heartbeater.cc:644] Failed to heartbeat to 127.21.255.62:37921 (6 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.21.255.62:37921 timed out after 15.000s (SENT)
I20260809 23:22:24.016083 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:23.953548  5643 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.024909 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.026441 24266 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:24.027539 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 194051us
I20260809 23:22:24.028445 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.028579 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 54716.9us
I20260809 23:22:24.029302 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:24.031395 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61115.8us
W20260809 23:22:24.036319 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80660.3us
I20260809 23:22:23.995239  6000 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:23.938246  5718 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.034350 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:23.976434  5752 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
W20260809 23:22:24.043547 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 164602us
I20260809 23:22:23.964524  5681 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.038452 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:23.997948  5484 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.032624  6011 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.036432  5889 raft_consensus.cc:2781] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:24.059067  6002 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.055159  6048 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:24.052137  5280 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:24.061259  6050 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260809 23:22:24.061748  6060 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:24.065778  5545 consensus_meta.cc:287] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.887s	user 0.009s	sys 0.001s
I20260809 23:22:24.068233 24542 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:24.070034 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.105535 24266 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.066607  6063 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.073510  6036 raft_consensus.cc:2781] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260809 23:22:24.074024  6011 raft_consensus.cc:3037] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.074069 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.096298 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.071476  5889 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.114408 24378 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.070658  6012 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.116148  5939 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.120935 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.123797 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.112726  6019 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:24.117069  5889 raft_consensus.cc:3037] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:24.129225 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60509.5us
I20260809 23:22:24.113128  5280 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.116531  6063 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.084865  5646 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.119472  5278 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.113014  6000 consensus_queue.cc:1046] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
I20260809 23:22:24.117997  6036 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.110424  5545 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.141469 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.129454  5813 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:24.128274  5891 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.130198  6019 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.160467 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.164692 25368 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:22:24.171254 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59932.7us
I20260809 23:22:24.173527 24269 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.162559 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.125793  6012 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260809 23:22:24.176153 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.129801  5939 consensus_queue.cc:1046] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260809 23:22:24.153739  6063 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.178469 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.173388 24427 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 33) took 1408 ms. Trace:
I20260809 23:22:24.180207 24422 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:24.183043 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.183672 25363 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:24.183772 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.180781 24427 rpcz_store.cc:276] 0809 23:22:22.755411 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.806244 (+ 50833us) service_pool.cc:227] Handling call
0809 23:22:24.160711 (+1354467us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.163386 (+  2675us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2259555200}
I20260809 23:22:24.159117  6036 raft_consensus.cc:3037] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:24.187577 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.161414  5813 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.193085 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.176807  6019 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:24.178776  6005 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.000s
I20260809 23:22:24.180047  6003 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:24.180080  5891 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.207235 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.187477  5903 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:24.196075  5813 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.218909 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.235951 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.223649 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.238463 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.244891 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.199496  6000 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.220818 25365 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:24.264786 25372 raft_consensus.cc:2396] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:24.184358  6033 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260809 23:22:24.201032  5718 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.274729 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.275638 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94594.2us
W20260809 23:22:24.223644  6066 log.cc:924] Time spent T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260809 23:22:24.207785  5643 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:24.207412  6003 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:24.248034  6024 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260809 23:22:24.281635 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.220800  5903 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.233178  5646 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:24.286664 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 99134us
I20260809 23:22:24.297780 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.298596 24425 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.229090  5545 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.291611 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.229233  6073 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.272275  5891 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:24.308753 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 191236us
I20260809 23:22:24.275291  6012 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.314291 25371 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: [23.18-25.19]
I20260809 23:22:24.315164 24427 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:24.323482 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.207731  6005 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.325003 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:24.287467  5571 consensus_meta.cc:287] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.872s	user 0.006s	sys 0.006s
I20260809 23:22:24.287508  6003 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:24.328286 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.331285 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.331328 24423 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
I20260809 23:22:24.302078  5903 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.336786 25365 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260809 23:22:24.224881  6069 consensus_queue.cc:1046] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.170s
W20260809 23:22:24.335988 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50922.7us
W20260809 23:22:24.344091 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.344867 24498 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.335704 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.348418 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.334024 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.285066  5961 consensus_queue.cc:1046] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260809 23:22:24.351318 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.351480 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 62321.3us
I20260809 23:22:24.352635 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.359344 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.326843  6073 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.372815 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.346814  5545 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.373667 25372 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:24.375422 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:24.375882 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 90983.7us
W20260809 23:22:24.378032 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83450.8us
I20260809 23:22:24.381245 25321 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.348673  5959 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:24.396077 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.341962  6068 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.399139 25321 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.339313  5571 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.402076 25365 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:24.374202  6073 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:24.404973 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 64233us
I20260809 23:22:24.349380  6012 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.410277 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.412885  6011 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:24.413573 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.422056 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.399029  5545 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.416059 25372 raft_consensus.cc:2362] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:24.422247 25363 raft_consensus.cc:2454] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
W20260809 23:22:24.432132 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 88092us
I20260809 23:22:24.420392  5885 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.424186 25370 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.430168 25365 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 30: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:24.403280  5959 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.401039  5194 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:24.433197 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.434235 25371 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.401600  6068 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:24.406957  6060 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election won for term 23
W20260809 23:22:24.447511 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.434736 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3594) took 1128 ms. Trace:
I20260809 23:22:24.399767  6033 consensus_queue.cc:1046] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260809 23:22:24.459204  6033 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:24.459812  6033 raft_consensus.cc:3032] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.460181  6033 raft_consensus.cc:729] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:24.461426  6012 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.464389 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.461336  6033 consensus_queue.cc:260] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.466575 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.451073  5194 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.467191 25375 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.467331  5194 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:24.471220  6057 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.471758  6057 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:24.474834  6007 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.565s	user 0.007s	sys 0.005s
I20260809 23:22:24.475457 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.475304  6007 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.476125 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:24.477576  6007 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.478567 24544 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.481992  5278 raft_consensus.cc:3032] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.482375  5278 raft_consensus.cc:729] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.483158  6060 raft_consensus.cc:686] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.483410 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.485285 25375 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:24.453421 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:24.454388  6088 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260809 23:22:24.491714 25366 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.488180 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.489041 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.434875  6086 raft_consensus.cc:3032] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.490540 24427 raft_consensus.cc:2454] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:24.453476 24429 rpcz_store.cc:276] 0809 23:22:23.306409 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.530895 (+224486us) service_pool.cc:227] Handling call
0809 23:22:23.588501 (+ 57606us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:23.588736 (+   235us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:23.588763 (+    27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:23.589247 (+   484us) log.cc:842] Serialized 26 byte log entry
0809 23:22:24.020204 (+430957us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:24.045691 (+ 25487us) log.cc:842] Serialized 14 byte log entry
0809 23:22:24.045889 (+   198us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.045960 (+    71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:24.127817 (+ 81857us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18001c7e60. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:24.423320 (+295503us) raft_consensus.cc:1671] finished
0809 23:22:24.434478 (+ 11158us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.434657 (+   179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2478,"prepare.queue_time_us":24535,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":1187,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6620,"spinlock_wait_cycles":15863552,"thread_start_us":430534,"threads_started":1,"wal-append.queue_time_us":505169}
I20260809 23:22:24.490691 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:24.499112 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.450642  6071 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:24.466418  6033 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.511029 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.446645  5885 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.515740  5885 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.516824  5885 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:24.519387 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71889.9us
I20260809 23:22:24.520542 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1857) took 1773 ms. Trace:
I20260809 23:22:24.521098 24431 rpcz_store.cc:276] 0809 23:22:22.747057 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.760569 (+ 13512us) service_pool.cc:227] Handling call
0809 23:22:22.761074 (+   505us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.646435 (+885361us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.520184 (+873749us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b540017fcd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.520470 (+   286us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2034286464}
W20260809 23:22:24.526400 24422 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 4054) took 1520 ms (client timeout 1895 ms). Trace:
I20260809 23:22:24.526117  5889 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.527076  5898 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:24.528982  6019 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.531033 24424 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:24.531312  6019 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.450523  6067 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.244s
I20260809 23:22:24.531973  6019 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:24.535341  5622 raft_consensus.cc:3032] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:24.535792  5622 raft_consensus.cc:729] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.536296  5545 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.536801  5622 consensus_queue.cc:260] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.538257  5622 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:24.538746  6019 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:24.539222  6019 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.522619 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.547537 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
W20260809 23:22:24.526760 24422 rpcz_store.cc:269] 0809 23:22:23.006257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.007283 (+  1026us) service_pool.cc:227] Handling call
0809 23:22:24.526117 (+1518834us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400062054. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.526255 (+   138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2640194432}
I20260809 23:22:24.541407 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:24.450253  6011 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.548919 24425 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:24.561161 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.507297 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.557014 25370 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:24.490407  6035 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.464761  5571 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.558504 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:24.576428 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 246984us
I20260809 23:22:24.562338 24498 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.579252 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.579633  6069 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:24.568590 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.544659 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50054.4us
I20260809 23:22:24.580689 24266 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.580886 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.453344  6068 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.581672 24269 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.581801  5796 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.582545 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.582749  5796 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.583535  5796 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:24.583724 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.584069  5796 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
W20260809 23:22:24.585412 24269 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713): Timed out: RequestConsensusVote RPC to 127.21.255.4:38713 timed out after 1.866s (SENT)
I20260809 23:22:24.586200  5801 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.587157  5801 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:24.588303  6000 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.585316  5796 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.589514  6069 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:24.581866 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.590583 24498 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.594862  6000 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:24.540233  6086 raft_consensus.cc:729] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:24.598492 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.599670 24266 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.600720 25321 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.434362  6092 consensus_queue.cc:1046] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.193s
I20260809 23:22:24.489707  6036 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.548513  5666 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:24.606287 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:24.617745 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.629662 24425 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
W20260809 23:22:24.596563  5946 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.003s	sys 0.005s
I20260809 23:22:24.612138 24430 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1863) took 1011 ms. Trace:
I20260809 23:22:24.631639 24425 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260809 23:22:24.565232  6005 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.483182  5278 consensus_queue.cc:260] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.564893  5545 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:24.646212 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.646669 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:24.647580  6078 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260809 23:22:24.647675  6031 raft_consensus.cc:2781] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:24.648434  6031 raft_consensus.cc:686] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.648882  5311 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.649163  6102 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.633524  5571 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:24.631088 24430 rpcz_store.cc:276] 0809 23:22:23.600406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.729071 (+128665us) service_pool.cc:227] Handling call
0809 23:22:23.729403 (+   332us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.826359 (+ 96956us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:23.854763 (+ 28404us) log.cc:842] Serialized 14 byte log entry
0809 23:22:24.519672 (+664909us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b180051b860. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:24.611693 (+ 92021us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:24.611762 (+    69us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:24.611823 (+    61us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.611920 (+    97us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.612067 (+   147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2381,"prepare.queue_time_us":28198,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":15352,"spinlock_wait_cycles":20539264,"thread_start_us":81837,"threads_started":1}
I20260809 23:22:24.649402  6031 consensus_queue.cc:237] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.639109  6094 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.649910  6102 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.651597  6094 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.652551  6102 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.653491 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.653669 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.679009 25372 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:24.654186  6057 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.655104 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3591) took 1349 ms. Trace:
I20260809 23:22:24.655522  5751 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:24.656423 24542 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2046) took 1578 ms. Trace:
I20260809 23:22:24.627854  5666 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:24.685667 24542 rpcz_store.cc:276] 0809 23:22:23.078501 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.078697 (+   196us) service_pool.cc:227] Handling call
0809 23:22:23.079088 (+   391us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.108747 (+1029659us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.656169 (+547422us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b54001d6d54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.656357 (+   188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2274281728}
I20260809 23:22:24.660604 24430 raft_consensus.cc:1261] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260809 23:22:24.663439  6074 log.cc:924] Time spent T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.504s	user 0.003s	sys 0.000s
I20260809 23:22:24.663749  5889 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.602046  5891 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260809 23:22:24.672951  6104 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.609860  6004 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260809 23:22:24.676756 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.678217 24425 raft_consensus.cc:1226] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
W20260809 23:22:24.636421  5946 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.003s	sys 0.005s
I20260809 23:22:24.680455 24535 raft_consensus.cc:1226] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:24.680866 25370 raft_consensus.cc:1226] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.678428 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.681777 24424 rpcz_store.cc:276] 0809 23:22:23.305549 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.493517 (+187968us) service_pool.cc:227] Handling call
0809 23:22:24.530820 (+1037303us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.531494 (+   674us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.654923 (+123429us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400062050. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.655054 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2384870784}
I20260809 23:22:24.684207 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.657505 24312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40856 (request call id 3301) took 1283 ms. Trace:
I20260809 23:22:24.608296  6091 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:24.563616  6011 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.606331  5670 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.068s
I20260809 23:22:24.641163  5652 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:24.701462 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.702544  6051 raft_consensus.cc:3032] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:24.714478 24312 rpcz_store.cc:276] 0809 23:22:23.374241 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.374398 (+   157us) service_pool.cc:227] Handling call
0809 23:22:23.374752 (+   354us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:23.628948 (+254196us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.657254 (+1028306us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b54000db1d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.657446 (+   192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":582845696}
I20260809 23:22:24.603063  6035 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.696179  5666 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.797135 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.637979  6060 consensus_queue.cc:237] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.798583 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:24.799556 25369 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:24.690661  6044 consensus_queue.cc:1046] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.110s
I20260809 23:22:24.801474  6044 raft_consensus.cc:3032] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.801836  6044 raft_consensus.cc:729] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:24.802314 24428 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47454 (request call id 1861) took 2053 ms (client timeout 1803 ms). Trace:
I20260809 23:22:24.637053  5813 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:24.802871 24428 rpcz_store.cc:269] 0809 23:22:22.748562 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.749693 (+  1131us) service_pool.cc:227] Handling call
0809 23:22:24.801773 (+2052080us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.802081 (+   308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2266494208}
I20260809 23:22:24.761210  5194 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.803439  6044 consensus_queue.cc:260] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.804483 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.805279  5194 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.804930  6086 consensus_queue.cc:260] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.806581  6086 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.808360  5666 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.809569 24427 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260809 23:22:24.810460  5666 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.643039  6019 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.813730  6126 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:24.724498  6104 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.820418  6104 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.776161  6048 raft_consensus.cc:3032] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.822036  6048 raft_consensus.cc:729] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.773519  6100 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:24.779129  5278 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.823284 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.822997  6048 consensus_queue.cc:260] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.824097  6048 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.684979  5545 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.827471  5545 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.781059 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.828992 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:24.829478 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 61609.3us
I20260809 23:22:24.830300 24425 raft_consensus.cc:1226] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.773012 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.795728  5670 raft_consensus.cc:3032] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:24.796233  6011 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.804594  5813 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.772446  6020 raft_consensus.cc:3032] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:24.805459 24428 raft_consensus.cc:2379] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:24.763118  6102 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.833637 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.834523 24425 raft_consensus.cc:1226] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 20. Current term is 21. Ops: []
I20260809 23:22:24.834456 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.781322  5959 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.775863  6073 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.810382  5903 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.840183 25371 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260809 23:22:24.797089  6036 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.854068 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.838538 25372 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:24.764936  5652 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:24.881768 24544 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 30. Current term is 31. Ops: []
I20260809 23:22:24.864243 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:24.870409 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.848927  6020 raft_consensus.cc:729] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:24.860397 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.766876  6004 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:24.769944  6051 raft_consensus.cc:729] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.885380  5903 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:24.886008  6003 consensus_meta.cc:287] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260809 23:22:24.885620  6051 consensus_queue.cc:260] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:24.760805  6002 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.652s	user 0.009s	sys 0.003s
I20260809 23:22:24.886462  6003 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.887171  6002 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:24.888298 24428 raft_consensus.cc:1226] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:24.888828  6003 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.862164  6102 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.889827 24534 raft_consensus.cc:1226] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.838380  6035 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.846416  5946 consensus_queue.cc:1046] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.046s
I20260809 23:22:24.893050  5946 raft_consensus.cc:3032] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:24.892660  6020 consensus_queue.cc:260] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.769564  5718 consensus_queue.cc:1035] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.110s
I20260809 23:22:24.894027  6044 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.895197  5652 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.896175  6073 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.896821 24429 raft_consensus.cc:1226] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.898459  5959 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:24.814445  6126 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:24.900617 24425 raft_consensus.cc:1261] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260809 23:22:24.901547  6126 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:24.901989 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.904217 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.886411  6053 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260809 23:22:24.909714 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 298225us
I20260809 23:22:24.910425 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:24.891335  6102 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:24.893409  5946 raft_consensus.cc:729] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:24.904541 24424 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
W20260809 23:22:24.859398  6063 consensus_meta.cc:287] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
I20260809 23:22:24.884119  6004 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.843010  5670 raft_consensus.cc:729] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:24.875473  5571 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:24.915602 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 51470us
I20260809 23:22:24.918031 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:24.920305 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:24.922505 24535 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260809 23:22:24.922895 24535 pending_rounds.cc:76] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 12
W20260809 23:22:24.920501 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 333796us
I20260809 23:22:24.847079  6005 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260809 23:22:24.894687  6020 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:24.923146 24535 pending_rounds.cc:97] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260809 23:22:24.967839 24535 raft_consensus.cc:2866] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:24.919656  6102 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:22:24.966336  6073 consensus_queue.cc:1046] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.006s
I20260809 23:22:24.971999 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.972472 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.025622 25366 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:24.972959 25375 raft_consensus.cc:1226] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260809 23:22:24.973595  6019 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:24.974004  6039 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260809 23:22:25.032121 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50333.6us
I20260809 23:22:24.975167 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:24.977278 24430 raft_consensus.cc:1226] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 25. Current term is 26. Ops: []
I20260809 23:22:24.978358 24428 raft_consensus.cc:1226] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:24.979458  5946 consensus_queue.cc:260] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:24.983553  6117 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:22:25.036968 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:24.823009  6100 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.988076  6000 raft_consensus.cc:3032] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Stepping down as leader of term 22
I20260809 23:22:24.992053 24269 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:24.997370 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1871) took 1012 ms. Trace:
I20260809 23:22:24.998284 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1867) took 1340 ms. Trace:
W20260809 23:22:25.044682 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52921.3us
I20260809 23:22:25.044867 24429 rpcz_store.cc:276] 0809 23:22:23.658350 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.556210 (+897860us) service_pool.cc:227] Handling call
0809 23:22:24.556616 (+   406us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.897206 (+340590us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.998062 (+100856us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54003336d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:24.998229 (+   167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":781969152}
I20260809 23:22:25.009339  6068 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.010062 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.012630  6007 consensus_queue.cc:1046] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:25.030558 25373 raft_consensus.cc:2379] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.031991 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.032882 24424 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:24.974535 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:25.034984 24378 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.043174 24428 raft_consensus.cc:1261] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:25.030351  5959 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.061503 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.044153 24422 rpcz_store.cc:276] 0809 23:22:23.984935 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.602390 (+617455us) service_pool.cc:227] Handling call
0809 23:22:24.832021 (+229631us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.832185 (+   164us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:24.832209 (+    24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:24.832248 (+    39us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:24.832276 (+    28us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:24.832329 (+    53us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:24.997187 (+164858us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540012fcd0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:24.997302 (+   115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":472936704}
I20260809 23:22:25.048017 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.053081 24429 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.045827 24269 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.064771 24269 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:25.066802  5646 consensus_queue.cc:1046] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:25.067062  5166 consensus_queue.cc:1035] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:25.054059 24378 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.075251 24378 leader_election.cc:334] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.075938 24378 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.077519 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.079284 24378 leader_election.cc:334] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.079946 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.081878 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.961104  5643 consensus_queue.cc:1046] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260809 23:22:25.084265 24423 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:24.963065  5571 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.086545 24378 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:24.918551  6063 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:24.965216  5545 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.087s
I20260809 23:22:25.043846  6000 raft_consensus.cc:729] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.091547 24269 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.100204 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.052446  6011 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:25.042716  6065 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.057s
W20260809 23:22:25.127907 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.128396 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.129372 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.129693 24378 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.097558  6095 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.050453  6067 consensus_queue.cc:1046] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
I20260809 23:22:25.083834  6073 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.139840 24542 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260809 23:22:25.084527  5946 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.096907  5965 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:25.138813 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 47349.7us
I20260809 23:22:25.206032 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.058591  6012 consensus_queue.cc:1046] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.079s
I20260809 23:22:25.208948 24423 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:25.098562  6039 raft_consensus.cc:3032] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Stepping down as leader of term 24
I20260809 23:22:25.098784  5752 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:25.103968  6036 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20260809 23:22:25.135289  5813 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.135111  6091 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:25.121917  5646 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.091s
W20260809 23:22:25.156288  5622 consensus_meta.cc:287] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.002s
I20260809 23:22:25.136912  6126 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:25.234280  6124 log.cc:924] Time spent T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260809 23:22:25.154836  6104 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260809 23:22:25.154620  6127 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.120817  6031 consensus_queue.cc:1035] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.095s
I20260809 23:22:25.256791  6012 raft_consensus.cc:3032] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 LEADER]: Stepping down as leader of term 30
W20260809 23:22:25.256959 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46363.6us
W20260809 23:22:25.263273  6033 consensus_meta.cc:287] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.719s	user 0.009s	sys 0.002s
I20260809 23:22:25.257316  6129 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:22:25.258009  5752 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.259572 25375 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260809 23:22:25.104329  5959 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.263723 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57777.8us
I20260809 23:22:25.299638 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.216070  6011 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.283572 25366 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:25.228576  6039 raft_consensus.cc:729] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:25.291905  6003 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.284951  6004 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.284948  6118 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:25.289320  6012 raft_consensus.cc:729] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.334530 25375 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.348094 24422 raft_consensus.cc:3032] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:25.354825 24269 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:25.356563  6136 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260809 23:22:25.356961  6135 log.cc:924] Time spent T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260809 23:22:25.357543  5670 consensus_queue.cc:260] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.331214  6048 consensus_meta.cc:287] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.003s
I20260809 23:22:25.359442 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1864) took 1756 ms. Trace:
W20260809 23:22:25.360515 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.361819  5278 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.000s
I20260809 23:22:25.245121  6000 consensus_queue.cc:260] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.362632  6000 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:25.350342 24422 raft_consensus.cc:729] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:25.369912 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 239343us
I20260809 23:22:25.359791 24435 rpcz_store.cc:276] 0809 23:22:23.603035 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.958118 (+355083us) service_pool.cc:227] Handling call
0809 23:22:23.970437 (+ 12319us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:23.980087 (+  9650us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:23.980125 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:24.665873 (+685748us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:24.693250 (+ 27377us) log.cc:842] Serialized 26 byte log entry
0809 23:22:25.091428 (+398178us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:25.091459 (+    31us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:25.091584 (+   125us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:25.359123 (+267539us) raft_consensus.cc:1671] finished
0809 23:22:25.359145 (+    22us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:25.359378 (+   233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6616960,"thread_start_us":386941,"threads_started":1,"wal-append.queue_time_us":626002}
W20260809 23:22:25.370560 24378 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.370971 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.361529  6011 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.374856 24424 raft_consensus.cc:2454] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
I20260809 23:22:25.381510 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:25.378832  5752 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.384654  6100 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.385445 25363 raft_consensus.cc:1226] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 22. Current term is 23. Ops: []
I20260809 23:22:25.381884 24496 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:25.386461 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.387138 25370 raft_consensus.cc:2362] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26. Re-sending same reply.
I20260809 23:22:25.391916 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.379817 24422 consensus_queue.cc:260] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.396400 24427 raft_consensus.cc:2379] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:25.395792 24424 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:25.397684 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:25.400019  5571 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.402482 24427 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
W20260809 23:22:25.372021 24378 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.393601 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.408923 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.409377 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.410915 24266 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.402827 24427 pending_rounds.cc:76] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 13
I20260809 23:22:25.379899  5796 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:25.387310  5813 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:25.418428 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46076.3us
I20260809 23:22:25.415444 24427 pending_rounds.cc:97] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260809 23:22:25.433226 24427 raft_consensus.cc:2866] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:25.424156 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.420279 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.396965  6068 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:25.439682 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.418378  6065 raft_consensus.cc:2781] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:25.449761 24435 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.444034 24378 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.430648  6091 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.448688 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.437501  5666 raft_consensus.cc:2781] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:25.431370  6063 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:25.441654 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 122370us
I20260809 23:22:25.402983  6095 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:25.412147  5718 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260809 23:22:25.478276 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.421095  5939 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260809 23:22:25.420961  6036 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
W20260809 23:22:25.493458 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44846.7us
I20260809 23:22:25.494973 24435 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:25.329501  6053 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.148s
W20260809 23:22:25.510407 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.511464 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:25.514034 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.515403 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.468344  5887 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.542894 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.545671 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.546463 25375 raft_consensus.cc:2354] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.441998  6069 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.050s
W20260809 23:22:25.547241 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.453421  5796 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:25.559722 24378 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.573454 24378 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:25.563376 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.451900  6068 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:25.473613  5666 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:25.574831 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.579310 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.581141 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.501663  6063 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.577734 24378 leader_election.cc:398] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.588132 24378 leader_election.cc:401] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:25.477273  6065 raft_consensus.cc:686] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:25.383620  6039 consensus_queue.cc:260] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.561865  6005 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:25.594308 25370 raft_consensus.cc:1226] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 28. Current term is 29. Ops: [28.20-28.20]
W20260809 23:22:25.574609  5652 consensus_meta.cc:287] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.002s
I20260809 23:22:25.578950  5611 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260809 23:22:25.576583  5813 consensus_queue.cc:1046] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260809 23:22:25.598305  5751 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:25.576306  5545 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:25.590324 24435 raft_consensus.cc:2379] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 23: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.440932  6012 consensus_queue.cc:260] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.604389 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 182502us
I20260809 23:22:25.560305  6057 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.559089  6099 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.559214  5959 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.576268  6033 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.581372  5666 raft_consensus.cc:3037] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:25.631911 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34784 (request call id 3610) took 1314 ms. Trace:
I20260809 23:22:25.658746 24423 rpcz_store.cc:276] 0809 23:22:24.317903 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.377891 (+ 59988us) service_pool.cc:227] Handling call
0809 23:22:25.631655 (+1253764us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540017f2d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:25.631826 (+   171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1577490816}
I20260809 23:22:25.585919  5887 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.636934 24539 raft_consensus.cc:1226] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:25.608466  6068 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:25.623915  5571 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.505908  6095 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.635612  6140 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
W20260809 23:22:25.656083 24498 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.632625  6004 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:25.635303  6146 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260809 23:22:25.655233  6039 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:25.633795  5652 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.644378  5813 raft_consensus.cc:3032] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Stepping down as leader of term 23
I20260809 23:22:25.661828  6012 raft_consensus.cc:3037] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:25.680563 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.674243  5571 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.688935  6095 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.689390  6004 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.677111  6065 consensus_queue.cc:237] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.700031 24498 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.705001 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:25.705965 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.706292 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.677666  6036 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.708307 25372 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:25.698474  5813 raft_consensus.cc:729] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:25.709190 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:25.721097 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.721557 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.659808  5801 consensus_queue.cc:1159] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 30 status { last_received { term: 28 index: 16 } last_committed_idx: 15 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:25.723484  5813 consensus_queue.cc:260] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.723959 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.724153 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:25.726207 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.726948 25363 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 24: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:25.726909 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.730947  6057 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.731465  6057 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.733554  6057 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:25.735230 25373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2124) took 1863 ms (client timeout 1866 ms). Trace:
W20260809 23:22:25.735761 25373 rpcz_store.cc:269] 0809 23:22:23.872257 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:23.872367 (+   110us) service_pool.cc:227] Handling call
0809 23:22:24.796983 (+924616us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b5800005198. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac287f279b 00007fac287f2821 00007fac287a21b0 00007fac287a1a94 00007fac21e89c04 00007fac21e89b92
0809 23:22:25.734989 (+938006us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54007f03d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:25.735163 (+   174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":514528256}
I20260809 23:22:25.738735  5882 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:22:25.738917  6005 consensus_queue.cc:1046] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:25.716236  6036 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:25.743655 24431 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1865) took 2131 ms. Trace:
I20260809 23:22:25.744369  6036 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.744660  6002 raft_consensus.cc:3032] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:25.745103  6002 raft_consensus.cc:729] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:25.746409  6019 consensus_meta.cc:287] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.708s	user 0.009s	sys 0.002s
I20260809 23:22:25.747277  5571 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.748808  5882 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:25.672811  5796 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.745986  6002 consensus_queue.cc:260] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.772034 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 208707us
W20260809 23:22:25.772938 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.773447 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.744043 24431 rpcz_store.cc:276] 0809 23:22:23.611995 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.557912 (+945917us) service_pool.cc:227] Handling call
0809 23:22:24.602914 (+ 45002us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:24.800625 (+197711us) raft_consensus.cc:1520] Early marking committed up to index 16
0809 23:22:24.801014 (+   389us) log.cc:842] Serialized 14 byte log entry
0809 23:22:25.097871 (+296857us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:25.097927 (+    56us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:25.097970 (+    43us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:25.743400 (+645430us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540018c754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:25.743448 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:25.743597 (+   149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2246,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":5690,"spinlock_wait_cycles":464166016,"thread_start_us":296364,"threads_started":1,"wal-append.queue_time_us":445493}
W20260809 23:22:25.766877 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 43531.4us
W20260809 23:22:25.767239 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46171.5us
I20260809 23:22:25.774411 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:25.775363 24498 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.674820  5887 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:25.775776 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:25.787992 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.789054 24545 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260809 23:22:25.787472 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:25.803912 24543 raft_consensus.cc:2396] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260809 23:22:25.806147 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:25.810793 25323 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.811277 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.771091  5946 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.812918 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.814211 24427 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:25.814141 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.744715  5652 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:25.818702 24498 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.815840 24425 raft_consensus.cc:2396] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260809 23:22:25.816681 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.802049 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.821014 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.773228  6069 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.822127 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.825371 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.834110 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:25.826870 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:25.835824 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.839972 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.836838 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.840606 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.824226  5751 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.818212  6002 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:25.823788  6086 consensus_meta.cc:287] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.979s	user 0.011s	sys 0.001s
W20260809 23:22:25.866729 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.824532  6005 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.836458  6165 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:25.854727  6167 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:25.857342  6057 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:25.905781 24543 raft_consensus.cc:3032] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Stepping down as leader of term 25
I20260809 23:22:25.859431  6032 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:25.914952 24543 raft_consensus.cc:729] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:25.876135  5882 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:25.927227 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110564us
I20260809 23:22:25.875154  5611 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:25.933710 25373 raft_consensus.cc:1226] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260809 23:22:25.919694 24543 consensus_queue.cc:260] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:25.936810 24543 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:25.944334 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:25.947237 24535 raft_consensus.cc:1226] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 30. Current term is 31. Ops: []
I20260809 23:22:25.945221 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
W20260809 23:22:25.952688 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 117222us
I20260809 23:22:25.953917 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.954715 24431 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260809 23:22:25.941565  6032 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:25.957662 25321 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.950461  6000 consensus_meta.cc:287] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.587s	user 0.008s	sys 0.016s
W20260809 23:22:25.960721 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.968174  5666 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:25.968855  6162 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260809 23:22:25.971304 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:25.971388  5666 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:25.972572 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:25.974032 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.974404 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46449.6us
W20260809 23:22:25.974987 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:25.975911 25323 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.933373  5611 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:25.976776 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.977921  5611 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:25.977187 25323 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:25.947606  6004 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:25.971869 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 371367us
I20260809 23:22:25.981712 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:25.982350 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:25.958565  6020 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.943s	user 0.012s	sys 0.009s
I20260809 23:22:25.990536 24430 raft_consensus.cc:1226] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:25.995108 24431 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:25.982134 24378 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:25.996799 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.003098  5571 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.003682  5571 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.006022  5571 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.006891 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.007503 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.007745 24430 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 29: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:26.008196 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.008692  5751 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260809 23:22:26.011672 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.012327  5882 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.014616  5751 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.013887 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.021502 25363 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260809 23:22:26.021930 25323 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.021380 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.022647  5751 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260809 23:22:26.023746 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.024199 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.025031  5611 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:26.024175 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.026394  5882 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:25.995631  5194 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.029892 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.031486 24541 pending_rounds.cc:76] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 16
I20260809 23:22:26.031414 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.031889 24541 pending_rounds.cc:97] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260809 23:22:26.020927  6167 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:26.032218 24541 raft_consensus.cc:2866] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260809 23:22:25.964375 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:26.033072 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 68766.5us
I20260809 23:22:26.035717 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.036384 24534 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:26.036464 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54370.3us
W20260809 23:22:26.037889 24378 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.038266 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.039176 24545 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
W20260809 23:22:26.041149 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.041112 25375 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260809 23:22:26.042145 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.042701 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.048487 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.034328  6181 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260809 23:22:26.034183  6189 log.cc:924] Time spent T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260809 23:22:26.033412  6163 log.cc:924] Time spent T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260809 23:22:26.063854 24378 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.033560  6178 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:22:26.059394 24431 raft_consensus.cc:2379] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 25: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:26.054836 25366 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.064378 24378 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.068352  6190 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260809 23:22:26.066203  6068 consensus_queue.cc:1046] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.005s
I20260809 23:22:26.071545  6068 raft_consensus.cc:3032] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:26.071887  6068 raft_consensus.cc:729] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:26.072718  6068 consensus_queue.cc:260] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.073946  6068 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.047317  5891 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.075624  6011 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:26.076290  5891 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.078121 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.078480  6188 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.079116  6126 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.079655 24378 leader_election.cc:334] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.035277  6031 consensus_queue.cc:1046] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.020s
I20260809 23:22:26.080155 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.083861 24543 raft_consensus.cc:2396] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260809 23:22:26.084272 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:26.085116 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.085729 24378 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.087157 24378 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.090606 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.091176 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.091245 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.093677 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.094276 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.048874  6167 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260809 23:22:26.047526  6020 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.097829 24270 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.047233  6035 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.040251  5887 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.122036 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.123507 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:26.124625 24378 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:26.124743 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.125056 24378 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.126148 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.126784 25323 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.137975 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.079195  6036 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.079681  5611 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.066623  5751 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:25.966459  6032 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.098544  6126 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.125891  6129 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.121737  6185 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.123306  6073 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.158638  6190 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.146191  5887 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.150565  6060 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.209049 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 122412us
I20260809 23:22:26.123939  6033 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.207374 25369 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.145541  6020 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.145308  6091 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.153077  5965 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260809 23:22:26.217408 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.204887  5194 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.204459  5611 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.212718  6129 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.229576  5194 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:26.232215  6195 log.cc:924] Time spent T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260809 23:22:26.235095  6158 log.cc:924] Time spent T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.600s	user 0.003s	sys 0.000s
I20260809 23:22:26.212862  6203 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.249240  6203 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.249584  6203 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.206068  5751 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.079191  6188 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.227540  6005 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.243362  6129 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.237946  6060 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.245522  5891 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.224191  6020 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.225440  5887 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.237694  6033 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260809 23:22:26.264904  6203 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.348747 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:26.343448  6012 consensus_meta.cc:287] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.638s	user 0.010s	sys 0.002s
I20260809 23:22:26.359288  6188 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.360332  6069 consensus_meta.cc:287] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.000s
I20260809 23:22:26.360778  6069 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.363766  6196 log.cc:924] Time spent T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260809 23:22:26.357985  5484 raft_consensus.cc:3032] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Stepping down as leader of term 20
I20260809 23:22:26.365393  5484 raft_consensus.cc:729] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:26.366955  5611 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.271785  6005 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.369042 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.369843  6005 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.372135 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.363864  6032 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.292749  6060 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.375646  6036 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.369817 24542 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
I20260809 23:22:26.377254  6060 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:26.378592 24498 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.366619  5484 consensus_queue.cc:260] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.380916 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:26.382505 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 338733us
W20260809 23:22:26.403331 24498 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.411551 25369 raft_consensus.cc:1226] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
W20260809 23:22:26.414179 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 187124us
W20260809 23:22:26.382771  6197 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260809 23:22:26.419355 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.424345 24542 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
W20260809 23:22:26.427316 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.427938 25323 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.431555 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.434180 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.435240 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.437953 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.418227  5965 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.439479  5965 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.441702  5965 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.442812 24429 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 23.
W20260809 23:22:26.447845 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 223930us
I20260809 23:22:26.438544 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:26.415200  6005 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:26.417445  6151 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.464941 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.450420 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.436025 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.466404 24431 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
W20260809 23:22:26.469517 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.465623 24427 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.469986 25323 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.440004  5484 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Advancing to term 21
I20260809 23:22:26.473564 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.429646  5959 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.447515  6091 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.495752 24427 raft_consensus.cc:1261] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260809 23:22:26.491595 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.463532  5611 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.465848  6005 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:22:26.493487 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.465458  6032 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.491151 25368 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260809 23:22:26.501549 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.492373 24545 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.506448 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.507469 24541 raft_consensus.cc:2396] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for term 28 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260809 23:22:26.509991  6031 consensus_queue.cc:1035] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:26.501588  5194 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.497586  5959 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.493399  6202 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260809 23:22:26.511520  6032 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.492391  5311 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.513028  5311 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.513342  5959 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.499650  6209 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.513358  5311 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.474328  6151 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.517042  6151 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.518594  5751 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.519024  6020 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.523034 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.500973  5611 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.528131 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:26.529229 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.529618 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:26.530751 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 60291.8us
I20260809 23:22:26.530773 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.532444 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:26.479496  6210 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.537077 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.501111  6005 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.538024 25368 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.538287 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.491801  5278 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.513470  5887 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.513622  6032 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.545688 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.546079 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.547521 24541 raft_consensus.cc:2379] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 30: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:26.534652  5952 raft_consensus.cc:3032] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Stepping down as leader of term 28
I20260809 23:22:26.551785 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.527762  6020 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.539732  6206 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:26.555538 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 83914.5us
W20260809 23:22:26.559520 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.554104  6210 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.565083 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.552289  6036 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:26.550737  6151 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.569526 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.500171  6091 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.570195 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.554751  5959 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:26.571403 25368 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:26.572880 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.574838 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.554473  5952 raft_consensus.cc:729] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:26.559180  6020 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.549576  5278 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.574949  6151 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.566129  6210 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.587141 24425 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:26.576674  6216 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260809 23:22:26.587808 25373 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260809 23:22:26.576767 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.588740 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.589398 25368 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 29.
I20260809 23:22:26.589951  6033 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.578998  5611 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.590435  6033 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.583799  6091 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.585341  6210 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.591809  6005 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.592830  5952 consensus_queue.cc:260] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.594091  5952 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:26.594605  6005 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.594651  6213 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.595508  6213 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.597482  6213 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.586556  6151 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.601900 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2146) took 1671 ms. Trace:
I20260809 23:22:26.604274 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.605733 25369 raft_consensus.cc:2354] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:26.586452  6068 consensus_meta.cc:287] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.002s
I20260809 23:22:26.564189  6202 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:26.602555 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.607388 24266 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:26.608225  5885 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260809 23:22:26.620834  6209 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:26.602269 25371 rpcz_store.cc:276] 0809 23:22:24.927050 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.927198 (+   148us) service_pool.cc:227] Handling call
0809 23:22:25.753084 (+825886us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:25.753413 (+   329us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:25.753441 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:25.779580 (+ 26139us) log.cc:842] Serialized 26 byte log entry
0809 23:22:25.962029 (+182449us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:25.987765 (+ 25736us) log.cc:842] Serialized 14 byte log entry
0809 23:22:26.034644 (+ 46879us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.034714 (+    70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:26.216045 (+181331us) raft_consensus.cc:1671] finished
0809 23:22:26.216065 (+    20us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:26.598722 (+382657us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b54007f03d0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac2385fa09 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:26.598865 (+   143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53945,"prepare.queue_time_us":66880,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":47100,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":18627,"spinlock_wait_cycles":1898590464,"thread_start_us":108937,"threads_started":1,"wal-append.queue_time_us":161683}
I20260809 23:22:26.581390  6020 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.624416 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.584048  5278 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.626397 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.614560 24427 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260809 23:22:26.627336 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.627558 24427 pending_rounds.cc:76] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 19
I20260809 23:22:26.627943 24427 pending_rounds.cc:97] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260809 23:22:26.628274 24427 raft_consensus.cc:2866] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:26.616664 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.629377 24541 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:26.617475  6129 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.590847  6033 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.622475  5670 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.537410  6154 consensus_queue.cc:1046] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260809 23:22:26.641607 25363 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
W20260809 23:22:26.642973 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.643723 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.644613 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.645392 24545 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
W20260809 23:22:26.651425 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.651738 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:26.646178 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.627246  6202 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.640343  5801 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.012s
W20260809 23:22:26.653309 24378 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:26.653785 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:26.655834 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.656186 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.627602  6005 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.640277  6216 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:26.640614  5611 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:26.660507 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.586913  5311 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.660574 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.652470  5494 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:26.662223  6216 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.662375  6005 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.662803  6068 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:26.671036  6166 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:26.683800  6129 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.655205 24428 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:26.655217  6202 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.622777  6210 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.661800  6057 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
W20260809 23:22:26.692162 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:26.664822  5611 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.687343  6095 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:26.689357  6216 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:26.690657  6213 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.693928  6220 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:26.705070 24266 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:26.698024  5611 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.708938 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.709800 24435 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:26.698000  6226 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:26.700547  5889 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.690060  5494 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.716972  6005 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.719712 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:26.704643  6213 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.739274 24422 raft_consensus.cc:3037] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:26.739400  6213 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.726904 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.740262 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.741729 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dad562edaec4eebaa047293c085c0c4" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.721287  5278 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:26.746728 25321 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.747902 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.749128 24544 raft_consensus.cc:2454] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:26.707650  6220 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.758706 24542 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.758563 24546 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:26.764753 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.765894  5311 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.768112 25371 raft_consensus.cc:1226] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260809 23:22:26.772297 24541 raft_consensus.cc:2454] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:26.752285  6011 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:26.772588 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.775683 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.776158 24431 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.772303 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:26.774428 25325 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:26.752126  6005 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:26.777446 24430 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.783133  6068 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.773053  5887 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.783635  6068 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.784322  5887 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.786634  5887 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.788292  6118 raft_consensus.cc:3032] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:26.793917 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.794692 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.796201 25371 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:26.797124 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.797300 24269 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.791004  6068 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.799963  5494 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.804500  6033 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.777311  6011 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.805727  6011 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.806196  5718 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:26.806746  5718 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.809455  5718 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.777841  6020 consensus_queue.cc:1046] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260809 23:22:26.810292  5946 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:26.810907  5646 raft_consensus.cc:2781] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:26.736714  5889 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.806649  6033 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.881119 24428 raft_consensus.cc:1226] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.813210  5611 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.885417 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.886252 24425 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.816830 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.888052 24544 raft_consensus.cc:2354] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
W20260809 23:22:26.887697 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 111954us
I20260809 23:22:26.817994 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.836622 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 4044) took 1424 ms. Trace:
I20260809 23:22:26.885460  6051 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.866421 24422 raft_consensus.cc:1261] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
W20260809 23:22:26.839192 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 180132us
I20260809 23:22:26.848384 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:26.863140 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.879956 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.816438 24541 raft_consensus.cc:1226] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Rejecting Update request from peer ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30. Ops: []
I20260809 23:22:26.818773 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.819235 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.909690 24542 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:26.909976 24534 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:26.820115 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.827320 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.832934 24431 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:26.890655 24535 rpcz_store.cc:276] 0809 23:22:25.412545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.413031 (+   486us) service_pool.cc:227] Handling call
0809 23:22:25.413327 (+   296us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.955135 (+541808us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.836393 (+881258us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54001fca54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:26.836565 (+   172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1227300224}
I20260809 23:22:26.838364 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.915912 24435 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 22.
I20260809 23:22:26.850448 24423 raft_consensus.cc:1261] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:22:26.863950 24378 leader_election.cc:398] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:26.840535 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 1984) took 1913 ms. Trace:
I20260809 23:22:26.831207 24430 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:26.878072 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.863387 25371 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
I20260809 23:22:26.919735 25373 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.917752 24378 leader_election.cc:401] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:26.906495 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 25.
I20260809 23:22:26.904896 24422 pending_rounds.cc:76] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 17
I20260809 23:22:26.925443 24422 pending_rounds.cc:97] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260809 23:22:26.925796 24422 raft_consensus.cc:2866] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:26.925828 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:26.926525 24544 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.926602 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4080) took 1598 ms. Trace:
I20260809 23:22:26.853412 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.891238 24270 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:26.929965 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66239.8us
I20260809 23:22:26.918067 25372 rpcz_store.cc:276] 0809 23:22:24.926865 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:24.927005 (+   140us) service_pool.cc:227] Handling call
0809 23:22:24.927293 (+   288us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.716586 (+789293us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.840204 (+1123618us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540064b754. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:26.840460 (+   256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1795622144}
I20260809 23:22:26.934592 24380 leader_election.cc:398] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:26.939769  6030 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.927052 24422 rpcz_store.cc:276] 0809 23:22:25.328264 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.328751 (+   487us) service_pool.cc:227] Handling call
0809 23:22:25.347960 (+ 19209us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:26.738747 (+1390787us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b5800a4fcb0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac237dbf63 00007fac2383faba 00007fac238573af 00007fac23858b4e 00007fac23862c89 00007fac23863556 00007fac2385fedf 00007fac2385f99e 00007fac287a12e9 00007fac21e87684
0809 23:22:26.926354 (+187607us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.926543 (+   189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":982050560}
I20260809 23:22:26.940598  6030 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.941264  6207 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260809 23:22:26.843704  6133 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 1.598s	user 0.005s	sys 0.005s
W20260809 23:22:26.873941  6199 log.cc:924] Time spent T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.520s	user 0.000s	sys 0.001s
I20260809 23:22:26.898430  5280 consensus_queue.cc:1046] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
W20260809 23:22:26.906514 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 58219.3us
I20260809 23:22:26.950079  6002 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.955979 25371 raft_consensus.cc:1226] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 23. Current term is 24. Ops: []
I20260809 23:22:26.955713  6002 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.957839  6002 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.855152  5643 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.945559 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:26.866106  5278 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.823637  5646 raft_consensus.cc:686] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:26.834095  6205 log.cc:924] Time spent T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.484s	user 0.001s	sys 0.000s
I20260809 23:22:26.862720  6011 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.976646  6011 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.977094  6030 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.947094 24266 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.824687  6220 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.951713 25366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37956 (request call id 2155) took 1228 ms. Trace:
I20260809 23:22:26.827070  6211 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:26.978644  6211 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:26.979077  6211 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.917166  6007 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.949906  6044 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.982157  6044 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.984234  6044 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:26.846019  5813 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:26.985824 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.986711 24427 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:26.987393 24539 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:26.870998  6067 raft_consensus.cc:3032] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:26.828019  5889 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.820073  6118 raft_consensus.cc:729] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:26.989114  6067 raft_consensus.cc:729] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:26.990123  6067 consensus_queue.cc:260] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.991850  6067 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.982175  6007 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.989084 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:26.995577 24427 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:26.995824  6007 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.996069  5959 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.996616  5959 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.990185  6118 consensus_queue.cc:260] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.997978  6118 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:26.999269  5611 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:26.943064  5965 raft_consensus.cc:3037] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 25
I20260809 23:22:27.000490 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.002169  6051 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:27.002573  6051 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.002883  6051 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:26.841444  6216 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.005187  6216 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:27.005836  6234 consensus_queue.cc:1159] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" responder_term: 30 status { last_received { term: 29 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 29. Current term is 30. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260809 23:22:26.867940  6057 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:27.007248  5646 consensus_queue.cc:237] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.007642 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 60678.3us
I20260809 23:22:27.009591  6216 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.010317  6216 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.010782  6216 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.011597  5278 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.929586  6003 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:26.945122 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 53964.5us
I20260809 23:22:26.843816  6086 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.872633  5891 raft_consensus.cc:3032] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Stepping down as leader of term 29
I20260809 23:22:26.810890  5946 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:26.890125  5752 raft_consensus.cc:2781] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.019585 25373 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:26.814989  6210 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:26.967182 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:26.971279  5643 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:26.977932 25366 rpcz_store.cc:276] 0809 23:22:25.723671 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.723833 (+   162us) service_pool.cc:227] Handling call
0809 23:22:26.951498 (+1227665us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b540063e554. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:26.951638 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584899456}
I20260809 23:22:27.033203 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.034148 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.034893 25366 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.001904  5951 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260809 23:22:26.930554  6071 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:26.963357 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:26.970549 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.040293 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.015923 25372 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:27.017014 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.042613 24430 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.017325 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.044839 24423 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:26.965966 25363 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.020576 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.021381 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.023164 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.023854 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.021457  5752 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.019428  5891 raft_consensus.cc:729] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.024721 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:26.966075  5882 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:26.961885  5796 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.038223 24269 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:27.057703 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 94407.9us
I20260809 23:22:26.868017  6202 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.050609 25373 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.059329 24378 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.050647 24535 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.052446 24541 raft_consensus.cc:2396] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.054482 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.050871 24427 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.060693 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:27.067288 24496 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.065541 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2156) took 1340 ms. Trace:
I20260809 23:22:27.024809  5311 consensus_queue.cc:1035] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260809 23:22:27.054328  5752 raft_consensus.cc:3037] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.079857 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.097288 24534 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.082206 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.086100 24544 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:27.070319 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.089243 24430 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:27.099242 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 61151.9us
I20260809 23:22:27.051499  6071 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:27.051735  6127 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.063407  5882 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.106474  5882 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.098404 25369 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.022073  6086 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.118083 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:27.122269 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 52063.9us
I20260809 23:22:27.123615 24427 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.125603 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.112478 24535 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:27.102563 25373 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.166396  5494 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.120400  5278 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.168994  5494 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.170302 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 59396.1us
I20260809 23:22:27.114059 25366 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
I20260809 23:22:27.019294  6003 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.170964 24266 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.133379  6007 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.115351 24534 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
I20260809 23:22:27.111383  6071 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:27.171582  6007 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.172013  6007 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.172845  6207 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.173344  6207 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.100365  6053 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.174186  6207 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.122045 25365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2162) took 1093 ms. Trace:
I20260809 23:22:27.141630  6086 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.072731  6033 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.118687  5946 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.151638 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.151829  6051 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.158299  5965 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 24. Current term is 25.
I20260809 23:22:27.158833  5891 consensus_queue.cc:260] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.071256  6020 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:27.163501 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.227296 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 63949.4us
I20260809 23:22:27.164183  5889 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:22:27.166429 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50688.9us
I20260809 23:22:27.171048  5494 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.174773  6053 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.118202  5796 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.167315  5813 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.177572 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.167320 24424 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260809 23:22:27.230034 24270 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.231971 24424 pending_rounds.cc:76] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 16
I20260809 23:22:27.232226 24424 pending_rounds.cc:97] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260809 23:22:27.232499 24424 raft_consensus.cc:2866] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:27.132525  6044 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:27.233110  6202 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.233678  6044 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:27.234115  6044 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:27.127919  6127 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.235066  6044 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.129130 25363 raft_consensus.cc:2454] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.179088 24269 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.235913  5813 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.236737  6127 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.236923  6220 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:27.237507  6220 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.237921  5813 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.238469  6051 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.223166  6210 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.239445  6210 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.240561 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.241190 24543 raft_consensus.cc:2362] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27. Re-sending same reply.
I20260809 23:22:27.241475  6210 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.246064 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.246464  5946 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260809 23:22:27.246969 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.247027  6053 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.246721 24543 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:27.248375  5889 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.248593 24435 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
I20260809 23:22:27.249336  5889 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.126552 24423 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.252599  6216 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.253674 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.252727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.254979 24431 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.134171  6211 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.257393  6211 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.257907  6007 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.183169 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.189651  6068 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.214500  5903 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.264369 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:27.239887  6142 log.cc:924] Time spent T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 1.611s	user 0.002s	sys 0.000s
I20260809 23:22:27.238402  6220 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.249719 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.236541 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 57536.5us
I20260809 23:22:27.262282 24427 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:27.171741  6102 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 30 is higher.
I20260809 23:22:27.226991  6020 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.180217 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.269451 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.285128 24423 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:27.274199 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:27.276258 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 46313.8us
I20260809 23:22:27.278584 24269 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.192844 25365 rpcz_store.cc:276] 0809 23:22:26.028618 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:26.029004 (+   386us) service_pool.cc:227] Handling call
0809 23:22:26.029344 (+   340us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:26.037943 (+  8599us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:26.037973 (+    30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:26.038349 (+   376us) log.cc:842] Serialized 26 byte log entry
0809 23:22:26.157990 (+119641us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:26.158050 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:26.158144 (+    94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:26.873380 (+715236us) spinlock_profiling.cc:243] Waited 9.75 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:27.121741 (+248361us) raft_consensus.cc:1671] finished
0809 23:22:27.121768 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:27.121982 (+   214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15570,"raft.queue_time_us":10568,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":8819,"spinlock_wait_cycles":22414592,"thread_start_us":74090,"threads_started":1,"wal-append.queue_time_us":211034}
I20260809 23:22:27.281342 24266 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.281400 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.278302  5903 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.246965  5946 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.305294 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.306406 24270 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.294250 24423 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.309007 24430 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.316486 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.317317 24534 raft_consensus.cc:2379] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.318501 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.318760 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.280498  6210 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:27.281008  6151 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:27.289043  6020 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:27.277910  6202 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:27.278028  6068 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.312688  6099 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.329631 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.315685 24431 raft_consensus.cc:2454] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:27.323463 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.336962 24544 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.311504  5946 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.338290 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.300395  6003 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.311714  6216 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.342658 24496 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.319005  5278 raft_consensus.cc:3037] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:27.324234  6086 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.331465  5494 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:27.320770  6057 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.327353  6210 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.331699  6151 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:27.353165 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.300701  5752 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.360955 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.376447 24498 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.377373 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.352046  6067 raft_consensus.cc:2781] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.364024 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.366636  6068 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.360477  6210 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.379889 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.380774 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.383531 25375 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.377976 25373 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.389361 25369 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:27.372256  6151 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.392168 25368 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:27.363150 24544 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.378517  6048 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.402838  6211 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:27.403672  6211 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.405264  6011 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.405738  6011 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.408226  6000 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:27.383122  6002 raft_consensus.cc:2781] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.412292  6002 raft_consensus.cc:686] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:27.412923 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.413319  6002 consensus_queue.cc:237] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.442863 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 63074.3us
I20260809 23:22:27.388803 24430 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:27.378666  5494 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.354790  6086 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.457765 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.469508 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.360221  6057 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.473090 24270 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:27.352156  5796 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.469079  6011 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.482170 24266 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:27.449719  5611 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.503568 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.476760  6000 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:27.508001 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.472666  5622 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.480939  6019 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
W20260809 23:22:27.513095 24378 leader_election.cc:334] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.510946 24430 raft_consensus.cc:2454] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 23.
I20260809 23:22:27.483435  5903 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.500226  5280 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:27.515218 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.513923  6211 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.517098  5622 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.524398  5622 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.529596  6204 raft_consensus.cc:2781] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:27.530122  6204 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.530519  6204 raft_consensus.cc:3037] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:27.519495  6000 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.531466  6000 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.532459  6053 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.533833  6000 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:27.550915 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 173391us
I20260809 23:22:27.554879 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.466634 25373 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 24.
I20260809 23:22:27.565858 25375 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
I20260809 23:22:27.518289  6033 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.507256  5611 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.549796  6053 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.548815  6019 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.577442 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.579700 25373 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.583222  6033 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.509938  5494 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.521446  6071 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.554674  5280 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:27.587288  5752 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.617576  5280 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:27.619117 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 729928us
I20260809 23:22:27.622367 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.624305  6086 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.630005 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:27.631238  5484 consensus_queue.cc:1035] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:27.632601  5484 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:27.633056  5484 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:27.633908  5484 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.633073 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.635515 24424 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3638) took 1512 ms. Trace:
I20260809 23:22:27.635841 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:27.635847 24424 rpcz_store.cc:276] 0809 23:22:26.123406 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:26.123953 (+   547us) service_pool.cc:227] Handling call
0809 23:22:26.124250 (+   297us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:27.232942 (+1108692us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:27.635278 (+402336us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54003331d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:27.635458 (+   180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2398001920}
I20260809 23:22:27.641500 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.646972 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.622778 25369 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 26.
W20260809 23:22:27.655084 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 211159us
I20260809 23:22:27.656610 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.657253 24424 raft_consensus.cc:3037] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.661680 24543 raft_consensus.cc:2396] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260809 23:22:27.661798 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.663581  6067 raft_consensus.cc:686] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:27.664507  6067 consensus_queue.cc:237] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.665344  5752 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.666137  5752 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:27.675750 24424 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:27.684237 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.685186 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.685497  5611 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.686213 24423 raft_consensus.cc:1261] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:27.688990 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.689857  6245 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:27.697700 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.659304  6068 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.590893  6019 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.595700  6053 consensus_queue.cc:1046] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.087s
I20260809 23:22:27.661393  6086 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.058s
I20260809 23:22:27.707428  6071 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.703325 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:27.744434 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.745371 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.757515 24544 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.706055  6057 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.679390  6020 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.760937  6020 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.708703  5946 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.762334 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.734298 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:27.767076 25369 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:27.747802 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.766299 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.785727 24422 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:27.787467  5889 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:27.754796 25373 raft_consensus.cc:2454] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:27.777843 25369 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
W20260809 23:22:27.790026 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 280123us
I20260809 23:22:27.790442 24378 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:27.768181 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 64964.3us
I20260809 23:22:27.767400 24424 raft_consensus.cc:2379] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.769131 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.738644  5646 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.733144  6234 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.706964  5752 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.772693 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.795215 24380 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.732599  6068 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.799039  6256 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:27.801898 24535 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:27.801898 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.795068  6020 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:27.809689 24424 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:27.796196  6003 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:27.808352 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:27.815506  6245 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:27.816020  6245 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.816403  6245 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:27.816519 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.816784 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.817485 25369 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:27.822417 24380 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.806437  6102 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.811311  6020 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.835125 24498 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.825721  6003 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:27.802973  5646 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.829972  6263 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260809 23:22:27.874572  5946 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.875301 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.810689  6256 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.831686  6020 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:27.870985  5278 consensus_meta.cc:287] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.007s
I20260809 23:22:27.888659 24269 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.804791  6234 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.892879 24429 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:27.893532 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 58507.1us
W20260809 23:22:27.910342 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:27.911124 24498 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:27.911913 24498 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.912154 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.877977  6102 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:27.906177 24269 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:27.913100  6207 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:27.913704 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.914768  6266 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.838754  6210 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.914077 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:27.928373 25369 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:27.903038  6251 raft_consensus.cc:2781] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:27.915700  5278 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:27.920601  5646 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.920939  6068 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.929528  6210 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.924149 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:27.895421  5751 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:27.931094  5751 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.931629  6003 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.933334  5751 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:27.935691 25375 raft_consensus.cc:1226] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 24. Current term is 25. Ops: []
I20260809 23:22:27.936501 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:27.937093 24546 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:27.943481 25369 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260809 23:22:27.944422 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 200971us
I20260809 23:22:27.944839 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.945474  5278 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:27.945830 24431 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:27.946153  5278 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.948239 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:27.948284  6204 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.948995  5278 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.956748  6204 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.956985  6256 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:27.903223  6057 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:27.957975  6057 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:27.960879  6057 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:27.071131 25370 rpcz_store.cc:276] 0809 23:22:25.725435 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.754635 (+ 29200us) service_pool.cc:227] Handling call
0809 23:22:25.805205 (+ 50570us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:25.805480 (+   275us) raft_consensus.cc:1520] Early marking committed up to index 19
0809 23:22:26.867056 (+1061576us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x7b180031bb80. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac234ef82c 00007fac2091a9dc 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20935155 00007fac27235167 00007fac2386a807 00007fac23879f0f
0809 23:22:26.867216 (+   160us) log.cc:842] Serialized 14 byte log entry
0809 23:22:27.065069 (+197853us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:27.065154 (+    85us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:27.065222 (+    68us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:27.065310 (+    88us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:27.065469 (+   159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25045,"prepare.queue_time_us":140245,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":1767,"raft.queue_time_us":175,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":15044,"spinlock_wait_cycles":11444480,"thread_start_us":309411,"threads_started":2}
I20260809 23:22:27.918118  6019 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:27.991389 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:27.915413  6207 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:28.001758 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.005379 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:27.939570  5889 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.007782 24546 raft_consensus.cc:2454] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:28.014519 25325 leader_election.cc:302] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:27.962026  6251 raft_consensus.cc:686] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
W20260809 23:22:28.022482 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:27.996630  6207 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:27.996193  6167 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.040274 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.041168 25323 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.056913 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:28.045334 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 222990us
I20260809 23:22:28.053983  5643 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.058179 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:28.066692 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.013628  6019 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:28.040169  6224 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.125578  6224 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.024307  5622 raft_consensus.cc:2781] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:28.127275  5622 raft_consensus.cc:686] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:28.128255  5622 consensus_queue.cc:237] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.130090  6007 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.131565 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.076460 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:28.133145 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 66511.9us
I20260809 23:22:28.134315  6091 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:28.077306 24380 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:28.136106 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 58907.6us
I20260809 23:22:28.064821 24269 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.059749  6129 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:22:28.084108 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 60030.5us
I20260809 23:22:28.089208  5952 raft_consensus.cc:3032] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Stepping down as leader of term 24
W20260809 23:22:28.093127  6151 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.695s	user 0.015s	sys 0.007s
I20260809 23:22:28.071702 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.022047  5889 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.039961  6207 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.123662  6245 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.120983  6251 consensus_queue.cc:237] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:28.139529 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 74822.4us
I20260809 23:22:28.078157  5494 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260809 23:22:28.144147 24270 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:28.146564 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 234638us
I20260809 23:22:28.150657 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.068101  5484 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:28.120431  6234 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.173913 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:28.176755 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.177309 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.177553 24545 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Advancing to term 22
I20260809 23:22:28.171448 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.180310 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.183236 24541 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260809 23:22:28.187727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.145339  5952 raft_consensus.cc:729] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:28.188736  6234 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:28.184090 25370 raft_consensus.cc:2454] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 21.
I20260809 23:22:28.143932  6033 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.155323  5959 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.152172  5280 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.152750  6151 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.197589 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.220537 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.203352 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.204216  6220 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:28.206159 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.194268  6086 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.210630 24431 raft_consensus.cc:2379] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:28.153321  6267 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.217854 24540 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:28.183027  6204 raft_consensus.cc:3037] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:28.257398  5952 consensus_queue.cc:260] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.201455  5643 consensus_queue.cc:1035] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260809 23:22:28.267256 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.270588 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.219424  6102 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.274564 24542 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 31.
I20260809 23:22:28.274212 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.251780  5751 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.252935  6168 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:28.295914 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 115642us
I20260809 23:22:28.285137 25368 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260809 23:22:28.350835 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.352185 24431 raft_consensus.cc:1261] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:28.266214  6086 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.274660  6224 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.344508  5952 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
W20260809 23:22:28.370949 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 195842us
I20260809 23:22:28.335400  6003 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.350783  6030 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:28.382095  6295 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260809 23:22:28.382433  6030 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.384661  6030 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.382126  6033 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.004s
I20260809 23:22:28.387401  6033 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.387856  6033 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.389786  6033 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:28.392584 24378 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:28.395186 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.269726  6267 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.421291 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.434533 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.435500 24544 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:28.439162 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:28.441965 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.442613 24544 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:28.339320  6168 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.452057 24544 raft_consensus.cc:2454] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 31.
I20260809 23:22:28.453454 24380 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:28.258697  6019 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.465204  6204 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:28.472409 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.473652  5952 consensus_queue.cc:1035] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:28.474100  5611 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Advancing to term 24
W20260809 23:22:28.474303  6293 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260809 23:22:28.474651  6012 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:28.476486  6102 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:28.477119  6102 consensus_queue.cc:1225] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260809 23:22:28.481667  5611 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.482735  5891 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:28.490298  6086 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.491328  5903 raft_consensus.cc:2781] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:28.491976  5903 raft_consensus.cc:686] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:28.492918  5903 consensus_queue.cc:237] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.500684 24378 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 146 similar messages]
I20260809 23:22:28.498771 24545 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:28.440974  6245 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.497515 25368 pending_rounds.cc:76] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 17
I20260809 23:22:28.512714 25368 pending_rounds.cc:97] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260809 23:22:28.513028 25368 raft_consensus.cc:2866] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:28.452557  5889 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.416946  6003 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.496945  6102 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:28.518695 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.519364  6316 raft_consensus.cc:482] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.512013  6057 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:28.520706  6267 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:28.541411 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:28.557477 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 136338us
I20260809 23:22:28.564745 24544 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:28.557772 24425 raft_consensus.cc:2396] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
W20260809 23:22:28.561934 24539 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4046) took 2846 ms (client timeout 1852 ms). Trace:
W20260809 23:22:28.516104  6295 log.cc:924] Time spent T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260809 23:22:28.566542 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:28.566210 24539 rpcz_store.cc:269] 0809 23:22:25.696806 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:25.696957 (+   151us) service_pool.cc:227] Handling call
0809 23:22:28.542673 (+2845716us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001fcf54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.542833 (+   160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2919915392}
I20260809 23:22:28.535822  6102 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.562881  6151 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.574476  6245 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.576292  6102 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:28.579164  5796 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.012s
W20260809 23:22:28.589347 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 136015us
I20260809 23:22:28.595448 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
W20260809 23:22:28.598832 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.590024 24424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.596902  6020 consensus_meta.cc:287] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.000s
I20260809 23:22:28.593117 25321 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.571201  6316 raft_consensus.cc:504] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.589416  5751 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:28.604923  5891 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260809 23:22:28.616772 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.588001  6319 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.619144 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.626758 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:28.627672  5891 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.612176  6020 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.634263  6207 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.643045  6319 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.599432  6245 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.621812  6320 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.611755  6168 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:28.673609 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.680601 24546 raft_consensus.cc:2379] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:28.669952 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.686165 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2662351f0e4b14bede1be0deaba6c8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.691102 24542 raft_consensus.cc:2354] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:28.666915  5891 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:28.694362 24422 raft_consensus.cc:2354] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
W20260809 23:22:28.688843 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 69877.4us
I20260809 23:22:28.706975  5889 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.724108 24380 leader_election.cc:398] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
I20260809 23:22:28.724448 24380 leader_election.cc:401] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:28.710436 24542 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:28.692081  6322 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.668921  6316 leader_election.cc:288] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.669024  6207 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.720750  6020 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.726416 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:28.727671 24378 leader_election.cc:334] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.734675 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.735795  6003 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.735803  5751 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.736475  5751 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.736724 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.738806  5751 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.745124  6303 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.018s
W20260809 23:22:28.755206 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.741375  6305 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:28.747293 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.766431 24541 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 24.
W20260809 23:22:28.767177 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:28.767719 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:28.770752 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34784 (request call id 3668) took 1502 ms. Trace:
I20260809 23:22:28.771174  6303 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:28.771142 24435 rpcz_store.cc:276] 0809 23:22:27.268658 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.277224 (+  8566us) service_pool.cc:227] Handling call
0809 23:22:27.306226 (+ 29002us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:27.350391 (+ 44165us) raft_consensus.cc:1520] Early marking committed up to index 13
0809 23:22:27.364557 (+ 14166us) log.cc:842] Serialized 14 byte log entry
0809 23:22:28.048153 (+683596us) spinlock_profiling.cc:243] Waited 7.47 ms on lock 0x7b180032f380. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e757b 00007fac2386a88a
0809 23:22:28.215585 (+167432us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0809 23:22:28.217210 (+  1625us) raft_consensus.cc:1631] Marking committed up to 13
0809 23:22:28.217280 (+    70us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:28.770535 (+553255us) spinlock_profiling.cc:243] Waited 43.9 ms on lock 0x7b540012edd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:28.770583 (+    48us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:28.770697 (+   114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":13879,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":3121,"raft.queue_time_us":2296,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":31924,"spinlock_wait_cycles":118229632,"thread_start_us":167133,"threads_started":1,"wal-append.queue_time_us":1010082}
W20260809 23:22:28.772575 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 597288us
W20260809 23:22:28.773224 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.773834 24535 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4091) took 1445 ms. Trace:
W20260809 23:22:28.774111 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.774224 24535 rpcz_store.cc:276] 0809 23:22:27.328767 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.329084 (+   317us) service_pool.cc:227] Handling call
0809 23:22:28.773641 (+1444557us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400116a54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.773772 (+   131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1069730688}
I20260809 23:22:28.775460 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38102 (request call id 4082) took 1459 ms. Trace:
I20260809 23:22:28.732752  6306 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:28.775403  5891 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.776026 24543 rpcz_store.cc:276] 0809 23:22:27.315685 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.318240 (+  2555us) service_pool.cc:227] Handling call
0809 23:22:28.775286 (+1457046us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400270bd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.775408 (+   122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":784302848}
I20260809 23:22:28.776407  6306 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.776881 24425 raft_consensus.cc:2354] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
I20260809 23:22:28.777864  5891 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.778458  6306 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:28.778786 24270 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.683671  6325 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:28.780524 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.782727 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:28.786743 24498 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:28.787911 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.766124  6305 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:28.755048 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.730187  6323 raft_consensus.cc:3037] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.747643  6044 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:28.799026 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:28.810324 24435 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:28.745177  5646 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:28.816355 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.793968  6099 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:28.819128 25373 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 30.
I20260809 23:22:28.771682  6303 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.820333 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.730816  5889 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.778744  6102 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:28.829128 22524 thread.cc:536] Waited for 1000ms trying to join with heartbeat (tid 24356)
W20260809 23:22:28.843526 24378 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.828908  6303 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:28.848399 24424 raft_consensus.cc:2379] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:28.795316  6325 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:28.856812  6102 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.858306 24423 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47454 (request call id 1920) took 1170 ms. Trace:
I20260809 23:22:28.864574  6325 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.863843 24423 rpcz_store.cc:276] 0809 23:22:27.687496 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.687630 (+   134us) service_pool.cc:227] Handling call
0809 23:22:27.687905 (+   275us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:27.688192 (+   287us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:27.688214 (+    22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:28.648197 (+959983us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x7b2c000002e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:28.648740 (+   543us) log.cc:842] Serialized 26 byte log entry
0809 23:22:28.781300 (+132560us) raft_consensus.cc:1631] Marking committed up to 15
0809 23:22:28.781360 (+    60us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:28.781452 (+    92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:28.857926 (+ 76474us) raft_consensus.cc:1671] finished
0809 23:22:28.857951 (+    25us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:28.858228 (+   277us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":324,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":11399424,"thread_start_us":132114,"threads_started":1,"wal-append.queue_time_us":172244}
I20260809 23:22:28.868345 25365 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260809 23:22:28.869563  5889 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.871151 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.871950 24541 raft_consensus.cc:2379] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 31: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:28.875771  6319 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:28.885749  6285 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.879s	user 0.000s	sys 0.006s
I20260809 23:22:28.896502  6305 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:28.899175 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 110382us
W20260809 23:22:28.900269 24498 leader_election.cc:334] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:28.826129  5646 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.904138 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.908155 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.908881 25369 raft_consensus.cc:2354] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:28.911520 25369 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260809 23:22:28.912200 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.912403 25370 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:28.919412 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.921950  5646 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.924355 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.925354 25371 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:28.927680 25366 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260809 23:22:28.929885 25366 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:28.931274 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:28.923039  6303 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:28.935571 25370 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:28.937703 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:28.936764 24269 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:28.943602 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 21 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:28.945454 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.945955 24266 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.956172 25363 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:28.960384 25371 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:28.971796 24541 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260809 23:22:28.972307 24541 pending_rounds.cc:76] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 15
I20260809 23:22:28.972579 24541 pending_rounds.cc:97] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260809 23:22:28.972855 24541 raft_consensus.cc:2866] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:28.978327 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:28.979553  6224 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:28.979924  6224 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:28.980247  6224 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:28.981228 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:28.983448  5611 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:28.984575  5278 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:28.985450  6303 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:28.987682  6207 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:22:28.990108 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 44291.5us
I20260809 23:22:28.991106  6213 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:28.996532 24422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34778 (request call id 4118) took 1125 ms. Trace:
I20260809 23:22:28.997298 24422 rpcz_store.cc:276] 0809 23:22:27.871335 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:27.871563 (+   228us) service_pool.cc:227] Handling call
0809 23:22:28.996163 (+1124600us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540023df54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:28.996309 (+   146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1833029376}
I20260809 23:22:28.999296 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:28.999914 24542 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:28.945173  6000 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260809 23:22:28.978539  6263 raft_consensus.cc:2781] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:29.003440  6263 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.003844  6263 raft_consensus.cc:3037] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:28.956609  6091 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260809 23:22:28.961369  5643 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.005681  5889 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:29.006372 24540 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:29.008004  5280 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260809 23:22:28.947809  6323 raft_consensus.cc:2726] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 27. Current term is 28.
W20260809 23:22:29.008734 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 162716us
I20260809 23:22:28.978370 24534 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 27 index: 24. (index mismatch)
I20260809 23:22:29.025513 24431 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260809 23:22:29.028427 24378 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.035451 24542 raft_consensus.cc:2454] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:29.037029 24269 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:29.036374 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.043553 25325 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:29.053077 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.055277 24269 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.010164  6277 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.070760  6266 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:29.075687  6086 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:29.076207  6086 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:29.076346 24496 leader_election.cc:398] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
I20260809 23:22:29.076686 24496 leader_election.cc:401] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.076563  6277 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.077132  6086 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.078773  6167 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:29.080726 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.089993 24498 leader_election.cc:398] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:29.090588 24498 leader_election.cc:401] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.068316  6166 raft_consensus.cc:2781] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:29.094683 24266 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.040155  6000 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
W20260809 23:22:29.115561 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 347778us
W20260809 23:22:29.117462 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:29.123816 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.132845  5484 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.117250  5311 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 21. Current term is 22.
I20260809 23:22:29.133409  5484 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.136276  5484 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.139683 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.149374 24496 leader_election.cc:398] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
I20260809 23:22:29.149745 24496 leader_election.cc:401] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260809 23:22:29.090816  5278 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260809 23:22:29.165365  6245 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.168174  6033 raft_consensus.cc:482] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.171653 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:29.171939 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 49081.7us
W20260809 23:22:29.174366 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 48376.2us
I20260809 23:22:29.180693 25371 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:29.117650  6166 raft_consensus.cc:686] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:29.183781 25368 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:29.124742  6211 raft_consensus.cc:3037] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:29.190130 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:29.194779 24424 raft_consensus.cc:1261] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:29.174623  6325 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.174319  5752 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260809 23:22:29.206270  6274 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:29.214988  6003 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260809 23:22:29.186625  6033 raft_consensus.cc:504] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.215294  6245 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:29.232446 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 50374.5us
I20260809 23:22:29.198280  6048 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.029s
I20260809 23:22:29.217775  6102 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.219681  5891 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:29.235083 24378 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.230804  6003 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:29.228585  6207 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.230248  6216 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.230229  5951 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260809 23:22:29.221923  6325 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.244133  6044 consensus_queue.cc:1035] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.028s
I20260809 23:22:29.245190  6245 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.245442  6102 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.251034  5891 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.279120 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.291311 24380 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:29.301491 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 532874us
W20260809 23:22:29.301928 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.262517  6003 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Advancing to term 31
W20260809 23:22:29.304397 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.305032 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.302237 24380 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:29.310909 24498 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.270689  6263 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.320116 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.313673  6262 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:29.282055  6207 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:29.328054 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.319904  6234 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.332024  6234 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.333470  5891 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.334303  6234 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:29.334534  6358 log.cc:924] Time spent T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260809 23:22:29.339645  6102 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.341349  6033 leader_election.cc:288] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.289639  6216 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.344710  6216 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.316192  6249 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.349792 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.351223  6211 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:29.352075  6325 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.352182 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.355916  6263 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.242332  6166 consensus_queue.cc:237] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.329753 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:29.366266 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.366753 25323 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.325383  6364 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:22:29.377243  6224 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.381821  6167 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 25. Current term is 26.
W20260809 23:22:29.385751 25323 leader_election.cc:334] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:29.387233 25323 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.387871 25323 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.392994 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.369594  6086 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.404321 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.415148 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:29.329864  6289 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:29.442260 24431 raft_consensus.cc:1261] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260809 23:22:29.382521  6277 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:29.422472 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.455144 24270 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.456943 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.478885 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:29.483618 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 176004us
W20260809 23:22:29.490628 24498 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.491226 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:29.490655 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.492779 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.398702  6305 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:22:29.492833 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:29.416991  6303 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.496940 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:29.504588  6033 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:29.508306 24428 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 27 index: 24. (index mismatch)
I20260809 23:22:29.418002  6319 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:29.518443 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.519110 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
W20260809 23:22:29.519516 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.519936 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.520310 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:29.521128 24542 raft_consensus.cc:3037] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:29.523476 24428 raft_consensus.cc:1261] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:29.523377 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.527539 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.517169 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:29.535363 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.537468 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.470103  6364 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.533131 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.620851 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.397507  6086 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.565984 24545 raft_consensus.cc:1261] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260809 23:22:29.453361  6314 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:29.633586 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 94783us
I20260809 23:22:29.508273  6216 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:29.509713  6366 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.619578  6224 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:29.680168 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 147129us
I20260809 23:22:29.675570 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:29.666838 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 159556us
I20260809 23:22:29.668076 24534 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2119) took 1294 ms. Trace:
I20260809 23:22:29.683622 24534 rpcz_store.cc:276] 0809 23:22:28.373544 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:28.373727 (+   183us) service_pool.cc:227] Handling call
0809 23:22:28.381563 (+  7836us) raft_consensus.cc:1457] Updating replica for 0 ops
0809 23:22:29.009993 (+628430us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:29.660601 (+650608us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b5400271fd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:29.668016 (+  7415us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1371737984}
I20260809 23:22:29.669800 24535 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260809 23:22:29.659588 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d375db793a4542808e1cff72fb6617" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:29.490082  6003 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.686534 24542 raft_consensus.cc:2454] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:29.627787  6325 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.695003 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:29.729732 24380 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.527725  6303 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.676772  6167 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.528160  6033 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.620339  6044 consensus_queue.cc:1035] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260809 23:22:29.652731  6319 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:29.786288 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:29.787987 24431 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 31: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
W20260809 23:22:29.792759 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 273240us
I20260809 23:22:29.758550  6102 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:29.800539 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:29.676522  6314 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.732182  6325 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:29.841614 22524 thread.cc:536] Waited for 2000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:29.688874  6366 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.677030  6364 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.792037  6319 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:29.948511 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 264971us
W20260809 23:22:29.943259 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 207260us
I20260809 23:22:29.767163  6167 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.958920 25370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37942 (request call id 2018) took 1040 ms. Trace:
I20260809 23:22:29.959335 25370 rpcz_store.cc:276] 0809 23:22:28.917999 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:28.931019 (+ 13020us) service_pool.cc:227] Handling call
0809 23:22:29.958709 (+1027690us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b54006d5954. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:29.958845 (+   136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8089856}
I20260809 23:22:29.960501 24380 leader_election.cc:302] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.963378  6086 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.963943  6086 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:29.964573  6086 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:29.966509  6086 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.963992  6224 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:29.972597 24378 leader_election.cc:334] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:29.975616 24378 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:29.959460  5718 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:29.988380  5718 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:29.988814  5718 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:29.992756  6224 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:29.997845 24380 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:29.939625  6207 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.003338  6365 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:30.003993  6376 consensus_queue.cc:1035] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.214s
W20260809 23:22:30.004859  6379 log.cc:924] Time spent T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
W20260809 23:22:30.014521  6370 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260809 23:22:30.028695 25371 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260809 23:22:30.025709 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.029778 25370 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Advancing to term 31
I20260809 23:22:29.951395  6057 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.063s
I20260809 23:22:30.044296  6366 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.044764  6057 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.045436  6057 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.046584  6319 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.047154  6319 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.047361  6003 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.047659  6319 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.049620  6319 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:29.999951  6091 raft_consensus.cc:2726] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:30.035944  6314 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.014873  6377 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.037338  6167 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.037880  6033 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.038094  6303 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.003548  6289 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260809 23:22:30.040325  6364 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.041396  6325 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.042060  6393 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.019601  6365 raft_consensus.cc:2781] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:30.019438  6376 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:30.013947  6298 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:30.074755 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.049896  6274 consensus_meta.cc:287] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Time spent flushing consensus metadata: real 0.817s	user 0.008s	sys 0.004s
I20260809 23:22:30.096638 25370 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 31.
W20260809 23:22:30.112140 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.114289 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:30.103006  6224 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.110958  6393 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.125324  6393 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.114042  6365 raft_consensus.cc:686] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:30.127018  6365 consensus_queue.cc:237] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.120285  6274 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 31. Current term is 32.
W20260809 23:22:30.133723  6378 log.cc:924] Time spent T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.331s	user 0.001s	sys 0.001s
I20260809 23:22:30.135874  5718 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.142364 24544 raft_consensus.cc:2454] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 32.
W20260809 23:22:30.146517 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:30.148890 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 176328us
I20260809 23:22:30.150846 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.145202  6057 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:30.153788 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 333202us
I20260809 23:22:30.153131 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.155292 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 335888us
W20260809 23:22:30.167338 24378 leader_election.cc:334] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.167799 24378 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.146167  6305 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.021s	sys 0.005s
I20260809 23:22:30.174012 24498 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:30.171269 24378 leader_election.cc:334] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.172328 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:30.171696 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.177520 24378 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.177795 24428 raft_consensus.cc:2454] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:30.178638 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.179766  6391 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.180486  6391 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.173967 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.181959 24422 raft_consensus.cc:2396] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260809 23:22:30.183424 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.184262 25375 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
I20260809 23:22:30.164703  6129 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260809 23:22:30.191965  6376 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:30.192364  6373 log.cc:924] Time spent T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.173s	user 0.002s	sys 0.000s
I20260809 23:22:30.155808  6310 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.196872  5718 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.197512  6289 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.207968 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.208840 25366 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 31.
W20260809 23:22:30.222159 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.223004 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.225497 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.227519 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.231652 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.234162  6204 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.238703 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.240710 24535 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 18. (term mismatch)
I20260809 23:22:30.241209 24535 pending_rounds.cc:76] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting all ops after (but not including) 17
I20260809 23:22:30.220878  6151 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.241562 24535 pending_rounds.cc:97] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260809 23:22:30.241876 24535 raft_consensus.cc:2866] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:30.242440 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.234813  6204 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.219775  6289 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
W20260809 23:22:30.251346  6369 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.609s	user 0.000s	sys 0.002s
I20260809 23:22:30.252264 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.253424 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.253569 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.256408 24539 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:30.236482  6393 raft_consensus.cc:482] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.257007  6393 raft_consensus.cc:504] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.260061  6393 leader_election.cc:288] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.260644  6289 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.262656  6289 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.262951 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 26 index: 23 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:30.182780  6391 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.268234 24422 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.264564  6204 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.265668 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.273166 24428 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 24.
I20260809 23:22:30.275051 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.277838 24422 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:30.283571 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.284569 24545 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:30.287079 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.287204 25325 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.288888 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.290755 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.291570 24545 raft_consensus.cc:2379] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 25: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:22:30.293288 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.294019 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.294126 24380 leader_election.cc:302] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.294797 24270 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.313452 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.335041 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.291353  6204 raft_consensus.cc:2781] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:30.341583 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.296168  6289 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.342124  6408 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.295223  5646 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:30.390167 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 51379.8us
I20260809 23:22:30.385376  6224 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:30.399238  6387 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260809 23:22:30.314400  6413 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:30.343627  6415 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:30.379622 24270 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.380391  6033 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.380162  5622 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:30.405004 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 375566us
I20260809 23:22:30.420284 24496 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.461480 24496 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:30.432926 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 222190us
I20260809 23:22:30.416371  6289 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.450454  6245 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.464824  6249 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:30.467716 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.468819  6408 consensus_queue.cc:1035] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:30.470077  6274 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
W20260809 23:22:30.471472 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 91925.4us
I20260809 23:22:30.394135  6401 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 26.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.113s
I20260809 23:22:30.420491  6204 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:30.337833  6364 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.454116  6415 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:30.462309  5796 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.493911 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.477044  6249 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:30.477108  6309 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:30.489100  6274 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:30.486721  6320 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.475520  5622 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.473891  6033 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.522477 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:30.484715  6204 raft_consensus.cc:3037] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.494741  6263 raft_consensus.cc:482] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.573974  6310 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.575886  6400 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260809 23:22:30.517730  6057 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.059s
I20260809 23:22:30.576323  6400 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.576893  6400 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:30.579653  5622 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.495834  6364 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.587651  6364 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.484040  6413 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.491158  6415 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:30.591080  6413 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.515374  6394 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:30.495252  5796 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.593947  5796 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.515373  6249 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.520745  5887 raft_consensus.cc:482] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.599334  5887 raft_consensus.cc:504] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.573349  6263 raft_consensus.cc:504] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.601696  5887 leader_election.cc:288] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.601752  6263 leader_election.cc:288] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.574505  6310 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.576264  6401 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:30.602900  6430 log.cc:924] Time spent T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260809 23:22:30.603266  6401 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.604964 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.605810  6401 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.606057  6410 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Stepping down as leader of term 27
I20260809 23:22:30.606554  6410 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:30.608321  6310 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:30.610900 24167 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 244) took 12342 ms (client timeout 15000 ms). Trace:
I20260809 23:22:30.612020  6289 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.616720  6033 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:30.617892  6427 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260809 23:22:30.619565  6204 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.620743 24544 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38092 (request call id 4050) took 1102 ms. Trace:
I20260809 23:22:30.517678  6391 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.621106 24544 rpcz_store.cc:276] 0809 23:22:29.518580 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.518836 (+   256us) service_pool.cc:227] Handling call
0809 23:22:30.620591 (+1101755us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400081e54. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac238642ae 00007fac287a1c27 00007fac21e89c04 00007fac21e89b92 00007fac21e89ac1 00007fac21e89a3c 00007fac21e88cf2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:30.620689 (+    98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1409682176}
I20260809 23:22:30.621526  6391 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.623610  6391 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:30.623692 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.624522 24539 raft_consensus.cc:2379] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 in current term 32: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:30.626145 24380 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:30.630682 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:30.572386 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 50034.5us
I20260809 23:22:30.633162 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.626751 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:30.612106 24167 rpcz_store.cc:269] 0809 23:22:18.268421 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:18.274317 (+  5896us) service_pool.cc:227] Handling call
0809 23:22:30.610793 (+12336476us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:23.822940 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:23.823572 (+   632us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:23.823625 (+    53us) write_op.cc:435] Acquired schema lock
0809 23:22:23.823639 (+    14us) tablet.cc:660] Decoding operations
0809 23:22:23.966958 (+143319us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:23.986759 (+ 19801us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:23.998275 (+ 11516us) tablet.cc:683] Acquiring locks for 21 operations
0809 23:22:24.226514 (+228239us) tablet.cc:699] Row locks acquired
0809 23:22:24.226565 (+    51us) write_op.cc:260] PREPARE: finished
0809 23:22:24.226746 (+   181us) write_op.cc:270] Start()
0809 23:22:24.226923 (+   177us) write_op.cc:276] Timestamp: P: 1786317744226673 usec, L: 0
0809 23:22:24.226987 (+    64us) op_driver.cc:331] REPLICATION: starting
0809 23:22:24.227756 (+   769us) log.cc:842] Serialized 8188 byte log entry
0809 23:22:25.211765 (+984009us) op_driver.cc:444] REPLICATION: finished
0809 23:22:25.666216 (+454451us) write_op.cc:301] APPLY: starting
0809 23:22:25.677042 (+ 10826us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:26.372663 (+695621us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:26.372780 (+   117us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:26.716034 (+343254us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:26.716479 (+   445us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=0,mrs_lookups=0
0809 23:22:26.716542 (+    63us) write_op.cc:312] APPLY: finished
0809 23:22:26.737139 (+ 20597us) log.cc:842] Serialized 188 byte log entry
0809 23:22:26.954289 (+217150us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:26.954502 (+   213us) write_op.cc:454] Released schema lock
0809 23:22:26.954577 (+    75us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:26.954958 (+   381us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:26.954986 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18263,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":399735,"apply.run_cpu_time_us":78820,"apply.run_wall_time_us":1310652,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91823,"num_ops":21,"prepare.queue_time_us":590547,"prepare.run_cpu_time_us":22360,"prepare.run_wall_time_us":483228,"raft.queue_time_us":584768,"raft.run_cpu_time_us":18002,"raft.run_wall_time_us":190163,"replication_time_us":930849,"thread_start_us":1070166,"threads_started":5,"wal-append.queue_time_us":537830,"wal-append.run_cpu_time_us":39929,"wal-append.run_wall_time_us":2446634}]]}
I20260809 23:22:30.638525 25372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2213) took 1564 ms. Trace:
I20260809 23:22:30.650219 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.651175 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:30.652544 24380 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:30.640066 25372 rpcz_store.cc:276] 0809 23:22:29.074691 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.074878 (+   187us) service_pool.cc:227] Handling call
0809 23:22:29.081180 (+  6302us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:29.082511 (+  1331us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:29.082578 (+    67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:29.083113 (+   535us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.250211 (+1167098us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b18001a49a0. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:30.409689 (+159478us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:30.417767 (+  8078us) log.cc:842] Serialized 14 byte log entry
0809 23:22:30.422013 (+  4246us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:30.422240 (+   227us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:30.627203 (+204963us) raft_consensus.cc:1671] finished
0809 23:22:30.638320 (+ 11117us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:30.638460 (+   140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3918,"prepare.queue_time_us":6109,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":4859,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":10005,"spinlock_wait_cycles":13354624,"thread_start_us":159276,"threads_started":1,"wal-append.queue_time_us":1390902}
I20260809 23:22:30.655925 24534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.680495 24539 raft_consensus.cc:1261] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:30.639293  5891 raft_consensus.cc:2781] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election won for term 32
I20260809 23:22:30.681834 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.678589  6033 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260809 23:22:30.685979 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:30.699777 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.702754  6319 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.712111  6319 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.675662  6262 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.675225  5952 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:30.698736  6033 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.703305  6401 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:30.672895  6415 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.708197  5891 raft_consensus.cc:686] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:30.726025  5887 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
W20260809 23:22:30.748376 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 229125us
I20260809 23:22:30.751593 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.738646  6033 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:30.754842 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.766450 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:30.767236 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.714816  6319 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:30.754976  6211 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.040s	sys 0.073s
I20260809 23:22:30.702594  6274 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 26.23, Last appended by leader: 23, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.774724 24431 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:30.776288 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.730525  6262 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.729816  5952 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:30.774638  6211 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.040s	sys 0.073s
I20260809 23:22:30.785993  6400 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.811599 25370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.783614 24430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1115ecb68bd4c399da85a680700bb7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.724414  6410 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 27.24, Last appended by leader: 24, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.785236 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.810597 25366 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260809 23:22:30.744638  6401 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:30.816665 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 66854.1us
I20260809 23:22:30.820720 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:30.825067 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:30.828670  6438 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260809 23:22:30.829162 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.829546  5952 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.753249  5887 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.819751  6204 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.792711  6210 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:30.676205  6249 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.835923 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34970da619114abc942b839d0862a300" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:30.842847 22524 thread.cc:536] Waited for 3000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:30.844666 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ed6a400f2643da95b8f744ebab6f4c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:30.845010  6435 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260809 23:22:30.835348  5891 consensus_queue.cc:237] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:30.848147 24269 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.850003 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.850306  6019 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader election won for term 31
I20260809 23:22:30.850890  6019 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:30.855798  6415 leader_election.cc:288] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.857884  6401 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:30.859423 24544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:30.860459  6262 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.778189  6437 consensus_queue.cc:1035] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.083s
I20260809 23:22:30.862895 24544 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:30.863456  6249 leader_election.cc:288] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.864506  6410 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:30.866971  6400 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.815371  6339 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:30.867606  6339 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:30.850421  5887 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:30.869706  6339 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
I20260809 23:22:30.873853 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.835850  6266 raft_consensus.cc:482] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:30.882328 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.887457 25373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:30.888442 25373 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.888592 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.884721 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:30.893685 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.893760 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
W20260809 23:22:30.899603 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.902673 25321 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.904194 25321 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.905962 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.835290  6129 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
W20260809 23:22:30.945564 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 50707.4us
I20260809 23:22:30.960678 25373 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
W20260809 23:22:30.968525 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 206221us
W20260809 23:22:30.973867 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67996.5us
I20260809 23:22:30.939066  5643 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:30.991780 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:30.991827 25366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:30.993304 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:30.944350  6339 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:30.944554  6437 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:30.997969 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:30.999934 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:30.851998  6019 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.008605 24431 raft_consensus.cc:3037] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:30.980954  6445 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260809 23:22:31.013556 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:30.906332  6266 raft_consensus.cc:504] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.041426  6266 leader_election.cc:288] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.016420 24269 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.024523  6298 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.025206 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.010126  5643 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:31.034796  6216 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.079082 24269 leader_election.cc:302] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.035948  6415 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.045794 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.005223  6166 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:22:31.081295 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64378.4us
I20260809 23:22:31.013229  6448 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:31.107822 24498 leader_election.cc:302] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:31.074162  6298 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:31.144768 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 65784.7us
I20260809 23:22:31.147352 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.149869 24435 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34778 (request call id 4143) took 1725 ms. Trace:
I20260809 23:22:31.150333 24435 rpcz_store.cc:276] 0809 23:22:29.424542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.424720 (+   178us) service_pool.cc:227] Handling call
0809 23:22:29.496986 (+ 72266us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.157410 (+660424us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:30.157454 (+    44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.220193 (+ 62739us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.426115 (+205922us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:30.426171 (+    56us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.084322 (+658151us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5471 00007fac23860f51 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772 00007fac21c8a94c 00007fac21c89d86
0809 23:22:31.084374 (+    52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.084419 (+    45us) raft_consensus.cc:1671] finished
0809 23:22:31.084431 (+    12us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:31.149786 (+ 65355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4005,"raft.queue_time_us":43000,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":1146937088,"thread_start_us":308665,"threads_started":2,"wal-append.queue_time_us":284961}
I20260809 23:22:31.082870  5494 raft_consensus.cc:3037] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:31.143747  6032 raft_consensus.cc:482] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.155614  6412 raft_consensus.cc:2781] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:31.186321 24540 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Rejecting Update request from peer b9b54436d0e94a5595cdded73fe33da9 for earlier term 27. Current term is 28. Ops: [27.24-27.24]
I20260809 23:22:31.048557  6204 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.119s
I20260809 23:22:31.157087  6118 raft_consensus.cc:2726] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:31.192795 24266 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.159673  6057 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:31.195014 24431 raft_consensus.cc:2454] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:31.169183  6263 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.091652  6216 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.212596  6216 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.193813  6032 raft_consensus.cc:504] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.204213  6412 raft_consensus.cc:686] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:31.215999  5280 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.215821  6412 consensus_queue.cc:237] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.216393  6032 leader_election.cc:288] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.216689  5280 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.218523  6032 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.220600 24266 leader_election.cc:302] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.208884  6298 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:31.144115  6166 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.222126 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.223088  6166 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.223362  6129 raft_consensus.cc:2726] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.224741  6449 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.224956 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:31.209930  6452 log.cc:924] Time spent T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260809 23:22:31.188084  6448 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.228896  6032 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.230139  6033 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.232718 24422 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:31.145380  6415 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.236526  6415 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.220829  6204 raft_consensus.cc:3037] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Advancing to term 32
W20260809 23:22:31.251200 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 96415.1us
W20260809 23:22:31.254530 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.255070 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:31.258095 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.263586 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.263748  6166 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:31.268221  6033 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.259423  6032 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.266788 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9429f7311dd744bea4301956e143e9a7" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.242056  6057 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:31.274833  6436 log.cc:924] Time spent T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.500s	user 0.000s	sys 0.005s
I20260809 23:22:31.256757  5280 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.250442  6263 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.285980 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.293576 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.312700 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.313758 25369 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.318058 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:31.322849  6263 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.324079 24378 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.324316 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d273006c869f4749852291bcee467158" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:31.325148 25372 raft_consensus.cc:2354] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.325532 24378 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.325953 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "974355bb93274291bb6be768b656cc67" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.326481 24378 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.288092  5494 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.328280 24378 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.328783 25369 raft_consensus.cc:2379] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 31: Already voted for candidate 60d5c69728134c8bb918a900eb6a8c21 in this term.
I20260809 23:22:31.331079 24378 leader_election.cc:401] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260809 23:22:31.332660 24378 leader_election.cc:302] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.335151 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.338094 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1955f952024cb2815f7065f861cf25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.342041 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:31.347718 24378 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.330001  5891 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.329509  6415 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:22:31.402470  6371 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:31.335440  6325 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
I20260809 23:22:31.336084  6316 raft_consensus.cc:2726] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:31.372613  6057 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.407027 24429 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:34030 (request call id 61) took 1649 ms. Trace:
I20260809 23:22:31.434265  6204 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:31.435807  6213 raft_consensus.cc:482] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.412611 24429 rpcz_store.cc:276] 0809 23:22:29.757518 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:29.757657 (+   139us) service_pool.cc:227] Handling call
0809 23:22:31.378805 (+1621148us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54003ea8d4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac287a5c6e 00007fac21e9c807 00007fac21e9c792 00007fac21e9c6c1 00007fac21e9c63c 00007fac21e9b8f2 00007fac21c8a94c 00007fac21c89d86 00007fac21c8972f
0809 23:22:31.406912 (+ 28107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1460627712}
I20260809 23:22:31.441056 25370 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260809 23:22:31.448740  6086 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:31.453197  6213 raft_consensus.cc:504] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:31.457629  6428 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.789s	user 0.003s	sys 0.015s
I20260809 23:22:31.462404 24543 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38102 (request call id 4138) took 1236 ms. Trace:
I20260809 23:22:31.462796 24543 rpcz_store.cc:276] 0809 23:22:30.226245 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.226393 (+   148us) service_pool.cc:227] Handling call
0809 23:22:30.226762 (+   369us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.232694 (+  5932us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:30.232722 (+    28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.233290 (+   568us) log.cc:842] Serialized 26 byte log entry
0809 23:22:30.471548 (+238258us) raft_consensus.cc:1631] Marking committed up to 19
0809 23:22:30.471590 (+    42us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:30.471658 (+    68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.456967 (+985309us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:31.462085 (+  5118us) raft_consensus.cc:1671] finished
0809 23:22:31.462113 (+    28us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:31.462343 (+   230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5518,"raft.queue_time_us":6771,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":30719232,"thread_start_us":237843,"threads_started":1,"wal-append.queue_time_us":434614}
W20260809 23:22:31.468988  6464 log.cc:924] Time spent T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260809 23:22:31.466743  6213 leader_election.cc:288] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:31.483464 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 225809us
W20260809 23:22:31.584039 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 420901us
I20260809 23:22:31.588675 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.589560 24422 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:31.634631 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:31.639411 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 373868us
I20260809 23:22:31.630987  6012 raft_consensus.cc:482] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.645416  6249 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:31.661893 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:31.662592 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 312337us
I20260809 23:22:31.673983  6249 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:31.672053  6086 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:31.706084 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.659147  6012 raft_consensus.cc:504] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.732679  6012 leader_election.cc:288] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:31.709218  6249 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:31.716955 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.680155  5752 consensus_queue.cc:1046] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260809 23:22:31.743691 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:31.748842 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.749204 24498 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:31.751905 24498 leader_election.cc:334] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.654309 24162 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:31.752383 24498 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.716180  6289 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:31.755102 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:31.759837  6274 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:31.735648  6057 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:31.746189  6303 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:31.752846  5311 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:31.753252  6262 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.787674 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:31.810770 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 50503us
W20260809 23:22:31.807423  6465 log.cc:924] Time spent T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260809 23:22:31.816216 24498 leader_election.cc:334] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.816649 24498 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.776801  6289 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:31.823452 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c57d179ede43aebb4ed26fdaf60001" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.823899 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.824349 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.891635 24431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3538d14b2274bf99cd72e3872fda1fa" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.825399 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:31.892419 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:31.830298 24430 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
W20260809 23:22:31.843751 22524 thread.cc:536] Waited for 4000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:31.893509 24430 pending_rounds.cc:76] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting all ops after (but not including) 19
I20260809 23:22:31.894009 24430 pending_rounds.cc:97] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260809 23:22:31.894277 24430 raft_consensus.cc:2866] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:31.783696  6057 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.898687 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:31.898898 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 82703us
W20260809 23:22:31.903329 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:31.903848 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:31.904580 24546 raft_consensus.cc:1261] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260809 23:22:31.906795 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.785027  6303 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.911687 24380 leader_election.cc:398] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 25. Current term is 26.
W20260809 23:22:31.916541 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:31.895766  6488 log.cc:924] Time spent T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260809 23:22:31.887404  6377 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:31.924420 24535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.928493 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.928053 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.895505  5643 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:31.958916 24540 raft_consensus.cc:1261] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260809 23:22:31.927839 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a92891fb40e8911ac15230653aab" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.952557  6412 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260809 23:22:31.977125  6303 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:31.979403  6057 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:31.980566 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1ef8fe019645f3b99232df6916cb2c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.981057  5643 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:31.955096  6491 consensus_queue.cc:1035] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260809 23:22:31.927345  5718 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:31.915144  6371 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:31.945238  6480 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260809 23:22:31.999755  5643 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.001638 24541 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46304 (request call id 2157) took 1119 ms. Trace:
I20260809 23:22:31.949745  6413 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:22:32.012591 24541 rpcz_store.cc:276] 0809 23:22:30.882489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.883063 (+   574us) service_pool.cc:227] Handling call
0809 23:22:30.883502 (+   439us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.883932 (+   430us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:30.883970 (+    38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.886585 (+  2615us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.413402 (+526817us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b18001c6720. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac2091a345 00007fac29751a39 00007fac209378bb 00007fac20935cbf 00007fac20937e22 00007fac234e15bb 00007fac234e7063 00007fac234e7411 00007fac238345cf
0809 23:22:31.686182 (+272780us) raft_consensus.cc:1631] Marking committed up to 17
0809 23:22:31.686226 (+    44us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.686291 (+    65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:31.986904 (+300613us) raft_consensus.cc:1671] finished
0809 23:22:31.986949 (+    45us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.001552 (+ 14603us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8581,"raft.queue_time_us":1562,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":9488,"spinlock_wait_cycles":47628160,"thread_start_us":272595,"threads_started":1,"wal-append.queue_time_us":1057083}
I20260809 23:22:31.949085  6277 raft_consensus.cc:482] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:32.019075 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c4d66939a48e59f84422216ede16f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.029834 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba754600fd94d2588497bb7f43a92a0" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:31.959936  6377 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.040191 24542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f974a8c0944480688bfdf1cf728da89" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.039973 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.055299 25323 leader_election.cc:334] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.058215 24546 raft_consensus.cc:1226] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 31. Current term is 32. Ops: []
I20260809 23:22:32.058847 24266 leader_election.cc:302] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.060729 24266 leader_election.cc:302] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:31.992762  6415 consensus_queue.cc:1035] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260809 23:22:32.075011 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.085650 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.098255 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.034492  6413 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.015528  6371 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.067047  6404 raft_consensus.cc:2726] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.052610  5752 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.049s
I20260809 23:22:32.037601  6277 raft_consensus.cc:504] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.123524  6210 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:32.130316 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.122807 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.100154  6220 raft_consensus.cc:2726] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:32.131712 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 222109us
W20260809 23:22:32.134404 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 223031us
I20260809 23:22:32.132354  6266 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:32.144351 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.140659  6210 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:32.149814 25325 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:32.152009  6494 log.cc:924] Time spent T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260809 23:22:32.151782  6413 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.150329  6289 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.152680  6277 leader_election.cc:288] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.163442  6371 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.215435 24270 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.217549 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.239564 24429 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:32.223052 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.162986  6377 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.236110 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:32.147791  6249 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.244287 24539 raft_consensus.cc:2379] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 27: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:22:32.251070 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.272132 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 122216us
I20260809 23:22:32.274757  6388 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.277601  6388 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.289733  6204 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:32.268785  6478 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:32.288909 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 115264us
I20260809 23:22:32.269348  5494 raft_consensus.cc:3032] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Stepping down as leader of term 26
I20260809 23:22:32.300254  5494 raft_consensus.cc:729] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:32.308593  5494 consensus_queue.cc:260] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.311282  5494 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:32.320770 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 112838us
I20260809 23:22:32.323115 24270 leader_election.cc:302] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.325167 24429 raft_consensus.cc:1226] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Rejecting Update request from peer 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27. Ops: []
I20260809 23:22:32.328109  6129 raft_consensus.cc:2726] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:32.332417  6388 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.332623 24428 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 23. Preceding OpId from leader: term: 28 index: 24. (index mismatch)
W20260809 23:22:32.335942  6487 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.254s	user 0.004s	sys 0.000s
I20260809 23:22:32.339568 25371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37942 (request call id 2033) took 1637 ms. Trace:
I20260809 23:22:32.340005 25371 rpcz_store.cc:276] 0809 23:22:30.702318 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:30.725754 (+ 23436us) service_pool.cc:227] Handling call
0809 23:22:30.726176 (+   422us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:30.734256 (+  8080us) raft_consensus.cc:1520] Early marking committed up to index 14
0809 23:22:30.734302 (+    46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:30.860474 (+126172us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b1800544c00. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20905f96 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a
0809 23:22:31.240378 (+379904us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b2c001379e8. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac22243e11 00007fac22253442 00007fac2389bc11 00007fac2386058a 00007fac2385f99e 00007fac287a12e9 00007fac21e87684 00007fac21e87612 00007fac21e87541 00007fac21e874bc 00007fac21e86772
0809 23:22:31.265576 (+ 25198us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.676823 (+411247us) raft_consensus.cc:1631] Marking committed up to 14
0809 23:22:31.676890 (+    67us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.677050 (+   160us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:32.334687 (+657637us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b44000232cc. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac2084aa95 00007fac2084b4b2 00007fac234f3511 00007fac234eee1a 00007fac234e2bee 00007fac234e1f36 00007fac234e1b58 00007fac234eb551 00007fac234eb509 00007fac234eb499
0809 23:22:32.339071 (+  4384us) raft_consensus.cc:1671] finished
0809 23:22:32.339098 (+    27us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.339343 (+   245us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8876,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":5523,"spinlock_wait_cycles":14096000,"thread_start_us":358560,"threads_started":1,"wal-append.queue_time_us":692731}
I20260809 23:22:32.346273  6249 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:32.361657 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.367313 24270 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.391145 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.393060 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.394433 25369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.395081 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 89541.9us
W20260809 23:22:32.396430 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.401791 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.404045 24270 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.393208  6466 log.cc:924] Time spent T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: Append to log took a long time: real 0.707s	user 0.008s	sys 0.016s
I20260809 23:22:32.368861  5611 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260809 23:22:32.436852 25368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37956 (request call id 2242) took 1265 ms. Trace:
W20260809 23:22:32.439419 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 67953.1us
I20260809 23:22:32.439785 24270 leader_election.cc:302] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.441371  6513 log.cc:924] Time spent T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260809 23:22:32.444420 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:32.445909  6305 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:32.446527  6305 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:32.447000  6305 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:32.453802  5571 raft_consensus.cc:2726] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:32.454244  6216 consensus_queue.cc:1046] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:32.467288  6224 raft_consensus.cc:3032] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Stepping down as leader of term 31
I20260809 23:22:32.467716  6224 raft_consensus.cc:729] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:32.467952  6404 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:32.470098 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 330510us
I20260809 23:22:32.469012  6224 consensus_queue.cc:260] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.463320 25372 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:32.464749 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.473834 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 84088.3us
I20260809 23:22:32.476781  6388 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:32.480340  6249 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.481158  6249 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.492116 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca74ab4e31434c79a52a994e74129545" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.492645  6249 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:32.437309 25368 rpcz_store.cc:276] 0809 23:22:31.171602 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:31.171770 (+   168us) service_pool.cc:227] Handling call
0809 23:22:31.172126 (+   356us) raft_consensus.cc:1457] Updating replica for 1 ops
0809 23:22:31.504328 (+332202us) raft_consensus.cc:1520] Early marking committed up to index 0
0809 23:22:31.504572 (+   244us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0809 23:22:31.506763 (+  2191us) log.cc:842] Serialized 26 byte log entry
0809 23:22:31.683203 (+176440us) raft_consensus.cc:1631] Marking committed up to 16
0809 23:22:31.772246 (+ 89043us) log.cc:842] Serialized 14 byte log entry
0809 23:22:31.783565 (+ 11319us) raft_consensus.cc:1683] Filling consensus response to leader.
0809 23:22:31.783654 (+    89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0809 23:22:32.280752 (+497098us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b180039c880. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938764 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:32.413932 (+133180us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b180039c880. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac297d78b1 00007fac297e4661 00007fac20938638 00007fac2093bb11 00007fac2093bac9 00007fac2093ba59 00007fac2093ba21 00007fac2093ad1d 00007fac29b9c204 00007fac29b9c039 00007fac2091adc6
0809 23:22:32.426653 (+ 12721us) raft_consensus.cc:1671] finished
0809 23:22:32.426674 (+    21us) raft_consensus.cc:1677] UpdateReplicas() finished
0809 23:22:32.436638 (+  9964us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26683,"prepare.queue_time_us":105757,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":3735,"raft.queue_time_us":48866,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":6065,"spinlock_wait_cycles":12565632,"thread_start_us":544814,"threads_started":3,"wal-append.queue_time_us":262547}
I20260809 23:22:32.434645  6509 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.500777 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.510437 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.514490 24435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.529086 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:32.530094 24429 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:32.530720 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 54753.1us
W20260809 23:22:32.532071 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:32.547485  6069 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:32.550037  6224 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:32.567332 24428 raft_consensus.cc:1261] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:32.596573  5796 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:32.610335 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 109681us
I20260809 23:22:32.612303 24498 leader_election.cc:302] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.612890 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.619158  6151 raft_consensus.cc:2726] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.628940 24429 raft_consensus.cc:2454] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:32.623279 24496 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.636260  5796 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.662389  5796 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.652503  6249 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.022s
I20260809 23:22:32.683290  6069 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:32.683676 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 74386.5us
I20260809 23:22:32.687310 24496 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.693315 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:32.698143 24270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 105408us
I20260809 23:22:32.710012  6151 consensus_queue.cc:1035] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:32.710150 24496 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:32.711360  6069 raft_consensus.cc:2781] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:32.711932  6069 raft_consensus.cc:686] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:32.696000  6249 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:32.713099  6069 consensus_queue.cc:237] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.716137  6305 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:32.716701 24378 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.717201 24378 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.717623 25368 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: []
I20260809 23:22:32.719986  6289 raft_consensus.cc:2726] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260809 23:22:32.743723 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:32.747092 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 55874.9us
I20260809 23:22:32.771229 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:32.777947 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 27 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
W20260809 23:22:32.754482 24498 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.756775  6305 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.791687 25321 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.787509 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.788736 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:32.799002 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.827929  6404 raft_consensus.cc:482] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:32.828626  6404 raft_consensus.cc:504] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:32.831250  5494 raft_consensus.cc:482] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:32.832180  6404 leader_election.cc:288] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.831923  5494 raft_consensus.cc:504] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.807327  6305 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260809 23:22:32.842675  6210 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:32.843317  6210 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:32.845826  6210 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:32.852869 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.869455 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:32.875296 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:32.877349  6535 log.cc:924] Time spent T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260809 23:22:32.853696  5494 leader_election.cc:288] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:32.889990  5311 consensus_queue.cc:1046] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.040s
W20260809 23:22:32.891348 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 80519.8us
I20260809 23:22:32.873498 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:32.903223 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:32.908583 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44954.5us
W20260809 23:22:32.912554 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.915452  6266 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260809 23:22:32.907150 22524 thread.cc:536] Waited for 5000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:32.918547 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:32.921921 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d3f3d2a169145ffad7fb65adf8b7d7f" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:32.922899  6537 log.cc:924] Time spent T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260809 23:22:32.927076 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:32.891348  5311 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 27.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260809 23:22:32.936424 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c831643e28e34208b6edf775607155e4" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:32.953984  6069 raft_consensus.cc:3032] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Stepping down as leader of term 28
W20260809 23:22:32.954291 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:32.959802  6540 log.cc:924] Time spent T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.004s
I20260809 23:22:32.911401  6151 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260809 23:22:32.986300 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 269661us
W20260809 23:22:32.960593  6536 log.cc:924] Time spent T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: Append to log took a long time: real 0.146s	user 0.000s	sys 0.005s
I20260809 23:22:32.979526 24270 leader_election.cc:302] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:32.983884  6069 raft_consensus.cc:729] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:33.007499  6069 consensus_queue.cc:260] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 28.24, Last appended by leader: 24, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.010692  6069 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:33.010910  5494 raft_consensus.cc:2726] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.008481 24378 leader_election.cc:334] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.026888 24269 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.028059  5611 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.028407 25368 raft_consensus.cc:1226] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Rejecting Update request from peer 808fed3ed3f04882ad757a3db9abfdd9 for earlier term 28. Current term is 29. Ops: [28.24-28.24]
W20260809 23:22:33.029014 24269 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 43777.3us
I20260809 23:22:33.030432  5494 raft_consensus.cc:482] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.031064  5494 raft_consensus.cc:504] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.033392  5494 leader_election.cc:288] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.035909 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.036337 25323 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.039675  6210 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.041451 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.042121 24378 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.052834 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a28a5ad332b42c2ba471779e78f3e25" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:33.060859 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 73370.6us
I20260809 23:22:33.062774  6547 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.063490 24378 leader_election.cc:334] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.067407 24428 raft_consensus.cc:2354] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.069443 24266 leader_election.cc:398] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.069774 24266 leader_election.cc:401] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:33.071059  6389 raft_consensus.cc:3037] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Advancing to term 27
W20260809 23:22:33.077822 24378 leader_election.cc:334] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.078313 24378 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.081027  6389 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) with higher term. Message: Invalid argument: T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d5c69728134c8bb918a900eb6a8c21 for earlier term 26. Current term is 27.
I20260809 23:22:33.103340  6547 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.104939 24378 leader_election.cc:334] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.105733 24378 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.107532  6547 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:33.108847 24378 leader_election.cc:334] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.109485 24378 leader_election.cc:302] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.110246  6547 raft_consensus.cc:2726] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:33.112910 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:33.116578 24378 leader_election.cc:334] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:33.117723 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.119830 24378 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
W20260809 23:22:33.126448 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.122603  6548 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.139048 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.154325 24378 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.156031 24425 raft_consensus.cc:1261] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:33.157546  6388 consensus_queue.cc:1035] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:33.163096 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.163568  6389 raft_consensus.cc:482] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:22:33.165787 24378 leader_election.cc:334] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.165974  6389 raft_consensus.cc:504] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.169703  6389 leader_election.cc:288] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.171317 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.176307 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed27208ddea495a9e5ab4132e38adbb" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:33.182981 24162 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 344) took 84335 ms (client timeout 15000 ms). Trace:
W20260809 23:22:33.197124 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.166697  6548 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:33.191025 24266 leader_election.cc:302] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.166587 24378 leader_election.cc:302] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.201431  6547 raft_consensus.cc:2726] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.203711  5611 raft_consensus.cc:2726] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:22:33.206966 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.207442 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
W20260809 23:22:33.197330 24162 rpcz_store.cc:269] 0809 23:21:08.847371 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.847795 (+   424us) service_pool.cc:227] Handling call
0809 23:22:33.182719 (+84334924us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:32.266446 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:32.295419 (+ 28973us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:32.295494 (+    75us) write_op.cc:435] Acquired schema lock
0809 23:22:32.295543 (+    49us) tablet.cc:660] Decoding operations
0809 23:22:32.314798 (+ 19255us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:32.369300 (+ 54502us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:32.371144 (+  1844us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:32.379373 (+  8229us) tablet.cc:699] Row locks acquired
0809 23:22:32.379408 (+    35us) write_op.cc:260] PREPARE: finished
0809 23:22:32.379596 (+   188us) write_op.cc:270] Start()
0809 23:22:32.379797 (+   201us) write_op.cc:276] Timestamp: P: 1786317752379513 usec, L: 0
0809 23:22:32.379858 (+    61us) op_driver.cc:331] REPLICATION: starting
0809 23:22:32.380657 (+   799us) log.cc:842] Serialized 538 byte log entry
0809 23:22:32.750848 (+370191us) op_driver.cc:444] REPLICATION: finished
0809 23:22:32.876396 (+125548us) write_op.cc:301] APPLY: starting
0809 23:22:32.876587 (+   191us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:32.878646 (+  2059us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:32.878792 (+   146us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:32.956589 (+ 77797us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:32.956936 (+   347us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:32.957008 (+    72us) write_op.cc:312] APPLY: finished
0809 23:22:32.957420 (+   412us) log.cc:842] Serialized 25 byte log entry
0809 23:22:32.957646 (+   226us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:32.957757 (+   111us) write_op.cc:454] Released schema lock
0809 23:22:32.957827 (+    70us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:32.958272 (+   445us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:32.958306 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":121662,"apply.run_cpu_time_us":5694,"apply.run_wall_time_us":85704,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"num_ops":1,"prepare.queue_time_us":49935,"prepare.run_cpu_time_us":14780,"prepare.run_wall_time_us":261550,"raft.queue_time_us":120455,"raft.run_cpu_time_us":28477,"raft.run_wall_time_us":69222,"replication_time_us":370866,"spinlock_wait_cycles":69504,"thread_start_us":271858,"threads_started":4,"wal-append.queue_time_us":216334}]]}
I20260809 23:22:33.210044  6547 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.213048 24160 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.211228 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.230824 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:33.233274 24378 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.233705 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.234567  6547 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:33.234823 24160 catalog_manager.cc:5495] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.246454 24160 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 21, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 21 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:33.248780 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.249289 24378 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:33.252386  6548 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.248389 24160 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 29 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.264187 24160 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 23 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.266642 24160 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 26. New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.283761 24160 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.296262 24160 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 22 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 25 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.304912 24160 catalog_manager.cc:5495] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.311363 24160 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.316315 24160 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 20 to 24, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.325826 24160 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.331486 24160 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 22, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 22 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.339792 24160 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.351215 24160 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.354609 24160 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.358318 24160 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.369275 24160 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.371737 24160 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:33.421535  5611 raft_consensus.cc:482] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:33.426075  5611 raft_consensus.cc:504] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.428381  5611 leader_election.cc:288] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.432139 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.433792 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a682e0d089354a44a680bd8ad9b4b789" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.438099 24266 leader_election.cc:302] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.441735  6310 raft_consensus.cc:2726] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:33.464329 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:33.511065 24498 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 181 similar messages]
W20260809 23:22:33.551164 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:33.563117  6476 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:33.563803  6476 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:33.570225 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.565973  6476 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:33.571396 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.572208 24428 raft_consensus.cc:2379] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 27: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:33.573170 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.574523  6476 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:33.575297  6151 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.575901  6151 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.578122  6151 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.580253 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:33.581096 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.584632 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.586999  6511 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260809 23:22:33.592298  6019 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.592808  6019 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.595399 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.596411 24428 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
I20260809 23:22:33.597617 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:33.594871  6019 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.599439 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.602469  5494 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:33.606878  5494 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:33.607272  5494 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.623620  6476 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:33.631971 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 27.
W20260809 23:22:33.636865 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:33.645004  5494 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.647739  5611 raft_consensus.cc:482] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b9b54436d0e94a5595cdded73fe33da9)
I20260809 23:22:33.648365  5611 raft_consensus.cc:504] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.650461  5611 leader_election.cc:288] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.650523 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:33.651225 24422 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.652364 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.660213 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2fc12c55ab24490a1ad12ed4b4b5c35" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.660899  6310 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.662225  6310 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.664343  6310 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.665740 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.666440 24540 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 32.
I20260809 23:22:33.667258 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:33.667735 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: 
I20260809 23:22:33.669189  5611 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260809 23:22:33.669301 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:33.669688  5611 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:33.670110 24540 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:33.670166  5611 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Advancing to term 33
I20260809 23:22:33.671049  5494 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.676338 24269 leader_election.cc:302] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.685043  6019 raft_consensus.cc:2726] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:33.715680 24422 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
W20260809 23:22:33.729779 24266 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68102.2us
I20260809 23:22:33.731758  5611 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.737345 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:33.738037 24543 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Advancing to term 33
I20260809 23:22:33.738463  6389 raft_consensus.cc:482] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.739032  6389 raft_consensus.cc:504] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.742165 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.743080  6389 leader_election.cc:288] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:33.746140  5611 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 election: Requested vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.718369 24540 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 28.
I20260809 23:22:33.747252 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:33.756639 24543 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d5c69728134c8bb918a900eb6a8c21 in term 33.
I20260809 23:22:33.735044 24266 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:33.760687 24270 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9; no voters: b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.761198  6310 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:33.761765  6310 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:33.762367  6019 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Leader election won for term 33
I20260809 23:22:33.762845  6310 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.762849  6019 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 LEADER]: Becoming Leader. State: Replica: 60d5c69728134c8bb918a900eb6a8c21, State: Running, Role: LEADER
I20260809 23:22:33.765111  6019 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.773491 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8542be4415fd4e648a44a76fdbc3e176" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.781836 24266 leader_election.cc:302] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.785145  5611 raft_consensus.cc:2726] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.871856  6310 raft_consensus.cc:482] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.872498  6310 raft_consensus.cc:504] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:33.876086 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.878782 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be87796c0684430b07c73073572ee50" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.880358 24266 leader_election.cc:302] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:33.881260  5611 raft_consensus.cc:2726] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:33.881780  6310 leader_election.cc:288] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.884275 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:33.886662 24160 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 346) took 22446 ms (client timeout 15000 ms). Trace:
I20260809 23:22:33.901090 24168 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 21 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 22 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.903200  6168 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:33.903781  6168 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:33.904572 24168 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 27, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.906558 24168 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:33.908046 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:33.911813 24539 raft_consensus.cc:2354] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.913343 25325 leader_election.cc:398] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.913657 25325 leader_election.cc:401] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:33.914244  6210 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Advancing to term 33
W20260809 23:22:33.915429 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:33.916522 22524 thread.cc:536] Waited for 6000ms trying to join with heartbeat (tid 24356)
I20260809 23:22:33.916232 24168 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.899339 24160 rpcz_store.cc:269] 0809 23:22:11.440358 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:11.443092 (+  2734us) service_pool.cc:227] Handling call
0809 23:22:33.886557 (+22443465us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:33.432698 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:33.433198 (+   500us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:33.433237 (+    39us) write_op.cc:435] Acquired schema lock
0809 23:22:33.433252 (+    15us) tablet.cc:660] Decoding operations
0809 23:22:33.454477 (+ 21225us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:33.454558 (+    81us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:33.454586 (+    28us) tablet.cc:683] Acquiring locks for 19 operations
0809 23:22:33.456049 (+  1463us) tablet.cc:699] Row locks acquired
0809 23:22:33.456068 (+    19us) write_op.cc:260] PREPARE: finished
0809 23:22:33.456206 (+   138us) write_op.cc:270] Start()
0809 23:22:33.456332 (+   126us) write_op.cc:276] Timestamp: P: 1786317753456162 usec, L: 0
0809 23:22:33.456357 (+    25us) op_driver.cc:331] REPLICATION: starting
0809 23:22:33.468360 (+ 12003us) log.cc:842] Serialized 7439 byte log entry
0809 23:22:33.497156 (+ 28796us) op_driver.cc:444] REPLICATION: finished
0809 23:22:33.528120 (+ 30964us) write_op.cc:301] APPLY: starting
0809 23:22:33.528214 (+    94us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:33.554960 (+ 26746us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:33.555080 (+   120us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:33.609570 (+ 54490us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:33.609904 (+   334us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0809 23:22:33.609955 (+    51us) write_op.cc:312] APPLY: finished
0809 23:22:33.618885 (+  8930us) log.cc:842] Serialized 172 byte log entry
0809 23:22:33.619089 (+   204us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:33.619292 (+   203us) write_op.cc:454] Released schema lock
0809 23:22:33.619358 (+    66us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:33.619829 (+   471us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:33.619866 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":208200,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":30613,"apply.run_cpu_time_us":25397,"apply.run_wall_time_us":92310,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83471,"num_ops":19,"prepare.queue_time_us":37878,"prepare.run_cpu_time_us":5972,"prepare.run_wall_time_us":35958,"replication_time_us":40691,"thread_start_us":54300,"threads_started":2}]]}
I20260809 23:22:33.920423  6310 raft_consensus.cc:482] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:33.924724  6310 raft_consensus.cc:504] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:33.918517 24150 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet cd2b577e2c3e4021bc8ebdfa3462437d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab5aefaeb3d14775b30a74a2a6d3357f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:33.927464  6310 leader_election.cc:288] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:33.928100 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:33.928798 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8039b4f7b5b04cd0a5fb1f9d6c999b5e" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:33.932441 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:33.906723  6168 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:33.954596 24164 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 356) took 49932 ms (client timeout 15000 ms). Trace:
W20260809 23:22:33.957064 24164 rpcz_store.cc:269] 0809 23:21:43.994543 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:43.995054 (+   511us) service_pool.cc:227] Handling call
0809 23:22:33.926771 (+49931717us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":390267,"reported_tablets":47}
I20260809 23:22:33.970374 24269 leader_election.cc:302] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:33.926033 24168 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 25 to 29, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:33.984217 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:33.946923  6210 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765) with higher term. Message: Invalid argument: T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 for earlier term 32. Current term is 33.
I20260809 23:22:33.988791 24168 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 20 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:33.988209  5611 raft_consensus.cc:2726] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:33.999294 24168 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 24 to 25, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.007360 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:34.018430 24168 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 22 to 27, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.017037 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 81797.1us
I20260809 23:22:34.020311 24168 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 17 to 24, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:34.024153 24168 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 23 to 29, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:34.029932 24168 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 reported cstate change: term changed from 16 to 19, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 19 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:34.038329 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.054908 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:22:34.059844 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.070209 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.078979 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.097782 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.103645 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.105207 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.113830 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.114900 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.120951 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.135015 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.139583 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.141441 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.157366 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.160562  6476 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.169438  6476 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.171576  6476 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.172479 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.173185 25363 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:34.174351 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:34.176972  6388 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:34.177361  6388 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:22:34.177685  6388 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.178741 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.192281 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.194655  5611 raft_consensus.cc:482] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.199609  5611 raft_consensus.cc:504] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:34.200837 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.201893  5611 leader_election.cc:288] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:34.200029  6388 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.204206 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.204676 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3913bd5adc0247eb847943b7e9a3ceb7" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.207928 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:34.211799 24266 leader_election.cc:302] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.219064  5611 raft_consensus.cc:2726] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:34.219703  6388 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.222671 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.224121 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.224833 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P 60d5c69728134c8bb918a900eb6a8c21 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.226666 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 30 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:34.227291 25363 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.250648 25365 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260809 23:22:34.264129  6310 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260809 23:22:34.271066 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.281069 25363 raft_consensus.cc:2454] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 30.
I20260809 23:22:34.282426 24498 leader_election.cc:302] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.272230 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.283747  6388 raft_consensus.cc:2781] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:34.284674  6388 raft_consensus.cc:686] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:34.285945  6388 consensus_queue.cc:237] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 28.24, Last appended by leader: 24, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.290477 24539 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260809 23:22:34.292757  6310 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260809 23:22:34.298154 24160 catalog_manager.cc:5495] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.300511 24160 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.310020 24160 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.318240 24422 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260809 23:22:34.319933  5611 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260809 23:22:34.325454 24160 catalog_manager.cc:5495] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.329660 24160 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.323369 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.336717 24160 catalog_manager.cc:5495] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.407714 24160 catalog_manager.cc:5495] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 28, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 28 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.424365 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:34.449347 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:34.456379 24160 catalog_manager.cc:5495] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.463016 24160 catalog_manager.cc:5495] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.339670 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.465458 24539 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer 60d5c69728134c8bb918a900eb6a8c21: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260809 23:22:34.473237 24160 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:34.348452 24168 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 375) took 15313 ms (client timeout 15000 ms). Trace:
I20260809 23:22:34.475011 24160 catalog_manager.cc:5495] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 30, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 30 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.482393 24160 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.489338 24160 catalog_manager.cc:5495] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 17 to 21, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 21 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.498561 24160 catalog_manager.cc:5495] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.501935 24160 catalog_manager.cc:5495] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.496490 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:34.480362 24168 rpcz_store.cc:269] 0809 23:22:19.035244 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:22.424771 (+3389527us) service_pool.cc:227] Handling call
0809 23:22:34.348356 (+11923585us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:34.087512 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:34.088040 (+   528us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:34.088079 (+    39us) write_op.cc:435] Acquired schema lock
0809 23:22:34.088099 (+    20us) tablet.cc:660] Decoding operations
0809 23:22:34.102776 (+ 14677us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:34.102859 (+    83us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:34.105608 (+  2749us) tablet.cc:683] Acquiring locks for 11 operations
0809 23:22:34.106378 (+   770us) tablet.cc:699] Row locks acquired
0809 23:22:34.106391 (+    13us) write_op.cc:260] PREPARE: finished
0809 23:22:34.106584 (+   193us) write_op.cc:270] Start()
0809 23:22:34.106690 (+   106us) write_op.cc:276] Timestamp: P: 1786317754106539 usec, L: 0
0809 23:22:34.106707 (+    17us) op_driver.cc:331] REPLICATION: starting
0809 23:22:34.107405 (+   698us) log.cc:842] Serialized 4359 byte log entry
0809 23:22:34.143278 (+ 35873us) op_driver.cc:444] REPLICATION: finished
0809 23:22:34.188441 (+ 45163us) write_op.cc:301] APPLY: starting
0809 23:22:34.188532 (+    91us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:34.216895 (+ 28363us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:34.216988 (+    93us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:34.228954 (+ 11966us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:34.229199 (+   245us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0809 23:22:34.229241 (+    42us) write_op.cc:312] APPLY: finished
0809 23:22:34.229528 (+   287us) log.cc:842] Serialized 105 byte log entry
0809 23:22:34.229693 (+   165us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:34.229830 (+   137us) write_op.cc:454] Released schema lock
0809 23:22:34.229876 (+    46us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:34.273203 (+ 43327us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:34.273233 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1558,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":44845,"apply.run_cpu_time_us":13745,"apply.run_wall_time_us":85206,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50063,"num_ops":11,"prepare.queue_time_us":33103,"prepare.run_cpu_time_us":4017,"prepare.run_wall_time_us":20280,"replication_time_us":36463,"thread_start_us":43993,"threads_started":2}]]}
I20260809 23:22:34.470132 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.516921 22524 pending_rounds.cc:62] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:34.517303 22524 pending_rounds.cc:68] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 24 index: 16 } timestamp: 7316757417841188864 op_type: NO_OP noop_request { }
I20260809 23:22:34.507297  6548 raft_consensus.cc:482] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:22:34.518114  6548 raft_consensus.cc:504] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.503242  6310 raft_consensus.cc:482] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.520364  6310 raft_consensus.cc:504] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.522692 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.523665 25363 raft_consensus.cc:2354] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
I20260809 23:22:34.524444 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.521692  6548 leader_election.cc:288] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.506659  6575 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:34.525832 24378 leader_election.cc:398] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
I20260809 23:22:34.526635 24378 leader_election.cc:401] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260809 23:22:34.527243  6289 raft_consensus.cc:3037] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 30
I20260809 23:22:34.529284 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c4cf897ac64cb8981b99dda67dbc92" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 24 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.529731 22524 raft_consensus.cc:2866] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:34.530160 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.535301  6289 raft_consensus.cc:2726] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713) with higher term. Message: Invalid argument: T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f for earlier term 29. Current term is 30.
W20260809 23:22:34.543311 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:34.527781 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a29ab049db4236a3198e8545cafb9c" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.545163 24270 leader_election.cc:302] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:34.546092  6576 raft_consensus.cc:2726] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:22:34.547153  6310 leader_election.cc:288] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.550236 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.551420 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.552910 24160 catalog_manager.cc:5495] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.556409 24160 catalog_manager.cc:5495] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 19 to 20, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 20 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.558145 24160 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.564730 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.559458 24160 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 28 to 31, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 31 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.628209 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.628741 24160 catalog_manager.cc:5495] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 26, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 26 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.632680 24160 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 27, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.630689 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.671950 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.670305 24160 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.677029 24160 catalog_manager.cc:5495] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 29, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 29 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:34.682758 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:34.691299 25368 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 24. Preceding OpId from leader: term: 30 index: 25. (index mismatch)
I20260809 23:22:34.693650  6511 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260809 23:22:34.695259  6511 consensus_queue.cc:1225] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Peer b9b54436d0e94a5595cdded73fe33da9 log is divergent from this leader: its last log entry 27.24 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 23
I20260809 23:22:34.698892  5611 raft_consensus.cc:482] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.699859  5611 raft_consensus.cc:504] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:34.702895  5611 leader_election.cc:288] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:34.704401 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.678807 24160 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.709007 25363 pending_rounds.cc:76] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Aborting all ops after (but not including) 23
I20260809 23:22:34.709429 25363 pending_rounds.cc:97] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: Aborting uncommitted NO_OP operation due to leader change: 27.24
I20260809 23:22:34.709699 25363 raft_consensus.cc:2866] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:22:34.709100 24160 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 25, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 25 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.719734 24160 catalog_manager.cc:5495] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:34.723531 24160 catalog_manager.cc:5495] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.720412 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:34.706739 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.713778 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e43c87eb09044548379ab2c35cce3c1" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.766287 24160 catalog_manager.cc:5495] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 28, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.778240 24160 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 20 to 22, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 22 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.782852 24160 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.785995 24160 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.786702 24270 leader_election.cc:302] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:34.787859 24160 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 23 to 26, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.789600 24160 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 22 to 28. New cstate: current_term: 28 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:34.799962  6575 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:34.800702  6575 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:34.802883  6575 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:34.804674 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 78685.7us
I20260809 23:22:34.810873  6576 raft_consensus.cc:2726] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:34.812700 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.813710 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:34.818921 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54afb87e83f14564b7c4ec761e92304a" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:34.820966 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.821338 24270 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:34.822506  5611 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260809 23:22:34.822569 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.825990 24422 raft_consensus.cc:1261] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 24. Preceding OpId from leader: term: 30 index: 25. (index mismatch)
I20260809 23:22:34.827529 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.830721 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.830960  6476 consensus_queue.cc:1035] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260809 23:22:34.831799 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.832520 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.843828 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.846410 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.848670 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P 60d5c69728134c8bb918a900eb6a8c21 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.858829 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.859973 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.861119 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.864145 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
W20260809 23:22:34.864353 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.865272 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.865927 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.869275 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.870375 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.871033 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P 60d5c69728134c8bb918a900eb6a8c21 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.874086 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.875133 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.876456 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.879393 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.880374 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.881433 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P 60d5c69728134c8bb918a900eb6a8c21 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.896770 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.897845 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.898507 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:34.943428 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.943956 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.945009 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.947778 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.950793 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.951663 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:34.952718 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:34.968154 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:34.975467 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.976557 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.977228 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:34.980154 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:34.980978 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:34.981473 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.004189 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.005425 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.009296 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.014169 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.014959 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.016088 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.035631 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.036562 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.037309 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:35.043433 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.068485 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.070729 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.071517 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.084659 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.085600 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.086125 22524 pending_rounds.cc:62] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.086355 22524 pending_rounds.cc:68] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7316757421635989504 op_type: NO_OP noop_request { }
I20260809 23:22:35.086725 22524 raft_consensus.cc:2866] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.087080 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.089757 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.090556 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Raft consensus shutting down.
W20260809 23:22:35.107147 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:35.111717 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.137553 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.138500 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.139192 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.142078 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.142840 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.160346 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.163899 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.164777 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.165933 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.173728 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.174503 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.175022 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P 60d5c69728134c8bb918a900eb6a8c21 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.177639 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.178387 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.178917 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.187598  6546 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.189232  6546 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.191287  6546 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.195863 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:35.202307 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.202744 25323 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.208686  6583 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:35.209959 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.212376 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.217483 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.225839 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.226766 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.234946  6578 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.235450 22524 pending_rounds.cc:62] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.235755 22524 pending_rounds.cc:68] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 21 index: 16 } timestamp: 7316757418382639104 op_type: NO_OP noop_request { }
I20260809 23:22:35.235602  6578 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.236261 22524 raft_consensus.cc:2866] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.236730 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P 60d5c69728134c8bb918a900eb6a8c21 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.240646 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.241293 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.241568 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.242435 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:35.252081 24378 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.252592 24378 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:35.253300  6289 raft_consensus.cc:2726] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:35.238222  6578 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.259779 24162 catalog_manager.cc:5495] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.261426 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
W20260809 23:22:35.254814 24151 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 48934.8us
I20260809 23:22:35.262295 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.262534 24162 catalog_manager.cc:5495] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 32, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 32 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:35.263720 22524 pending_rounds.cc:62] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
W20260809 23:22:35.264333 24244 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.268397 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.266391 24160 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49570 (request call id 245) took 2168 ms. Trace:
I20260809 23:22:35.270129 22524 pending_rounds.cc:68] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 27 index: 16 } timestamp: 7316757451460698112 op_type: NO_OP noop_request { }
I20260809 23:22:35.277489 22524 raft_consensus.cc:2866] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.277961 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.270037 24162 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.281641 24162 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 21, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 21 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.283018 24160 rpcz_store.cc:276] 0809 23:22:33.098300 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:33.929738 (+831438us) service_pool.cc:227] Handling call
0809 23:22:35.266286 (+1336548us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:34.918474 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:34.918991 (+   517us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:34.919020 (+    29us) write_op.cc:435] Acquired schema lock
0809 23:22:34.919036 (+    16us) tablet.cc:660] Decoding operations
0809 23:22:34.921413 (+  2377us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:34.921497 (+    84us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:34.921526 (+    29us) tablet.cc:683] Acquiring locks for 33 operations
0809 23:22:34.923901 (+  2375us) tablet.cc:699] Row locks acquired
0809 23:22:34.923917 (+    16us) write_op.cc:260] PREPARE: finished
0809 23:22:34.924158 (+   241us) write_op.cc:270] Start()
0809 23:22:34.924274 (+   116us) write_op.cc:276] Timestamp: P: 1786317754924105 usec, L: 0
0809 23:22:34.924298 (+    24us) op_driver.cc:331] REPLICATION: starting
0809 23:22:34.925018 (+   720us) log.cc:842] Serialized 12814 byte log entry
0809 23:22:34.996697 (+ 71679us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.007085 (+ 10388us) write_op.cc:301] APPLY: starting
0809 23:22:35.007179 (+    94us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.124837 (+117658us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.124961 (+   124us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.196594 (+ 71633us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.197049 (+   455us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=66,key_file_lookups=66,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.197099 (+    50us) write_op.cc:312] APPLY: finished
0809 23:22:35.197602 (+   503us) log.cc:842] Serialized 284 byte log entry
0809 23:22:35.197791 (+   189us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.198078 (+   287us) write_op.cc:454] Released schema lock
0809 23:22:35.198142 (+    64us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.198489 (+   347us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.198516 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25119,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":10080,"apply.run_cpu_time_us":42262,"apply.run_wall_time_us":216551,"cfile_cache_hit":134,"cfile_cache_hit_bytes":141935,"num_ops":33,"prepare.queue_time_us":19039,"prepare.run_cpu_time_us":6912,"prepare.run_wall_time_us":27450,"replication_time_us":72298,"thread_start_us":25991,"threads_started":2}]]}
I20260809 23:22:35.286283 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.293496 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.294718 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P 60d5c69728134c8bb918a900eb6a8c21 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.299090 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.300700 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.301681 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.304896 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.291471 24162 catalog_manager.cc:5495] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.305661 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.306314 22524 pending_rounds.cc:62] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.306541 22524 pending_rounds.cc:68] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 21 index: 16 } timestamp: 7316757424355483648 op_type: NO_OP noop_request { }
I20260809 23:22:35.306913 22524 raft_consensus.cc:2866] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.309151 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P 60d5c69728134c8bb918a900eb6a8c21 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.311884 24162 catalog_manager.cc:5495] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.318416 24162 catalog_manager.cc:5495] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.320855 24162 catalog_manager.cc:5495] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 30, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 30 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:35.336479 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.340116 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.340792 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P 60d5c69728134c8bb918a900eb6a8c21 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.342522 24162 catalog_manager.cc:5495] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 31, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 31 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:35.346459 24162 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.355672 24162 catalog_manager.cc:5495] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 28 to 31, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 31 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.356930  6590 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.357523  6590 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.358466 24162 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.359537  6590 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.360682 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.360231 24162 catalog_manager.cc:5495] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 31, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 31 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.361375 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:35.362690 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:35.363768  6511 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:35.364192  6511 raft_consensus.cc:482] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.364583  6511 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:35.366276  6575 raft_consensus.cc:482] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:22:35.366884  6575 raft_consensus.cc:504] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } attrs { replace: true } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.369247 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.371471 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.372548 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P 60d5c69728134c8bb918a900eb6a8c21 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.373456 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.373839  6575 leader_election.cc:288] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.374027 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.375664 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2b577e2c3e4021bc8ebdfa3462437d" candidate_uuid: "60d5c69728134c8bb918a900eb6a8c21" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.378789 24270 leader_election.cc:302] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 60d5c69728134c8bb918a900eb6a8c21; no voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.381240  5494 raft_consensus.cc:2726] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:35.383103  6590 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.383703  6590 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.378711 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.385547 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:35.385967 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.386277 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P 60d5c69728134c8bb918a900eb6a8c21 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.389950 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.390201 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.391129 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:35.391690 22524 pending_rounds.cc:62] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Trying to abort 1 pending ops.
I20260809 23:22:35.391916 22524 pending_rounds.cc:68] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7316757361437347840 op_type: NO_OP noop_request { }
I20260809 23:22:35.392222 22524 raft_consensus.cc:2866] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:22:35.392534 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.395521 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21: stopping tablet replica
I20260809 23:22:35.396215 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:35.397284 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 60d5c69728134c8bb918a900eb6a8c21 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:35.385716  6590 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.414729  6511 raft_consensus.cc:504] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.450383  6578 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.451052  6578 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.453132  6578 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.455536 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.457063 24160 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:35.459204 24378 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.458737 24160 catalog_manager.cc:5495] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 30, leader changed from b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 30 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:35.459692 24378 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.460813  6578 raft_consensus.cc:2726] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:22:35.467430 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:35.472098 24498 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.487828 24498 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:35.487991 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827f95d8d9f946f6b673db8272386c5f" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:35.485352  6511 leader_election.cc:288] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.492914 24428 raft_consensus.cc:3037] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:35.502970  6511 raft_consensus.cc:2726] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:35.507578 24428 raft_consensus.cc:2454] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:35.508711 24496 leader_election.cc:302] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:35.509583  6511 raft_consensus.cc:2781] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:35.512233  6511 raft_consensus.cc:686] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:35.513407  6578 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.513958  6578 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.513226  6511 consensus_queue.cc:237] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.516213  6578 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.518570 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.519200 25363 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:35.520252 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.521492  6578 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:35.521880  6578 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.522257  6578 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.534355 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.536732  6578 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.542387  6546 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.543262  6583 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.543774  6583 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.545656 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deaa6cc935614b7bbb64f5ee8faf4c74" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.546646 25363 raft_consensus.cc:3037] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:35.547940  6546 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.552997 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.553603 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:35.545899  6583 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.556864 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.554803  6546 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.561569 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.560022 25363 raft_consensus.cc:2454] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:35.538774  6578 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.565009 24378 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.566811  6583 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:35.567649  6583 raft_consensus.cc:482] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.567152  6289 raft_consensus.cc:2781] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:35.568080  6583 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.568519 24378 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.576258  6583 raft_consensus.cc:504] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.584391 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.616110 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb0467d784e841868c4a1188ee0af4bc" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:35.616767 24428 raft_consensus.cc:3037] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.625474 24498 leader_election.cc:334] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.626462 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:35.611150  6590 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.624655 24428 raft_consensus.cc:2454] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
W20260809 23:22:35.628047 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.597301  6583 leader_election.cc:288] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.634922 24167 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 17 to 26, leader changed from 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3) to 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1). New cstate: current_term: 26 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:35.627224  6590 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.638888  6590 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.639832 25321 leader_election.cc:302] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.648012 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.648675 25363 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:35.649897 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:35.648633 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.650529 25323 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:35.655076 24162 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35216 (request call id 347) took 2868 ms. Trace:
I20260809 23:22:35.655686 24162 rpcz_store.cc:276] 0809 23:22:32.774420 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:33.233555 (+459135us) service_pool.cc:227] Handling call
0809 23:22:35.642213 (+2408658us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:35.377969 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:35.379344 (+  1375us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:35.379393 (+    49us) write_op.cc:435] Acquired schema lock
0809 23:22:35.379411 (+    18us) tablet.cc:660] Decoding operations
0809 23:22:35.381792 (+  2381us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:35.381854 (+    62us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:35.382057 (+   203us) tablet.cc:683] Acquiring locks for 13 operations
0809 23:22:35.386723 (+  4666us) tablet.cc:699] Row locks acquired
0809 23:22:35.386746 (+    23us) write_op.cc:260] PREPARE: finished
0809 23:22:35.386911 (+   165us) write_op.cc:270] Start()
0809 23:22:35.387050 (+   139us) write_op.cc:276] Timestamp: P: 1786317755386864 usec, L: 0
0809 23:22:35.387074 (+    24us) op_driver.cc:331] REPLICATION: starting
0809 23:22:35.396204 (+  9130us) log.cc:842] Serialized 5159 byte log entry
0809 23:22:35.399868 (+  3664us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.402489 (+  2621us) write_op.cc:301] APPLY: starting
0809 23:22:35.402585 (+    96us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.409404 (+  6819us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.409498 (+    94us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.434380 (+ 24882us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.434606 (+   226us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.434649 (+    43us) write_op.cc:312] APPLY: finished
0809 23:22:35.434977 (+   328us) log.cc:842] Serialized 121 byte log entry
0809 23:22:35.435159 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.435316 (+   157us) write_op.cc:454] Released schema lock
0809 23:22:35.435376 (+    60us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.441910 (+  6534us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.441951 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":43714,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":2375,"apply.run_cpu_time_us":16524,"apply.run_wall_time_us":40439,"cfile_cache_hit":52,"cfile_cache_hit_bytes":54288,"num_ops":13,"prepare.queue_time_us":122,"prepare.run_cpu_time_us":4848,"prepare.run_wall_time_us":21837,"replication_time_us":12698}]]}
W20260809 23:22:35.656059 24498 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.659755  6615 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:35.651388  6590 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:35.660712  6590 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.661090  6590 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:35.651957  6583 raft_consensus.cc:2781] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:35.684553  6583 raft_consensus.cc:686] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:35.685562  6583 consensus_queue.cc:237] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.688078 24167 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 358) took 3766 ms. Trace:
I20260809 23:22:35.688515 24167 rpcz_store.cc:276] 0809 23:22:31.945153 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:31.945882 (+   729us) service_pool.cc:227] Handling call
0809 23:22:35.688013 (+3742131us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:35.642644 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:35.643749 (+  1105us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:35.643786 (+    37us) write_op.cc:435] Acquired schema lock
0809 23:22:35.643801 (+    15us) tablet.cc:660] Decoding operations
0809 23:22:35.644151 (+   350us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:35.644223 (+    72us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:35.644263 (+    40us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:35.644502 (+   239us) tablet.cc:699] Row locks acquired
0809 23:22:35.644530 (+    28us) write_op.cc:260] PREPARE: finished
0809 23:22:35.644696 (+   166us) write_op.cc:270] Start()
0809 23:22:35.644834 (+   138us) write_op.cc:276] Timestamp: P: 1786317755644641 usec, L: 0
0809 23:22:35.644860 (+    26us) op_driver.cc:331] REPLICATION: starting
0809 23:22:35.645495 (+   635us) log.cc:842] Serialized 538 byte log entry
0809 23:22:35.647463 (+  1968us) op_driver.cc:444] REPLICATION: finished
0809 23:22:35.647858 (+   395us) write_op.cc:301] APPLY: starting
0809 23:22:35.647935 (+    77us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:35.648758 (+   823us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:35.648854 (+    96us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:35.649536 (+   682us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:35.649686 (+   150us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:35.649726 (+    40us) write_op.cc:312] APPLY: finished
0809 23:22:35.649935 (+   209us) log.cc:842] Serialized 25 byte log entry
0809 23:22:35.650117 (+   182us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.650187 (+    70us) write_op.cc:454] Released schema lock
0809 23:22:35.650221 (+    34us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:35.650450 (+   229us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:35.650479 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2205,"reported_tablets":47,"child_traces":[["op",{"apply.queue_time_us":217,"apply.run_cpu_time_us":3006,"apply.run_wall_time_us":3003,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4176,"num_ops":1,"prepare.queue_time_us":137,"prepare.run_cpu_time_us":2576,"prepare.run_wall_time_us":4874,"replication_time_us":2520}]]}
W20260809 23:22:35.705089 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:35.652345  6289 raft_consensus.cc:686] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:35.719385  6546 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.727190  6546 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.729336  6546 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.730911 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:35.730777  6289 consensus_queue.cc:237] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.731971 24539 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
I20260809 23:22:35.733510 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.734653  6546 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:35.735102  6546 raft_consensus.cc:482] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.735564  6546 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.751287 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 102681us
W20260809 23:22:35.755052 25323 leader_election.cc:334] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.751421 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 85197.6us
W20260809 23:22:35.793335 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.801627  6615 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.803282  6615 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.804795  6289 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.805621  6615 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.805388  6289 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.801209 24167 catalog_manager.cc:5495] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:35.810520 24378 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.812413  6546 raft_consensus.cc:504] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.813706 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.817761  6476 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.818284  6476 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.820315  6476 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.821280 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.813717  6289 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.827795 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:35.827749  6604 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.829000  6604 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.831158  6604 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:35.831667 25363 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 29.
I20260809 23:22:35.833423 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.834594  6289 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260809 23:22:35.835019  6289 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:35.835335 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.835388  6289 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:35.835798 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.836977  6476 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260809 23:22:35.850704  6546 leader_election.cc:288] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:35.825924  6590 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:35.860164 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:35.852492 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac3c46a49aea44a2b2721eb2a750f055" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:35.853210  6289 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.860805 24540 raft_consensus.cc:3037] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
W20260809 23:22:35.863574 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.867177 25323 leader_election.cc:334] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.876185  6590 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.877460 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:38882
I20260809 23:22:35.880326 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3962e082dc274290a1f20e2fa64877a9" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.895244 25363 raft_consensus.cc:3037] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:35.884996 24378 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:35.888592 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 68793us
W20260809 23:22:35.889873 24498 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.900758 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:35.901346 25368 raft_consensus.cc:3037] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Advancing to term 30
I20260809 23:22:35.911319 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f20a6f5f014e45d3a917a197db7a0998" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.912151 24422 raft_consensus.cc:2379] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 30: Already voted for candidate ab5aefaeb3d14775b30a74a2a6d3357f in this term.
I20260809 23:22:35.912784 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.930006 24540 raft_consensus.cc:2454] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:35.924342 25363 raft_consensus.cc:2454] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:35.927707 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:35.931485 25325 leader_election.cc:302] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.935614 25321 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.934485 24498 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.899040  6289 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.939424 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:35.940418  6623 raft_consensus.cc:2781] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:35.943574  6615 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:35.940850  6611 raft_consensus.cc:2781] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:35.945495  6611 raft_consensus.cc:686] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:35.941056  6623 raft_consensus.cc:686] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:35.946400  6611 consensus_queue.cc:237] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.946689  6623 consensus_queue.cc:237] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.971499  6604 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.972123  6604 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.974252  6604 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:35.976137 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:22:35.983026 25368 raft_consensus.cc:2454] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 30.
I20260809 23:22:35.985600  6621 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260809 23:22:35.985627 24170 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 354) took 133327 ms (client timeout 15000 ms). Trace:
W20260809 23:22:35.986274 24170 rpcz_store.cc:269] 0809 23:20:22.635593 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:20:22.635901 (+   308us) service_pool.cc:227] Handling call
0809 23:20:23.344092 (+708191us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400002dd4. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac237e5af0 00007fac23868f2a 00007fac297bad6c 00007fac298b4c3e 00007fac27bc6fb4 00007fac27bc6f42 00007fac27bc6e71 00007fac27bc6dec 00007fac27bc60a2 00007fac21c8a94c 00007fac21c89d86
0809 23:22:35.953004 (+132608912us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23270,"reported_tablets":47,"spinlock_wait_cycles":349312768}
I20260809 23:22:35.987324 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.988260 24422 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 31.
I20260809 23:22:35.989625 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:35.990280 24378 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:35.990609  6623 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260809 23:22:35.991242  6578 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:35.992123  6605 raft_consensus.cc:2781] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Leader election won for term 30
I20260809 23:22:35.991833  6578 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:35.992877  6605 raft_consensus.cc:686] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:35.993351 24422 raft_consensus.cc:1261] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260809 23:22:35.994372  6578 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:35.994998  6476 consensus_queue.cc:1035] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:35.992306  6590 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:36.001398 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.999325 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:22:36.002326 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.003082 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:36.003120  6590 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:35.991065  6623 raft_consensus.cc:482] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.004484  6623 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:36.005896  6590 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.008117 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:35.993909  6605 consensus_queue.cc:237] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.012653 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:35.958181 24167 catalog_manager.cc:5495] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 19 to 27, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.015493 24528 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ed6a400f2643da95b8f744ebab6f4c with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:38882
W20260809 23:22:36.016188 24171 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 353) took 161118 ms (client timeout 15000 ms). Trace:
I20260809 23:22:36.017973 24152 catalog_manager.cc:4841] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): tablet e4ed6a400f2643da95b8f744ebab6f4c (table test-workload [id=80f516be4b44458f9a25e80f3cd47e1e]) successfully deleted
I20260809 23:22:36.015727  6623 raft_consensus.cc:504] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.018563 24171 rpcz_store.cc:269] 0809 23:19:54.897182 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:19:54.897457 (+   275us) service_pool.cc:227] Handling call
0809 23:22:36.015976 (+161118519us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119814,"reported_tablets":47}
I20260809 23:22:36.021250 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.014207 24167 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "60d5c69728134c8bb918a900eb6a8c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.027637  6578 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:36.028069 24167 catalog_manager.cc:5495] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.036207 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcb08685ecd486c8f1d951336f39982" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.037565 24428 raft_consensus.cc:3037] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Advancing to term 32
I20260809 23:22:36.028033  6578 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.038506  6578 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:36.036712  6623 leader_election.cc:288] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.045794 25323 leader_election.cc:334] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.051602  6604 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.051259 24171 catalog_manager.cc:5495] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 26 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:36.052141  6604 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.054399  6604 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.055579 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.070263 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.070752 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.071091  6623 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.071837  6623 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.072467  6621 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:22:36.076023 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.082443 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.087495  6609 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.088160  6609 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.081195  6289 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.089676  6289 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.090298  6609 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.091645  6289 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.094422 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.098614 24428 raft_consensus.cc:2454] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 32.
W20260809 23:22:36.098677 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:36.096995 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.101558 25321 leader_election.cc:302] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.101734 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.100521 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.105122  6604 raft_consensus.cc:2781] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Leader election won for term 32
I20260809 23:22:36.105702  6604 raft_consensus.cc:686] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:36.106614  6604 consensus_queue.cc:237] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.104352  6621 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.081449  6623 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.099349 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.116750 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.119626  6511 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:22:36.106655  6578 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.156307 24378 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.156899 24378 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.163854  6605 raft_consensus.cc:2726] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:22:36.165706 24378 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.168547  6578 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.152434  6590 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.169392  6590 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.170135  6511 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.170778  6511 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.172830  6511 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.175721 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.176280 24427 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:36.166184 24378 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.177191 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.180070  6620 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:36.180481  6620 raft_consensus.cc:482] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.180820  6620 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.183884 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.184044 24378 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.188560  6605 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.188885 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3063532c2cd49f8b7075e771a2e818a" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.189091  6605 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.189474 25368 raft_consensus.cc:3037] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:36.197721 24428 raft_consensus.cc:1261] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260809 23:22:36.198321 24170 catalog_manager.cc:5495] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 31 to 32, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 32 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260809 23:22:36.200611 24498 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:22:36.201310 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.201746  6604 consensus_queue.cc:1035] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260809 23:22:36.187054  6590 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.177799  6578 raft_consensus.cc:2726] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:36.203331  6605 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.207998 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.208714 24539 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:36.210381 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:22:36.223490 25368 raft_consensus.cc:2454] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:36.225907 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.228029 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.232362  6476 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:36.230269  6605 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:22:36.232918  6476 raft_consensus.cc:482] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.233100  6605 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.235208  6476 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.235462  6605 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:36.215785  6590 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.237402  6590 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.228766  6623 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.240052  6590 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.240192  6623 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.242966 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.245167 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.250202  6476 raft_consensus.cc:504] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.251993  6623 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:22:36.255337 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260809 23:22:36.255761 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.257892  6623 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.272430  6605 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.273008  6289 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.273583  6289 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.276041  6289 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.276110  6620 raft_consensus.cc:504] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.276057 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b804168f1e458cae6cd72cbdb5370e" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.277482 24545 raft_consensus.cc:3037] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:22:36.281287  6620 leader_election.cc:288] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.283687  6476 leader_election.cc:288] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.286051 24422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e16e58b6ff4df4bcb9bb7ca0300f4d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.286693 24422 raft_consensus.cc:3037] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.276126  6605 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.290731  6623 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.291280  6623 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.293473  6511 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.298276  6511 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.300885  6511 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.301213  6621 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.306694 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904621b566f64064bc1a7411ab27c677" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.307230 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.308606 24422 raft_consensus.cc:2454] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:36.309873 24496 leader_election.cc:302] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.310992  6590 raft_consensus.cc:2781] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:36.311619  6590 raft_consensus.cc:686] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.312804  6590 consensus_queue.cc:237] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.307482 24427 raft_consensus.cc:3037] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:36.307428 24545 raft_consensus.cc:2454] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.308156 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:36.317111 24380 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.304934  6621 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.318444  6605 raft_consensus.cc:2781] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader election won for term 26
I20260809 23:22:36.320678  6621 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.320338 24425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.318996  6605 raft_consensus.cc:686] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.322899  6605 consensus_queue.cc:237] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.304996  6623 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.344298 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.345630 25368 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:36.347309 24167 catalog_manager.cc:5495] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:36.348569 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.347831 24378 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.351965 25363 raft_consensus.cc:1261] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260809 23:22:36.348745 24167 catalog_manager.cc:5495] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 29 to 30, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 30 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.358831 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.359948 25365 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
W20260809 23:22:36.365483 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.373005 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.362408  6609 raft_consensus.cc:2781] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
I20260809 23:22:36.355793  6578 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.384415  6578 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.379403  6289 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.408126  6578 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.412528 24427 raft_consensus.cc:2454] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
W20260809 23:22:36.420725 24161 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 357) took 27370 ms (client timeout 15000 ms). Trace:
W20260809 23:22:36.429005 24161 rpcz_store.cc:269] 0809 23:22:09.052426 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:09.052627 (+   201us) service_pool.cc:227] Handling call
0809 23:22:36.420343 (+27367716us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":920363,"reported_tablets":47}
W20260809 23:22:36.433378 25323 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.436643 25323 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.441308 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.431569  6609 raft_consensus.cc:686] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
W20260809 23:22:36.446395 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.448930 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.449303 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.403915  6289 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.457369 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.447872  6604 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.462124 24496 leader_election.cc:302] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.445593  6649 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.459635  6622 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.511236  6649 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.511426  6622 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.513427  6649 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.514315  6622 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.514091  6609 consensus_queue.cc:237] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.472074  6578 consensus_queue.cc:1035] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:36.473799  6289 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.462003  6623 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260809 23:22:36.518330 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:36.498306 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.519246 24543 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 26.
I20260809 23:22:36.521322 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.522488  6622 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.523104  6622 raft_consensus.cc:482] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.523658  6622 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:36.525760 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.528643 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.531880 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 86830.8us
I20260809 23:22:36.532441  6643 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.535041  6643 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.536703  6622 raft_consensus.cc:504] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.537602 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 75555.7us
I20260809 23:22:36.539111  6511 raft_consensus.cc:2781] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:36.539170  6622 leader_election.cc:288] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.539767  6511 raft_consensus.cc:686] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.541102  6511 consensus_queue.cc:237] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.547849 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66997a772ef46a5ae07d530d26ab8a0" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.548503 24539 raft_consensus.cc:3037] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.552343  6638 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.552882  6638 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.554833  6611 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.555316  6611 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.555959  6638 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.557065 24539 raft_consensus.cc:2454] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:22:36.557173  6611 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:36.533838 24498 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.557402 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.558146 24427 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:36.557977 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.558058 25323 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.559263 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
W20260809 23:22:36.560199 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:36.560945 24498 leader_election.cc:334] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.561822 24498 leader_election.cc:334] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.548656 24543 raft_consensus.cc:1261] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260809 23:22:36.537173  6643 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.564546  6611 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.565011 25325 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.586023  6604 raft_consensus.cc:2781] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:36.565111  6638 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:36.586555  6604 raft_consensus.cc:686] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:22:36.586848  6638 raft_consensus.cc:482] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.587329  6638 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.587697  6604 consensus_queue.cc:237] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.571209 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 119753us
I20260809 23:22:36.590644 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.568430 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.593303 24543 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 21.
W20260809 23:22:36.593689 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:36.594177 24378 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.594815 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.594604  6623 consensus_queue.cc:1035] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260809 23:22:36.599346 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.600178  6638 raft_consensus.cc:504] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.603680 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc822924176247c7abbc4a4756daf3a6" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:36.604434 24427 raft_consensus.cc:3037] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.625108 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.627120  6605 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260809 23:22:36.627553  6605 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.627928  6605 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:36.625864 24427 raft_consensus.cc:2454] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 29.
I20260809 23:22:36.632215 24496 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.633133  6511 raft_consensus.cc:2781] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:36.635079  6611 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.635603  6611 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.636973 25363 raft_consensus.cc:1261] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260809 23:22:36.636245 24167 catalog_manager.cc:5495] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 25 to 26, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 26 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:36.562849 24498 leader_election.cc:334] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.638917 24498 leader_election.cc:302] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:36.638266 24167 catalog_manager.cc:5495] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:36.640466 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:36.640312 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 77500us
I20260809 23:22:36.641043  6289 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.641340  6652 raft_consensus.cc:2726] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.642187  6289 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.643447  6611 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.645609  6289 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.641197  6609 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.651358  6609 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.651741  6609 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.654838 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:22:36.655917 24378 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.656436 24378 leader_election.cc:302] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:36.655553 25363 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 23.
I20260809 23:22:36.657531 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.658427 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.637017  6605 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.663558  6643 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.664115  6643 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.665546  6605 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.666079  6643 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.667531 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf691f757af44e6a7cc16228b7424e3" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.668042 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:36.668133 24540 raft_consensus.cc:3037] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Advancing to term 22
I20260809 23:22:36.672418 25368 raft_consensus.cc:1261] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260809 23:22:36.673659 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.674311 25375 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 33.
I20260809 23:22:36.675932 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.658847  6289 raft_consensus.cc:2726] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:36.656126  6511 raft_consensus.cc:686] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:36.681953  6511 consensus_queue.cc:237] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.684221  6546 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.688051 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.688912 25371 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.690215 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:36.690810 25363 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 26.
I20260809 23:22:36.691114 24540 raft_consensus.cc:2454] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 22.
I20260809 23:22:36.692076 24380 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.641054  6638 leader_election.cc:288] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:36.697469 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:36.684777  6546 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.699760  6546 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:36.696456  6605 raft_consensus.cc:2781] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Leader election won for term 22
I20260809 23:22:36.700994  6605 raft_consensus.cc:686] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.701911  6605 consensus_queue.cc:237] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.710964 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.713560 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.714217 24540 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:22:36.715358 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.718187  6546 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260809 23:22:36.720659  6546 raft_consensus.cc:482] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.721179  6546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:36.719022  6638 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.719712  6604 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.727371  6604 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.727829  6578 consensus_queue.cc:1035] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260809 23:22:36.730055  6604 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.716178  6289 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
W20260809 23:22:36.718673 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:36.732263 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
W20260809 23:22:36.733609 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.734402 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.732203 24425 raft_consensus.cc:1261] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260809 23:22:36.735098 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.735172  6609 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.735589 24498 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.735545  6289 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:36.736280 24498 leader_election.cc:334] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.736076  6611 consensus_queue.cc:1035] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:22:36.737833 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.736692  6638 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:36.732752 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.745674  6578 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260809 23:22:36.745275 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.746107  6578 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.746555  6578 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:36.745285 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.749089  6620 consensus_queue.cc:1035] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260809 23:22:36.749820  6658 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.750195  6658 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.750486  6653 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260809 23:22:36.750560  6658 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.750998  6653 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.751415  6653 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 33 FOLLOWER]: Advancing to term 34
I20260809 23:22:36.756464  6578 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:36.767854 24378 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.768330  6289 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:36.768676 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:36.769649 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:36.769989 24425 raft_consensus.cc:1261] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260809 23:22:36.757956  6658 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:36.776819 25371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ad5d0ab888b41229481d32d207c7466" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.775240 24498 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:22:36.777480 25371 raft_consensus.cc:3037] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.780891 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:36.781744 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:36.782526 24545 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.744694  6609 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.798133 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:22:36.799088 24539 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 26.
I20260809 23:22:36.805835 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:36.822595 24540 raft_consensus.cc:1261] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260809 23:22:36.824067 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a4c69723b934fd5869e2f079439b2f9" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.824853 25368 raft_consensus.cc:3037] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Advancing to term 24
I20260809 23:22:36.758693  6578 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:36.831981 25371 raft_consensus.cc:2454] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:36.844065 24378 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:36.783200  6652 raft_consensus.cc:2726] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:22:36.809720  6476 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.828142 24545 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:36.883495  6656 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260809 23:22:36.884140  6656 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.884593  6656 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Advancing to term 27
W20260809 23:22:36.887162 24165 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 355) took 88418 ms (client timeout 15000 ms). Trace:
W20260809 23:22:36.887580 24165 rpcz_store.cc:269] 0809 23:21:08.468468 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:08.470072 (+  1604us) service_pool.cc:227] Handling call
0809 23:22:36.876112 (+88406040us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87933,"reported_tablets":47}
I20260809 23:22:36.873402  6476 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.888443  6476 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.856004  6609 raft_consensus.cc:2781] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
W20260809 23:22:36.903390 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.903980 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:22:36.915472 25365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:36.916160 25365 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:36.872614  6658 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:36.900673 24498 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.894639 25368 raft_consensus.cc:2454] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
W20260809 23:22:36.900228 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.853753  6649 consensus_queue.cc:1035] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260809 23:22:36.903615 24161 catalog_manager.cc:5495] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 27 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:36.827741  6616 consensus_queue.cc:1035] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.098s
W20260809 23:22:36.923344 24378 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:36.914608  6623 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:22:36.917814 24498 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:22:36.929026 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 120411us
I20260809 23:22:36.916891  6651 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.917742  6609 raft_consensus.cc:686] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:36.961910 24378 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:36.964490 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 63842.5us
W20260809 23:22:36.977522 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 102696us
I20260809 23:22:36.961272  6658 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:36.966743  6638 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:22:36.968549  6643 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:22:36.977360  6647 raft_consensus.cc:2781] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader election won for term 24
I20260809 23:22:37.029688 24428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" is_pre_election: true
I20260809 23:22:37.037101 25365 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:37.043982 24378 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.046363 24171 catalog_manager.cc:5495] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.051702  6578 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.052278  6578 raft_consensus.cc:686] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.053314  6578 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.070477 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 97094.2us
I20260809 23:22:37.043732 24165 catalog_manager.cc:5495] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 21 to 22, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 22 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:36.966183  6651 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.075839 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.037113  6647 raft_consensus.cc:686] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.006388  6658 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.081799  6668 log.cc:924] Time spent T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
W20260809 23:22:37.087627 24378 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.091292 24378 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.092007 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.093029 24428 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:37.093523 24378 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.020138  6546 raft_consensus.cc:504] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.104875 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.110121  6643 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:22:37.110556  6643 raft_consensus.cc:482] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:37.110960  6643 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:22:37.111582  6687 log.cc:924] Time spent T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260809 23:22:37.114306  6658 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:22:37.115340 24163 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 373) took 77836 ms (client timeout 15000 ms). Trace:
W20260809 23:22:37.115875 24163 rpcz_store.cc:269] 0809 23:21:19.284711 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:21:19.304198 (+ 19487us) service_pool.cc:227] Handling call
0809 23:22:37.115099 (+77810901us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":48454,"reported_tablets":47}
I20260809 23:22:37.042243  6609 consensus_queue.cc:237] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.129631  6647 consensus_queue.cc:237] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.147698 24425 raft_consensus.cc:1261] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
W20260809 23:22:37.130959 24149 connection.cc:441] server connection from 127.0.0.1:35202 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35202 (request call id 373) could send its response
I20260809 23:22:37.155722  6620 consensus_queue.cc:1035] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260809 23:22:37.158643  6546 leader_election.cc:288] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:37.196631 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.197602 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.198143  6476 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.202077  6653 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.205868 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.206501 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.203009  6476 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.242575 24428 raft_consensus.cc:1261] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260809 23:22:37.243073  6651 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:22:37.244982 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:22:37.248607 25323 leader_election.cc:334] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.249529  6653 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.251345  6653 consensus_queue.cc:1035] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260809 23:22:37.263649  6643 raft_consensus.cc:504] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:22:37.264909 24543 raft_consensus.cc:2379] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab5aefaeb3d14775b30a74a2a6d3357f in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:22:37.265594 25368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48af4d51aa8f41a486c4fe6e7541700e" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.266211 25368 raft_consensus.cc:3037] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:37.265735  6643 leader_election.cc:288] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:22:37.283957 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.284893 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddd2e2b2ca834a8a967d719c1ce9f6b8" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f"
I20260809 23:22:37.290516 24423 raft_consensus.cc:3037] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:22:37.291605 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cadfc1bde6cb4d6997f3fb215dc184ec" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.292212 25363 raft_consensus.cc:3037] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 33 FOLLOWER]: Advancing to term 34
I20260809 23:22:37.302677 25368 raft_consensus.cc:2454] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
W20260809 23:22:37.319564 22524 thread.cc:536] Waited for 1000ms trying to join with diag-logger (tid 24352)
I20260809 23:22:37.355049  6656 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.357618 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 255775us
I20260809 23:22:37.359997 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:22:37.360692 25372 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 24.
I20260809 23:22:37.363230 24546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90dd68b3876b4b75aea408edfa68d5e5" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:37.363651 24539 raft_consensus.cc:1261] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260809 23:22:37.363801 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.364410 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.371512 25372 raft_consensus.cc:1261] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260809 23:22:37.375608 24543 raft_consensus.cc:1261] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260809 23:22:37.379331  6649 consensus_queue.cc:1035] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260809 23:22:37.363858 24546 raft_consensus.cc:3037] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Advancing to term 29
I20260809 23:22:37.393527 24423 raft_consensus.cc:2454] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 28.
I20260809 23:22:37.386433 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
W20260809 23:22:37.399029 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 84979.1us
I20260809 23:22:37.401827 24498 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.403983 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba62e1cbbfd84b86a817f9118be39b00" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:22:37.404633 24543 raft_consensus.cc:3037] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Advancing to term 27
I20260809 23:22:37.407214 24496 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.414678  6706 consensus_queue.cc:1035] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:37.401058  6656 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:22:37.423245 24378 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.423441 25363 raft_consensus.cc:2454] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 34.
I20260809 23:22:37.401861  6605 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260809 23:22:37.466519 24543 raft_consensus.cc:2454] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 27.
I20260809 23:22:37.467694 24380 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 
I20260809 23:22:37.469203  6710 raft_consensus.cc:2781] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.469887  6710 raft_consensus.cc:686] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.470853  6710 consensus_queue.cc:237] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.471802 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 64669.2us
I20260809 23:22:37.472134  6620 raft_consensus.cc:2781] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:22:37.472723  6620 raft_consensus.cc:686] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.473672  6620 consensus_queue.cc:237] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:22:37.478853 24498 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.479689 24498 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.435071  6643 raft_consensus.cc:2781] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader election won for term 27
I20260809 23:22:37.465390  6605 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 60d5c69728134c8bb918a900eb6a8c21)
I20260809 23:22:37.494369 24498 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:37.477784  6697 consensus_queue.cc:1035] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260809 23:22:37.496851  6605 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:37.518592  6638 raft_consensus.cc:2781] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Leader election won for term 34
I20260809 23:22:37.519358  6638 raft_consensus.cc:686] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.492197  6643 raft_consensus.cc:686] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:22:37.520453  6643 consensus_queue.cc:237] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:37.520485  6638 consensus_queue.cc:237] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.590126 24380 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 62443.5us
W20260809 23:22:37.590215 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 54528.7us
W20260809 23:22:37.617717 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 83704.4us
W20260809 23:22:37.621906 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.636514 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.637068 24498 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:37.638075 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.638489 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.643994 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.645466 24378 leader_election.cc:334] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.645942 24378 leader_election.cc:302] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:22:37.620541 24546 raft_consensus.cc:2454] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 29.
I20260809 23:22:37.672513  6691 raft_consensus.cc:2726] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:22:37.672003 25325 leader_election.cc:302] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21
I20260809 23:22:37.683614 24546 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260809 23:22:37.685977 24378 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.688076  6712 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260809 23:22:37.696260 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:37.699887 25363 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260809 23:22:37.680020  6676 raft_consensus.cc:2781] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Leader election won for term 29
I20260809 23:22:37.707556  6676 raft_consensus.cc:686] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
W20260809 23:22:37.708454 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:37.708544  6676 consensus_queue.cc:237] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.701891 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 80103.7us
I20260809 23:22:37.751214  6691 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260809 23:22:37.732060  6605 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.766626 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:37.767501 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.769934 25363 raft_consensus.cc:1261] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260809 23:22:37.796846  6605 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Requested vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:22:37.777441  6656 consensus_queue.cc:1035] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260809 23:22:37.814085 25363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9932f35c1d8a431f9c035a848f63cb0b" candidate_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:22:37.814733 25363 raft_consensus.cc:3037] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Advancing to term 25
I20260809 23:22:37.817849 25372 raft_consensus.cc:1261] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260809 23:22:37.829679 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:37.831710 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.832746 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:37.845273 25363 raft_consensus.cc:2454] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab5aefaeb3d14775b30a74a2a6d3357f in term 25.
I20260809 23:22:37.849568 24378 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9; no voters: 
I20260809 23:22:37.850126 24171 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35222 (request call id 359) took 1684 ms. Trace:
I20260809 23:22:37.859333  6656 consensus_queue.cc:1035] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260809 23:22:37.859534  6651 raft_consensus.cc:2781] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Leader election won for term 25
W20260809 23:22:37.882771 24378 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:22:37.850490 24171 rpcz_store.cc:276] 0809 23:22:36.166218 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:22:36.166372 (+   154us) service_pool.cc:227] Handling call
0809 23:22:37.850073 (+1683701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0809 23:22:37.203714 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0809 23:22:37.204162 (+   448us) write_op.cc:432] Acquiring schema lock in shared mode
0809 23:22:37.204190 (+    28us) write_op.cc:435] Acquired schema lock
0809 23:22:37.204201 (+    11us) tablet.cc:660] Decoding operations
0809 23:22:37.229239 (+ 25038us) write_op.cc:620] Acquiring the partition lock for write op
0809 23:22:37.234030 (+  4791us) write_op.cc:641] Partition lock acquired for write op
0809 23:22:37.239407 (+  5377us) tablet.cc:683] Acquiring locks for 1 operations
0809 23:22:37.271128 (+ 31721us) tablet.cc:699] Row locks acquired
0809 23:22:37.274608 (+  3480us) write_op.cc:260] PREPARE: finished
0809 23:22:37.291973 (+ 17365us) write_op.cc:270] Start()
0809 23:22:37.297712 (+  5739us) write_op.cc:276] Timestamp: P: 1786317757291903 usec, L: 0
0809 23:22:37.305003 (+  7291us) op_driver.cc:331] REPLICATION: starting
0809 23:22:37.351626 (+ 46623us) log.cc:842] Serialized 533 byte log entry
0809 23:22:37.695950 (+344324us) op_driver.cc:444] REPLICATION: finished
0809 23:22:37.756071 (+ 60121us) write_op.cc:301] APPLY: starting
0809 23:22:37.757452 (+  1381us) tablet.cc:1365] starting BulkCheckPresence
0809 23:22:37.795413 (+ 37961us) tablet.cc:1368] finished BulkCheckPresence
0809 23:22:37.795559 (+   146us) tablet.cc:1370] starting ApplyRowOperation cycle
0809 23:22:37.796363 (+   804us) tablet.cc:1381] finished ApplyRowOperation cycle
0809 23:22:37.796570 (+   207us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0809 23:22:37.796615 (+    45us) write_op.cc:312] APPLY: finished
0809 23:22:37.796811 (+   196us) log.cc:842] Serialized 25 byte log entry
0809 23:22:37.796980 (+   169us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:37.797068 (+    88us) write_op.cc:454] Released schema lock
0809 23:22:37.797104 (+    36us) write_op.cc:341] FINISH: Updating metrics
0809 23:22:37.797424 (+   320us) write_op.cc:489] Releasing partition, row and schema locks
0809 23:22:37.797450 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":78877,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":5071,"apply.run_cpu_time_us":3943,"apply.run_wall_time_us":67148,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8303,"mutex_wait_us":23908,"num_ops":1,"prepare.queue_time_us":127342,"prepare.run_cpu_time_us":5442,"prepare.run_wall_time_us":149267,"replication_time_us":376419}]]}
I20260809 23:22:37.896363 24161 catalog_manager.cc:5495] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.916560 24161 catalog_manager.cc:5495] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 28 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 29 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.892040  6651 raft_consensus.cc:686] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Becoming Leader. State: Replica: ab5aefaeb3d14775b30a74a2a6d3357f, State: Running, Role: LEADER
I20260809 23:22:37.918140 24161 catalog_manager.cc:5495] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.935763  6651 consensus_queue.cc:237] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:22:37.929203 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:37.940235 24161 catalog_manager.cc:5495] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } }
I20260809 23:22:37.946978 24161 catalog_manager.cc:5495] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 23 to 24, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 24 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.952870 24161 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:37.954839 24161 catalog_manager.cc:5495] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:37.981855 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:37.992295 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:37.995965 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.080034 24165 catalog_manager.cc:5495] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 28 to 29, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 29 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:22:38.081002 25363 raft_consensus.cc:1261] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260809 23:22:38.087561  6655 consensus_queue.cc:1035] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260809 23:22:38.095188 24543 raft_consensus.cc:1261] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260809 23:22:38.096500  6691 consensus_queue.cc:1035] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260809 23:22:38.102751 24171 catalog_manager.cc:5495] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 27 to 28, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 28 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } }
I20260809 23:22:38.104940 24171 catalog_manager.cc:5495] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 33 to 34, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 34 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:38.117668 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.136118 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.141448 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.106550 24171 catalog_manager.cc:5495] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 26 to 27, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 27 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:22:38.169816 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:38.173801 24427 raft_consensus.cc:1261] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260809 23:22:38.174258 25363 raft_consensus.cc:1261] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260809 23:22:38.176508  6620 consensus_queue.cc:1035] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260809 23:22:38.177851  6620 consensus_queue.cc:1035] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260809 23:22:38.194178 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260809 23:22:38.197789 24165 catalog_manager.cc:5495] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f reported cstate change: term changed from 24 to 25, leader changed from 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) to ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2). New cstate: current_term: 25 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260809 23:22:38.238234 24546 raft_consensus.cc:1261] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260809 23:22:38.241605  6675 consensus_queue.cc:1035] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260809 23:22:38.320088 22524 thread.cc:536] Waited for 2000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:38.397346 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:38.410494 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260809 23:22:38.436255 25375 raft_consensus.cc:1261] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260809 23:22:38.437789  6691 consensus_queue.cc:1035] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260809 23:22:38.489917 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:38.540879 25323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 369 similar messages]
W20260809 23:22:38.632645 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:38.666527 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:38.723593 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:38.828394 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.844589 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.907812 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:38.929491 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:38.935875 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:39.030452 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.069355 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.121943 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.163829 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.215929 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.246104 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.320679 22524 thread.cc:536] Waited for 3000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:39.325034 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:39.332139 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.334118 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.340541 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.374466 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.486325 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.497105 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.541884 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 45124.4us
W20260809 23:22:39.631208 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.647257 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.691574 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.762539 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:39.779362 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:39.840782 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:39.903658 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.018432 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.019141 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.293251 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.305536 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.321506 22524 thread.cc:536] Waited for 4000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:40.331434 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.342473 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.365185 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.382272 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:40.414088 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.504138 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.522156 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.533262 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:40.542294 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.550172 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.653672 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.654194 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:40.679430 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.771974 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:22:40.983645 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.048519 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.067188 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:41.204190 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.257122 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.265594 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.322227 22524 thread.cc:536] Waited for 5000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:41.378995 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.379693 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.523564 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 76082.5us
W20260809 23:22:41.537828 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.545823 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.644548 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.656816 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.699756 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.738806 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.774240 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.858539 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:41.872458 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:41.906867 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.915017 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:41.916898 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.929023 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.932312 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:41.984566 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.026434 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.050009 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.264060 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.270288 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.358246 22524 thread.cc:536] Waited for 6000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:42.400383 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.424820 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:42.444995 24378 consensus_peers.cc:487] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.507004 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.513021 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.642519 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.653707 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.776948 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:42.785567 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:42.787222 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:42.796777 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:42.995898 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:43.043123 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:43.067287 24427 consensus_queue.cc:237] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:22:43.111157 25363 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260809 23:22:43.116122  6763 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260809 23:22:43.122372 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.142885 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260809 23:22:43.152338 24544 raft_consensus.cc:1261] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Refusing update from remote peer ab5aefaeb3d14775b30a74a2a6d3357f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260809 23:22:43.163671 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 23:22:43.169204  6763 consensus_queue.cc:1035] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260809 23:22:43.173656  6786 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:43.175432 25375 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:43.194276 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260809 23:22:43.214780 24171 catalog_manager.cc:5495] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New cstate: current_term: 27 leader_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
I20260809 23:22:43.224139 24150 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c3876fa8158d49ec9c1128fbebe6f994 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 15)
I20260809 23:22:43.234522 24544 raft_consensus.cc:2932] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 6 to 18, VOTER 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } } }
W20260809 23:22:43.246618 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.277037 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:43.348836 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.360848 22524 thread.cc:536] Waited for 7000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:43.465983 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260809 23:22:43.481552 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:43.506009 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:43.546617 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 475 similar messages]
W20260809 23:22:43.613065 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.695175 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.734637 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:43.762704 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:43.949922 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.971514 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:43.981755 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.071445 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.185603 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.204026 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.205547 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.214458 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.251688 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.288883 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.295830 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.323282 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.361466 22524 thread.cc:536] Waited for 8000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:44.422550 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.429253 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:44.524637 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.534207 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.590778 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.639392 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.650266 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.656983 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:44.703169 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.913066 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:44.916213 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:44.975437 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:44.976068 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.031840 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.065470 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:45.102443 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.223168 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.272696 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.274536 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.362315 22524 thread.cc:536] Waited for 9000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:45.403436 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.410341 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.547221 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.603015 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.645450 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.650213 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.694408 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.772217 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:45.799011 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.801237 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:45.802277 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:45.921953 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 48313us
W20260809 23:22:45.922760 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260809 23:22:45.945344 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.011148 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.203291 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.220029 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.322443 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.345176 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.363065 22524 thread.cc:536] Waited for 10000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:46.373409 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.377466 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.398969 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.649892 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.679776 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:46.717887 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.719462 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.880457 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:46.908936 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.939949 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:46.954097 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.019589 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.028900 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.029526 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.032310 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.047176 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.222010 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.262892 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.318364 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.363655 22524 thread.cc:536] Waited for 11000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:47.383950 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 52034.5us
W20260809 23:22:47.448067 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.507507 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.536587 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.540005 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.652731 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:47.685117 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.687077 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.777523 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:47.791504 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.847868 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:47.972888 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:47.981439 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.025175 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.032135 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:48.211882 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 66920.4us
W20260809 23:22:48.223740 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.248937 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.260950 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08004028c0 after lost signal to thread 22536
W20260809 23:22:48.262603 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc3a0 after lost signal to thread 24220
W20260809 23:22:48.267364 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.267450 25405 debug-util.cc:398] Leaking SignalData structure 0x7b0800439ca0 after lost signal to thread 24469
W20260809 23:22:48.269533 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08003723a0 after lost signal to thread 24579
W20260809 23:22:48.270995 25405 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb860 after lost signal to thread 25410
W20260809 23:22:48.307157 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.334534 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260809 23:22:48.347944 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:48.357765 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:48.364295 22524 thread.cc:536] Waited for 12000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:48.394030 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.549158 24498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 463 similar messages]
W20260809 23:22:48.551823 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.585500 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.750501 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:48.753178 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:48.806869 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.823016 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:48.893935 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:48.953387 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.100961 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.112106 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.123922 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.308080 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.364880 22524 thread.cc:536] Waited for 13000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:49.367897 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.389302 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:49.446261 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:49.471419 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.543403 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.528348 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.554752 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.604717 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.631367 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:49.694960 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:49.775035 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.853561 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:49.866576 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:50.043990 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:50.077615 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.091914 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.166442 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.247498 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 59173.2us
W20260809 23:22:50.293711 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 95726.8us
W20260809 23:22:50.297850 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.330900 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.333119 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.365419 22524 thread.cc:536] Waited for 14000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:50.381793 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.382426 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.447394 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.511991 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.607074 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.635382 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.643216 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.775123 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:50.787914 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:50.838515 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:50.972340 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:51.011555 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 46063.7us
W20260809 23:22:51.066491 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:51.133636 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.220157 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.229753 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260809 23:22:51.287300 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 46827.5us
W20260809 23:22:51.290987 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.365943 22524 thread.cc:536] Waited for 15000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:51.389916 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08003383e0 after lost signal to thread 22536
W20260809 23:22:51.391295 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08004045c0 after lost signal to thread 24220
W20260809 23:22:51.392832 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080024d400 after lost signal to thread 24469
W20260809 23:22:51.394222 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08002669e0 after lost signal to thread 24579
W20260809 23:22:51.395615 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad4c0 after lost signal to thread 25410
W20260809 23:22:51.517055 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 67637.4us
W20260809 23:22:51.542796 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.595595 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:51.613942 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.665479 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.673713 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.675976 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.676581 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.674353 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.701221 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.792834 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.812414 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:51.958516 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:51.966392 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.003068 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.003926 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.118799 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.142284 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.155673 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.260723 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.287057 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.366530 22524 thread.cc:536] Waited for 16000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:52.376441 24151 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 179514us
W20260809 23:22:52.424111 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.434791 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:22:52.478682 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.497992 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.501741 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.699298 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.760114 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.775743 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:52.848161 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:52.870779 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.878947 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.897091 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:52.917240 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:52.925702 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.050067 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.068535 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.261554 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.278112 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.315799 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.367496 22524 thread.cc:536] Waited for 17000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:53.418872 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.544785 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.545728 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:53.550076 25323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 447 similar messages]
W20260809 23:22:53.552659 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:53.581354 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.649751 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.651990 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260809 23:22:53.670410 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.689635 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:53.884833 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.925707 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:53.974997 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:54.009560 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.044349 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.119601 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.138975 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.216213 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.235695 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.368498 22524 thread.cc:536] Waited for 18000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:54.386703 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.443899 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.451130 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.507561 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:54.567802 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.652784 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.664502 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:54.686590 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:54.690105 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.718998 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 53248.7us
W20260809 23:22:54.808887 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.814916 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.961126 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:54.998842 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.060998 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.136780 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.145308 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.200248 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.210507 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.243661 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.369002 22524 thread.cc:536] Waited for 19000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:55.393031 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:55.420848 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.439136 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.491663 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.496644 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.506549 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:55.587136 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.664705 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.887292 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:55.972971 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:55.975260 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 108193us
W20260809 23:22:55.999130 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:56.003700 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:56.012493 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:56.089093 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 56306.2us
W20260809 23:22:56.099349 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.109369 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.203042 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 71090.3us
W20260809 23:22:56.203872 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.295034 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 157058us
W20260809 23:22:56.369570 22524 thread.cc:536] Waited for 20000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:56.367870 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.422451 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.423388 25321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 53443.7us
W20260809 23:22:56.443636 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:56.482096 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.583734 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260809 23:22:56.621840 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.649557 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.681615 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.823266 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:56.825407 24378 consensus_peers.cc:487] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.976714 24378 consensus_peers.cc:487] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:56.988842 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:57.019461 25325 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:37765, user_credentials={real_user=slave}} blocked reactor thread for 52922.8us
W20260809 23:22:57.035686 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.036978 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.081753 25323 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 44833.6us
W20260809 23:22:57.077755 24378 consensus_peers.cc:487] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.087638 24496 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42099, user_credentials={real_user=slave}} blocked reactor thread for 127227us
W20260809 23:22:57.086755 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:36681, user_credentials={real_user=slave}} blocked reactor thread for 51815.1us
W20260809 23:22:57.100490 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.158269 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.205722 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.207216 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.207732 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.313390 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.370642 22524 thread.cc:536] Waited for 21000ms trying to join with diag-logger (tid 24352)
W20260809 23:22:57.464855 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.553798 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.638461 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.651748 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:57.720824 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.843542 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:57.843678 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08002011a0 after lost signal to thread 22536
W20260809 23:22:57.845160 24464 debug-util.cc:398] Leaking SignalData structure 0x7b080005db60 after lost signal to thread 24220
W20260809 23:22:57.846634 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800346d60 after lost signal to thread 24469
W20260809 23:22:57.848953 24464 debug-util.cc:398] Leaking SignalData structure 0x7b0800318160 after lost signal to thread 24579
W20260809 23:22:57.850350 24464 debug-util.cc:398] Leaking SignalData structure 0x7b08001b97a0 after lost signal to thread 25410
W20260809 23:22:57.919503 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:57.940189 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:57.967974 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.235687 24378 consensus_peers.cc:487] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.282364 24378 consensus_peers.cc:487] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.306668 24378 consensus_peers.cc:487] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.332921 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.365753 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.375756 24378 consensus_peers.cc:487] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.421674 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.604270 24378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 114436us
W20260809 23:22:58.606513 24378 consensus_peers.cc:487] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.607425 24378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 432 similar messages]
W20260809 23:22:58.741597 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:58.748865 24498 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:38713, user_credentials={real_user=slave}} blocked reactor thread for 319200us
W20260809 23:22:58.776238 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:58.799105 24378 consensus_peers.cc:487] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.845772 24378 consensus_peers.cc:487] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:22:58.896803 24378 consensus_peers.cc:487] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:58.921192 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:58.975860 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.979249 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:58.982692 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:59.103904 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.114744 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260809 23:22:59.115378 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
W20260809 23:22:59.131803 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.160400 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.239082 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260809 23:22:59.260784 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.309881 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.388428 24378 consensus_peers.cc:487] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:22:59.407191 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
W20260809 23:22:59.420207 24496 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.464097 24496 consensus_peers.cc:487] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.470050 25321 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.495548 24496 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.496939 24496 consensus_peers.cc:487] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.499890 25321 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.514068 25321 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.516753 24496 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.522607 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:22:59.525205 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.526414 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.529441 24496 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.530012 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.533202 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.537729 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:22:59.540827 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08001718e0 after lost signal to thread 22536
I20260809 23:22:59.543989 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.544320 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08002524e0 after lost signal to thread 24220
I20260809 23:22:59.544680 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.545922 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08003b5360 after lost signal to thread 24469
I20260809 23:22:59.547734 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.548183 24217 debug-util.cc:398] Leaking SignalData structure 0x7b08003ce780 after lost signal to thread 24579
I20260809 23:22:59.548617 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.550341 24217 debug-util.cc:398] Leaking SignalData structure 0x7b0800365ac0 after lost signal to thread 25410
I20260809 23:22:59.550669 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.554395 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.559439 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.561132 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.570411 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.571326 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.572031 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.579250 24496 consensus_peers.cc:487] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.581585 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.582652 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.583367 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.589406 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.591629 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.597581 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:22:59.597692 25321 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.602469 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.605757 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.606658 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.607934 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.610740 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.615568 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.616289 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P ab5aefaeb3d14775b30a74a2a6d3357f [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.618481 25321 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.623008 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.625131 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.625599 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:22:59.626675 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.631801 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.636312 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.640152 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.643150 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.643992 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.644650 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.647476 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.648921 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.650209 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.652860 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.653716 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.654804 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.655813 24496 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.657420 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.658298 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.659260 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.662096 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.662906 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.664022 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.664988 24496 consensus_peers.cc:487] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.666893 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.667900 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.669112 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.671836 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.672650 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.673326 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.676057 24496 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.683148 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.694794 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.696729 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:22:59.698987 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.713644 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.714725 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 LEADER]: Raft consensus shutting down.
W20260809 23:22:59.716053 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:22:59.716061 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.719242 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.721617 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.722434 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.725143 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.728137 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.729169 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.731905 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.732774 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus shutting down.
W20260809 23:22:59.733330 25321 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.734033 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P ab5aefaeb3d14775b30a74a2a6d3357f [term 29 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.736668 24496 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.736820 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.737768 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.738510 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.741452 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.742200 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.743096 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.746194 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.747056 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.751068 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.753513 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.754313 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.755576 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.758246 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.760141 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.761615 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.768427 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.770994 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.772274 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.772671 25321 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.777433 24496 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.777819 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.778599 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.783110 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.788234 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.789211 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.789986 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.795145 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.800843 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 23:22:59.802321 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.803095 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.807219 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.808132 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.813192 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P ab5aefaeb3d14775b30a74a2a6d3357f [term 31 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.817631 24496 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.815850 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.821504 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.822872 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.826555 24378 consensus_peers.cc:487] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:22:59.827100 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.828007 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.829569 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.834688 24496 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.844429 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.845830 24496 consensus_peers.cc:487] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.846338 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.847942 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.852559 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.853350 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.854038 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.858114 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.858958 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.860250 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P ab5aefaeb3d14775b30a74a2a6d3357f [term 22 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.861634 24496 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.866024 25321 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.869925 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.871073 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 LEADER]: Raft consensus shutting down.
W20260809 23:22:59.879765 25321 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.872406 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P ab5aefaeb3d14775b30a74a2a6d3357f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.885178 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.887179 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.887974 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.890738 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
W20260809 23:22:59.891458 24496 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.891503 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.896991 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P ab5aefaeb3d14775b30a74a2a6d3357f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.899775 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.900633 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:22:59.901857 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P ab5aefaeb3d14775b30a74a2a6d3357f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.905283 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.906450 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.907262 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.913030 25321 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:22:59.914628 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.915482 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.916203 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P ab5aefaeb3d14775b30a74a2a6d3357f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.918972 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.919704 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.920341 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.925096 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.925803 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.926590 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P ab5aefaeb3d14775b30a74a2a6d3357f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:22:59.929589 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f: stopping tablet replica
I20260809 23:22:59.930240 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:22:59.930915 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P ab5aefaeb3d14775b30a74a2a6d3357f [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:22:59.936053 24496 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.936856 24498 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:22:59.945518 25321 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:22:59.979723 24496 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:00.151129 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:00.166225 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.263809 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.300781 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:00.404243 24498 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:00.428694 24498 consensus_peers.cc:487] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:23:00.429443 24498 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:00.430051 24498 consensus_peers.cc:487] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:00.471663 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260809 23:23:00.541488  6882 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.542562  6882 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.553126  6882 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.555037 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:00.555732 25321 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.556270 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.557637  6882 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:00.635797  6902 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.636442  6902 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:00.638468  6902 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.649678 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:00.652194 24496 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.652643 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.654052  6887 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260809 23:23:00.732260 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.739192  6902 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.739708  6902 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.743423  6902 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:00.745087 25375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
W20260809 23:23:00.751788 24496 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.754261 24498 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:00.755484  6887 raft_consensus.cc:2726] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:00.771255  6898 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.772006  6898 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.785213  6887 raft_consensus.cc:482] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.774710  6898 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:00.785531 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:00.789062 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:00.789583 25321 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.791497  6898 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:00.795977  6887 raft_consensus.cc:504] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:00.797997  6887 leader_election.cc:288] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.888159  6903 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.888962  6903 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.891700  6903 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.892477  6903 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.893114  6903 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.895524  6903 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.891587  6898 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.896302  6898 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.898367  6898 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:00.903151 25321 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.903923 25321 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.904454 25321 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.907500 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.907630  6898 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:00.908051 25323 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.908196  6898 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:00.908888  6903 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260809 23:23:00.909850 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.911001 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.910209  6898 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:00.911926  6898 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260809 23:23:00.912752 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.913380 25323 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:00.915791 25321 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:00.920619 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:00.922425  6898 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:00.922449 25323 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:00.924201  6898 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:01.233134  6921 raft_consensus.cc:482] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.245956  6903 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.253419  6907 raft_consensus.cc:482] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.254019  6907 raft_consensus.cc:504] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.256237  6907 leader_election.cc:288] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.247227  6903 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.259202  6903 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:01.260947 24498 consensus_peers.cc:487] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:23:01.266809  6882 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.267458  6882 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.268177  6920 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.268901  6920 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.272619  6920 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.279363  6887 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.283774  6887 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.286469  6887 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
W20260809 23:23:01.282877 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260809 23:23:01.269464  6882 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:01.289903 25321 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.296285 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.297075 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.297610 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.298184 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.293606  6903 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.299203  6903 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.301291  6903 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:01.305308 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.233789  6921 raft_consensus.cc:504] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.316179  6921 leader_election.cc:288] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.298640 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.317857  6903 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:23:01.322595  6933 raft_consensus.cc:482] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.323462  6933 raft_consensus.cc:504] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.330801  6887 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:23:01.331334 24498 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.331377  6887 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.333429  6887 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.341257  6921 raft_consensus.cc:482] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.341847  6921 raft_consensus.cc:504] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:01.329253  6933 leader_election.cc:288] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.327505  6938 raft_consensus.cc:482] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
W20260809 23:23:01.347081 24496 leader_election.cc:334] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.348506 24496 leader_election.cc:302] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.357219 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.358171 25372 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 27.
I20260809 23:23:01.347823  6938 raft_consensus.cc:504] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
W20260809 23:23:01.362879 24498 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:23:01.363848  6938 leader_election.cc:288] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.364384  6938 raft_consensus.cc:2726] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.365386 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
I20260809 23:23:01.365628 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.366092 24543 raft_consensus.cc:2454] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:23:01.366197 25372 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 25.
W20260809 23:23:01.367014 24496 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.367293 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:01.367820 24496 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.368237 24543 raft_consensus.cc:2454] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 25.
W20260809 23:23:01.368726 24496 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.369769 24496 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.371579 24496 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.363823 24498 consensus_peers.cc:487] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260809 23:23:01.347234  6921 leader_election.cc:288] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:01.372778 24540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9" is_pre_election: true
W20260809 23:23:01.376107 24498 leader_election.cc:334] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.376518 24498 leader_election.cc:302] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.376653 24540 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 27.
I20260809 23:23:01.377327  6931 raft_consensus.cc:2726] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260809 23:23:01.382838 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.383948  6920 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.384330  6920 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.384675  6920 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
W20260809 23:23:01.385229 24496 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:01.385910 24496 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.388934 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.389747  6931 raft_consensus.cc:2781] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.390152  6931 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.390553  6931 raft_consensus.cc:3037] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.400089 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.400449 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.400974  6903 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:23:01.401242  6921 raft_consensus.cc:2781] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260809 23:23:01.401329  6903 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.401736  6921 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.401799  6903 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:23:01.407898  6921 raft_consensus.cc:3037] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Advancing to term 26
I20260809 23:23:01.397464  6920 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.421435  6921 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.416770 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.416721  6903 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.427883 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.430756  6924 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260809 23:23:01.431238  6924 raft_consensus.cc:482] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.431596  6924 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.433043  6903 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.398324  6931 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.438524 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.440888  6938 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.441431  6938 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.446748 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.447562 24543 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 26: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
I20260809 23:23:01.451465  6938 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.454375  6924 raft_consensus.cc:504] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:01.453603 24496 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.453441 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.459767  6903 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260809 23:23:01.462616 24496 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.469880 24539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.470679 24539 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate b9b54436d0e94a5595cdded73fe33da9 in current term 28: Already voted for candidate 808fed3ed3f04882ad757a3db9abfdd9 in this term.
W20260809 23:23:01.463440 24498 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.472978 24498 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260809 23:23:01.476796 24498 leader_election.cc:334] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.477190 24498 leader_election.cc:302] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.477878  6938 raft_consensus.cc:2726] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:23:01.478281 24498 leader_election.cc:334] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.478703 24498 leader_election.cc:302] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.479323  6938 raft_consensus.cc:2726] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260809 23:23:01.480362 24498 leader_election.cc:334] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.480741 24498 leader_election.cc:302] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.481357  6938 raft_consensus.cc:2726] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:01.457006  6921 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.474148 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:01.485071 25321 leader_election.cc:334] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.475162 24543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37e848050ec44069438f4c88a76f9ef" candidate_uuid: "b9b54436d0e94a5595cdded73fe33da9" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "808fed3ed3f04882ad757a3db9abfdd9"
I20260809 23:23:01.485726 24543 raft_consensus.cc:3037] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Advancing to term 28
I20260809 23:23:01.485816 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9" is_pre_election: true
I20260809 23:23:01.486472 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 22.
I20260809 23:23:01.489825 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ef03e5e7d0443d69a2228dd175bd010" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.490489 25372 raft_consensus.cc:2379] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 26: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
W20260809 23:23:01.477517 24496 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.474565  6924 leader_election.cc:288] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.485209  6903 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.467020  6931 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.492944 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3876fa8158d49ec9c1128fbebe6f994" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.493623 25372 raft_consensus.cc:2379] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 808fed3ed3f04882ad757a3db9abfdd9 in current term 28: Already voted for candidate b9b54436d0e94a5595cdded73fe33da9 in this term.
I20260809 23:23:01.495401 24543 raft_consensus.cc:2454] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9b54436d0e94a5595cdded73fe33da9 in term 28.
I20260809 23:23:01.467523  6920 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:01.503159 25325 leader_election.cc:302] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:01.503731 25323 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.499194 24498 leader_election.cc:334] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:01.504487 24498 leader_election.cc:302] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.505231  6931 raft_consensus.cc:2726] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:01.506047  6920 raft_consensus.cc:2781] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader election won for term 28
I20260809 23:23:01.506793  6920 raft_consensus.cc:686] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Becoming Leader. State: Replica: b9b54436d0e94a5595cdded73fe33da9, State: Running, Role: LEADER
I20260809 23:23:01.507714  6920 consensus_queue.cc:237] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.509847 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.510622  6931 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260809 23:23:01.510989  6931 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:01.511337  6931 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Advancing to term 23
I20260809 23:23:01.531504 24498 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:01.532779 24498 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f, b9b54436d0e94a5595cdded73fe33da9
I20260809 23:23:01.535176  6931 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.536233 24163 catalog_manager.cc:5495] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 reported cstate change: term changed from 27 to 28, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4). New cstate: current_term: 28 leader_uuid: "b9b54436d0e94a5595cdded73fe33da9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260809 23:23:01.537259  6931 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Requested vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), b9b54436d0e94a5595cdded73fe33da9 (127.21.255.4:38713)
I20260809 23:23:01.537858  6938 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260809 23:23:01.538487  6938 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260809 23:23:01.540844 25372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec68a6991264e72a23a8750a420795d" candidate_uuid: "808fed3ed3f04882ad757a3db9abfdd9" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b9b54436d0e94a5595cdded73fe33da9"
I20260809 23:23:01.541396 25372 raft_consensus.cc:3037] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Advancing to term 23
W20260809 23:23:01.546856 24496 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:01.552910 25372 raft_consensus.cc:2454] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 808fed3ed3f04882ad757a3db9abfdd9 in term 23.
I20260809 23:23:01.554466 24498 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9, b9b54436d0e94a5595cdded73fe33da9; no voters: ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:01.558862  6921 raft_consensus.cc:2781] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Leader election won for term 23
I20260809 23:23:01.559374  6921 raft_consensus.cc:686] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Becoming Leader. State: Replica: 808fed3ed3f04882ad757a3db9abfdd9, State: Running, Role: LEADER
I20260809 23:23:01.560575  6921 consensus_queue.cc:237] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:01.610781 24165 catalog_manager.cc:5495] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 reported cstate change: term changed from 22 to 23, leader changed from ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2) to 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3). New cstate: current_term: 23 leader_uuid: "808fed3ed3f04882ad757a3db9abfdd9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:01.785490 25323 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.818131 25323 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:01.880503 25323 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.898439 24498 consensus_peers.cc:487] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260809 23:23:01.949636 24496 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:01.950184 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080023c280 after lost signal to thread 22536
W20260809 23:23:01.953676 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080023c600 after lost signal to thread 24220
W20260809 23:23:01.955268 24574 debug-util.cc:398] Leaking SignalData structure 0x7b080031acc0 after lost signal to thread 24579
W20260809 23:23:01.960033 24574 debug-util.cc:398] Leaking SignalData structure 0x7b0800415660 after lost signal to thread 25410
W20260809 23:23:01.963610 24498 consensus_peers.cc:487] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.964488 24498 consensus_peers.cc:487] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:01.952307 25321 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:01.997169 25323 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.005687 24496 consensus_peers.cc:487] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.012259 25321 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.017490 24543 raft_consensus.cc:1261] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Refusing update from remote peer b9b54436d0e94a5595cdded73fe33da9: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260809 23:23:02.020702  6903 consensus_queue.cc:1035] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260809 23:23:02.041153 24496 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.077796 25321 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.118309 24498 consensus_peers.cc:487] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.140836 25321 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.158324 25372 raft_consensus.cc:1261] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Refusing update from remote peer 808fed3ed3f04882ad757a3db9abfdd9: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260809 23:23:02.160393  6954 consensus_queue.cc:1035] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260809 23:23:02.183859 24498 consensus_peers.cc:487] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.217566 25321 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.232977 25321 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.233074 24496 consensus_peers.cc:487] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.245766 24496 consensus_peers.cc:487] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.248039 24498 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260809 23:23:02.249629 25323 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:02.258755 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
W20260809 23:23:02.282390 24496 consensus_peers.cc:487] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.354059 24496 consensus_peers.cc:487] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.359382  6931 raft_consensus.cc:482] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.359952  6931 raft_consensus.cc:504] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } }
I20260809 23:23:02.362468  6931 leader_election.cc:288] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:02.368746 24498 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:02.379698 25323 consensus_peers.cc:487] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.381786 24498 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.383431 24496 leader_election.cc:334] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.383806 24496 leader_election.cc:302] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 808fed3ed3f04882ad757a3db9abfdd9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.384569  6931 raft_consensus.cc:2726] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260809 23:23:02.395857 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
W20260809 23:23:02.404567 24244 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:23:02.404946 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:23:02.415123 24496 consensus_peers.cc:487] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.417028 25325 consensus_peers.cc:487] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.417162 24498 consensus_peers.cc:487] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260809 23:23:02.421355 25325 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.432889 25321 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.461719 25321 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.464066 24496 consensus_peers.cc:487] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.467670 24496 consensus_peers.cc:487] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.478345 24496 consensus_peers.cc:487] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.482869 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:23:02.486171 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.487167 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.489130 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.492143 24496 consensus_peers.cc:487] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:02.493817 25321 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.495223 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.496100 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.497228 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P 808fed3ed3f04882ad757a3db9abfdd9 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.498071  6903 raft_consensus.cc:482] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.498652  6903 raft_consensus.cc:504] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.499749 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.500732 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.500854  6903 leader_election.cc:288] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.501562 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.505592 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.506458 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.506920 25323 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.507616  6924 raft_consensus.cc:482] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.508133  6924 raft_consensus.cc:504] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:02.508841 24498 consensus_peers.cc:487] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:02.510320  6924 leader_election.cc:288] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.513873 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.514623 25321 leader_election.cc:334] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.515173 25321 leader_election.cc:302] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
W20260809 23:23:02.515508 24496 consensus_peers.cc:487] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.516013  6924 raft_consensus.cc:2726] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:02.516911 22524 tablet_replica.cc:330] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.516980 25321 consensus_peers.cc:487] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.517609 22524 raft_consensus.cc:2229] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.518627 22524 raft_consensus.cc:2258] T 7be87796c0684430b07c73073572ee50 P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.520867 22524 tablet_replica.cc:330] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.521485 22524 raft_consensus.cc:2229] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.522327 22524 raft_consensus.cc:2258] T b1115ecb68bd4c399da85a680700bb7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.513881 25323 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:02.522807 25321 leader_election.cc:334] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.525238 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.525821 25325 consensus_peers.cc:487] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.525730 25321 leader_election.cc:302] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.526111 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.526906  6924 raft_consensus.cc:2726] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:02.527261 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.528868  6924 raft_consensus.cc:482] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.529390  6924 raft_consensus.cc:504] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.531260  6924 leader_election.cc:288] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.529377 22524 tablet_replica.cc:330] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.532474 22524 raft_consensus.cc:2229] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.533516 22524 raft_consensus.cc:2258] T 4d3f3d2a169145ffad7fb65adf8b7d7f P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.535673 25323 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.536342 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.537185 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.537748 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.539853 25321 leader_election.cc:334] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.540105 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.540343 25321 leader_election.cc:302] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.540879 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.541404  6903 raft_consensus.cc:2726] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:02.541581 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.544180 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.545547 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.546146 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P 808fed3ed3f04882ad757a3db9abfdd9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.548327 22524 tablet_replica.cc:330] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.548904 22524 raft_consensus.cc:2229] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.549449 22524 raft_consensus.cc:2258] T bcf691f757af44e6a7cc16228b7424e3 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.551319 22524 tablet_replica.cc:330] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.552006 22524 raft_consensus.cc:2229] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.552765 22524 raft_consensus.cc:2258] T c9b804168f1e458cae6cd72cbdb5370e P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.554922 25325 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.555051 22524 tablet_replica.cc:330] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.555778 22524 raft_consensus.cc:2229] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.556672 22524 raft_consensus.cc:2258] T 904621b566f64064bc1a7411ab27c677 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.558598 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.559332 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.559949 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.562451 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
W20260809 23:23:02.562410 25321 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.563325 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.563829 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.566565 22524 tablet_replica.cc:330] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.567341 22524 raft_consensus.cc:2229] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.568039 22524 raft_consensus.cc:2258] T eba754600fd94d2588497bb7f43a92a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.570415 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.571271 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.572266 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P 808fed3ed3f04882ad757a3db9abfdd9 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.573915 22524 tablet_replica.cc:330] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.574421 22524 raft_consensus.cc:2229] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.575316 22524 raft_consensus.cc:2258] T c6e16e58b6ff4df4bcb9bb7ca0300f4d P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.577057 22524 tablet_replica.cc:330] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.577598 22524 raft_consensus.cc:2229] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.578392 22524 raft_consensus.cc:2258] T 3913bd5adc0247eb847943b7e9a3ceb7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.580183 22524 tablet_replica.cc:330] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.580725 22524 raft_consensus.cc:2229] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.581781 22524 raft_consensus.cc:2258] T 3ed27208ddea495a9e5ab4132e38adbb P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.585114 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.586175 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.587050 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.587308 25325 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.587371  6924 raft_consensus.cc:482] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.588732  6924 raft_consensus.cc:504] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.590152 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.591303 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.592159 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.592218  6924 leader_election.cc:288] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.595479 22524 tablet_replica.cc:330] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.596372 22524 raft_consensus.cc:2229] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.597744 25323 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.597846 22524 raft_consensus.cc:2258] T 1a28a5ad332b42c2ba471779e78f3e25 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.598364 25321 leader_election.cc:334] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.598829 25321 leader_election.cc:302] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.599622  6924 raft_consensus.cc:2726] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260809 23:23:02.643556 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.652478  6903 raft_consensus.cc:482] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:02.653954  6903 raft_consensus.cc:504] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:02.658028  6903 leader_election.cc:288] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
I20260809 23:23:02.659961 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:02.661852 25325 consensus_peers.cc:487] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.663939 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.671710 25321 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:02.672771 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.674644 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 LEADER]: Raft consensus shutting down.
W20260809 23:23:02.675028 25323 leader_election.cc:334] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:02.675698 25323 leader_election.cc:302] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:02.676906  6903 raft_consensus.cc:2726] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:02.678380 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.683221 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.684638 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.688923 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.702219 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.704466 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.707968 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.713065 22524 tablet_replica.cc:330] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.714759 22524 raft_consensus.cc:2229] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.717519 22524 raft_consensus.cc:2258] T 7f974a8c0944480688bfdf1cf728da89 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.721788 22524 tablet_replica.cc:330] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.722970 22524 raft_consensus.cc:2229] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.727010 22524 raft_consensus.cc:2258] T bc822924176247c7abbc4a4756daf3a6 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.731318 22524 tablet_replica.cc:330] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.732726 22524 raft_consensus.cc:2229] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.734470 22524 raft_consensus.cc:2258] T 137c4d66939a48e59f84422216ede16f P 808fed3ed3f04882ad757a3db9abfdd9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.738582 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.739535 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.743685 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.746297 25323 consensus_peers.cc:487] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260809 23:23:02.747494 22524 tablet_replica.cc:330] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.748739 22524 raft_consensus.cc:2229] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.751199 22524 raft_consensus.cc:2258] T 827f95d8d9f946f6b673db8272386c5f P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.754685 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.755591 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.758775 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P 808fed3ed3f04882ad757a3db9abfdd9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.762682 22524 tablet_replica.cc:330] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.763784 22524 raft_consensus.cc:2229] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.766157 22524 raft_consensus.cc:2258] T ddd2e2b2ca834a8a967d719c1ce9f6b8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.766827 24496 consensus_peers.cc:487] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.769929 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.770967 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:02.771200 24496 consensus_peers.cc:487] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.772348 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P 808fed3ed3f04882ad757a3db9abfdd9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.775756 22524 tablet_replica.cc:330] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.776618 22524 raft_consensus.cc:2229] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.778648 22524 raft_consensus.cc:2258] T 07c57d179ede43aebb4ed26fdaf60001 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.783241 22524 tablet_replica.cc:330] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.784272 22524 raft_consensus.cc:2229] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.785516 22524 raft_consensus.cc:2258] T ba62e1cbbfd84b86a817f9118be39b00 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.787022 25325 consensus_peers.cc:487] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:02.789552 22524 tablet_replica.cc:330] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.790602 22524 raft_consensus.cc:2229] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.792835 22524 raft_consensus.cc:2258] T 8542be4415fd4e648a44a76fdbc3e176 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.796162 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.797034 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.799695 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P 808fed3ed3f04882ad757a3db9abfdd9 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.803699 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.804515 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.805473 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.806208 25321 consensus_peers.cc:487] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 -> Peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Couldn't send request to peer ab5aefaeb3d14775b30a74a2a6d3357f. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:02.810025 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.811159 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:02.814358 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P 808fed3ed3f04882ad757a3db9abfdd9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.817891 22524 tablet_replica.cc:330] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.818888 22524 raft_consensus.cc:2229] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.820811 22524 raft_consensus.cc:2258] T 54afb87e83f14564b7c4ec761e92304a P 808fed3ed3f04882ad757a3db9abfdd9 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.824559 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.825394 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.826297 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P 808fed3ed3f04882ad757a3db9abfdd9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.830163 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.830960 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.831774 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P 808fed3ed3f04882ad757a3db9abfdd9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:02.835052 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9: stopping tablet replica
I20260809 23:23:02.835568 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:02.836292 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P 808fed3ed3f04882ad757a3db9abfdd9 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:02.854787 25325 consensus_peers.cc:487] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.857161 25325 consensus_peers.cc:487] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.864897 25325 consensus_peers.cc:487] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.874090 25325 consensus_peers.cc:487] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.886818 25325 consensus_peers.cc:487] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Couldn't send request to peer 808fed3ed3f04882ad757a3db9abfdd9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:02.912813 25323 consensus_peers.cc:487] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:02.938782 25323 consensus_peers.cc:487] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260809 23:23:02.994361 25323 consensus_peers.cc:487] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:03.132210 25323 consensus_peers.cc:487] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260809 23:23:03.386475  6973 raft_consensus.cc:482] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:03.387143  6973 raft_consensus.cc:504] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.389140  6973 leader_election.cc:288] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.392467 25321 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.392908 25325 leader_election.cc:334] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.393323 25325 leader_election.cc:302] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.394191  6973 raft_consensus.cc:2726] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260809 23:23:03.491757 24244 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.498018 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.498311 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.503813 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.504073 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.507915 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.508167 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.512689 24244 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.512943 24244 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.522011 25323 consensus_peers.cc:487] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260809 23:23:03.555186  6973 raft_consensus.cc:482] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:03.555928  6973 raft_consensus.cc:504] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.557705  6973 leader_election.cc:288] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.558435  6973 raft_consensus.cc:482] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ab5aefaeb3d14775b30a74a2a6d3357f)
I20260809 23:23:03.559001  6973 raft_consensus.cc:504] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
W20260809 23:23:03.561254 25321 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.561653  6973 leader_election.cc:288] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099)
W20260809 23:23:03.561803 25325 leader_election.cc:334] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.562348 25325 leader_election.cc:302] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.565169  6973 raft_consensus.cc:2726] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260809 23:23:03.565759 25321 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.567560 25323 leader_election.cc:334] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.567965 25323 leader_election.cc:302] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.568823  6973 raft_consensus.cc:2726] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260809 23:23:03.614835 25321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111) [suppressed 612 similar messages]
I20260809 23:23:03.655536 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:23:03.701689  6973 raft_consensus.cc:482] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.702499  6973 raft_consensus.cc:504] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.705071  6973 leader_election.cc:288] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.709823 25321 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.710400 25325 leader_election.cc:334] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.711035 25325 leader_election.cc:302] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.712066  6972 raft_consensus.cc:2726] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260809 23:23:03.714859  6972 raft_consensus.cc:482] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.715619  6972 raft_consensus.cc:504] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.718274  6972 leader_election.cc:288] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.722141 25323 consensus_peers.cc:487] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 -> Peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Couldn't send request to peer 60d5c69728134c8bb918a900eb6a8c21. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260809 23:23:03.725029 25325 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.726444 25321 leader_election.cc:334] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.726981 25321 leader_election.cc:302] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.727857  6972 raft_consensus.cc:2726] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260809 23:23:03.728405  6973 raft_consensus.cc:482] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.729110  6973 raft_consensus.cc:504] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.731532  6973 leader_election.cc:288] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.735973 25321 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.736557 25325 leader_election.cc:334] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.736943 25325 leader_election.cc:302] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.737687  6972 raft_consensus.cc:2726] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260809 23:23:03.746222  6972 raft_consensus.cc:482] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.747133  6972 raft_consensus.cc:504] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.749418  6972 leader_election.cc:288] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
W20260809 23:23:03.752835 25323 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:03.753433 25325 leader_election.cc:334] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.753911 25325 leader_election.cc:302] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.754755  6973 raft_consensus.cc:2726] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:03.767558 22524 tablet_server.cc:178] TabletServer@127.21.255.4:0 shutting down...
I20260809 23:23:03.812103  6972 raft_consensus.cc:482] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.813555  6972 raft_consensus.cc:504] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.816099  6972 leader_election.cc:288] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765), 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681)
W20260809 23:23:03.821002 25325 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.821513 25323 leader_election.cc:334] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.821942 25323 leader_election.cc:302] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.822822  6972 raft_consensus.cc:2726] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260809 23:23:03.840520 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:23:03.841497 22524 tablet_replica.cc:330] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.842864 22524 raft_consensus.cc:2229] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.844300 22524 raft_consensus.cc:2258] T ac3c46a49aea44a2b2721eb2a750f055 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.846683  6972 raft_consensus.cc:482] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.847556 22524 tablet_replica.cc:330] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.847313  6972 raft_consensus.cc:504] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.848791 22524 raft_consensus.cc:2229] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.849376  6972 leader_election.cc:288] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.850423 22524 raft_consensus.cc:2258] T 4e2662351f0e4b14bede1be0deaba6c8 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:03.853408 25321 leader_election.cc:334] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
W20260809 23:23:03.854099 25325 leader_election.cc:334] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.854467 25325 leader_election.cc:302] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.854560 22524 tablet_replica.cc:330] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.855373  6972 raft_consensus.cc:482] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.855865 22524 raft_consensus.cc:2229] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.855957  6972 raft_consensus.cc:504] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.855834  6924 raft_consensus.cc:2726] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260809 23:23:03.857800 22524 raft_consensus.cc:2258] T d3a8a92891fb40e8911ac15230653aab P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.858179  6972 leader_election.cc:288] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.861308 22524 tablet_replica.cc:330] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
W20260809 23:23:03.861378 25325 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:03.862289 25323 leader_election.cc:334] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
I20260809 23:23:03.862330 22524 raft_consensus.cc:2229] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.862794 25323 leader_election.cc:302] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.863499  6972 raft_consensus.cc:2726] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260809 23:23:03.863798 22524 raft_consensus.cc:2258] T 8e43c87eb09044548379ab2c35cce3c1 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.866340 22524 tablet_replica.cc:330] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.866984 22524 raft_consensus.cc:2229] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.867901 22524 raft_consensus.cc:2258] T 4a4c69723b934fd5869e2f079439b2f9 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.870309 22524 tablet_replica.cc:330] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.871071 22524 raft_consensus.cc:2229] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.871683 22524 raft_consensus.cc:2258] T cadfc1bde6cb4d6997f3fb215dc184ec P b9b54436d0e94a5595cdded73fe33da9 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.874653 22524 tablet_replica.cc:330] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.874336  6972 raft_consensus.cc:482] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.875242  6972 raft_consensus.cc:504] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "60d5c69728134c8bb918a900eb6a8c21" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 36681 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.875691 22524 raft_consensus.cc:2229] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.876426 22524 raft_consensus.cc:2258] T ef1955f952024cb2815f7065f861cf25 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.877514  6972 leader_election.cc:288] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.879837 22524 tablet_replica.cc:330] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.880636 22524 raft_consensus.cc:2229] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:03.880837 25323 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:03.881567 25325 leader_election.cc:334] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.881814 22524 raft_consensus.cc:2258] T cd2b577e2c3e4021bc8ebdfa3462437d P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.882278 25325 leader_election.cc:302] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 60d5c69728134c8bb918a900eb6a8c21, 808fed3ed3f04882ad757a3db9abfdd9
I20260809 23:23:03.883231  6972 raft_consensus.cc:2726] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260809 23:23:03.884093  6972 raft_consensus.cc:482] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 808fed3ed3f04882ad757a3db9abfdd9)
I20260809 23:23:03.885387 22524 tablet_replica.cc:330] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.884683  6972 raft_consensus.cc:504] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ab5aefaeb3d14775b30a74a2a6d3357f" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42099 } } peers { permanent_uuid: "808fed3ed3f04882ad757a3db9abfdd9" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 37765 } } peers { permanent_uuid: "b9b54436d0e94a5595cdded73fe33da9" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 38713 } attrs { promote: false } }
I20260809 23:23:03.886879 22524 raft_consensus.cc:2229] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.888149  6972 leader_election.cc:288] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099), 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765)
I20260809 23:23:03.888338 22524 raft_consensus.cc:2258] T c831643e28e34208b6edf775607155e4 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.891499 22524 tablet_replica.cc:330] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
W20260809 23:23:03.891474 25321 leader_election.cc:334] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer ab5aefaeb3d14775b30a74a2a6d3357f (127.21.255.2:42099): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42099: connect: Connection refused (error 111)
I20260809 23:23:03.892321 22524 raft_consensus.cc:2229] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.893085 22524 raft_consensus.cc:2258] T 48af4d51aa8f41a486c4fe6e7541700e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:03.893620 25325 leader_election.cc:334] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:03.893991 25325 leader_election.cc:302] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9b54436d0e94a5595cdded73fe33da9; no voters: 808fed3ed3f04882ad757a3db9abfdd9, ab5aefaeb3d14775b30a74a2a6d3357f
I20260809 23:23:03.894609  6972 raft_consensus.cc:2726] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260809 23:23:03.896085 22524 tablet_replica.cc:330] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.896886 22524 raft_consensus.cc:2229] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.898033 22524 raft_consensus.cc:2258] T 65d375db793a4542808e1cff72fb6617 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.900614 22524 tablet_replica.cc:330] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.901175 22524 raft_consensus.cc:2229] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.902297 22524 raft_consensus.cc:2258] T c37e848050ec44069438f4c88a76f9ef P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.904667 22524 tablet_replica.cc:330] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.905371 22524 raft_consensus.cc:2229] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.906469 22524 raft_consensus.cc:2258] T e4ed6a400f2643da95b8f744ebab6f4c P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.908741 22524 tablet_replica.cc:330] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.909269 22524 raft_consensus.cc:2229] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.909816 22524 raft_consensus.cc:2258] T deaa6cc935614b7bbb64f5ee8faf4c74 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.911885 22524 tablet_replica.cc:330] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.912432 22524 raft_consensus.cc:2229] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.913350 22524 raft_consensus.cc:2258] T b3063532c2cd49f8b7075e771a2e818a P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.915619 22524 tablet_replica.cc:330] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.916184 22524 raft_consensus.cc:2229] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.917100 22524 raft_consensus.cc:2258] T 8039b4f7b5b04cd0a5fb1f9d6c999b5e P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.919489 22524 tablet_replica.cc:330] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.920024 22524 raft_consensus.cc:2229] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.921113 22524 raft_consensus.cc:2258] T b8a29ab049db4236a3198e8545cafb9c P b9b54436d0e94a5595cdded73fe33da9 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.923267 22524 tablet_replica.cc:330] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.923748 22524 raft_consensus.cc:2229] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.924211 22524 raft_consensus.cc:2258] T f20a6f5f014e45d3a917a197db7a0998 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.925987 22524 tablet_replica.cc:330] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.926509 22524 raft_consensus.cc:2229] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.927449 22524 raft_consensus.cc:2258] T b3538d14b2274bf99cd72e3872fda1fa P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.929385 22524 tablet_replica.cc:330] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.929978 22524 raft_consensus.cc:2229] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.930434 22524 raft_consensus.cc:2258] T 2ef03e5e7d0443d69a2228dd175bd010 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.932400 22524 tablet_replica.cc:330] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.932934 22524 raft_consensus.cc:2229] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.933436 22524 raft_consensus.cc:2258] T af1ef8fe019645f3b99232df6916cb2c P b9b54436d0e94a5595cdded73fe33da9 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.935463 22524 tablet_replica.cc:330] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.936008 22524 raft_consensus.cc:2229] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.936525 22524 raft_consensus.cc:2258] T 9dad562edaec4eebaa047293c085c0c4 P b9b54436d0e94a5595cdded73fe33da9 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.938468 22524 tablet_replica.cc:330] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.939014 22524 raft_consensus.cc:2229] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.939683 22524 raft_consensus.cc:2258] T b2c4cf897ac64cb8981b99dda67dbc92 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.941700 22524 tablet_replica.cc:330] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.942199 22524 raft_consensus.cc:2229] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.942730 22524 raft_consensus.cc:2258] T 3962e082dc274290a1f20e2fa64877a9 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.945461 22524 tablet_replica.cc:330] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.946228 22524 raft_consensus.cc:2229] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.947672 22524 raft_consensus.cc:2258] T 34970da619114abc942b839d0862a300 P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.950340 22524 tablet_replica.cc:330] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.951021 22524 raft_consensus.cc:2229] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.951655 22524 raft_consensus.cc:2258] T 3ad5d0ab888b41229481d32d207c7466 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.953715 22524 tablet_replica.cc:330] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.954239 22524 raft_consensus.cc:2229] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.955259 22524 raft_consensus.cc:2258] T f2fc12c55ab24490a1ad12ed4b4b5c35 P b9b54436d0e94a5595cdded73fe33da9 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.957158 22524 tablet_replica.cc:330] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.957700 22524 raft_consensus.cc:2229] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.958184 22524 raft_consensus.cc:2258] T 9932f35c1d8a431f9c035a848f63cb0b P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.960191 22524 tablet_replica.cc:330] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.960877 22524 raft_consensus.cc:2229] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.962131 22524 raft_consensus.cc:2258] T 5fcb08685ecd486c8f1d951336f39982 P b9b54436d0e94a5595cdded73fe33da9 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.964967 22524 tablet_replica.cc:330] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.965566 22524 raft_consensus.cc:2229] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.966723 22524 raft_consensus.cc:2258] T fb0467d784e841868c4a1188ee0af4bc P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.968778 22524 tablet_replica.cc:330] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.969347 22524 raft_consensus.cc:2229] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.969908 22524 raft_consensus.cc:2258] T cec68a6991264e72a23a8750a420795d P b9b54436d0e94a5595cdded73fe33da9 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.971910 22524 tablet_replica.cc:330] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.972414 22524 raft_consensus.cc:2229] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.973270 22524 raft_consensus.cc:2258] T a682e0d089354a44a680bd8ad9b4b789 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.975214 22524 tablet_replica.cc:330] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.975742 22524 raft_consensus.cc:2229] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.976370 22524 raft_consensus.cc:2258] T c3876fa8158d49ec9c1128fbebe6f994 P b9b54436d0e94a5595cdded73fe33da9 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.978348 22524 tablet_replica.cc:330] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.978826 22524 raft_consensus.cc:2229] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.979804 22524 raft_consensus.cc:2258] T 90dd68b3876b4b75aea408edfa68d5e5 P b9b54436d0e94a5595cdded73fe33da9 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.982896 22524 tablet_replica.cc:330] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.983464 22524 raft_consensus.cc:2229] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.984067 22524 raft_consensus.cc:2258] T d273006c869f4749852291bcee467158 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.986003 22524 tablet_replica.cc:330] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.986510 22524 raft_consensus.cc:2229] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:03.987120 22524 raft_consensus.cc:2258] T 9429f7311dd744bea4301956e143e9a7 P b9b54436d0e94a5595cdded73fe33da9 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.989279 22524 tablet_replica.cc:330] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.989765 22524 raft_consensus.cc:2229] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.990675 22524 raft_consensus.cc:2258] T c66997a772ef46a5ae07d530d26ab8a0 P b9b54436d0e94a5595cdded73fe33da9 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.993103 22524 tablet_replica.cc:330] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.993916 22524 raft_consensus.cc:2229] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.995169 22524 raft_consensus.cc:2258] T 974355bb93274291bb6be768b656cc67 P b9b54436d0e94a5595cdded73fe33da9 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:03.997691 22524 tablet_replica.cc:330] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9: stopping tablet replica
I20260809 23:23:03.998189 22524 raft_consensus.cc:2229] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 LEADER]: Raft consensus shutting down.
I20260809 23:23:03.999676 22524 raft_consensus.cc:2258] T ca74ab4e31434c79a52a994e74129545 P b9b54436d0e94a5595cdded73fe33da9 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:04.401579 22524 tablet_server.cc:194] TabletServer@127.21.255.4:0 shutdown complete.
I20260809 23:23:04.473778 22524 master.cc:537] Master@127.21.255.62:37921 shutting down...
W20260809 23:23:17.123988 24151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20260809 23:23:17.127803 24151 catalog_manager.cc:4584] TS 60d5c69728134c8bb918a900eb6a8c21 (127.21.255.1:36681): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3876fa8158d49ec9c1128fbebe6f994: Network error: Client connection negotiation failed: client connection to 127.21.255.1:36681: connect: Connection refused (error 111)
W20260809 23:23:33.968999 24152 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
W20260809 23:23:33.972368 24152 catalog_manager.cc:4584] TS 808fed3ed3f04882ad757a3db9abfdd9 (127.21.255.3:37765): ChangeConfig:REMOVE_PEER RPC failed for tablet cd2b577e2c3e4021bc8ebdfa3462437d: Network error: Client connection negotiation failed: client connection to 127.21.255.3:37765: connect: Connection refused (error 111)
I20260809 23:23:37.689265 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:37.692821 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:37.693310 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P e3357d0fe4ef46db9a4739044fa527d5: stopping tablet replica
I20260809 23:23:37.924127 22524 master.cc:552] Master@127.21.255.62:37921 shutdown complete.
I20260809 23:23:37.988503 22524 test_util.cc:175] -----------------------------------------------
I20260809 23:23:37.988709 22524 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1786317384654715-22524-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (371429 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260809 23:23:37.992539 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:37409
I20260809 23:23:37.993654 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:38.001034  6993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.001413  6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.002216  6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.003960 22524 server_base.cc:1034] running on GCE node
I20260809 23:23:38.009282 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:38.009479 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:38.009585 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317818009578 us; error 0 us; skew 500 ppm
I20260809 23:23:38.010038 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:38.017393 22524 webserver.cc:460] Webserver started at http://127.21.255.62:44969/ using document root <none> and password file <none>
I20260809 23:23:38.017884 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:38.018047 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:23:38.018285 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:23:38.020148 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "de4b75404640471e847c3177bd586290"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.024897 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260809 23:23:38.028882  7005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.029959 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260809 23:23:38.030296 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "de4b75404640471e847c3177bd586290"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.030591 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:38.059715 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:38.061017 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:38.107506 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:37409
I20260809 23:23:38.107640  7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:37409 every 8 connection(s)
I20260809 23:23:38.111688  7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:38.122869  7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290: Bootstrap starting.
I20260809 23:23:38.127566  7080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:38.132159  7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290: No bootstrap required, opened a new log
I20260809 23:23:38.134459  7080 raft_consensus.cc:348] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER }
I20260809 23:23:38.134968  7080 raft_consensus.cc:374] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:38.135157  7080 raft_consensus.cc:729] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4b75404640471e847c3177bd586290, State: Initialized, Role: FOLLOWER
I20260809 23:23:38.135653  7080 consensus_queue.cc:260] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER }
I20260809 23:23:38.136086  7080 raft_consensus.cc:388] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:23:38.136289  7080 raft_consensus.cc:482] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:23:38.136500  7080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:38.140758  7080 raft_consensus.cc:504] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER }
I20260809 23:23:38.141232  7080 leader_election.cc:302] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de4b75404640471e847c3177bd586290; no voters: 
I20260809 23:23:38.142515  7080 leader_election.cc:288] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:23:38.143291  7083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:38.144986  7083 raft_consensus.cc:686] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 1 LEADER]: Becoming Leader. State: Replica: de4b75404640471e847c3177bd586290, State: Running, Role: LEADER
I20260809 23:23:38.145666  7083 consensus_queue.cc:237] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER }
I20260809 23:23:38.145989  7080 sys_catalog.cc:564] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:23:38.153119  7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de4b75404640471e847c3177bd586290. Latest consensus state: current_term: 1 leader_uuid: "de4b75404640471e847c3177bd586290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER } }
I20260809 23:23:38.153406  7084 sys_catalog.cc:455] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de4b75404640471e847c3177bd586290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de4b75404640471e847c3177bd586290" member_type: VOTER } }
I20260809 23:23:38.154078  7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [sys.catalog]: This master's current role is: LEADER
I20260809 23:23:38.154390  7084 sys_catalog.cc:458] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [sys.catalog]: This master's current role is: LEADER
I20260809 23:23:38.156288  7095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:23:38.163115  7095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:23:38.164335 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:23:38.172386  7095 catalog_manager.cc:1347] Generated new cluster ID: b776419b6ad34288848c21b205adee5b
I20260809 23:23:38.172683  7095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:23:38.186069  7095 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:23:38.187803  7095 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:23:38.201262  7095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290: Generated new TSK 0
I20260809 23:23:38.201903  7095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:23:38.231663 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:38.239130  7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.240698  7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.242293  7113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.245273 22524 server_base.cc:1034] running on GCE node
I20260809 23:23:38.249141 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:38.249336 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:38.249521 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317818249507 us; error 0 us; skew 500 ppm
I20260809 23:23:38.250142 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:38.252658 22524 webserver.cc:460] Webserver started at http://127.21.255.1:46231/ using document root <none> and password file <none>
I20260809 23:23:38.253150 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:38.253587 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:23:38.253973 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:23:38.255755 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.262091 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.001s	sys 0.004s
I20260809 23:23:38.265841  7119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.266808 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260809 23:23:38.267208 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.267614 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:38.280665 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:38.281824 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:38.283520 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:23:38.285949 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:23:38.286123 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.286413 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:23:38.286567 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.342201 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:37549
I20260809 23:23:38.342451  7207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:37549 every 8 connection(s)
I20260809 23:23:38.348699 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:38.357322  7215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.359670  7216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.362021  7218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.364758 22524 server_base.cc:1034] running on GCE node
I20260809 23:23:38.368530  7210 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37409
I20260809 23:23:38.369076  7210 heartbeater.cc:464] Registering TS with master...
I20260809 23:23:38.369758 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:38.370029 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:38.370208 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317818370190 us; error 0 us; skew 500 ppm
I20260809 23:23:38.370357  7210 heartbeater.cc:511] Master 127.21.255.62:37409 requested a full tablet report, sending...
I20260809 23:23:38.370850 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:38.374330 22524 webserver.cc:460] Webserver started at http://127.21.255.2:37759/ using document root <none> and password file <none>
I20260809 23:23:38.374215  7030 ts_manager.cc:194] Registered new tserver with Master: ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:38.375240 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:38.375583 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:23:38.375962 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:23:38.377344  7030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59448
I20260809 23:23:38.377473 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "ec555693f39b444f95a05ba0bd6b37e7"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.382831 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260809 23:23:38.386518  7225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.387413 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:23:38.387717 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "ec555693f39b444f95a05ba0bd6b37e7"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.387990 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:38.405047 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:38.406373 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:38.407975 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:23:38.410264 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:23:38.410490 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.410745 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:23:38.410924 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.466712 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:42321
I20260809 23:23:38.466838  7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:42321 every 8 connection(s)
I20260809 23:23:38.472537 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:38.481495  7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.482333  7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.486820 22524 server_base.cc:1034] running on GCE node
W20260809 23:23:38.486888  7327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.488474 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:38.488727 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:38.488937 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317818488921 us; error 0 us; skew 500 ppm
I20260809 23:23:38.489744 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:38.492692  7312 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37409
I20260809 23:23:38.493026 22524 webserver.cc:460] Webserver started at http://127.21.255.3:43193/ using document root <none> and password file <none>
I20260809 23:23:38.493216  7312 heartbeater.cc:464] Registering TS with master...
I20260809 23:23:38.493698 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:38.493978 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:23:38.494133  7312 heartbeater.cc:511] Master 127.21.255.62:37409 requested a full tablet report, sending...
I20260809 23:23:38.494328 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:23:38.496454 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "081930af26fd4f7fab29988df3e8b9aa"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.497217  7030 ts_manager.cc:194] Registered new tserver with Master: ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:38.499399  7030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59464
I20260809 23:23:38.502713 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.003s
I20260809 23:23:38.506999  7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.508095 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:23:38.508647 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "081930af26fd4f7fab29988df3e8b9aa"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.508967 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:38.526784 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:38.528246 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:38.529512 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:23:38.532395 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:23:38.532833 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.533105 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:23:38.533272 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.589483 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:46099
I20260809 23:23:38.589825  7414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:46099 every 8 connection(s)
I20260809 23:23:38.595819 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:38.606823  7423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:38.609162  7421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.610877 22524 server_base.cc:1034] running on GCE node
W20260809 23:23:38.611975  7425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:38.613245 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:38.613562 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:38.613821 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317818613801 us; error 0 us; skew 500 ppm
I20260809 23:23:38.614449 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:38.615525  7415 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37409
I20260809 23:23:38.615972  7415 heartbeater.cc:464] Registering TS with master...
I20260809 23:23:38.616886  7415 heartbeater.cc:511] Master 127.21.255.62:37409 requested a full tablet report, sending...
I20260809 23:23:38.617879 22524 webserver.cc:460] Webserver started at http://127.21.255.4:35753/ using document root <none> and password file <none>
I20260809 23:23:38.618525 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:38.618897 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:23:38.619499 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:23:38.619622  7030 ts_manager.cc:194] Registered new tserver with Master: 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:38.621620 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-3-root/instance:
uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.622393  7030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59470
I20260809 23:23:38.627556 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20260809 23:23:38.631780  7434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.632824 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:23:38.633147 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:38.633443 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:38.649361 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:38.650800 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:38.652848 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:23:38.655174 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:23:38.655481 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.655707 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:23:38.655874 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:38.710223 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.4:42323
I20260809 23:23:38.710526  7525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.4:42323 every 8 connection(s)
I20260809 23:23:38.728544  7526 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37409
I20260809 23:23:38.729014  7526 heartbeater.cc:464] Registering TS with master...
I20260809 23:23:38.729730  7526 heartbeater.cc:511] Master 127.21.255.62:37409 requested a full tablet report, sending...
I20260809 23:23:38.731820  7030 ts_manager.cc:194] Registered new tserver with Master: a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:38.732082 22524 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.018446053s
I20260809 23:23:38.733915  7030 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59482
I20260809 23:23:38.735884 22524 test_util.cc:268] Using random seed: -2029343495
I20260809 23:23:38.766847  7030 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59486:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260809 23:23:39.045315  7162 tablet_service.cc:1467] Processing CreateTablet for tablet 73788985062f47c397a6b02115c97a1e (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:23:39.047161  7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73788985062f47c397a6b02115c97a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.047477  7160 tablet_service.cc:1467] Processing CreateTablet for tablet 4468dee129594149bc011deeaef72185 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:23:39.048906  7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4468dee129594149bc011deeaef72185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.052179  7159 tablet_service.cc:1467] Processing CreateTablet for tablet 02f4e2e6c1834f09905dc17072c08a53 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:23:39.053043  7155 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:23:39.053683  7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f4e2e6c1834f09905dc17072c08a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.054446  7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.055974  7158 tablet_service.cc:1467] Processing CreateTablet for tablet d542b5134ed04d51b8f1e946e99c69a5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:23:39.056486  7151 tablet_service.cc:1467] Processing CreateTablet for tablet ec734a4c9690454f8c3986ae773f6e9f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:23:39.057449  7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d542b5134ed04d51b8f1e946e99c69a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.060068  7150 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7f029122e441ad9d3194829e439141 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:23:39.060783  7157 tablet_service.cc:1467] Processing CreateTablet for tablet bf9f1e8974a54b8b8c1d3a790e28a058 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:23:39.062443  7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7f029122e441ad9d3194829e439141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.045504  7161 tablet_service.cc:1467] Processing CreateTablet for tablet 0400132440374a1c88fa8fb7d7c5a6b4 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:23:39.067307  7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec734a4c9690454f8c3986ae773f6e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.069415  7148 tablet_service.cc:1467] Processing CreateTablet for tablet 03a1d03c208b4696910c2b7dfe1469ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 23:23:39.069418  7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0400132440374a1c88fa8fb7d7c5a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.073354  7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9f1e8974a54b8b8c1d3a790e28a058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.073765  7265 tablet_service.cc:1467] Processing CreateTablet for tablet 0400132440374a1c88fa8fb7d7c5a6b4 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:23:39.074899  7262 tablet_service.cc:1467] Processing CreateTablet for tablet 1632dcfee20a4fc0a39347412a4dcc42 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:23:39.076402  7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1632dcfee20a4fc0a39347412a4dcc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.076192  7259 tablet_service.cc:1467] Processing CreateTablet for tablet 02f4e2e6c1834f09905dc17072c08a53 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:23:39.077962  7258 tablet_service.cc:1467] Processing CreateTablet for tablet d542b5134ed04d51b8f1e946e99c69a5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:23:39.080439  7257 tablet_service.cc:1467] Processing CreateTablet for tablet ce32c0f77b0c4572a2e2bf042e987cc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:23:39.084208  7254 tablet_service.cc:1467] Processing CreateTablet for tablet e279157f9a0a4fb696c9bc01ae212b96 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:23:39.086768  7253 tablet_service.cc:1467] Processing CreateTablet for tablet 03a1d03c208b4696910c2b7dfe1469ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 23:23:39.088380  7259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f4e2e6c1834f09905dc17072c08a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.088792  7255 tablet_service.cc:1467] Processing CreateTablet for tablet ec734a4c9690454f8c3986ae773f6e9f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:23:39.090173  7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec734a4c9690454f8c3986ae773f6e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.091332  7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0400132440374a1c88fa8fb7d7c5a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.089774  7256 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:23:39.092413  7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.092199  7264 tablet_service.cc:1467] Processing CreateTablet for tablet 4468dee129594149bc011deeaef72185 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:23:39.093568  7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4468dee129594149bc011deeaef72185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.098636  7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32c0f77b0c4572a2e2bf042e987cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.103376  7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e279157f9a0a4fb696c9bc01ae212b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.106527  7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a1d03c208b4696910c2b7dfe1469ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.110209  7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d542b5134ed04d51b8f1e946e99c69a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.123442  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a1d03c208b4696910c2b7dfe1469ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:39.159364  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:39.171936  7474 tablet_service.cc:1467] Processing CreateTablet for tablet 73788985062f47c397a6b02115c97a1e (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:23:39.173471  7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73788985062f47c397a6b02115c97a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.179186  7473 tablet_service.cc:1467] Processing CreateTablet for tablet 4468dee129594149bc011deeaef72185 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260809 23:23:39.181329  7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4468dee129594149bc011deeaef72185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.184590  7472 tablet_service.cc:1467] Processing CreateTablet for tablet 1632dcfee20a4fc0a39347412a4dcc42 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:23:39.186096  7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1632dcfee20a4fc0a39347412a4dcc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.186365  7466 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7f029122e441ad9d3194829e439141 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:23:39.187744  7466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7f029122e441ad9d3194829e439141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.188138  7471 tablet_service.cc:1467] Processing CreateTablet for tablet d542b5134ed04d51b8f1e946e99c69a5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260809 23:23:39.189540  7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d542b5134ed04d51b8f1e946e99c69a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.192165  7470 tablet_service.cc:1467] Processing CreateTablet for tablet bf9f1e8974a54b8b8c1d3a790e28a058 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:23:39.193624  7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9f1e8974a54b8b8c1d3a790e28a058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.196689  7469 tablet_service.cc:1467] Processing CreateTablet for tablet ce32c0f77b0c4572a2e2bf042e987cc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:23:39.197068  7464 tablet_service.cc:1467] Processing CreateTablet for tablet e279157f9a0a4fb696c9bc01ae212b96 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:23:39.198194  7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32c0f77b0c4572a2e2bf042e987cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.199875  7463 tablet_service.cc:1467] Processing CreateTablet for tablet 1f13e532e7304ece940fbe319522775f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:23:39.200174  7468 tablet_service.cc:1467] Processing CreateTablet for tablet ec734a4c9690454f8c3986ae773f6e9f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260809 23:23:39.201334  7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f13e532e7304ece940fbe319522775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.207336  7468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec734a4c9690454f8c3986ae773f6e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.210913  7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e279157f9a0a4fb696c9bc01ae212b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.216810  7370 tablet_service.cc:1467] Processing CreateTablet for tablet 73788985062f47c397a6b02115c97a1e (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260809 23:23:39.219084  7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73788985062f47c397a6b02115c97a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.231869  7366 tablet_service.cc:1467] Processing CreateTablet for tablet 1632dcfee20a4fc0a39347412a4dcc42 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260809 23:23:39.233456  7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1632dcfee20a4fc0a39347412a4dcc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.231869  7367 tablet_service.cc:1467] Processing CreateTablet for tablet 0400132440374a1c88fa8fb7d7c5a6b4 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260809 23:23:39.236195  7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0400132440374a1c88fa8fb7d7c5a6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.245011  7365 tablet_service.cc:1467] Processing CreateTablet for tablet 02f4e2e6c1834f09905dc17072c08a53 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260809 23:23:39.247380  7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f4e2e6c1834f09905dc17072c08a53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.247922  7264 tablet_service.cc:1467] Processing CreateTablet for tablet 1f13e532e7304ece940fbe319522775f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:23:39.249538  7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f13e532e7304ece940fbe319522775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.263010  7256 tablet_service.cc:1467] Processing CreateTablet for tablet f9f47812f9fe462abdd14952f3aeb72b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:23:39.264484  7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f47812f9fe462abdd14952f3aeb72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.264940  7364 tablet_service.cc:1467] Processing CreateTablet for tablet bf9f1e8974a54b8b8c1d3a790e28a058 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260809 23:23:39.266331  7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf9f1e8974a54b8b8c1d3a790e28a058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.268194  7363 tablet_service.cc:1467] Processing CreateTablet for tablet ce32c0f77b0c4572a2e2bf042e987cc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260809 23:23:39.269660  7363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce32c0f77b0c4572a2e2bf042e987cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.271557  7558 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:39.272127  7362 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260809 23:23:39.274452  7361 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7f029122e441ad9d3194829e439141 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260809 23:23:39.276748  7360 tablet_service.cc:1467] Processing CreateTablet for tablet e279157f9a0a4fb696c9bc01ae212b96 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260809 23:23:39.279166  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 03a1d03c208b4696910c2b7dfe1469ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260809 23:23:39.285593  7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6db5f7d4fa4aa3973cc96d9c5fbdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.287168  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a1d03c208b4696910c2b7dfe1469ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.289882  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7f029122e441ad9d3194829e439141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.295317  7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e279157f9a0a4fb696c9bc01ae212b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.297210  7150 tablet_service.cc:1467] Processing CreateTablet for tablet 1f13e532e7304ece940fbe319522775f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260809 23:23:39.298652  7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f13e532e7304ece940fbe319522775f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.308084  7155 tablet_service.cc:1467] Processing CreateTablet for tablet 615321f1b8064c90892728123be05c59 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:23:39.312101  7159 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb27ac1684d2bac75f40ba93712ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:23:39.346390  7253 tablet_service.cc:1467] Processing CreateTablet for tablet 7102c5cad94641438ac6c66721ce193c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:23:39.348281  7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102c5cad94641438ac6c66721ce193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.348778  7559 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:39.359261  7160 tablet_service.cc:1467] Processing CreateTablet for tablet 7102c5cad94641438ac6c66721ce193c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:23:39.360009  7148 tablet_service.cc:1467] Processing CreateTablet for tablet 10b54da326434a3398b667f5d2955e1c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:23:39.362574  7257 tablet_service.cc:1467] Processing CreateTablet for tablet 10b54da326434a3398b667f5d2955e1c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:23:39.364020  7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b54da326434a3398b667f5d2955e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.365203  7161 tablet_service.cc:1467] Processing CreateTablet for tablet a9505b9c3f0342279113f757c317a868 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:23:39.313722  7558 tablet_bootstrap.cc:654] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:39.321900  7157 tablet_service.cc:1467] Processing CreateTablet for tablet c18336f4a3644c798d5f8bb715908fd0 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:23:39.325974  7162 tablet_service.cc:1467] Processing CreateTablet for tablet 0590a10d16ff4d12aa9edfc6985ebede (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:23:39.337939  7254 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb27ac1684d2bac75f40ba93712ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:23:39.340131  7262 tablet_service.cc:1467] Processing CreateTablet for tablet c18336f4a3644c798d5f8bb715908fd0 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:23:39.378125  7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18336f4a3644c798d5f8bb715908fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.341143  7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615321f1b8064c90892728123be05c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.342625  7255 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1578784c042f68a1e8d865c5583aa (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:23:39.380848  7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1578784c042f68a1e8d865c5583aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.381820  7210 heartbeater.cc:503] Master 127.21.255.62:37409 was elected leader, sending a full tablet report...
I20260809 23:23:39.386150  7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18336f4a3644c798d5f8bb715908fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.386873  7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0590a10d16ff4d12aa9edfc6985ebede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.397704  7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb27ac1684d2bac75f40ba93712ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.399310  7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102c5cad94641438ac6c66721ce193c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.399612  7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb27ac1684d2bac75f40ba93712ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.403797  7472 tablet_service.cc:1467] Processing CreateTablet for tablet c18336f4a3644c798d5f8bb715908fd0 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260809 23:23:39.405176  7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18336f4a3644c798d5f8bb715908fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.413194  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b54da326434a3398b667f5d2955e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.416172  7265 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7beeffe6914835b84417315f44dd34 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:23:39.417558  7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7beeffe6914835b84417315f44dd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.366448  7466 tablet_service.cc:1467] Processing CreateTablet for tablet f9f47812f9fe462abdd14952f3aeb72b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:23:39.421814  7466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f47812f9fe462abdd14952f3aeb72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.421875  7474 tablet_service.cc:1467] Processing CreateTablet for tablet 615321f1b8064c90892728123be05c59 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:23:39.422394  7473 tablet_service.cc:1467] Processing CreateTablet for tablet 0590a10d16ff4d12aa9edfc6985ebede (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:23:39.423271  7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615321f1b8064c90892728123be05c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.424662  7463 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1578784c042f68a1e8d865c5583aa (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:23:39.425889  7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1578784c042f68a1e8d865c5583aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.432297  7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9505b9c3f0342279113f757c317a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.439283  7151 tablet_service.cc:1467] Processing CreateTablet for tablet 327e1d8704e64831b8ebda9aaf3710b5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:23:39.448606  7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327e1d8704e64831b8ebda9aaf3710b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.450174  7469 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7beeffe6914835b84417315f44dd34 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:23:39.438716  7365 tablet_service.cc:1467] Processing CreateTablet for tablet f9f47812f9fe462abdd14952f3aeb72b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260809 23:23:39.451709  7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9f47812f9fe462abdd14952f3aeb72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.451757  7363 tablet_service.cc:1467] Processing CreateTablet for tablet 615321f1b8064c90892728123be05c59 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260809 23:23:39.452596  7464 tablet_service.cc:1467] Processing CreateTablet for tablet bb2091b41c5341ebb239c60326eb8a22 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:23:39.453224  7363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 615321f1b8064c90892728123be05c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.443692  7158 tablet_service.cc:1467] Processing CreateTablet for tablet bb2091b41c5341ebb239c60326eb8a22 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:23:39.455525  7471 tablet_service.cc:1467] Processing CreateTablet for tablet c0e3fe783b214e9ba0376a444a7306ce (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:23:39.456171  7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2091b41c5341ebb239c60326eb8a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.456653  7370 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1578784c042f68a1e8d865c5583aa (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260809 23:23:39.456784  7366 tablet_service.cc:1467] Processing CreateTablet for tablet 0590a10d16ff4d12aa9edfc6985ebede (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260809 23:23:39.457931  7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1578784c042f68a1e8d865c5583aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.460724  7561 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:39.442855  7470 tablet_service.cc:1467] Processing CreateTablet for tablet 10b54da326434a3398b667f5d2955e1c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260809 23:23:39.465453  7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0590a10d16ff4d12aa9edfc6985ebede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.438454  7259 tablet_service.cc:1467] Processing CreateTablet for tablet 28c7eaaf2fea43f7aae67f0524f08d94 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 23:23:39.466840  7259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c7eaaf2fea43f7aae67f0524f08d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.468112  7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10b54da326434a3398b667f5d2955e1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.475272  7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7beeffe6914835b84417315f44dd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.475755  7258 tablet_service.cc:1467] Processing CreateTablet for tablet 327e1d8704e64831b8ebda9aaf3710b5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:23:39.477155  7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327e1d8704e64831b8ebda9aaf3710b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.479624  7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2091b41c5341ebb239c60326eb8a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.440659  7566 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:39.454715  7364 tablet_service.cc:1467] Processing CreateTablet for tablet cffdb27ac1684d2bac75f40ba93712ab (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260809 23:23:39.491169  7559 tablet_bootstrap.cc:654] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:39.493309  7566 tablet_bootstrap.cc:654] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:39.496044  7558 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:39.496610  7558 ts_tablet_manager.cc:1397] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.225s	user 0.008s	sys 0.010s
I20260809 23:23:39.500964  7468 tablet_service.cc:1467] Processing CreateTablet for tablet a9505b9c3f0342279113f757c317a868 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:23:39.504880  7312 heartbeater.cc:503] Master 127.21.255.62:37409 was elected leader, sending a full tablet report...
I20260809 23:23:39.506450  7148 tablet_service.cc:1467] Processing CreateTablet for tablet c0e3fe783b214e9ba0376a444a7306ce (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:23:39.511499  7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e3fe783b214e9ba0376a444a7306ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.522982  7367 tablet_service.cc:1467] Processing CreateTablet for tablet 7102c5cad94641438ac6c66721ce193c (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260809 23:23:39.528436  7468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9505b9c3f0342279113f757c317a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.534039  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e3fe783b214e9ba0376a444a7306ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.543160  7558 raft_consensus.cc:348] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:39.543967  7558 raft_consensus.cc:374] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:39.544275  7558 raft_consensus.cc:729] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:39.545022  7558 consensus_queue.cc:260] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:39.527464  7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0590a10d16ff4d12aa9edfc6985ebede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.553848  7561 tablet_bootstrap.cc:654] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:39.563009  7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cffdb27ac1684d2bac75f40ba93712ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.590692  7558 ts_tablet_manager.cc:1428] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.094s	user 0.007s	sys 0.011s
I20260809 23:23:39.590996  7256 tablet_service.cc:1467] Processing CreateTablet for tablet bb2091b41c5341ebb239c60326eb8a22 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260809 23:23:39.591784  7558 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:39.592342  7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb2091b41c5341ebb239c60326eb8a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.600045  7558 tablet_bootstrap.cc:654] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:39.603508  7155 tablet_service.cc:1467] Processing CreateTablet for tablet 356a2f292a694ebbab1424f68bfe3dc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:23:39.603756  7361 tablet_service.cc:1467] Processing CreateTablet for tablet a9505b9c3f0342279113f757c317a868 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260809 23:23:39.585640  7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7102c5cad94641438ac6c66721ce193c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:39.568104  7009 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 54803.6us
I20260809 23:23:39.631309  7415 heartbeater.cc:503] Master 127.21.255.62:37409 was elected leader, sending a full tablet report...
I20260809 23:23:39.586410  7466 tablet_service.cc:1467] Processing CreateTablet for tablet 64ff74f37a38496b8bf542b0df4d74f7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:23:39.631639  7362 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7beeffe6914835b84417315f44dd34 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260809 23:23:39.633437  7255 tablet_service.cc:1467] Processing CreateTablet for tablet 65b04e56cb2d4321a0b5e6317136d6f2 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:23:39.636871  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 327e1d8704e64831b8ebda9aaf3710b5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260809 23:23:39.643754  7365 tablet_service.cc:1467] Processing CreateTablet for tablet c0e3fe783b214e9ba0376a444a7306ce (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260809 23:23:39.648599  7360 tablet_service.cc:1467] Processing CreateTablet for tablet 64ff74f37a38496b8bf542b0df4d74f7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:23:39.577728  7150 tablet_service.cc:1467] Processing CreateTablet for tablet 65b04e56cb2d4321a0b5e6317136d6f2 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:23:39.670442  7572 raft_consensus.cc:482] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:39.671991  7466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ff74f37a38496b8bf542b0df4d74f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.661864  7253 tablet_service.cc:1467] Processing CreateTablet for tablet 356a2f292a694ebbab1424f68bfe3dc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:23:39.685945  7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b04e56cb2d4321a0b5e6317136d6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.671396  7572 raft_consensus.cc:504] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:39.659555  7264 tablet_service.cc:1467] Processing CreateTablet for tablet 64ff74f37a38496b8bf542b0df4d74f7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260809 23:23:39.661794  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9505b9c3f0342279113f757c317a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.695892  7254 tablet_service.cc:1467] Processing CreateTablet for tablet c9c4f8d91c504cdd8f1fd9dd8f707602 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 23:23:39.696367  7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7beeffe6914835b84417315f44dd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.690208  7566 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:39.684614  7157 tablet_service.cc:1467] Processing CreateTablet for tablet edbd59811ea94e13b359f7c9db694683 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:23:39.697291  7566 ts_tablet_manager.cc:1397] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.257s	user 0.006s	sys 0.011s
I20260809 23:23:39.661993  7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356a2f292a694ebbab1424f68bfe3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.698478  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 327e1d8704e64831b8ebda9aaf3710b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.698729  7463 tablet_service.cc:1467] Processing CreateTablet for tablet 356a2f292a694ebbab1424f68bfe3dc5 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260809 23:23:39.700021  7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356a2f292a694ebbab1424f68bfe3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.720549  7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ff74f37a38496b8bf542b0df4d74f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.724215  7464 tablet_service.cc:1467] Processing CreateTablet for tablet c9c4f8d91c504cdd8f1fd9dd8f707602 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260809 23:23:39.725548  7317 tablet.cc:2369] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 23:23:39.726203  7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c4f8d91c504cdd8f1fd9dd8f707602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.673345  7472 tablet_service.cc:1467] Processing CreateTablet for tablet edbd59811ea94e13b359f7c9db694683 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:23:39.704373  7262 tablet_service.cc:1467] Processing CreateTablet for tablet 3c057787ab114bba890199f5b36b37b7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:23:39.728828  7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbd59811ea94e13b359f7c9db694683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.725697  7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b04e56cb2d4321a0b5e6317136d6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.733179  7559 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:39.734231  7559 ts_tablet_manager.cc:1397] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.386s	user 0.012s	sys 0.032s
I20260809 23:23:39.738255  7526 heartbeater.cc:503] Master 127.21.255.62:37409 was elected leader, sending a full tablet report...
I20260809 23:23:39.745100  7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 356a2f292a694ebbab1424f68bfe3dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.745854  7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ff74f37a38496b8bf542b0df4d74f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.747622  7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c057787ab114bba890199f5b36b37b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.749507  7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c4f8d91c504cdd8f1fd9dd8f707602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.750387  7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0e3fe783b214e9ba0376a444a7306ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.753196  7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbd59811ea94e13b359f7c9db694683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.755646  7558 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:39.756245  7558 ts_tablet_manager.cc:1397] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.165s	user 0.018s	sys 0.010s
I20260809 23:23:39.768716  7561 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:39.769387  7561 ts_tablet_manager.cc:1397] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.309s	user 0.015s	sys 0.011s
I20260809 23:23:39.731261  7151 tablet_service.cc:1467] Processing CreateTablet for tablet 3066e3b731424266bfd6155e1dddc90b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:23:39.779058  7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066e3b731424266bfd6155e1dddc90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.815202  7159 tablet_service.cc:1467] Processing CreateTablet for tablet 3c057787ab114bba890199f5b36b37b7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:23:39.832410  7572 leader_election.cc:288] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:39.837292  7561 raft_consensus.cc:348] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:39.935371  7561 raft_consensus.cc:374] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:39.935858  7561 raft_consensus.cc:729] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:39.937359  7561 consensus_queue.cc:260] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:39.839964  7566 raft_consensus.cc:348] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:39.939194  7566 raft_consensus.cc:374] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:39.939690  7566 raft_consensus.cc:729] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:39.940202  7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c057787ab114bba890199f5b36b37b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.843148  7473 tablet_service.cc:1467] Processing CreateTablet for tablet 28c7eaaf2fea43f7aae67f0524f08d94 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 23:23:39.845542  7558 raft_consensus.cc:348] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:39.942056  7558 raft_consensus.cc:374] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:39.942443  7558 raft_consensus.cc:729] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:39.943044  7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c7eaaf2fea43f7aae67f0524f08d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.943396  7558 consensus_queue.cc:260] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:23:39.846647  7009 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 126138us
I20260809 23:23:39.847143  7258 tablet_service.cc:1467] Processing CreateTablet for tablet d87722cb5bdb4aa797b10938f4e1a640 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:23:39.862840  7161 tablet_service.cc:1467] Processing CreateTablet for tablet 4f841b906f634c05afe5a33637229c8f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:23:39.865625  7162 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa78046012472d8e5472478adbe1b9 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:23:39.866518  7158 tablet_service.cc:1467] Processing CreateTablet for tablet 28c7eaaf2fea43f7aae67f0524f08d94 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260809 23:23:39.874981  7366 tablet_service.cc:1467] Processing CreateTablet for tablet 65b04e56cb2d4321a0b5e6317136d6f2 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260809 23:23:39.948045  7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f841b906f634c05afe5a33637229c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.948359  7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b04e56cb2d4321a0b5e6317136d6f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.877465  7468 tablet_service.cc:1467] Processing CreateTablet for tablet d87722cb5bdb4aa797b10938f4e1a640 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:23:39.950865  7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87722cb5bdb4aa797b10938f4e1a640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.951112  7468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87722cb5bdb4aa797b10938f4e1a640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.953676  7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa78046012472d8e5472478adbe1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.956007  7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c7eaaf2fea43f7aae67f0524f08d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:39.921665  7470 tablet_service.cc:1467] Processing CreateTablet for tablet 4f841b906f634c05afe5a33637229c8f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:23:39.940747  7566 consensus_queue.cc:260] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:39.995734  7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f841b906f634c05afe5a33637229c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.075126  7559 raft_consensus.cc:348] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:40.075978  7559 raft_consensus.cc:374] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:40.076264  7559 raft_consensus.cc:729] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:40.076994  7559 consensus_queue.cc:260] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
W20260809 23:23:40.083889  7257 consensus_meta.cc:287] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: Time spent flushing consensus metadata: real 0.531s	user 0.033s	sys 0.015s
I20260809 23:23:40.086659  7257 tablet_service.cc:1467] Processing CreateTablet for tablet 8ded669d4bcc4a7a81c44ad0b578c899 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 23:23:40.100970  7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ded669d4bcc4a7a81c44ad0b578c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.101248  7561 ts_tablet_manager.cc:1428] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.331s	user 0.008s	sys 0.025s
I20260809 23:23:40.102751  7561 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:40.116747  7558 ts_tablet_manager.cc:1428] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.360s	user 0.009s	sys 0.011s
I20260809 23:23:40.118237  7558 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:40.120277  7464 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa78046012472d8e5472478adbe1b9 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:23:40.121618  7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa78046012472d8e5472478adbe1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.130724  7474 tablet_service.cc:1467] Processing CreateTablet for tablet b3e08fe5d08748ba98111cd243575917 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:23:40.135665  7256 tablet_service.cc:1467] Processing CreateTablet for tablet 3066e3b731424266bfd6155e1dddc90b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:23:40.137003  7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066e3b731424266bfd6155e1dddc90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.144512  7566 ts_tablet_manager.cc:1428] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.447s	user 0.006s	sys 0.034s
I20260809 23:23:40.146035  7566 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:40.132129  7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e08fe5d08748ba98111cd243575917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.158830  7148 tablet_service.cc:1467] Processing CreateTablet for tablet b3e08fe5d08748ba98111cd243575917 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:23:40.160310  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e08fe5d08748ba98111cd243575917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.165928  7361 tablet_service.cc:1467] Processing CreateTablet for tablet edbd59811ea94e13b359f7c9db694683 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260809 23:23:40.167384  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbd59811ea94e13b359f7c9db694683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.178390  7561 tablet_bootstrap.cc:654] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:40.188570  7559 ts_tablet_manager.cc:1428] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.454s	user 0.043s	sys 0.060s
I20260809 23:23:40.189754  7559 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:40.200033  7558 tablet_bootstrap.cc:654] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:40.203898  7148 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5b4438b1d84d7198189995b2d2d409 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:23:40.209035  7463 tablet_service.cc:1467] Processing CreateTablet for tablet e89212541eac4b91975dbfbe73e23c3a (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:23:40.210253  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5b4438b1d84d7198189995b2d2d409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.210592  7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89212541eac4b91975dbfbe73e23c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.212433  7265 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 17) took 1066 ms. Trace:
I20260809 23:23:40.212848  7265 rpcz_store.cc:276] 0809 23:23:39.146301 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.415314 (+269013us) service_pool.cc:227] Handling call
0809 23:23:39.417101 (+  1787us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.417203 (+   102us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:39.657469 (+240266us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.174873 (+517404us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.212366 (+ 37493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.224928  7265 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5b4438b1d84d7198189995b2d2d409 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:23:40.227221  7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5b4438b1d84d7198189995b2d2d409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.227993  7255 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa78046012472d8e5472478adbe1b9 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260809 23:23:40.229614  7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa78046012472d8e5472478adbe1b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.239758  7160 consensus_meta.cc:287] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent flushing consensus metadata: real 0.511s	user 0.004s	sys 0.016s
I20260809 23:23:40.239383  7472 tablet_service.cc:1467] Processing CreateTablet for tablet 8ded669d4bcc4a7a81c44ad0b578c899 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
W20260809 23:23:40.246564  7009 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 296499us
W20260809 23:23:40.248132  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:40.248987  7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ded669d4bcc4a7a81c44ad0b578c899. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.246476  7008 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 530639us
I20260809 23:23:40.251176  7364 tablet_service.cc:1467] Processing CreateTablet for tablet c9c4f8d91c504cdd8f1fd9dd8f707602 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260809 23:23:40.262209  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 3c057787ab114bba890199f5b36b37b7 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260809 23:23:40.264608  7160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 15) took 1179 ms. Trace:
I20260809 23:23:40.272387  7160 rpcz_store.cc:276] 0809 23:23:39.084775 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.332191 (+247416us) service_pool.cc:227] Handling call
0809 23:23:39.360342 (+ 28151us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.360440 (+    98us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:39.726270 (+365830us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.264078 (+537808us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.264546 (+   468us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13898}
I20260809 23:23:40.285465  7160 tablet_service.cc:1467] Processing CreateTablet for tablet e89212541eac4b91975dbfbe73e23c3a (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:23:40.291989  7365 tablet_service.cc:1467] Processing CreateTablet for tablet d87722cb5bdb4aa797b10938f4e1a640 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260809 23:23:40.293578  7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d87722cb5bdb4aa797b10938f4e1a640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.297720  7469 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 17) took 1092 ms. Trace:
I20260809 23:23:40.303050  7469 rpcz_store.cc:276] 0809 23:23:39.205710 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.449301 (+243591us) service_pool.cc:227] Handling call
0809 23:23:39.451048 (+  1747us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.451138 (+    90us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.134226 (+683088us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.297174 (+162948us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.297659 (+   485us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.306159  7361 tablet_service.cc:1467] Processing CreateTablet for tablet 3066e3b731424266bfd6155e1dddc90b (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260809 23:23:40.308476  7155 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 22) took 1129 ms. Trace:
I20260809 23:23:40.311906  7257 tablet_service.cc:1467] Processing CreateTablet for tablet e89212541eac4b91975dbfbe73e23c3a (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260809 23:23:40.312767  7558 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:40.313387  7558 ts_tablet_manager.cc:1397] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.195s	user 0.016s	sys 0.007s
I20260809 23:23:40.313681  7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89212541eac4b91975dbfbe73e23c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.316663  7558 raft_consensus.cc:348] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.317351  7262 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 24) took 1119 ms. Trace:
I20260809 23:23:40.319303  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c057787ab114bba890199f5b36b37b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.320228  7159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 24) took 1130 ms. Trace:
I20260809 23:23:40.320570  7254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 23) took 1123 ms. Trace:
I20260809 23:23:40.342274  7370 tablet_service.cc:1467] Processing CreateTablet for tablet 4f841b906f634c05afe5a33637229c8f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260809 23:23:40.347106  7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f841b906f634c05afe5a33637229c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.321470  7566 tablet_bootstrap.cc:700] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: Time spent opening tablet: real 0.103s	user 0.000s	sys 0.005s
I20260809 23:23:40.309032  7155 rpcz_store.cc:276] 0809 23:23:39.179532 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.602389 (+422857us) service_pool.cc:227] Handling call
0809 23:23:39.604661 (+  2272us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.621739 (+ 17078us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.205832 (+584093us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.272229 (+ 66397us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.308392 (+ 36163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.325976  7148 tablet_service.cc:1467] Processing CreateTablet for tablet 3f126def92864a43a94113cdd599ee4f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:23:40.328944  7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89212541eac4b91975dbfbe73e23c3a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.278249  7259 consensus_meta.cc:287] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.012s
I20260809 23:23:40.361236  7155 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecdc375af4a473ca3c09c02da2a2594 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:23:40.343228  7572 raft_consensus.cc:482] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.362366  7259 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 25) took 1164 ms. Trace:
I20260809 23:23:40.363049  7150 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 21) took 1184 ms. Trace:
I20260809 23:23:40.363487  7150 rpcz_store.cc:276] 0809 23:23:39.178412 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.545153 (+366741us) service_pool.cc:227] Handling call
0809 23:23:39.683983 (+138830us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.684105 (+   122us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.200824 (+516719us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.357625 (+156801us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.362985 (+  5360us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.292393  7473 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 24) took 1056 ms. Trace:
I20260809 23:23:40.365229  7473 rpcz_store.cc:276] 0809 23:23:39.236315 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.842173 (+605858us) service_pool.cc:227] Handling call
0809 23:23:39.942149 (+ 99976us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.942300 (+   151us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.138696 (+196396us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.291842 (+153146us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.292332 (+   490us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.365899  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:40.367420  7581 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.367750  7362 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 17) took 1055 ms. Trace:
I20260809 23:23:40.368009  7581 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.368139  7150 tablet_service.cc:1467] Processing CreateTablet for tablet 46a6ff8a5b7e4c73ae66ec2560453733 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:23:40.371052  7566 tablet_bootstrap.cc:654] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:40.371510  7228 leader_election.cc:341] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.375149  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3066e3b731424266bfd6155e1dddc90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.376111  7581 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:40.377961  7367 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 15) took 1066 ms. Trace:
I20260809 23:23:40.342531  7254 rpcz_store.cc:276] 0809 23:23:39.197125 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.656151 (+459026us) service_pool.cc:227] Handling call
0809 23:23:39.736155 (+ 80004us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b58004aa098. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900b7c 00007fac2090073b 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:39.745021 (+  8866us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.745202 (+   181us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.165917 (+420715us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.319839 (+153922us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.320498 (+   659us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":60966272}
I20260809 23:23:40.354306  7466 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 20) took 1147 ms. Trace:
I20260809 23:23:40.361591  7151 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 27) took 1162 ms. Trace:
I20260809 23:23:40.362519  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f126def92864a43a94113cdd599ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.362181  7572 raft_consensus.cc:504] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.337558  7262 rpcz_store.cc:276] 0809 23:23:39.197477 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.702982 (+505505us) service_pool.cc:227] Handling call
0809 23:23:39.744721 (+ 41739us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.744842 (+   121us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.086449 (+341607us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.316622 (+230173us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.317272 (+   650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.353731  7559 tablet_bootstrap.cc:654] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:40.368191  7362 rpcz_store.cc:276] 0809 23:23:39.311935 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.630628 (+318693us) service_pool.cc:227] Handling call
0809 23:23:39.668811 (+ 38183us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.668922 (+   111us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.081867 (+412945us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.367245 (+285378us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.367691 (+   446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.362884  7259 rpcz_store.cc:276] 0809 23:23:39.197833 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.410056 (+212223us) service_pool.cc:227] Handling call
0809 23:23:39.466225 (+ 56169us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.466347 (+   122us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:39.640690 (+174343us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.361682 (+720992us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.362307 (+   625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.391040  7161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 26) took 1194 ms. Trace:
I20260809 23:23:40.390890  7362 tablet_service.cc:1467] Processing CreateTablet for tablet b3e08fe5d08748ba98111cd243575917 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260809 23:23:40.391613  7161 rpcz_store.cc:276] 0809 23:23:39.196542 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.861932 (+665390us) service_pool.cc:227] Handling call
0809 23:23:39.947410 (+ 85478us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.947567 (+   157us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.179953 (+232386us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.338865 (+158912us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.352746 (+ 13881us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12160}
I20260809 23:23:40.343500  7469 tablet_service.cc:1467] Processing CreateTablet for tablet 3f126def92864a43a94113cdd599ee4f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:23:40.392911  7151 rpcz_store.cc:276] 0809 23:23:39.199568 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.730022 (+530454us) service_pool.cc:227] Handling call
0809 23:23:39.778381 (+ 48359us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.778516 (+   135us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.222514 (+443998us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.322820 (+100306us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.361530 (+ 38710us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.382762  7572 leader_election.cc:288] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:40.383976  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:40.379969  7466 rpcz_store.cc:276] 0809 23:23:39.206702 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.550450 (+343748us) service_pool.cc:227] Handling call
0809 23:23:39.658511 (+108061us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.663172 (+  4661us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.073239 (+410067us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.353764 (+280525us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.354238 (+   474us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:23:40.400976  7228 leader_election.cc:341] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.341924  7159 rpcz_store.cc:276] 0809 23:23:39.189762 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.813838 (+624076us) service_pool.cc:227] Handling call
0809 23:23:39.939362 (+125524us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.939576 (+   214us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.206342 (+266766us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.319748 (+113406us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.320172 (+   424us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15258}
I20260809 23:23:40.353605  7471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 19) took 1147 ms. Trace:
I20260809 23:23:40.336829  7558 raft_consensus.cc:374] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:40.403158  7558 raft_consensus.cc:729] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:40.403239  7159 tablet_service.cc:1467] Processing CreateTablet for tablet 8d22e37cd9d24996abca51e457d761cc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 23:23:40.393939  7161 tablet_service.cc:1467] Processing CreateTablet for tablet 207ede1fd2664d3a871bbb0a5b854a64 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:23:40.396173  7262 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecdc375af4a473ca3c09c02da2a2594 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:23:40.397416  7473 tablet_service.cc:1467] Processing CreateTablet for tablet 895d1e78ae154da4a1c299f234d88965 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:23:40.397959  7258 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 26) took 1199 ms. Trace:
I20260809 23:23:40.377236  7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9c4f8d91c504cdd8f1fd9dd8f707602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.404294  7558 consensus_queue.cc:260] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.404891  7254 tablet_service.cc:1467] Processing CreateTablet for tablet 46a6ff8a5b7e4c73ae66ec2560453733 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:23:40.361251  7464 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 27) took 1062 ms. Trace:
I20260809 23:23:40.335880  7162 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 28) took 1049 ms. Trace:
I20260809 23:23:40.391331  7259 tablet_service.cc:1467] Processing CreateTablet for tablet 895d1e78ae154da4a1c299f234d88965 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:23:40.382509  7468 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 26) took 1086 ms. Trace:
I20260809 23:23:40.393829  7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f126def92864a43a94113cdd599ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.325417  7363 consensus_meta.cc:287] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: Time spent flushing consensus metadata: real 0.631s	user 0.006s	sys 0.011s
I20260809 23:23:40.378480  7367 rpcz_store.cc:276] 0809 23:23:39.311173 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.516492 (+205319us) service_pool.cc:227] Handling call
0809 23:23:39.540111 (+ 23619us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.556296 (+ 16185us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.138167 (+581871us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.377285 (+239118us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.377891 (+   606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.402459  7471 rpcz_store.cc:276] 0809 23:23:39.206360 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.454562 (+248202us) service_pool.cc:227] Handling call
0809 23:23:39.456478 (+  1916us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.456569 (+    91us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.163158 (+706589us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.340528 (+177370us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.353536 (+ 13008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.418412  7366 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 21) took 1105 ms. Trace:
I20260809 23:23:40.426539  7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a6ff8a5b7e4c73ae66ec2560453733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.418351  7258 rpcz_store.cc:276] 0809 23:23:39.198539 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.846231 (+647692us) service_pool.cc:227] Handling call
0809 23:23:39.946916 (+100685us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.950389 (+  3473us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.261496 (+311107us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.397441 (+135945us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.397900 (+   459us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.437991  7561 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:40.438130  7558 ts_tablet_manager.cc:1428] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.124s	user 0.006s	sys 0.001s
I20260809 23:23:40.439612  7471 tablet_service.cc:1467] Processing CreateTablet for tablet 207ede1fd2664d3a871bbb0a5b854a64 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:23:40.448802  7561 ts_tablet_manager.cc:1397] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.346s	user 0.035s	sys 0.061s
I20260809 23:23:40.449651  7566 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:40.450266  7566 ts_tablet_manager.cc:1397] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.305s	user 0.010s	sys 0.015s
I20260809 23:23:40.451074  7367 tablet_service.cc:1467] Processing CreateTablet for tablet 8ded669d4bcc4a7a81c44ad0b578c899 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260809 23:23:40.452193  7360 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 20) took 1139 ms. Trace:
I20260809 23:23:40.452549  7360 rpcz_store.cc:276] 0809 23:23:39.312980 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.647678 (+334698us) service_pool.cc:227] Handling call
0809 23:23:39.719888 (+ 72210us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b58008ee398. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:39.720041 (+   153us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.720154 (+   113us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.205570 (+485416us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.428598 (+223028us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.452128 (+ 23530us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2476800}
I20260809 23:23:40.453204  7566 raft_consensus.cc:348] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.454012  7566 raft_consensus.cc:374] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:40.454308  7566 raft_consensus.cc:729] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:40.432852  7366 rpcz_store.cc:276] 0809 23:23:39.313331 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.873407 (+560076us) service_pool.cc:227] Handling call
0809 23:23:39.947740 (+ 74333us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.947887 (+   147us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.185345 (+237458us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.379772 (+194427us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.415558 (+ 35786us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.456070  7566 consensus_queue.cc:260] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.456980  7363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 10) took 1188 ms. Trace:
I20260809 23:23:40.456853  7366 tablet_service.cc:1467] Processing CreateTablet for tablet 3f126def92864a43a94113cdd599ee4f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260809 23:23:40.457379  7363 rpcz_store.cc:276] 0809 23:23:39.268939 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.436312 (+167373us) service_pool.cc:227] Handling call
0809 23:23:39.452702 (+ 16390us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.452827 (+   125us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:39.693483 (+240656us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.444002 (+750519us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.456920 (+ 12918us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.425140  7162 rpcz_store.cc:276] 0809 23:23:39.269880 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.864061 (+594181us) service_pool.cc:227] Handling call
0809 23:23:39.953072 (+ 89011us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.953204 (+   132us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.184725 (+231521us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.318379 (+133654us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.318867 (+   488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3456}
I20260809 23:23:40.459877  7566 ts_tablet_manager.cc:1428] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.005s
I20260809 23:23:40.460569  7162 tablet_service.cc:1467] Processing CreateTablet for tablet b64607aa57f542cda93ba367af662006 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:23:40.462486  7558 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:40.462553  7264 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 20) took 1274 ms. Trace:
I20260809 23:23:40.463225  7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895d1e78ae154da4a1c299f234d88965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.463069  7264 rpcz_store.cc:276] 0809 23:23:39.187768 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.598319 (+410551us) service_pool.cc:227] Handling call
0809 23:23:39.744183 (+145864us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b58004aa098. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900b7c 00007fac2090073b 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:39.745412 (+  1229us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.745499 (+    87us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.181379 (+435880us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.431381 (+250002us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.462492 (+ 31111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":64788864}
I20260809 23:23:40.463794  7253 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 22) took 1275 ms. Trace:
I20260809 23:23:40.464529  7253 rpcz_store.cc:276] 0809 23:23:39.188513 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.633991 (+445478us) service_pool.cc:227] Handling call
0809 23:23:39.744522 (+110531us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.744682 (+   160us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.302901 (+558219us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.463060 (+160159us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.463721 (+   661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.454485  7360 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5b4438b1d84d7198189995b2d2d409 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260809 23:23:40.468185  7258 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a40c34e514b9fbcec1c0000466ecc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:23:40.469432  7151 tablet_service.cc:1467] Processing CreateTablet for tablet 43caf538bbac4e0da20c274fad4d7f1f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:23:40.470059  7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecdc375af4a473ca3c09c02da2a2594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.476073  7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e08fe5d08748ba98111cd243575917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.471055  7259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895d1e78ae154da4a1c299f234d88965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.471053  7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207ede1fd2664d3a871bbb0a5b854a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.434286  7256 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 27) took 1169 ms. Trace:
I20260809 23:23:40.484247  7363 tablet_service.cc:1467] Processing CreateTablet for tablet 895d1e78ae154da4a1c299f234d88965 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260809 23:23:40.480933  7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207ede1fd2664d3a871bbb0a5b854a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.425220  7464 rpcz_store.cc:276] 0809 23:23:39.296018 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.119344 (+823326us) service_pool.cc:227] Handling call
0809 23:23:40.121150 (+  1806us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.121249 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.261245 (+139996us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.358195 (+ 96950us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.358645 (+   450us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.506013  7579 raft_consensus.cc:482] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.506616  7579 raft_consensus.cc:504] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:40.507395  7464 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a40c34e514b9fbcec1c0000466ecc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:23:40.509193  7464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a40c34e514b9fbcec1c0000466ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.427646  7468 rpcz_store.cc:276] 0809 23:23:39.295668 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.876250 (+580582us) service_pool.cc:227] Handling call
0809 23:23:39.950513 (+ 74263us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.950637 (+   124us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.229247 (+278610us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.381926 (+152679us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.382436 (+   510us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.512445  7157 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 23) took 1331 ms. Trace:
I20260809 23:23:40.523624  7566 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:40.523286  7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecdc375af4a473ca3c09c02da2a2594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.533174  7470 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 25) took 1238 ms. Trace:
I20260809 23:23:40.534863  7470 rpcz_store.cc:276] 0809 23:23:39.295229 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.854696 (+559467us) service_pool.cc:227] Handling call
0809 23:23:39.995028 (+140332us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.995177 (+   149us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.352028 (+356851us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.532611 (+180583us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.533112 (+   501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.536914  7470 tablet_service.cc:1467] Processing CreateTablet for tablet 43caf538bbac4e0da20c274fad4d7f1f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:23:40.533023  7468 tablet_service.cc:1467] Processing CreateTablet for tablet b64607aa57f542cda93ba367af662006 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:23:40.532716  7264 tablet_service.cc:1467] Processing CreateTablet for tablet 8d22e37cd9d24996abca51e457d761cc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 23:23:40.541463  7158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 25) took 1350 ms. Trace:
I20260809 23:23:40.541893  7158 rpcz_store.cc:276] 0809 23:23:39.190845 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.858122 (+667277us) service_pool.cc:227] Handling call
0809 23:23:39.955444 (+ 97322us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.955579 (+   135us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.241680 (+286101us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.525158 (+283478us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.541397 (+ 16239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.543995  7158 tablet_service.cc:1467] Processing CreateTablet for tablet f23623f6580149f19787b8924d48ff24 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:23:40.546116  7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23623f6580149f19787b8924d48ff24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.547027  7160 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 31) took 1212 ms. Trace:
I20260809 23:23:40.547451  7160 rpcz_store.cc:276] 0809 23:23:39.334887 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.284234 (+949347us) service_pool.cc:227] Handling call
0809 23:23:40.286570 (+  2336us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.286705 (+   135us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.439468 (+152763us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.546433 (+106965us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.546964 (+   531us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.548557  7159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d22e37cd9d24996abca51e457d761cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.559791  7466 tablet_service.cc:1467] Processing CreateTablet for tablet 46a6ff8a5b7e4c73ae66ec2560453733 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260809 23:23:40.564993  7253 tablet_service.cc:1467] Processing CreateTablet for tablet 43caf538bbac4e0da20c274fad4d7f1f (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260809 23:23:40.527060  7157 rpcz_store.cc:276] 0809 23:23:39.180714 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:39.650786 (+470072us) service_pool.cc:227] Handling call
0809 23:23:39.697492 (+ 46706us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:39.697732 (+   240us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.177969 (+480237us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.441209 (+263240us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.510259 (+ 69050us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.569252  7583 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.569039  7561 raft_consensus.cc:348] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.569998  7561 raft_consensus.cc:374] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:40.569849  7583 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:40.571161  7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43caf538bbac4e0da20c274fad4d7f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.572674  7583 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:40.570353  7561 raft_consensus.cc:729] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:40.524493  7258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a40c34e514b9fbcec1c0000466ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.524842  7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f126def92864a43a94113cdd599ee4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.488391  7256 rpcz_store.cc:276] 0809 23:23:39.264746 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.090764 (+826018us) service_pool.cc:227] Handling call
0809 23:23:40.136537 (+ 45773us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.136631 (+    94us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.312567 (+175936us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.420157 (+107590us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.432228 (+ 12071us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.592600  7463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 30) took 1260 ms. Trace:
I20260809 23:23:40.594234  7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43caf538bbac4e0da20c274fad4d7f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.597295  7255 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 28) took 1321 ms. Trace:
I20260809 23:23:40.593022  7579 leader_election.cc:288] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:40.613045  7359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 24) took 1298 ms. Trace:
I20260809 23:23:40.613548  7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43caf538bbac4e0da20c274fad4d7f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.618678  7472 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 29) took 1322 ms. Trace:
I20260809 23:23:40.604111  7463 rpcz_store.cc:276] 0809 23:23:39.332652 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.208148 (+875496us) service_pool.cc:227] Handling call
0809 23:23:40.210075 (+  1927us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.210174 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.365462 (+155288us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.591982 (+226520us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.592529 (+   547us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.661001  7463 tablet_service.cc:1467] Processing CreateTablet for tablet 2e008aea8ad64b2b9fa356cca2300ea1 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:23:40.619896  7561 consensus_queue.cc:260] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.663811  7561 ts_tablet_manager.cc:1428] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.215s	user 0.006s	sys 0.002s
I20260809 23:23:40.664811  7561 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:40.667580  7265 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 30) took 1337 ms. Trace:
I20260809 23:23:40.668004  7265 rpcz_store.cc:276] 0809 23:23:39.330489 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.214161 (+883672us) service_pool.cc:227] Handling call
0809 23:23:40.226143 (+ 11982us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.226388 (+   245us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.397942 (+171554us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.667068 (+269126us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.667520 (+   452us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.670655  7155 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 33) took 1232 ms. Trace:
I20260809 23:23:40.671106  7155 rpcz_store.cc:276] 0809 23:23:39.438276 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.360331 (+922055us) service_pool.cc:227] Handling call
0809 23:23:40.363039 (+  2708us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.363132 (+    93us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.566223 (+203091us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.670101 (+103878us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.670590 (+   489us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.624244  7559 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:40.673017  7559 ts_tablet_manager.cc:1397] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.483s	user 0.036s	sys 0.039s
I20260809 23:23:40.626546  7474 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 28) took 1330 ms. Trace:
I20260809 23:23:40.675176  7474 rpcz_store.cc:276] 0809 23:23:39.296391 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.129842 (+833451us) service_pool.cc:227] Handling call
0809 23:23:40.131627 (+  1785us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.131721 (+    94us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.546450 (+414729us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.625992 (+ 79542us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.626484 (+   492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.676487  7559 raft_consensus.cc:348] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.677268  7559 raft_consensus.cc:374] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:40.677050  7474 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f251a895a462abb71b8668cf0aa80 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:23:40.677551  7559 raft_consensus.cc:729] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:40.677755  7364 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 23) took 1363 ms. Trace:
I20260809 23:23:40.628968  7365 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 26) took 1284 ms. Trace:
I20260809 23:23:40.631799  7255 rpcz_store.cc:276] 0809 23:23:39.275620 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.184156 (+908536us) service_pool.cc:227] Handling call
0809 23:23:40.229115 (+ 44959us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.229216 (+   101us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.437010 (+207794us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.596728 (+159718us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.597223 (+   495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.678315  7559 consensus_queue.cc:260] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.678310  7365 rpcz_store.cc:276] 0809 23:23:39.344945 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.291097 (+946152us) service_pool.cc:227] Handling call
0809 23:23:40.293063 (+  1966us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.293217 (+   154us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.551316 (+258099us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.628440 (+ 77124us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.628903 (+   463us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.681048  7365 tablet_service.cc:1467] Processing CreateTablet for tablet 207ede1fd2664d3a871bbb0a5b854a64 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260809 23:23:40.643751  7359 rpcz_store.cc:276] 0809 23:23:39.314462 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.261273 (+946811us) service_pool.cc:227] Handling call
0809 23:23:40.276230 (+ 14957us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.280584 (+  4354us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.547740 (+267156us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.612471 (+ 64731us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.612986 (+   515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.683333  7559 ts_tablet_manager.cc:1428] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260809 23:23:40.647069  7472 rpcz_store.cc:276] 0809 23:23:39.296733 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.195047 (+898314us) service_pool.cc:227] Handling call
0809 23:23:40.244719 (+ 49672us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.246224 (+  1505us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.424496 (+178272us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.618097 (+193601us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.618614 (+   517us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.672147  7161 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 35) took 1170 ms. Trace:
I20260809 23:23:40.622264  7363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895d1e78ae154da4a1c299f234d88965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.607729  7466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a6ff8a5b7e4c73ae66ec2560453733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.689604  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecdc375af4a473ca3c09c02da2a2594 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260809 23:23:40.689895  7255 tablet_service.cc:1467] Processing CreateTablet for tablet 58137ea1624f4b1db96f7a6b9ada7f68 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:23:40.694154  7361 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 27) took 1348 ms. Trace:
I20260809 23:23:40.696504  7257 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 31) took 1344 ms. Trace:
I20260809 23:23:40.696609  7256 tablet_service.cc:1467] Processing CreateTablet for tablet 2e008aea8ad64b2b9fa356cca2300ea1 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:23:40.721410  7155 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f251a895a462abb71b8668cf0aa80 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:23:40.670439  7265 tablet_service.cc:1467] Processing CreateTablet for tablet f23623f6580149f19787b8924d48ff24 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:23:40.697774  7162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64607aa57f542cda93ba367af662006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.702523  7148 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 32) took 1339 ms. Trace:
I20260809 23:23:40.703464  7472 tablet_service.cc:1467] Processing CreateTablet for tablet 29afbcdad59b414fa48484b70ff56499 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:23:40.707257  7474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f251a895a462abb71b8668cf0aa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.707342  7159 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 36) took 1068 ms. Trace:
I20260809 23:23:40.709518  7160 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2e2de3a9d45a8a776ce5d568059bb (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:23:40.710048  7157 tablet_service.cc:1467] Processing CreateTablet for tablet 29afbcdad59b414fa48484b70ff56499 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:23:40.716976  7150 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 34) took 1268 ms. Trace:
I20260809 23:23:40.731364  7150 rpcz_store.cc:276] 0809 23:23:39.448599 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.364466 (+915867us) service_pool.cc:227] Handling call
0809 23:23:40.369554 (+  5088us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.369675 (+   121us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.592531 (+222856us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.716429 (+123898us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.716915 (+   486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.732084  7160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2e2de3a9d45a8a776ce5d568059bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.719251  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecdc375af4a473ca3c09c02da2a2594. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.734529  7579 raft_consensus.cc:482] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.720669  7361 rpcz_store.cc:276] 0809 23:23:39.345396 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.305246 (+959850us) service_pool.cc:227] Handling call
0809 23:23:40.337457 (+ 32211us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.338362 (+   905us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.648602 (+310240us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.693569 (+ 44967us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.694093 (+   524us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.735172  7579 raft_consensus.cc:504] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.725807  7572 raft_consensus.cc:482] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.736776  7157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29afbcdad59b414fa48484b70ff56499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.737969  7579 leader_election.cc:288] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:40.737794  7361 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1a40c34e514b9fbcec1c0000466ecc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260809 23:23:40.725701  7148 rpcz_store.cc:276] 0809 23:23:39.363626 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.325054 (+961428us) service_pool.cc:227] Handling call
0809 23:23:40.362017 (+ 36963us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.362116 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.536882 (+174766us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.696980 (+160098us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.702460 (+  5480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.736915  7572 raft_consensus.cc:504] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:40.734555  7150 tablet_service.cc:1467] Processing CreateTablet for tablet 7d68de750e444f43a513619eac515b03 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:23:40.739871  7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d68de750e444f43a513619eac515b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.739815  7148 tablet_service.cc:1467] Processing CreateTablet for tablet 964bd1390e2d43b9ab381a795bdddedf (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 23:23:40.740442  7572 leader_election.cc:288] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:40.741962  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:40.730378  7370 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 25) took 1415 ms. Trace:
I20260809 23:23:40.743078  7370 rpcz_store.cc:276] 0809 23:23:39.314807 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.278386 (+963579us) service_pool.cc:227] Handling call
0809 23:23:40.346613 (+ 68227us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.346709 (+    96us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.629032 (+282323us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.729821 (+100789us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.730319 (+   498us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.743669  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:40.741444  7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964bd1390e2d43b9ab381a795bdddedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.745186  7370 tablet_service.cc:1467] Processing CreateTablet for tablet 8d22e37cd9d24996abca51e457d761cc (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260809 23:23:40.746640  7468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64607aa57f542cda93ba367af662006. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.747323  7227 leader_election.cc:341] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.747874  7227 leader_election.cc:302] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:40.747524  7472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29afbcdad59b414fa48484b70ff56499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.749254  7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e008aea8ad64b2b9fa356cca2300ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.723105  7257 rpcz_store.cc:276] 0809 23:23:39.351891 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.267654 (+915763us) service_pool.cc:227] Handling call
0809 23:23:40.313030 (+ 45376us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.313166 (+   136us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.560638 (+247472us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.695991 (+135353us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.696445 (+   454us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.756671  7257 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7f251a895a462abb71b8668cf0aa80 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260809 23:23:40.726012  7254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46a6ff8a5b7e4c73ae66ec2560453733. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.761135  7256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e008aea8ad64b2b9fa356cca2300ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.735162  7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ded669d4bcc4a7a81c44ad0b578c899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.763592  7365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207ede1fd2664d3a871bbb0a5b854a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.762833  7255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58137ea1624f4b1db96f7a6b9ada7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.727319  7159 rpcz_store.cc:276] 0809 23:23:39.639262 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.402365 (+763103us) service_pool.cc:227] Handling call
0809 23:23:40.487101 (+ 84736us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b58001e3f98. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:40.498768 (+ 11667us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.516914 (+ 18146us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.631264 (+114350us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.706639 (+ 75375us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.707267 (+   628us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11887360}
I20260809 23:23:40.698781  7161 rpcz_store.cc:276] 0809 23:23:39.502227 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.393134 (+890907us) service_pool.cc:227] Handling call
0809 23:23:40.423688 (+ 30554us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.428847 (+  5159us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.619796 (+190949us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.658503 (+ 38707us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.672079 (+ 13576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.814314  7155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f251a895a462abb71b8668cf0aa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.814471  7559 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:40.815205  7151 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 37) took 1173 ms. Trace:
I20260809 23:23:40.815753  7151 rpcz_store.cc:276] 0809 23:23:39.641742 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.418591 (+776849us) service_pool.cc:227] Handling call
0809 23:23:40.470359 (+ 51768us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.470468 (+   109us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.689796 (+219328us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.814243 (+124447us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.815125 (+   882us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.818840  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:40.821712  7464 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 35) took 1161 ms. Trace:
I20260809 23:23:40.821847  7471 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 34) took 1320 ms. Trace:
I20260809 23:23:40.832603  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:40.833433  7173 raft_consensus.cc:2454] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:40.829115  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:40.827443  7338 leader_election.cc:341] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.830667  7262 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 33) took 1383 ms. Trace:
I20260809 23:23:40.836946  7473 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 32) took 1458 ms. Trace:
I20260809 23:23:40.837404  7473 rpcz_store.cc:276] 0809 23:23:39.378159 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.372871 (+994712us) service_pool.cc:227] Handling call
0809 23:23:40.425489 (+ 52618us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.430278 (+  4789us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.631438 (+201160us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.836399 (+204961us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.836885 (+   486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.839094  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:40.846441  7338 leader_election.cc:341] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.847306  7338 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:40.848639  7581 raft_consensus.cc:2726] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:40.698251  7364 rpcz_store.cc:276] 0809 23:23:39.314095 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.245322 (+931227us) service_pool.cc:227] Handling call
0809 23:23:40.339999 (+ 94677us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b58008ee398. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:40.340157 (+   158us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.340257 (+   100us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.618366 (+278109us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.677225 (+ 58859us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.677695 (+   470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6235904}
I20260809 23:23:40.852162  7473 tablet_service.cc:1467] Processing CreateTablet for tablet 58137ea1624f4b1db96f7a6b9ada7f68 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:23:40.853116  7364 tablet_service.cc:1467] Processing CreateTablet for tablet b64607aa57f542cda93ba367af662006 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260809 23:23:40.822234  7471 rpcz_store.cc:276] 0809 23:23:39.501539 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.438666 (+937127us) service_pool.cc:227] Handling call
0809 23:23:40.447796 (+  9130us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.447893 (+    97us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.678143 (+230250us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.821399 (+143256us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.821792 (+   393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.855178  7469 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 31) took 1498 ms. Trace:
I20260809 23:23:40.855579  7469 rpcz_store.cc:276] 0809 23:23:39.357274 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.342447 (+985173us) service_pool.cc:227] Handling call
0809 23:23:40.393315 (+ 50868us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.393414 (+    99us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.692944 (+299530us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.854614 (+161670us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.855115 (+   501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.856570  7471 tablet_service.cc:1467] Processing CreateTablet for tablet 7d68de750e444f43a513619eac515b03 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:23:40.857954  7469 tablet_service.cc:1467] Processing CreateTablet for tablet 964bd1390e2d43b9ab381a795bdddedf (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260809 23:23:40.865900  7581 raft_consensus.cc:482] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:40.866890  7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23623f6580149f19787b8924d48ff24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.853562  7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58137ea1624f4b1db96f7a6b9ada7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.866701  7253 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 37) took 1224 ms. Trace:
I20260809 23:23:40.875942  7581 raft_consensus.cc:504] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:40.940537  7253 rpcz_store.cc:276] 0809 23:23:39.642839 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.563833 (+920994us) service_pool.cc:227] Handling call
0809 23:23:40.566266 (+  2433us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.566377 (+   111us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.822762 (+256385us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.866172 (+ 43410us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.866639 (+   467us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.942344  7581 leader_election.cc:288] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:40.836881  7262 rpcz_store.cc:276] 0809 23:23:39.447080 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.395390 (+948310us) service_pool.cc:227] Handling call
0809 23:23:40.429963 (+ 34573us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.435126 (+  5163us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.679755 (+244629us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.718833 (+ 39078us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.830296 (+111463us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b58004aa098. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:23:40.830606 (+   310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":100226304}
I20260809 23:23:40.878742  7572 raft_consensus.cc:2726] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:40.879294  7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5b4438b1d84d7198189995b2d2d409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.882302  7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d22e37cd9d24996abca51e457d761cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.958227  7259 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 32) took 1594 ms. Trace:
I20260809 23:23:40.946879  7262 tablet_service.cc:1467] Processing CreateTablet for tablet 7d68de750e444f43a513619eac515b03 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260809 23:23:40.961028  7259 rpcz_store.cc:276] 0809 23:23:39.364334 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.390424 (+1026090us) service_pool.cc:227] Handling call
0809 23:23:40.438434 (+ 48010us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b58004aa098. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900702 00007fac297f0b09 00007fac2872c730 00007fac2880ed21 00007fac287ec1e3 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2 00007fac21d9fb01
0809 23:23:40.461975 (+ 23541us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.462080 (+   105us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.947518 (+485438us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.957457 (+  9939us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.958153 (+   696us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3409792}
I20260809 23:23:40.947669  7253 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2e2de3a9d45a8a776ce5d568059bb (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:23:40.963680  7253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2e2de3a9d45a8a776ce5d568059bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.822130  7464 rpcz_store.cc:276] 0809 23:23:39.660545 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.506108 (+845563us) service_pool.cc:227] Handling call
0809 23:23:40.508546 (+  2438us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.508672 (+   126us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.669752 (+161080us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.821145 (+151393us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.821649 (+   504us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.869593  7559 tablet_bootstrap.cc:654] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:40.896680  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 149399us
W20260809 23:23:40.965073  7227 leader_election.cc:341] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.966009  7227 leader_election.cc:302] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
W20260809 23:23:40.967326  7227 leader_election.cc:341] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.967698  7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964bd1390e2d43b9ab381a795bdddedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.967821  7227 leader_election.cc:302] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:40.973379  7572 raft_consensus.cc:2781] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:40.973776  7572 raft_consensus.cc:482] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:40.974061  7572 raft_consensus.cc:3037] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:40.974400  7558 tablet_bootstrap.cc:654] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:40.975265  7370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d22e37cd9d24996abca51e457d761cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.981388  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:40.921916  7258 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 35) took 1417 ms. Trace:
I20260809 23:23:40.982771  7257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7f251a895a462abb71b8668cf0aa80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.927103  7470 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 36) took 1258 ms. Trace:
I20260809 23:23:40.983058  7258 rpcz_store.cc:276] 0809 23:23:39.504961 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.466985 (+962024us) service_pool.cc:227] Handling call
0809 23:23:40.469369 (+  2384us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.469497 (+   128us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.835792 (+366295us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.921344 (+ 85552us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.921840 (+   496us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.983794  7470 rpcz_store.cc:276] 0809 23:23:39.668667 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.535943 (+867276us) service_pool.cc:227] Handling call
0809 23:23:40.537806 (+  1863us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.537921 (+   115us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.713389 (+175468us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:40.890219 (+176830us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:40.927021 (+ 36802us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:40.983546  7262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d68de750e444f43a513619eac515b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.932349  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:40.939610  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:40.987766  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a1a40c34e514b9fbcec1c0000466ecc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:23:40.986796  7126 leader_election.cc:341] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:40.989063  7437 leader_election.cc:341] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.981388  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:40.985913  7338 leader_election.cc:341] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:40.990670  7124 leader_election.cc:341] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:40.991267  7124 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:40.992662  7583 raft_consensus.cc:2726] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:23:40.868026  7008 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 122324us
I20260809 23:23:40.993584  7364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b64607aa57f542cda93ba367af662006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:40.991246  7471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d68de750e444f43a513619eac515b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.015915  7559 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.016587  7559 ts_tablet_manager.cc:1397] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.205s	user 0.030s	sys 0.005s
I20260809 23:23:41.019743  7559 raft_consensus.cc:348] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.020617  7559 raft_consensus.cc:374] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.020977  7559 raft_consensus.cc:729] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.021904  7559 consensus_queue.cc:260] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.025346  7559 ts_tablet_manager.cc:1428] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260809 23:23:41.037642  7566 tablet_bootstrap.cc:654] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.039314  7559 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.046489  7559 tablet_bootstrap.cc:654] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.060039  7359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 33) took 1612 ms. Trace:
I20260809 23:23:41.060550  7359 rpcz_store.cc:276] 0809 23:23:39.447896 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.683605 (+1235709us) service_pool.cc:227] Handling call
0809 23:23:40.703145 (+ 19540us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.703239 (+    94us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.846330 (+143091us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.059478 (+213148us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.059973 (+   495us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.063054  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 2e008aea8ad64b2b9fa356cca2300ea1 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260809 23:23:41.064424  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e008aea8ad64b2b9fa356cca2300ea1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.064615  7162 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:32942 (request call id 38) took 1418 ms. Trace:
I20260809 23:23:41.065056  7162 rpcz_store.cc:276] 0809 23:23:39.646245 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.459741 (+813496us) service_pool.cc:227] Handling call
0809 23:23:40.461434 (+  1693us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.461535 (+   101us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.849883 (+388348us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.064073 (+214190us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.064553 (+   480us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.074620  7604 raft_consensus.cc:2726] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:23:41.077885  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 110596us
I20260809 23:23:41.082535  7363 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 32) took 1717 ms. Trace:
I20260809 23:23:41.085295  7561 tablet_bootstrap.cc:654] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.099335  7466 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 33) took 1688 ms. Trace:
I20260809 23:23:41.100474  7572 raft_consensus.cc:504] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.102025  7359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 38) took 1418 ms. Trace:
I20260809 23:23:41.112684  7463 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 38) took 1429 ms. Trace:
I20260809 23:23:41.114616  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:41.114710  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:41.116799  7466 rpcz_store.cc:276] 0809 23:23:39.411252 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.418139 (+1006887us) service_pool.cc:227] Handling call
0809 23:23:40.560701 (+142562us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.560801 (+   100us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.840998 (+280197us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.098743 (+257745us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.099265 (+   522us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.130575  7359 rpcz_store.cc:276] 0809 23:23:39.683668 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:41.062014 (+1378346us) service_pool.cc:227] Handling call
0809 23:23:41.063928 (+  1914us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:41.064028 (+   100us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.093282 (+ 29254us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.101528 (+  8246us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.101965 (+   437us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.131799  7463 rpcz_store.cc:276] 0809 23:23:39.683443 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.660033 (+976590us) service_pool.cc:227] Handling call
0809 23:23:40.661835 (+  1802us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.661947 (+   112us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.024989 (+363042us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.111979 (+ 86990us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.112606 (+   627us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.129449  7362 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 28) took 1783 ms. Trace:
I20260809 23:23:41.137677  7359 tablet_service.cc:1467] Processing CreateTablet for tablet 67a2e2de3a9d45a8a776ce5d568059bb (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260809 23:23:41.138356  7360 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 30) took 1791 ms. Trace:
I20260809 23:23:41.126834  7370 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 36) took 1487 ms. Trace:
I20260809 23:23:41.154237  7361 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 35) took 1603 ms. Trace:
W20260809 23:23:41.134743  7437 leader_election.cc:341] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.155395  7437 leader_election.cc:302] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.155308  7361 rpcz_store.cc:276] 0809 23:23:39.551290 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.736416 (+1185126us) service_pool.cc:227] Handling call
0809 23:23:40.796355 (+ 59939us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.796468 (+   113us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.085083 (+288615us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.150504 (+ 65421us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.154170 (+  3666us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260809 23:23:41.157040  7437 leader_election.cc:341] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.157734  7366 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 31) took 1800 ms. Trace:
I20260809 23:23:41.156009  7254 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 34) took 1698 ms. Trace:
I20260809 23:23:41.157804  7579 raft_consensus.cc:2726] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.146098  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.146380  7359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67a2e2de3a9d45a8a776ce5d568059bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.138178  7362 rpcz_store.cc:276] 0809 23:23:39.345810 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.389704 (+1043894us) service_pool.cc:227] Handling call
0809 23:23:40.392188 (+  2484us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.392339 (+   151us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.045571 (+653232us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.110463 (+ 64892us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.117921 (+  7458us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.158397  7361 tablet_service.cc:1467] Processing CreateTablet for tablet f23623f6580149f19787b8924d48ff24 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260809 23:23:41.157986  7558 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.146780  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:41.144956  7572 leader_election.cc:288] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.147804  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.187395  7558 ts_tablet_manager.cc:1397] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.725s	user 0.027s	sys 0.005s
W20260809 23:23:41.188424  7339 leader_election.cc:341] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.192306  7339 leader_election.cc:302] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:41.167443  7254 rpcz_store.cc:276] 0809 23:23:39.457531 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.404071 (+946540us) service_pool.cc:227] Handling call
0809 23:23:40.435396 (+ 31325us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.444460 (+  9064us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.093039 (+648579us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.155519 (+ 62480us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.155944 (+   425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.193715  7581 raft_consensus.cc:2726] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.138921  7360 rpcz_store.cc:276] 0809 23:23:39.346509 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.453653 (+1107144us) service_pool.cc:227] Handling call
0809 23:23:40.486288 (+ 32635us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.486404 (+   116us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.062748 (+576344us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.137808 (+ 75060us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.138299 (+   491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.146235  7367 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 29) took 1800 ms. Trace:
I20260809 23:23:41.172686  7174 raft_consensus.cc:3037] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.199059  7367 rpcz_store.cc:276] 0809 23:23:39.346147 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.443810 (+1097663us) service_pool.cc:227] Handling call
0809 23:23:40.480851 (+ 37041us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.485800 (+  4949us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.009872 (+524072us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.145696 (+135824us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.146175 (+   479us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.201330  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.201459  7367 tablet_service.cc:1467] Processing CreateTablet for tablet 964bd1390e2d43b9ab381a795bdddedf (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260809 23:23:41.202972  7437 leader_election.cc:341] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.203269  7364 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 37) took 1556 ms. Trace:
I20260809 23:23:41.203537  7437 leader_election.cc:302] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:41.203801  7364 rpcz_store.cc:276] 0809 23:23:39.646813 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.852198 (+1205385us) service_pool.cc:227] Handling call
0809 23:23:40.854062 (+  1864us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.854156 (+    94us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.095201 (+241045us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.183663 (+ 88462us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.202895 (+ 19232us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b58008ee398. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900e82 00007fac298198b9 00007fac2872c5d8 00007fac287f0063 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:23:41.203213 (+   318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":35851520}
I20260809 23:23:41.204598  7566 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.204841  7579 raft_consensus.cc:2726] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.205147  7566 ts_tablet_manager.cc:1397] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.690s	user 0.032s	sys 0.017s
W20260809 23:23:41.177685  7227 leader_election.cc:341] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.139163  7370 rpcz_store.cc:276] 0809 23:23:39.639717 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.744321 (+1104604us) service_pool.cc:227] Handling call
0809 23:23:40.746031 (+  1710us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.746146 (+   115us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.050707 (+304561us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.126320 (+ 75613us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.126771 (+   451us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.179267  7583 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.206770  7174 raft_consensus.cc:2454] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:41.180444  7256 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 38) took 1512 ms. Trace:
I20260809 23:23:41.181002  7559 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.208285  7566 raft_consensus.cc:348] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.208650  7227 leader_election.cc:302] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:41.209120  7559 ts_tablet_manager.cc:1397] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.170s	user 0.018s	sys 0.033s
I20260809 23:23:41.181234  7362 tablet_service.cc:1467] Processing CreateTablet for tablet 29afbcdad59b414fa48484b70ff56499 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260809 23:23:41.116392  7363 rpcz_store.cc:276] 0809 23:23:39.366209 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.458367 (+1092158us) service_pool.cc:227] Handling call
0809 23:23:40.512306 (+ 53939us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.519245 (+  6939us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:40.860564 (+341319us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.081985 (+221421us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.082471 (+   486us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.182473  7264 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40638 (request call id 36) took 1542 ms. Trace:
I20260809 23:23:41.182559  7468 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:33320 (request call id 37) took 1513 ms. Trace:
I20260809 23:23:41.167264  7366 rpcz_store.cc:276] 0809 23:23:39.357831 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.456009 (+1098178us) service_pool.cc:227] Handling call
0809 23:23:40.457900 (+  1891us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.458018 (+   118us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.144209 (+686191us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.153599 (+  9390us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.157676 (+  4077us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.217134  7468 rpcz_store.cc:276] 0809 23:23:39.669096 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.527040 (+857944us) service_pool.cc:227] Handling call
0809 23:23:40.538462 (+ 11422us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.538560 (+    98us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.143327 (+604767us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.174544 (+ 31217us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.182479 (+  7935us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.216765  7264 rpcz_store.cc:276] 0809 23:23:39.640151 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.487754 (+847603us) service_pool.cc:227] Handling call
0809 23:23:40.540647 (+ 52893us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.540768 (+   121us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.127055 (+586287us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.179914 (+ 52859us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.182411 (+  2497us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.209204  7566 raft_consensus.cc:374] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.219592  7566 raft_consensus.cc:729] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.198335  7360 tablet_service.cc:1467] Processing CreateTablet for tablet 58137ea1624f4b1db96f7a6b9ada7f68 (DEFAULT_TABLE table=test-workload [id=7ab647f19e0d4eebbf5d9b0415763a3a]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260809 23:23:41.208240  7256 rpcz_store.cc:276] 0809 23:23:39.667820 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.646005 (+978185us) service_pool.cc:227] Handling call
0809 23:23:40.722044 (+ 76039us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.722150 (+   106us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.106809 (+384659us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.179867 (+ 73058us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.180385 (+   518us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260809 23:23:41.207753  7583 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.220363  7566 consensus_queue.cc:260] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.222756  7566 ts_tablet_manager.cc:1428] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.002s
I20260809 23:23:41.225831  7566 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.237979  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.211022  7572 raft_consensus.cc:2781] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.213409  7559 raft_consensus.cc:348] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.240477  7559 raft_consensus.cc:374] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.215565  7365 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:41138 (request call id 34) took 1755 ms. Trace:
I20260809 23:23:41.215960  7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964bd1390e2d43b9ab381a795bdddedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.215456  7558 raft_consensus.cc:348] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.241196  7558 raft_consensus.cc:374] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.240728  7381 raft_consensus.cc:2454] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:41.241535  7561 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.241031  7365 rpcz_store.cc:276] 0809 23:23:39.459777 (+     0us) service_pool.cc:170] Inserting onto call queue
0809 23:23:40.680110 (+1220333us) service_pool.cc:227] Handling call
0809 23:23:40.681814 (+  1704us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0809 23:23:40.681906 (+    92us) ts_tablet_manager.cc:676] Creating new metadata...
0809 23:23:41.119747 (+437841us) tablet_metadata.cc:641] Metadata flushed
0809 23:23:41.209132 (+ 89385us) tablet_replica.cc:205] Creating consensus instance
0809 23:23:41.215259 (+  6127us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b58008ee398. stack: 00007fac20906033 00007fac20905f30 00007fac20264534 00007fac29f708b4 00007fac29f70849 00007fac2077cf11 00007fac20900851 00007fac20900fee 00007fac298198f9 00007fac2872c6a1 00007fac287f020b 00007fac287ebe2f 00007fac287ec5ed 00007fac2879d869 00007fac21d9fc44 00007fac21d9fbd2
0809 23:23:41.215508 (+   249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10216320}
I20260809 23:23:41.242287  7561 ts_tablet_manager.cc:1397] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.578s	user 0.038s	sys 0.022s
I20260809 23:23:41.240815  7559 raft_consensus.cc:729] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.242869  7124 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:41.241640  7558 raft_consensus.cc:729] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.243592  7559 consensus_queue.cc:260] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.239881  7583 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.245404  7583 raft_consensus.cc:2781] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.244820  7558 consensus_queue.cc:260] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.245487  7561 raft_consensus.cc:348] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.244848  7572 raft_consensus.cc:686] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:41.246410  7561 raft_consensus.cc:374] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.246726  7561 raft_consensus.cc:729] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.245816  7583 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.247645  7561 consensus_queue.cc:260] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:23:41.251700  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:41.250174  7561 ts_tablet_manager.cc:1428] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.005s
I20260809 23:23:41.254755  7561 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.255723  7558 ts_tablet_manager.cc:1428] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.064s	user 0.008s	sys 0.001s
I20260809 23:23:41.258129  7558 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.248536  7583 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.247416  7572 consensus_queue.cc:237] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.262194  7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58137ea1624f4b1db96f7a6b9ada7f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.262305  7561 tablet_bootstrap.cc:654] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.281329  7362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29afbcdad59b414fa48484b70ff56499. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.282172  7361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f23623f6580149f19787b8924d48ff24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:23:41.287672  7581 raft_consensus.cc:482] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.288215  7581 raft_consensus.cc:504] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.292984  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.293676  7174 raft_consensus.cc:2454] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:41.294684  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.300885  7338 leader_election.cc:302] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:41.301745  7559 ts_tablet_manager.cc:1428] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.092s	user 0.011s	sys 0.016s
W20260809 23:23:41.304735  7339 leader_election.cc:341] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.305094  7559 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.299435  7581 leader_election.cc:288] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.306249  7581 raft_consensus.cc:2781] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.307143  7581 raft_consensus.cc:482] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.311700  7581 raft_consensus.cc:3037] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.324409  7558 tablet_bootstrap.cc:654] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.325851  7583 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.329649  7583 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.330986  7561 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.332916  7561 ts_tablet_manager.cc:1397] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.078s	user 0.012s	sys 0.015s
I20260809 23:23:41.330106  7027 catalog_manager.cc:5495] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:41.312288  7124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 69504.5us
I20260809 23:23:41.338145  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:41.338554  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:41.339231  7381 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.343219  7559 tablet_bootstrap.cc:654] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.344338  7561 raft_consensus.cc:348] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.327284  7581 raft_consensus.cc:504] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.345180  7561 raft_consensus.cc:374] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.345572  7561 raft_consensus.cc:729] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.346421  7566 tablet_bootstrap.cc:654] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.346503  7561 consensus_queue.cc:260] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.349097  7381 raft_consensus.cc:2454] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:41.351231  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
W20260809 23:23:41.357863  7339 leader_election.cc:341] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.352314  7581 leader_election.cc:288] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.353979  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.359160  7174 raft_consensus.cc:3037] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.361341  7561 ts_tablet_manager.cc:1428] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.002s
I20260809 23:23:41.355666  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:41.369760  7174 raft_consensus.cc:2454] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:41.372517  7561 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.372274  7338 leader_election.cc:302] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:41.374043  7581 raft_consensus.cc:2781] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
W20260809 23:23:41.355671  7124 leader_election.cc:341] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.376044  7124 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
W20260809 23:23:41.377424  7124 leader_election.cc:341] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.377612  7558 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.379148  7558 ts_tablet_manager.cc:1397] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.121s	user 0.013s	sys 0.006s
I20260809 23:23:41.383140  7558 raft_consensus.cc:348] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.384168  7558 raft_consensus.cc:374] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.384589  7558 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.386178  7561 tablet_bootstrap.cc:654] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.378059  7583 raft_consensus.cc:2781] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.387064  7583 raft_consensus.cc:686] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:41.374730  7581 raft_consensus.cc:686] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:41.386060  7558 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.390825  7581 consensus_queue.cc:237] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.390847  7583 consensus_queue.cc:237] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.398811  7566 tablet_bootstrap.cc:492] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.400071  7558 ts_tablet_manager.cc:1428] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.003s
I20260809 23:23:41.399394  7566 ts_tablet_manager.cc:1397] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.174s	user 0.019s	sys 0.020s
I20260809 23:23:41.401614  7558 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.403611  7559 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.404368  7559 ts_tablet_manager.cc:1397] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.100s	user 0.005s	sys 0.029s
I20260809 23:23:41.410518  7566 raft_consensus.cc:348] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.411711  7566 raft_consensus.cc:374] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.411954  7558 tablet_bootstrap.cc:654] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.412099  7566 raft_consensus.cc:729] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.413483  7030 catalog_manager.cc:5495] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.417085  7566 consensus_queue.cc:260] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.421980  7561 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.422688  7561 ts_tablet_manager.cc:1397] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.050s	user 0.017s	sys 0.010s
I20260809 23:23:41.423954  7559 raft_consensus.cc:348] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.424968  7559 raft_consensus.cc:374] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.425251  7559 raft_consensus.cc:729] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.424705  7025 catalog_manager.cc:5495] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.427706  7561 raft_consensus.cc:348] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.428773  7561 raft_consensus.cc:374] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.429186  7561 raft_consensus.cc:729] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.422013  7566 ts_tablet_manager.cc:1428] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.007s
I20260809 23:23:41.432226  7566 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.435155  7558 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.435807  7558 ts_tablet_manager.cc:1397] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.007s
I20260809 23:23:41.436769  7559 consensus_queue.cc:260] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.439415  7566 tablet_bootstrap.cc:654] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.439755  7559 ts_tablet_manager.cc:1428] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.034s	user 0.001s	sys 0.006s
I20260809 23:23:41.439244  7558 raft_consensus.cc:348] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.440269  7558 raft_consensus.cc:374] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.440665  7558 raft_consensus.cc:729] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.440840  7559 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.441537  7558 consensus_queue.cc:260] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.430570  7561 consensus_queue.cc:260] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.444824  7558 ts_tablet_manager.cc:1428] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.006s
I20260809 23:23:41.445878  7558 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.449797  7561 ts_tablet_manager.cc:1428] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.027s	user 0.010s	sys 0.000s
I20260809 23:23:41.451853  7561 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.454499  7558 tablet_bootstrap.cc:654] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.458197  7566 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.458859  7566 ts_tablet_manager.cc:1397] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.027s	user 0.007s	sys 0.004s
I20260809 23:23:41.465659  7561 tablet_bootstrap.cc:654] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.467666  7559 tablet_bootstrap.cc:654] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.469563  7558 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.470078  7558 ts_tablet_manager.cc:1397] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.024s	user 0.016s	sys 0.000s
I20260809 23:23:41.462671  7566 raft_consensus.cc:348] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.473584  7558 raft_consensus.cc:348] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.474108  7566 raft_consensus.cc:374] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.474298  7558 raft_consensus.cc:374] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.474450  7566 raft_consensus.cc:729] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.474632  7558 raft_consensus.cc:729] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.475625  7558 consensus_queue.cc:260] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.478248  7558 ts_tablet_manager.cc:1428] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260809 23:23:41.479472  7558 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.475320  7566 consensus_queue.cc:260] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.486308  7558 tablet_bootstrap.cc:654] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.487468  7566 ts_tablet_manager.cc:1428] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.001s
I20260809 23:23:41.492179  7566 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.492470  7581 raft_consensus.cc:482] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.493024  7581 raft_consensus.cc:504] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.499969  7581 leader_election.cc:288] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:41.501147  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:41.501897  7274 raft_consensus.cc:2454] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:41.503546  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.504263  7490 raft_consensus.cc:2454] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:41.503706  7579 raft_consensus.cc:482] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.505158  7561 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.505707  7561 ts_tablet_manager.cc:1397] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.054s	user 0.009s	sys 0.003s
I20260809 23:23:41.505468  7579 raft_consensus.cc:504] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.507614  7579 leader_election.cc:288] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.509125  7561 raft_consensus.cc:348] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.511296  7561 raft_consensus.cc:374] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.511292  7339 leader_election.cc:302] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:41.511837  7561 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.511605  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.511842  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf9f1e8974a54b8b8c1d3a790e28a058" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.512475  7581 raft_consensus.cc:2781] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.513680  7581 raft_consensus.cc:482] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.513939  7174 raft_consensus.cc:2379] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 1: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
I20260809 23:23:41.514034  7581 raft_consensus.cc:3037] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.516201  7437 leader_election.cc:302] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.517391  7579 raft_consensus.cc:3037] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.518002  7617 raft_consensus.cc:482] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.518592  7617 raft_consensus.cc:504] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.520129  7559 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.520702  7559 ts_tablet_manager.cc:1397] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.080s	user 0.014s	sys 0.008s
I20260809 23:23:41.520747  7617 leader_election.cc:288] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:41.521588  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73788985062f47c397a6b02115c97a1e" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.522243  7174 raft_consensus.cc:2454] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:41.523576  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73788985062f47c397a6b02115c97a1e" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.523980  7559 raft_consensus.cc:348] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.524809  7338 leader_election.cc:302] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:41.525651  7617 raft_consensus.cc:2781] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.526382  7617 raft_consensus.cc:482] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.526741  7617 raft_consensus.cc:3037] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.527653  7558 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.528203  7558 ts_tablet_manager.cc:1397] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.049s	user 0.015s	sys 0.006s
I20260809 23:23:41.524267  7490 raft_consensus.cc:2454] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:41.531093  7558 raft_consensus.cc:348] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.531957  7558 raft_consensus.cc:374] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.513396  7561 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.532276  7558 raft_consensus.cc:729] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.533150  7579 raft_consensus.cc:2726] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.531744  7581 raft_consensus.cc:504] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.533304  7558 consensus_queue.cc:260] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.525050  7559 raft_consensus.cc:374] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.518121  7566 tablet_bootstrap.cc:654] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.535866  7559 raft_consensus.cc:729] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.536535  7558 ts_tablet_manager.cc:1428] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260809 23:23:41.537009  7559 consensus_queue.cc:260] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.537750  7561 ts_tablet_manager.cc:1428] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.004s
I20260809 23:23:41.539619  7572 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.540144  7572 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.541996  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:41.542239  7572 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.542590  7274 raft_consensus.cc:3037] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.544531  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.547768  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:41.550385  7227 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:41.551263  7227 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.551681  7227 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.552012  7617 raft_consensus.cc:504] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.553359  7274 raft_consensus.cc:2454] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:41.554602  7338 leader_election.cc:302] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:41.555612  7558 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.556213  7572 raft_consensus.cc:2726] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.557957  7617 leader_election.cc:288] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:41.558537  7617 raft_consensus.cc:2781] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.559008  7617 raft_consensus.cc:686] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:41.559836  7617 consensus_queue.cc:237] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.560207  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73788985062f47c397a6b02115c97a1e" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:41.560810  7490 raft_consensus.cc:3037] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.568138  7558 tablet_bootstrap.cc:654] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.571033  7581 leader_election.cc:288] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:41.572587  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1632dcfee20a4fc0a39347412a4dcc42" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:41.572798  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73788985062f47c397a6b02115c97a1e" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.573277  7488 raft_consensus.cc:3037] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.573563  7174 raft_consensus.cc:3037] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.575289  7559 ts_tablet_manager.cc:1428] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.000s
I20260809 23:23:41.576445  7559 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.582441  7488 raft_consensus.cc:2454] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:41.582707  7559 tablet_bootstrap.cc:654] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.583788  7561 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.583707  7490 raft_consensus.cc:2454] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:41.590746  7339 leader_election.cc:302] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:41.591987  7561 tablet_bootstrap.cc:654] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.593384  7174 raft_consensus.cc:2454] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:41.591817  7581 raft_consensus.cc:2781] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.596474  7581 raft_consensus.cc:686] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:41.597612  7581 consensus_queue.cc:237] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.610601  7558 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.611366  7566 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.611382  7558 ts_tablet_manager.cc:1397] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.072s	user 0.019s	sys 0.003s
I20260809 23:23:41.612136  7566 ts_tablet_manager.cc:1397] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.120s	user 0.024s	sys 0.001s
I20260809 23:23:41.618829  7558 raft_consensus.cc:348] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.619609  7558 raft_consensus.cc:374] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.619892  7558 raft_consensus.cc:729] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.620031  7559 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.620561  7558 consensus_queue.cc:260] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.622799  7558 ts_tablet_manager.cc:1428] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260809 23:23:41.620539  7559 ts_tablet_manager.cc:1397] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.003s
I20260809 23:23:41.624089  7648 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.624586  7648 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.622102  7566 raft_consensus.cc:348] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.627039  7566 raft_consensus.cc:374] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.626740  7559 raft_consensus.cc:348] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.627364  7566 raft_consensus.cc:729] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.627553  7559 raft_consensus.cc:374] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.627919  7559 raft_consensus.cc:729] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.628631  7558 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.628147  7566 consensus_queue.cc:260] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.629928  7027 catalog_manager.cc:5495] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.630913  7566 ts_tablet_manager.cc:1428] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.002s
I20260809 23:23:41.630591  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.631938  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.632226  7566 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.632649  7174 raft_consensus.cc:2454] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:41.634773  7437 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
W20260809 23:23:41.636864  7437 leader_election.cc:341] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.637620  7579 raft_consensus.cc:2781] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.638051  7579 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.638397  7579 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.639124  7559 consensus_queue.cc:260] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.642019  7566 tablet_bootstrap.cc:654] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.626619  7648 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.645658  7559 ts_tablet_manager.cc:1428] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260809 23:23:41.647612  7583 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.647759  7572 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.648136  7583 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.648401  7572 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.650187  7583 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.650427  7572 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.650285  7579 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.650745  7604 raft_consensus.cc:482] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.651317  7604 raft_consensus.cc:504] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.652917  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.653326  7604 leader_election.cc:288] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.653680  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:41.654063  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:41.654780  7274 raft_consensus.cc:2454] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:41.655277  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.655619  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.655977  7490 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:41.656009  7558 tablet_bootstrap.cc:654] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.666671  7228 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
W20260809 23:23:41.668291  7124 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.668627  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0400132440374a1c88fa8fb7d7c5a6b4" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:41.669494  7227 leader_election.cc:341] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.669955  7174 raft_consensus.cc:2454] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:41.670611  7124 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:41.671270  7572 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.671819  7572 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.672163  7604 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.672303  7572 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.672724  7604 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.675832  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:41.676332  7604 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.676431  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0400132440374a1c88fa8fb7d7c5a6b4" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.677109  7583 raft_consensus.cc:2781] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.677578  7583 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.678040  7583 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.679029  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.680161  7382 raft_consensus.cc:2379] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:41.683909  7227 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.685487  7227 leader_election.cc:302] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:41.677377  7381 raft_consensus.cc:2454] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
W20260809 23:23:41.671880  7437 leader_election.cc:341] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.687587  7604 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.694840  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.695544  7566 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.695508  7174 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.695127  7027 catalog_manager.cc:5495] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.696166  7566 ts_tablet_manager.cc:1397] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.064s	user 0.025s	sys 0.026s
I20260809 23:23:41.691928  7579 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.699239  7566 raft_consensus.cc:348] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.699971  7566 raft_consensus.cc:374] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.700237  7566 raft_consensus.cc:729] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.700958  7566 consensus_queue.cc:260] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.703408  7174 raft_consensus.cc:2454] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:41.704725  7437 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:41.707372  7561 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.707902  7561 ts_tablet_manager.cc:1397] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.138s	user 0.021s	sys 0.011s
I20260809 23:23:41.708580  7566 ts_tablet_manager.cc:1428] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260809 23:23:41.709118  7579 raft_consensus.cc:2781] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.709643  7579 raft_consensus.cc:686] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:41.710431  7566 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.710695  7559 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.711192  7579 consensus_queue.cc:237] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.716050  7561 raft_consensus.cc:348] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.717059  7561 raft_consensus.cc:374] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.721195  7572 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.722432  7583 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.724066  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:41.724682  7490 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.725899  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
W20260809 23:23:41.727325  7124 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.727895  7583 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.717361  7561 raft_consensus.cc:729] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.730253  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:41.731127  7274 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.732101  7566 tablet_bootstrap.cc:654] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.733011  7572 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.734531  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.735543  7572 raft_consensus.cc:2781] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.735819  7604 raft_consensus.cc:2726] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:23:41.736789  7227 leader_election.cc:341] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.738000  7572 raft_consensus.cc:482] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.739588  7572 raft_consensus.cc:3037] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.743546  7490 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:41.745082  7561 consensus_queue.cc:260] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.747377  7228 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.748559  7604 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.748867  7561 ts_tablet_manager.cc:1428] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260809 23:23:41.749204  7604 raft_consensus.cc:686] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:41.749956  7561 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.750164  7604 consensus_queue.cc:237] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.755178  7559 tablet_bootstrap.cc:654] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.756846  7561 tablet_bootstrap.cc:654] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.758172  7274 raft_consensus.cc:2454] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:41.763387  7124 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:41.765431  7583 raft_consensus.cc:2781] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.766069  7583 raft_consensus.cc:686] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:41.767207  7558 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.769088  7572 raft_consensus.cc:504] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.768962  7027 catalog_manager.cc:5495] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.767694  7558 ts_tablet_manager.cc:1397] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.144s	user 0.009s	sys 0.005s
I20260809 23:23:41.771121  7572 leader_election.cc:288] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:41.773654  7558 raft_consensus.cc:348] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.775552  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0400132440374a1c88fa8fb7d7c5a6b4" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:41.775864  7558 raft_consensus.cc:374] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.776724  7558 raft_consensus.cc:729] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.776203  7174 raft_consensus.cc:3037] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.777611  7558 consensus_queue.cc:260] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:23:41.779359  7227 consensus_peers.cc:487] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:41.780771  7558 ts_tablet_manager.cc:1428] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260809 23:23:41.783622  7558 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.785346  7174 raft_consensus.cc:2454] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:41.774523  7566 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.775789  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0400132440374a1c88fa8fb7d7c5a6b4" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:41.787047  7381 raft_consensus.cc:3037] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.787084  7566 ts_tablet_manager.cc:1397] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.077s	user 0.011s	sys 0.012s
I20260809 23:23:41.767683  7583 consensus_queue.cc:237] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.790833  7566 raft_consensus.cc:348] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.792152  7566 raft_consensus.cc:374] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.792541  7566 raft_consensus.cc:729] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.793129  7227 leader_election.cc:302] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:41.793253  7566 consensus_queue.cc:260] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.794559  7381 raft_consensus.cc:2454] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:41.795928  7572 raft_consensus.cc:2781] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:41.796890  7561 tablet_bootstrap.cc:492] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.796700  7572 raft_consensus.cc:686] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:41.797420  7561 ts_tablet_manager.cc:1397] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.048s	user 0.014s	sys 0.000s
I20260809 23:23:41.797888  7572 consensus_queue.cc:237] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.800189  7561 raft_consensus.cc:348] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.800879  7561 raft_consensus.cc:374] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.801141  7561 raft_consensus.cc:729] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.801791  7561 consensus_queue.cc:260] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.802632  7027 catalog_manager.cc:5495] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.805181  7559 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.806052  7559 ts_tablet_manager.cc:1397] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.096s	user 0.015s	sys 0.001s
I20260809 23:23:41.812595  7663 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.813182  7663 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.815486  7663 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:41.817481  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.818818  7558 tablet_bootstrap.cc:654] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:41.819052  7228 leader_election.cc:341] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.820299  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:41.821502  7227 leader_election.cc:341] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:41.821997  7227 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:41.822878  7566 ts_tablet_manager.cc:1428] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.001s
I20260809 23:23:41.810393  7559 raft_consensus.cc:348] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.823246  7604 raft_consensus.cc:2726] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:41.823529  7559 raft_consensus.cc:374] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.824015  7030 catalog_manager.cc:5495] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:41.829028  7566 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:41.824055  7559 raft_consensus.cc:729] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.832134  7561 ts_tablet_manager.cc:1428] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.001s
I20260809 23:23:41.835263  7561 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.833926  7559 consensus_queue.cc:260] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:41.839521  7559 ts_tablet_manager.cc:1428] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.005s
I20260809 23:23:41.840657  7559 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:41.845530  7558 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.846587  7558 ts_tablet_manager.cc:1397] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.002s
I20260809 23:23:41.845920  7604 consensus_queue.cc:1035] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:41.849948  7561 tablet_bootstrap.cc:654] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.849622  7558 raft_consensus.cc:348] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.851658  7558 raft_consensus.cc:374] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.852061  7558 raft_consensus.cc:729] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.852697  7558 consensus_queue.cc:260] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.856246  7583 consensus_queue.cc:1035] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:41.868224  7558 ts_tablet_manager.cc:1428] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260809 23:23:41.869313  7558 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:41.870769  7566 tablet_bootstrap.cc:654] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.877912  7559 tablet_bootstrap.cc:654] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.881878  7030 catalog_manager.cc:5495] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:41.889384  7617 consensus_queue.cc:1035] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:41.915238  7558 tablet_bootstrap.cc:654] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:41.928134  7561 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:41.928783  7561 ts_tablet_manager.cc:1397] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.094s	user 0.008s	sys 0.011s
I20260809 23:23:41.932114  7561 raft_consensus.cc:348] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.932952  7561 raft_consensus.cc:374] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.933277  7561 raft_consensus.cc:729] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.934031  7561 consensus_queue.cc:260] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.936473  7566 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:41.937098  7566 ts_tablet_manager.cc:1397] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.108s	user 0.018s	sys 0.000s
I20260809 23:23:41.940492  7559 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:41.951654  7572 raft_consensus.cc:482] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:41.952198  7572 raft_consensus.cc:504] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.954198  7572 leader_election.cc:288] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:41.954285  7558 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:41.955482  7617 consensus_queue.cc:1035] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:41.955875  7558 ts_tablet_manager.cc:1397] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.087s	user 0.016s	sys 0.011s
I20260809 23:23:41.957571  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:41.968786  7381 raft_consensus.cc:2454] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:41.957664  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:41.970194  7227 leader_election.cc:302] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:41.948813  7583 consensus_queue.cc:1035] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260809 23:23:41.971241  7558 raft_consensus.cc:348] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.972179  7558 raft_consensus.cc:374] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.973320  7490 raft_consensus.cc:2454] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:41.943199  7561 ts_tablet_manager.cc:1428] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260809 23:23:41.941321  7566 raft_consensus.cc:348] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:41.974696  7558 raft_consensus.cc:729] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.975082  7566 raft_consensus.cc:374] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.941272  7559 ts_tablet_manager.cc:1397] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.101s	user 0.012s	sys 0.006s
I20260809 23:23:41.971203  7572 raft_consensus.cc:2781] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:41.975391  7566 raft_consensus.cc:729] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.976951  7558 consensus_queue.cc:260] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:41.979152  7561 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:41.976545  7572 raft_consensus.cc:482] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:41.986352  7572 raft_consensus.cc:3037] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:41.996323  7559 raft_consensus.cc:348] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:41.997330  7559 raft_consensus.cc:374] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:41.997620  7559 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:41.999886  7559 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.002079  7559 ts_tablet_manager.cc:1428] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.000s
I20260809 23:23:42.003160  7558 ts_tablet_manager.cc:1428] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
I20260809 23:23:42.003235  7559 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:42.004309  7558 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:42.009119  7566 consensus_queue.cc:260] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.010128  7579 raft_consensus.cc:482] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.010861  7579 raft_consensus.cc:504] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.011523  7566 ts_tablet_manager.cc:1428] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.074s	user 0.011s	sys 0.000s
I20260809 23:23:42.012274  7566 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:42.013274  7579 leader_election.cc:288] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:42.015363  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.015918  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.016018  7275 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:42.016613  7561 tablet_bootstrap.cc:654] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.016713  7174 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:42.017757  7437 leader_election.cc:302] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.018601  7579 raft_consensus.cc:2781] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.019037  7579 raft_consensus.cc:482] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.019433  7579 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.019774  7572 raft_consensus.cc:504] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.021926  7572 leader_election.cc:288] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:42.022696  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:42.023312  7381 raft_consensus.cc:3037] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.033783  7617 consensus_queue.cc:1035] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.041358  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce32c0f77b0c4572a2e2bf042e987cc5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.042074  7488 raft_consensus.cc:3037] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.045075  7381 raft_consensus.cc:2454] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:42.047268  7227 leader_election.cc:302] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.048565  7663 raft_consensus.cc:2781] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.049161  7663 raft_consensus.cc:686] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:42.050509  7663 consensus_queue.cc:237] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.054128  7566 tablet_bootstrap.cc:654] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.057848  7579 raft_consensus.cc:504] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.061290  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.062048  7274 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.058374  7617 raft_consensus.cc:482] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.063678  7617 raft_consensus.cc:504] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.066730  7617 leader_election.cc:288] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.068490  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.077304  7572 raft_consensus.cc:482] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.074353  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:42.078176  7488 raft_consensus.cc:2454] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:42.078640  7173 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.073832  7617 consensus_queue.cc:1035] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 23:23:42.077803  7572 raft_consensus.cc:504] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.068079  7559 tablet_bootstrap.cc:654] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.066896  7558 tablet_bootstrap.cc:654] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.095466  7572 leader_election.cc:288] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:42.068935  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6db5f7d4fa4aa3973cc96d9c5fbdec" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.096073  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.074430  7579 leader_election.cc:288] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:42.089394  7173 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
W20260809 23:23:42.097545  7228 leader_election.cc:341] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.098065  7338 leader_election.cc:302] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.098632  7437 leader_election.cc:302] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:42.096958  7649 consensus_queue.cc:1035] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:42.099788  7579 raft_consensus.cc:2781] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.100128  7649 raft_consensus.cc:2726] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.100548  7579 raft_consensus.cc:686] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:42.101529  7579 consensus_queue.cc:237] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.101989  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:42.103272  7227 leader_election.cc:341] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.103720  7227 leader_election.cc:302] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:42.104874  7572 raft_consensus.cc:2726] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.129287  7561 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.129808  7561 ts_tablet_manager.cc:1397] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.153s	user 0.017s	sys 0.015s
I20260809 23:23:42.130110  7274 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:42.110567  7027 catalog_manager.cc:5495] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:42.134888  7648 consensus_queue.cc:1035] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.133334  7561 raft_consensus.cc:348] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.139842  7561 raft_consensus.cc:374] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.140098  7561 raft_consensus.cc:729] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.140770  7617 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.140821  7561 consensus_queue.cc:260] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.141490  7617 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.143567  7561 ts_tablet_manager.cc:1428] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260809 23:23:42.145022  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.145521  7559 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:42.143616  7617 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:42.146551  7559 ts_tablet_manager.cc:1397] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.144s	user 0.016s	sys 0.003s
I20260809 23:23:42.147004  7561 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.150977  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.154495  7339 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.159034  7581 raft_consensus.cc:2726] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.155181  7566 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:42.163116  7566 ts_tablet_manager.cc:1397] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.151s	user 0.017s	sys 0.006s
I20260809 23:23:42.166138  7581 consensus_queue.cc:1035] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.157618  7558 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:42.166968  7559 raft_consensus.cc:348] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.167721  7559 raft_consensus.cc:374] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.168114  7559 raft_consensus.cc:729] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.166190  7566 raft_consensus.cc:348] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.168867  7566 raft_consensus.cc:374] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.167461  7558 ts_tablet_manager.cc:1397] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.163s	user 0.013s	sys 0.006s
I20260809 23:23:42.169705  7566 raft_consensus.cc:729] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.170519  7566 consensus_queue.cc:260] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.171144  7648 raft_consensus.cc:482] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.172219  7559 consensus_queue.cc:260] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.174525  7648 raft_consensus.cc:504] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.176937  7648 leader_election.cc:288] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.178165  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.180048  7274 raft_consensus.cc:2454] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:42.179924  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:42.181159  7437 leader_election.cc:302] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.181938  7559 ts_tablet_manager.cc:1428] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20260809 23:23:42.177255  7558 raft_consensus.cc:348] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
W20260809 23:23:42.184038  7437 leader_election.cc:341] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.184150  7558 raft_consensus.cc:374] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.184592  7558 raft_consensus.cc:729] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.185305  7648 raft_consensus.cc:2781] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.185708  7648 raft_consensus.cc:482] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.185530  7558 consensus_queue.cc:260] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.186050  7648 raft_consensus.cc:3037] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.186347  7566 ts_tablet_manager.cc:1428] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260809 23:23:42.187391  7566 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:42.192914  7381 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.192651  7561 tablet_bootstrap.cc:654] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.193822  7613 consensus_queue.cc:1035] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:42.202191  7566 tablet_bootstrap.cc:654] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.211246  7559 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:42.212790  7561 tablet_bootstrap.cc:492] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.213301  7561 ts_tablet_manager.cc:1397] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.069s	user 0.009s	sys 0.002s
I20260809 23:23:42.216130  7561 raft_consensus.cc:348] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.216732  7561 raft_consensus.cc:374] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.216962  7561 raft_consensus.cc:729] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.217568  7561 consensus_queue.cc:260] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.218425  7648 raft_consensus.cc:504] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.220288  7712 consensus_queue.cc:1035] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:42.221302  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.222033  7275 raft_consensus.cc:3037] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.230196  7275 raft_consensus.cc:2454] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:42.231652  7437 leader_election.cc:302] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.232873  7712 raft_consensus.cc:2781] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.233649  7712 raft_consensus.cc:686] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:42.234745  7712 consensus_queue.cc:237] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.222470  7561 ts_tablet_manager.cc:1428] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260809 23:23:42.238350  7561 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.241225  7558 ts_tablet_manager.cc:1428] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.072s	user 0.006s	sys 0.001s
I20260809 23:23:42.242050  7558 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:42.244568  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
W20260809 23:23:42.247409  7437 leader_election.cc:341] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.248610  7566 tablet_bootstrap.cc:492] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:42.249081  7566 ts_tablet_manager.cc:1397] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.062s	user 0.008s	sys 0.005s
I20260809 23:23:42.250041  7172 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.251806  7559 tablet_bootstrap.cc:654] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.251969  7566 raft_consensus.cc:348] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.252776  7566 raft_consensus.cc:374] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.253031  7566 raft_consensus.cc:729] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.253777  7566 consensus_queue.cc:260] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.257198  7027 catalog_manager.cc:5495] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:42.263383  7561 tablet_bootstrap.cc:654] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.269395  7572 consensus_queue.cc:1035] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.279141  7558 tablet_bootstrap.cc:654] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.281538  7566 ts_tablet_manager.cc:1428] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.001s
I20260809 23:23:42.282873  7566 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:42.259229  7648 leader_election.cc:288] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
W20260809 23:23:42.301512  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:42.306027  7380 raft_consensus.cc:3037] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.315286  7581 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.316031  7581 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.318889  7648 raft_consensus.cc:482] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.319799  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.319569  7648 raft_consensus.cc:504] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.323904  7581 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.326752  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.333096  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:42.333647  7572 consensus_queue.cc:1035] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:42.334156  7377 raft_consensus.cc:2454] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:42.331708  7338 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.341423  7437 leader_election.cc:302] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:42.342984  7712 raft_consensus.cc:2781] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.343539  7712 raft_consensus.cc:482] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.344074  7712 raft_consensus.cc:3037] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.347225  7581 raft_consensus.cc:2726] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.349172  7648 leader_election.cc:288] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.351426  7572 raft_consensus.cc:482] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.353091  7572 raft_consensus.cc:504] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.364239  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:42.365140  7377 raft_consensus.cc:2454] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:42.356236  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.367264  7227 leader_election.cc:302] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.369622  7663 raft_consensus.cc:2781] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.370085  7663 raft_consensus.cc:482] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.370477  7663 raft_consensus.cc:3037] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.368289  7381 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.396432  7561 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.397002  7561 ts_tablet_manager.cc:1397] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.159s	user 0.017s	sys 0.003s
I20260809 23:23:42.406155  7558 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:42.407388  7558 ts_tablet_manager.cc:1397] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.165s	user 0.051s	sys 0.064s
I20260809 23:23:42.405964  7663 raft_consensus.cc:504] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.410255  7275 raft_consensus.cc:2379] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:42.410758  7561 raft_consensus.cc:348] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.411442  7663 leader_election.cc:288] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:42.411654  7561 raft_consensus.cc:374] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.412184  7561 raft_consensus.cc:729] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.413498  7561 consensus_queue.cc:260] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.406139  7712 raft_consensus.cc:504] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.420120  7613 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.420243  7712 leader_election.cc:288] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.420712  7613 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.422274  7561 ts_tablet_manager.cc:1428] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.001s
I20260809 23:23:42.423403  7561 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.423324  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.425372  7583 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.425841  7583 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.427310  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:42.427879  7583 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.403661  7604 raft_consensus.cc:482] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.429188  7604 raft_consensus.cc:504] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.430524  7561 tablet_bootstrap.cc:654] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.427943  7377 raft_consensus.cc:3037] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.433123  7558 raft_consensus.cc:348] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.433911  7558 raft_consensus.cc:374] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.434170  7558 raft_consensus.cc:729] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.434576  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.434815  7558 consensus_queue.cc:260] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.435699  7488 raft_consensus.cc:2379] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:42.422765  7613 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.441301  7604 leader_election.cc:288] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.438627  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.446959  7490 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:42.451211  7126 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:42.454156  7559 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:42.454321  7558 ts_tablet_manager.cc:1428] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.002s
I20260809 23:23:42.454866  7559 ts_tablet_manager.cc:1397] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.244s	user 0.016s	sys 0.004s
I20260809 23:23:42.455722  7558 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:42.452315  7583 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:42.459264  7583 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.459059  7559 raft_consensus.cc:348] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.459970  7559 raft_consensus.cc:374] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.460271  7559 raft_consensus.cc:729] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.461527  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:42.461741  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.459980  7583 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:42.463104  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:42.461095  7559 consensus_queue.cc:260] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.472888  7566 tablet_bootstrap.cc:654] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.473935  7559 ts_tablet_manager.cc:1428] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260809 23:23:42.474895  7559 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:42.476629  7028 catalog_manager.cc:5495] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:42.483916  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.484582  7487 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:42.444764  7663 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.041s
I20260809 23:23:42.492918  7613 consensus_queue.cc:1035] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.490722  7604 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:23:42.487072  7126 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:42.496803  7613 raft_consensus.cc:2781] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.497154  7613 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.497557  7558 tablet_bootstrap.cc:654] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.497489  7613 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.500442  7561 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.500949  7561 ts_tablet_manager.cc:1397] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.078s	user 0.031s	sys 0.014s
I20260809 23:23:42.503695  7561 raft_consensus.cc:348] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.504238  7572 leader_election.cc:288] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:42.487896  7377 raft_consensus.cc:2454] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:42.506191  7437 leader_election.cc:302] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:42.508247  7712 raft_consensus.cc:2781] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.508863  7712 raft_consensus.cc:686] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:42.488318  7583 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.511415  7712 consensus_queue.cc:237] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.512605  7583 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.504400  7561 raft_consensus.cc:374] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.518595  7561 raft_consensus.cc:729] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.519825  7561 consensus_queue.cc:260] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.524957  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.528292  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.528561  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9f47812f9fe462abdd14952f3aeb72b" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.529225  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.529996  7275 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:42.530041  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.535576  7561 ts_tablet_manager.cc:1428] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.002s
I20260809 23:23:42.539934  7561 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.543176  7613 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.547372  7173 raft_consensus.cc:1226] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Rejecting Update request from peer ec555693f39b444f95a05ba0bd6b37e7 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 23:23:42.547426  7648 consensus_queue.cc:1035] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:23:42.549566  7227 leader_election.cc:341] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.550891  7559 tablet_bootstrap.cc:654] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.552008  7227 leader_election.cc:302] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:42.553509  7572 raft_consensus.cc:2726] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.553356  7663 consensus_queue.cc:1046] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.555841  7663 raft_consensus.cc:3032] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:23:42.560199  7663 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:42.560988  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.561090  7663 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.558820  7583 raft_consensus.cc:482] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.561647  7487 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.562148  7583 raft_consensus.cc:504] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.563448  7561 tablet_bootstrap.cc:654] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.563264  7025 catalog_manager.cc:5495] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:42.565104  7663 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:42.564265  7583 leader_election.cc:288] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.558058  7566 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:42.567360  7566 ts_tablet_manager.cc:1397] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.285s	user 0.021s	sys 0.010s
I20260809 23:23:42.570685  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:42.553352  7227 leader_election.cc:341] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:42.572124  7126 leader_election.cc:341] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.572567  7227 leader_election.cc:302] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.574210  7487 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:42.575248  7124 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:42.575790  7613 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.576388  7604 raft_consensus.cc:2726] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.577663  7126 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:42.576347  7613 raft_consensus.cc:2726] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:23:42.579787  7694 raft_consensus.cc:2781] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.580310  7694 raft_consensus.cc:686] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:42.583271  7694 consensus_queue.cc:237] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.588054  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.589787  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.590423  7273 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.593489  7558 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:42.595501  7566 raft_consensus.cc:348] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.596583  7566 raft_consensus.cc:374] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.596765  7558 ts_tablet_manager.cc:1397] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.141s	user 0.015s	sys 0.011s
I20260809 23:23:42.606221  7566 raft_consensus.cc:729] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
W20260809 23:23:42.602137  7124 leader_election.cc:341] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.607151  7124 leader_election.cc:302] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:42.607443  7566 consensus_queue.cc:260] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.608741  7583 raft_consensus.cc:2726] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.615317  7561 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.616382  7561 ts_tablet_manager.cc:1397] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.077s	user 0.015s	sys 0.000s
I20260809 23:23:42.617756  7566 ts_tablet_manager.cc:1428] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20260809 23:23:42.614539  7558 raft_consensus.cc:348] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.623804  7558 raft_consensus.cc:374] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.625684  7558 raft_consensus.cc:729] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.619108  7561 raft_consensus.cc:348] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.626408  7561 raft_consensus.cc:374] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.624428  7566 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:42.626699  7561 raft_consensus.cc:729] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.627440  7561 consensus_queue.cc:260] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.631809  7558 consensus_queue.cc:260] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.632738  7579 consensus_queue.cc:1035] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:42.636816  7561 ts_tablet_manager.cc:1428] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.002s
I20260809 23:23:42.638994  7566 tablet_bootstrap.cc:654] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.641988  7561 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.653226  7561 tablet_bootstrap.cc:654] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.644968  7558 ts_tablet_manager.cc:1428] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.001s
I20260809 23:23:42.645217  7273 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:42.655687  7558 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:42.662088  7559 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:42.662752  7559 ts_tablet_manager.cc:1397] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.188s	user 0.023s	sys 0.026s
I20260809 23:23:42.667217  7559 raft_consensus.cc:348] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.668157  7559 raft_consensus.cc:374] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.668485  7559 raft_consensus.cc:729] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.669718  7559 consensus_queue.cc:260] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.675453  7027 catalog_manager.cc:5495] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:42.682636  7663 consensus_queue.cc:1035] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.685006  7649 raft_consensus.cc:482] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.685657  7558 tablet_bootstrap.cc:654] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.685513  7649 raft_consensus.cc:504] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.692159  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.693099  7273 raft_consensus.cc:2379] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:42.693599  7730 consensus_queue.cc:1035] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260809 23:23:42.696822  7649 leader_election.cc:288] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:42.697490  7380 raft_consensus.cc:3037] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.697896  7648 consensus_queue.cc:1035] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.700132  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e279157f9a0a4fb696c9bc01ae212b96" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.701824  7339 leader_election.cc:302] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:42.704298  7559 ts_tablet_manager.cc:1428] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260809 23:23:42.705298  7559 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:42.717399  7566 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:42.718111  7566 ts_tablet_manager.cc:1397] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.094s	user 0.011s	sys 0.018s
I20260809 23:23:42.738866  7581 raft_consensus.cc:2726] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.751998  7566 raft_consensus.cc:348] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.752751  7566 raft_consensus.cc:374] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.753018  7566 raft_consensus.cc:729] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.753293  7712 consensus_queue.cc:1035] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:42.753721  7566 consensus_queue.cc:260] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.758744  7559 tablet_bootstrap.cc:654] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.765873  7561 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:42.771095  7566 ts_tablet_manager.cc:1428] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260809 23:23:42.773018  7566 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:42.771376  7561 ts_tablet_manager.cc:1397] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.130s	user 0.032s	sys 0.003s
I20260809 23:23:42.780107  7583 raft_consensus.cc:482] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.784832  7663 raft_consensus.cc:482] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.783536  7583 raft_consensus.cc:504] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.786378  7663 raft_consensus.cc:504] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.789016  7583 leader_election.cc:288] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.792228  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.793286  7273 raft_consensus.cc:2454] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:42.793637  7561 raft_consensus.cc:348] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.794449  7561 raft_consensus.cc:374] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.794780  7561 raft_consensus.cc:729] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.796509  7561 consensus_queue.cc:260] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.802188  7124 leader_election.cc:302] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:42.809666  7663 leader_election.cc:288] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:42.792215  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.814359  7487 raft_consensus.cc:2454] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:42.815397  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:42.816041  7488 raft_consensus.cc:2454] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:42.817201  7228 leader_election.cc:302] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:42.818688  7572 raft_consensus.cc:2781] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.819164  7572 raft_consensus.cc:482] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.819542  7572 raft_consensus.cc:3037] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.821535  7712 raft_consensus.cc:482] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.813642  7583 raft_consensus.cc:2781] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.822244  7583 raft_consensus.cc:482] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.822048  7712 raft_consensus.cc:504] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.826531  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.822649  7583 raft_consensus.cc:3037] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.827839  7566 tablet_bootstrap.cc:654] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:42.830204  7559 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:42.830705  7559 ts_tablet_manager.cc:1397] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.126s	user 0.006s	sys 0.024s
I20260809 23:23:42.832273  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.833209  7274 raft_consensus.cc:2454] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:42.834613  7579 raft_consensus.cc:482] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.835266  7579 raft_consensus.cc:504] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.836248  7561 ts_tablet_manager.cc:1428] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.061s	user 0.006s	sys 0.001s
I20260809 23:23:42.837809  7579 leader_election.cc:288] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.840049  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.840727  7437 leader_election.cc:302] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
W20260809 23:23:42.842039  7437 leader_election.cc:341] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.842556  7572 raft_consensus.cc:504] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.846556  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:42.848274  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.848493  7559 raft_consensus.cc:348] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.848632  7712 leader_election.cc:288] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.848923  7488 raft_consensus.cc:3037] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.849457  7712 raft_consensus.cc:2781] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:42.849864  7712 raft_consensus.cc:482] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:42.849573  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:42.850243  7712 raft_consensus.cc:3037] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:42.820254  7558 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:42.854112  7558 ts_tablet_manager.cc:1397] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.199s	user 0.020s	sys 0.005s
I20260809 23:23:42.854687  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:42.856043  7437 leader_election.cc:341] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:42.857837  7437 leader_election.cc:341] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:42.858191  7712 raft_consensus.cc:504] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:42.858542  7488 raft_consensus.cc:2454] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:42.858392  7437 leader_election.cc:302] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.846580  7561 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:42.865803  7648 raft_consensus.cc:2726] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:42.859576  7572 leader_election.cc:288] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:42.917881  7228 leader_election.cc:302] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:42.864938  7559 raft_consensus.cc:374] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.860560  7663 raft_consensus.cc:482] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.862843  7558 raft_consensus.cc:348] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.860308  7583 raft_consensus.cc:504] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:42.974812  7558 raft_consensus.cc:374] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:42.975334  7558 raft_consensus.cc:729] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.974581  7663 raft_consensus.cc:504] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:42.976578  7558 consensus_queue.cc:260] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:23:42.972662  7228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 55461.4us
I20260809 23:23:42.977399  7583 leader_election.cc:288] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:42.978108  7663 leader_election.cc:288] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:42.958328  7763 raft_consensus.cc:482] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:42.968212  7566 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:42.973379  7559 raft_consensus.cc:729] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:42.978837  7173 raft_consensus.cc:2379] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:42.957644  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:42.979103  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.981215  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:42.982473  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:42.984002  7558 ts_tablet_manager.cc:1428] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.125s	user 0.008s	sys 0.000s
I20260809 23:23:42.983959  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f13e532e7304ece940fbe319522775f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:42.993839  7488 raft_consensus.cc:2379] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:42.990424  7763 raft_consensus.cc:504] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.989854  7559 consensus_queue.cc:260] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:42.996778  7274 raft_consensus.cc:3037] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:42.997081  7437 leader_election.cc:341] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.002134  7559 ts_tablet_manager.cc:1428] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.171s	user 0.005s	sys 0.003s
I20260809 23:23:43.004592  7712 leader_election.cc:288] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:43.003376  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:42.999562  7730 raft_consensus.cc:2781] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:42.998868  7763 leader_election.cc:288] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.005669  7558 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.010030  7566 ts_tablet_manager.cc:1397] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.237s	user 0.038s	sys 0.015s
I20260809 23:23:43.001370  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.007715  7613 raft_consensus.cc:482] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.017473  7488 raft_consensus.cc:2454] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:43.017673  7613 raft_consensus.cc:504] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.019073  7339 leader_election.cc:302] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:43.009538  7648 consensus_queue.cc:1035] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260809 23:23:43.020103  7763 raft_consensus.cc:2781] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.020498  7763 raft_consensus.cc:482] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.020833  7763 raft_consensus.cc:3037] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.020987  7613 leader_election.cc:288] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:43.015086  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.008720  7730 raft_consensus.cc:686] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:43.026799  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.023990  7730 consensus_queue.cc:237] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:23:43.029310  7126 leader_election.cc:341] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:43.012703  7227 leader_election.cc:341] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:43.031217  7227 leader_election.cc:341] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.035454  7227 leader_election.cc:302] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.036572  7712 consensus_queue.cc:1035] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 23:23:43.033974  7338 leader_election.cc:341] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.039536  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.015833  7559 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:43.043442  7763 raft_consensus.cc:504] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.044262  7558 tablet_bootstrap.cc:654] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:43.046641  7124 leader_election.cc:341] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.047175  7124 leader_election.cc:302] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:43.047405  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:43.048126  7488 raft_consensus.cc:3037] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.034778  7566 raft_consensus.cc:348] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.053640  7566 raft_consensus.cc:374] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.054643  7566 raft_consensus.cc:729] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.063038  7613 raft_consensus.cc:2726] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.066107  7561 tablet_bootstrap.cc:654] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.056432  7572 raft_consensus.cc:2726] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.055532  7763 leader_election.cc:288] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.068186  7566 consensus_queue.cc:260] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.072921  7488 raft_consensus.cc:2454] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:43.073652  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.074435  7339 leader_election.cc:302] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
W20260809 23:23:43.075726  7338 leader_election.cc:341] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.083199  7559 tablet_bootstrap.cc:654] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.086668  7581 raft_consensus.cc:2781] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.088797  7648 raft_consensus.cc:482] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.090538  7648 raft_consensus.cc:504] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.093317  7558 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.093871  7558 ts_tablet_manager.cc:1397] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.089s	user 0.016s	sys 0.014s
I20260809 23:23:43.094542  7581 raft_consensus.cc:686] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:43.095867  7581 consensus_queue.cc:237] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.099858  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.103148  7124 leader_election.cc:302] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
W20260809 23:23:43.103895  7437 leader_election.cc:341] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.104465  7613 raft_consensus.cc:2726] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.107722  7648 leader_election.cc:288] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.112989  7613 consensus_queue.cc:1035] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:43.108771  7566 ts_tablet_manager.cc:1428] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.091s	user 0.003s	sys 0.007s
I20260809 23:23:43.116513  7558 raft_consensus.cc:348] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.117333  7558 raft_consensus.cc:374] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.117594  7558 raft_consensus.cc:729] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.119045  7274 raft_consensus.cc:2454] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:43.119638  7613 consensus_queue.cc:1035] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:43.122394  7566 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.118263  7558 consensus_queue.cc:260] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.122550  7437 leader_election.cc:302] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:43.126173  7558 ts_tablet_manager.cc:1428] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.001s
I20260809 23:23:43.127526  7558 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.128692  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:43.133502  7437 leader_election.cc:341] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.134037  7437 leader_election.cc:302] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.127188  7027 catalog_manager.cc:5495] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:43.135108  7579 raft_consensus.cc:2726] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.132439  7648 raft_consensus.cc:2781] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.149071  7558 tablet_bootstrap.cc:654] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.149725  7648 raft_consensus.cc:686] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:43.151732  7648 consensus_queue.cc:237] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.156957  7561 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.171345  7561 ts_tablet_manager.cc:1397] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.325s	user 0.018s	sys 0.005s
I20260809 23:23:43.175361  7566 tablet_bootstrap.cc:654] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.182076  7559 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:43.182616  7559 ts_tablet_manager.cc:1397] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.167s	user 0.017s	sys 0.012s
I20260809 23:23:43.185750  7559 raft_consensus.cc:348] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.186512  7559 raft_consensus.cc:374] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.187100  7559 raft_consensus.cc:729] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.190429  7559 consensus_queue.cc:260] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.192087  7561 raft_consensus.cc:348] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.193061  7561 raft_consensus.cc:374] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.193337  7561 raft_consensus.cc:729] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.194146  7561 consensus_queue.cc:260] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.194604  7558 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.195343  7558 ts_tablet_manager.cc:1397] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.068s	user 0.013s	sys 0.010s
I20260809 23:23:43.199697  7558 raft_consensus.cc:348] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.200170  7558 raft_consensus.cc:374] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.200449  7558 raft_consensus.cc:729] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.204118  7561 ts_tablet_manager.cc:1428] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260809 23:23:43.207677  7581 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.208226  7581 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.204813  7558 consensus_queue.cc:260] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.211720  7561 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:43.216863  7558 ts_tablet_manager.cc:1428] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.002s
I20260809 23:23:43.225220  7617 raft_consensus.cc:482] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.225834  7559 ts_tablet_manager.cc:1428] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.000s
I20260809 23:23:43.228097  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:43.231724  7339 leader_election.cc:341] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.225970  7617 raft_consensus.cc:504] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.239118  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.239416  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.240526  7275 raft_consensus.cc:2454] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
W20260809 23:23:43.241312  7338 leader_election.cc:341] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.242193  7581 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.242367  7338 leader_election.cc:302] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.237849  7617 leader_election.cc:288] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.243597  7617 raft_consensus.cc:2781] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.244622  7617 raft_consensus.cc:482] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.245841  7617 raft_consensus.cc:3037] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.246984  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.253546  7566 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.254106  7566 ts_tablet_manager.cc:1397] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.132s	user 0.013s	sys 0.012s
I20260809 23:23:43.257272  7566 raft_consensus.cc:348] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.257992  7566 raft_consensus.cc:374] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.258287  7566 raft_consensus.cc:729] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.259121  7566 consensus_queue.cc:260] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.260686  7581 raft_consensus.cc:482] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.260787  7712 raft_consensus.cc:482] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.261181  7581 raft_consensus.cc:504] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.261283  7712 raft_consensus.cc:504] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.261756  7566 ts_tablet_manager.cc:1428] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260809 23:23:43.263001  7566 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.262107  7028 catalog_manager.cc:5495] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:43.263641  7581 leader_election.cc:288] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
W20260809 23:23:43.265156  7338 leader_election.cc:341] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.269104  7712 leader_election.cc:288] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.268185  7338 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.272051  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.275184  7763 raft_consensus.cc:2726] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.275269  7172 raft_consensus.cc:2454] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:43.276866  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.277632  7273 raft_consensus.cc:2454] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:43.279439  7437 leader_election.cc:302] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:43.280293  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.281472  7579 raft_consensus.cc:2781] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.281929  7579 raft_consensus.cc:482] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.282356  7579 raft_consensus.cc:3037] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:43.286109  7338 leader_election.cc:341] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.287073  7561 tablet_bootstrap.cc:654] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.286886  7338 leader_election.cc:302] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.288421  7763 raft_consensus.cc:2781] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.288863  7763 raft_consensus.cc:482] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.289196  7763 raft_consensus.cc:3037] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.292436  7566 tablet_bootstrap.cc:654] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.293910  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.311509  7558 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.303489  7579 raft_consensus.cc:504] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.297302  7763 raft_consensus.cc:504] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.317265  7559 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:43.319482  7437 leader_election.cc:341] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.327678  7030 catalog_manager.cc:5495] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:43.332500  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.330911  7579 leader_election.cc:288] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.330947  7763 leader_election.cc:288] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.339512  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:43.342314  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:43.343384  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.343664  7275 raft_consensus.cc:3037] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.344000  7173 raft_consensus.cc:3037] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:43.346777  7437 leader_election.cc:341] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:43.354533  7338 leader_election.cc:341] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.358493  7558 tablet_bootstrap.cc:654] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.362869  7173 raft_consensus.cc:2454] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:43.368561  7559 tablet_bootstrap.cc:654] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.389611  7566 tablet_bootstrap.cc:492] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.389793  7558 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.390347  7558 ts_tablet_manager.cc:1397] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.123s	user 0.002s	sys 0.011s
I20260809 23:23:43.390177  7566 ts_tablet_manager.cc:1397] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.127s	user 0.015s	sys 0.006s
I20260809 23:23:43.393702  7566 raft_consensus.cc:348] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.394443  7566 raft_consensus.cc:374] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.390156  7437 leader_election.cc:302] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:43.394713  7566 raft_consensus.cc:729] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.395354  7579 raft_consensus.cc:2781] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.395875  7579 raft_consensus.cc:686] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:43.396792  7558 raft_consensus.cc:348] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.396740  7579 consensus_queue.cc:237] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.397516  7558 raft_consensus.cc:374] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.397800  7558 raft_consensus.cc:729] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.395486  7566 consensus_queue.cc:260] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.400504  7566 ts_tablet_manager.cc:1428] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260809 23:23:43.401798  7566 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.363431  7617 raft_consensus.cc:504] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.386267  7561 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.406862  7572 raft_consensus.cc:482] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.407733  7572 raft_consensus.cc:504] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.411265  7572 leader_election.cc:288] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.412119  7561 ts_tablet_manager.cc:1397] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.207s	user 0.017s	sys 0.036s
I20260809 23:23:43.414575  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.415678  7173 raft_consensus.cc:2454] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:43.415253  7561 raft_consensus.cc:348] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.416260  7561 raft_consensus.cc:374] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.416618  7561 raft_consensus.cc:729] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.417536  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.417454  7561 consensus_queue.cc:260] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.418416  7488 raft_consensus.cc:2454] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:43.398439  7558 consensus_queue.cc:260] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.421291  7558 ts_tablet_manager.cc:1428] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260809 23:23:43.422220  7227 leader_election.cc:302] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:43.423172  7561 ts_tablet_manager.cc:1428] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:23:43.423252  7572 raft_consensus.cc:2781] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.423758  7572 raft_consensus.cc:482] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.424119  7572 raft_consensus.cc:3037] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.424474  7561 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:43.426968  7558 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.430508  7566 tablet_bootstrap.cc:654] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:43.432214  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:43.369498  7275 raft_consensus.cc:2454] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:43.435266  7338 leader_election.cc:302] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.435891  7617 leader_election.cc:288] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.437227  7792 raft_consensus.cc:2781] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.444905  7792 raft_consensus.cc:686] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:43.446031  7792 consensus_queue.cc:237] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.447597  7566 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.448268  7566 ts_tablet_manager.cc:1397] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.000s
I20260809 23:23:43.438364  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.451472  7566 raft_consensus.cc:348] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.452183  7566 raft_consensus.cc:374] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.452435  7566 raft_consensus.cc:729] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
W20260809 23:23:43.452546  7338 leader_election.cc:341] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.453130  7566 consensus_queue.cc:260] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.453615  7572 raft_consensus.cc:504] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.455869  7566 ts_tablet_manager.cc:1428] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260809 23:23:43.456404  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:43.456930  7572 leader_election.cc:288] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.437655  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:43.444999  7561 tablet_bootstrap.cc:654] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.457579  7275 raft_consensus.cc:3037] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.457947  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18336f4a3644c798d5f8bb715908fd0" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.458685  7173 raft_consensus.cc:3037] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.457110  7488 raft_consensus.cc:3037] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.459921  7027 catalog_manager.cc:5495] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:43.437631  7558 tablet_bootstrap.cc:654] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.467346  7173 raft_consensus.cc:2454] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:43.457180  7566 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.469046  7227 leader_election.cc:302] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:43.470773  7730 raft_consensus.cc:2781] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.471298  7730 raft_consensus.cc:686] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:43.471966  7488 raft_consensus.cc:2454] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:43.472155  7730 consensus_queue.cc:237] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.474826  7275 raft_consensus.cc:2454] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:43.476264  7338 leader_election.cc:302] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.477216  7617 raft_consensus.cc:2781] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.477689  7617 raft_consensus.cc:686] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:43.478506  7617 consensus_queue.cc:237] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.482700  7732 raft_consensus.cc:482] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.483238  7561 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.483867  7561 ts_tablet_manager.cc:1397] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.003s
I20260809 23:23:43.483361  7732 raft_consensus.cc:504] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.486114  7732 leader_election.cc:288] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.486706  7561 raft_consensus.cc:348] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.487491  7561 raft_consensus.cc:374] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.487746  7561 raft_consensus.cc:729] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.488310  7559 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:43.488430  7561 consensus_queue.cc:260] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.488891  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.489964  7488 raft_consensus.cc:2454] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:43.491505  7228 leader_election.cc:302] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:43.492847  7732 raft_consensus.cc:2781] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.493251  7732 raft_consensus.cc:482] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.493642  7732 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.494221  7559 ts_tablet_manager.cc:1397] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.177s	user 0.014s	sys 0.024s
I20260809 23:23:43.490342  7566 tablet_bootstrap.cc:654] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.497148  7559 raft_consensus.cc:348] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.497799  7559 raft_consensus.cc:374] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.498055  7559 raft_consensus.cc:729] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.491370  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:43.499604  7227 leader_election.cc:341] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.498661  7559 consensus_queue.cc:260] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.510293  7030 catalog_manager.cc:5495] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:43.525624  7558 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.526306  7558 ts_tablet_manager.cc:1397] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.100s	user 0.021s	sys 0.015s
I20260809 23:23:43.500077  7028 catalog_manager.cc:5495] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:43.530476  7579 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.499632  7561 ts_tablet_manager.cc:1428] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260809 23:23:43.531008  7579 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.532169  7561 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:43.522620  7732 raft_consensus.cc:504] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.533303  7579 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.506482  7559 ts_tablet_manager.cc:1428] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260809 23:23:43.535950  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.536249  7559 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:43.536164  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.536669  7380 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:43.529646  7025 catalog_manager.cc:5495] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
W20260809 23:23:43.537734  7437 leader_election.cc:341] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.538664  7437 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.539211  7561 tablet_bootstrap.cc:654] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.539830  7579 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.540236  7579 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.540591  7579 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.548521  7566 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.548242  7579 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.549173  7566 ts_tablet_manager.cc:1397] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.092s	user 0.026s	sys 0.017s
I20260809 23:23:43.551772  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:43.552582  7376 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.552577  7566 raft_consensus.cc:348] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.553427  7566 raft_consensus.cc:374] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.553726  7566 raft_consensus.cc:729] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.554514  7566 consensus_queue.cc:260] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.556493  7572 consensus_queue.cc:1035] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:43.560607  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:43.556493  7566 ts_tablet_manager.cc:1428] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:23:43.561334  7487 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.562417  7566 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.563030  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
W20260809 23:23:43.564522  7227 leader_election.cc:341] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.560734  7732 leader_election.cc:288] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.566897  7559 tablet_bootstrap.cc:654] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.579504  7617 raft_consensus.cc:482] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.580024  7617 raft_consensus.cc:504] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.576659  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:43.582309  7617 leader_election.cc:288] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:43.569108  7566 tablet_bootstrap.cc:654] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.529053  7558 raft_consensus.cc:348] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.587842  7712 raft_consensus.cc:482] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.600427  7712 raft_consensus.cc:504] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.602649  7712 leader_election.cc:288] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
W20260809 23:23:43.611430  7437 leader_election.cc:341] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.611492  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.613106  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:43.614526  7437 leader_election.cc:341] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:43.615345  7437 leader_election.cc:341] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.588896  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.592608  7380 raft_consensus.cc:3037] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.599974  7558 raft_consensus.cc:374] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.588378  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1578784c042f68a1e8d865c5583aa" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.616460  7558 raft_consensus.cc:729] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.616858  7275 raft_consensus.cc:2379] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:43.603111  7730 consensus_queue.cc:1035] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:43.617278  7558 consensus_queue.cc:260] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.619148  7566 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.619498  7558 ts_tablet_manager.cc:1428] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.093s	user 0.007s	sys 0.000s
I20260809 23:23:43.619620  7566 ts_tablet_manager.cc:1397] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.058s	user 0.010s	sys 0.004s
I20260809 23:23:43.620615  7558 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.622345  7566 raft_consensus.cc:348] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.623093  7566 raft_consensus.cc:374] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.623392  7566 raft_consensus.cc:729] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.624168  7566 consensus_queue.cc:260] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.626396  7558 tablet_bootstrap.cc:654] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.627583  7339 leader_election.cc:302] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:43.629727  7566 ts_tablet_manager.cc:1428] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.004s
I20260809 23:23:43.629454  7613 raft_consensus.cc:482] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.630543  7613 raft_consensus.cc:504] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.631208  7566 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.633005  7617 raft_consensus.cc:2726] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.635344  7559 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:43.635809  7559 ts_tablet_manager.cc:1397] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.100s	user 0.025s	sys 0.019s
I20260809 23:23:43.638823  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.638756  7559 raft_consensus.cc:348] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.639714  7559 raft_consensus.cc:374] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.639964  7566 tablet_bootstrap.cc:654] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.640009  7559 raft_consensus.cc:729] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.640935  7617 consensus_queue.cc:1035] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:43.641039  7559 consensus_queue.cc:260] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.641515  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "327e1d8704e64831b8ebda9aaf3710b5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.642339  7275 raft_consensus.cc:2379] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
I20260809 23:23:43.643669  7124 leader_election.cc:302] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:43.644220  7559 ts_tablet_manager.cc:1428] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:23:43.644572  7789 raft_consensus.cc:3037] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.644851  7561 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.648169  7561 ts_tablet_manager.cc:1397] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.116s	user 0.011s	sys 0.004s
I20260809 23:23:43.645641  7559 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:43.652768  7561 raft_consensus.cc:348] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.653761  7561 raft_consensus.cc:374] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.654060  7561 raft_consensus.cc:729] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.655102  7561 consensus_queue.cc:260] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.640133  7613 leader_election.cc:288] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:43.572729  7487 raft_consensus.cc:2454] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:43.573781  7376 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:43.657970  7561 ts_tablet_manager.cc:1428] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260809 23:23:43.659904  7228 leader_election.cc:302] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.666501  7732 raft_consensus.cc:2781] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.667197  7559 tablet_bootstrap.cc:654] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.667637  7561 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:43.672209  7566 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.672896  7566 ts_tablet_manager.cc:1397] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.042s	user 0.006s	sys 0.018s
I20260809 23:23:43.667129  7732 raft_consensus.cc:686] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:43.677213  7732 consensus_queue.cc:237] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.574746  7579 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.616060  7437 leader_election.cc:302] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:43.687577  7566 raft_consensus.cc:348] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.688721  7566 raft_consensus.cc:374] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.689721  7579 raft_consensus.cc:2726] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.685343  7789 raft_consensus.cc:2726] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.690135  7566 raft_consensus.cc:729] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
W20260809 23:23:43.691140  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 75836.1us
W20260809 23:23:43.697284  7338 consensus_peers.cc:487] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:43.698002  7437 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.701761  7617 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.701475  7566 consensus_queue.cc:260] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.707283  7559 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:43.707791  7559 ts_tablet_manager.cc:1397] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.063s	user 0.014s	sys 0.009s
I20260809 23:23:43.705292  7712 consensus_queue.cc:1035] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:23:43.708532  7566 ts_tablet_manager.cc:1428] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.035s	user 0.011s	sys 0.000s
I20260809 23:23:43.702271  7617 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.709549  7566 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.710912  7617 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.705078  7579 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:43.712069  7579 raft_consensus.cc:686] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:43.713299  7579 consensus_queue.cc:237] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.714095  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.717672  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:43.717837  7339 leader_election.cc:341] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:43.719043  7338 leader_election.cc:341] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.719067  7566 tablet_bootstrap.cc:654] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.719619  7338 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.720883  7617 raft_consensus.cc:2726] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.725234  7559 raft_consensus.cc:348] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.726203  7559 raft_consensus.cc:374] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.726541  7559 raft_consensus.cc:729] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.727349  7559 consensus_queue.cc:260] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.730104  7559 ts_tablet_manager.cc:1428] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20260809 23:23:43.731180  7559 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:43.736152  7820 raft_consensus.cc:482] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.736685  7820 raft_consensus.cc:504] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.737594  7558 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.738065  7558 ts_tablet_manager.cc:1397] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.118s	user 0.028s	sys 0.018s
I20260809 23:23:43.740331  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.740820  7558 raft_consensus.cc:348] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.741536  7558 raft_consensus.cc:374] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.741853  7558 raft_consensus.cc:729] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.738706  7820 leader_election.cc:288] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.751191  7558 consensus_queue.cc:260] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.753470  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10b54da326434a3398b667f5d2955e1c" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.756676  7172 raft_consensus.cc:2379] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:43.759457  7227 leader_election.cc:302] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.760717  7820 raft_consensus.cc:3037] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.766438  7561 tablet_bootstrap.cc:654] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.767273  7558 ts_tablet_manager.cc:1428] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.004s
I20260809 23:23:43.766305  7028 catalog_manager.cc:5495] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:43.776002  7558 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.779526  7820 raft_consensus.cc:2726] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.773695  7579 consensus_queue.cc:1035] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260809 23:23:43.791303  7559 tablet_bootstrap.cc:654] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.788501  7558 tablet_bootstrap.cc:654] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.794535  7579 raft_consensus.cc:482] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.806722  7566 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.807435  7566 ts_tablet_manager.cc:1397] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.098s	user 0.015s	sys 0.013s
I20260809 23:23:43.821661  7581 raft_consensus.cc:482] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.822383  7581 raft_consensus.cc:504] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.826484  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:43.828419  7339 leader_election.cc:341] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.831238  7789 raft_consensus.cc:482] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.831974  7789 raft_consensus.cc:504] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.817799  7566 raft_consensus.cc:348] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.833647  7566 raft_consensus.cc:374] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.832473  7581 leader_election.cc:288] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.834034  7566 raft_consensus.cc:729] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.839943  7566 consensus_queue.cc:260] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.841442  7789 leader_election.cc:288] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:43.813570  7579 raft_consensus.cc:504] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.845181  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.846118  7274 raft_consensus.cc:2379] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
I20260809 23:23:43.848600  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.852169  7558 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.852682  7558 ts_tablet_manager.cc:1397] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.079s	user 0.016s	sys 0.022s
I20260809 23:23:43.856479  7579 leader_election.cc:288] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.857543  7559 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:43.858040  7559 ts_tablet_manager.cc:1397] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.127s	user 0.020s	sys 0.011s
I20260809 23:23:43.860538  7558 raft_consensus.cc:348] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.861279  7558 raft_consensus.cc:374] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.861509  7558 raft_consensus.cc:729] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.862161  7558 consensus_queue.cc:260] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:43.863729  7558 ts_tablet_manager.cc:1428] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.004s
I20260809 23:23:43.864746  7558 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
W20260809 23:23:43.867434  7338 leader_election.cc:341] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.868260  7338 leader_election.cc:302] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:43.870661  7763 raft_consensus.cc:2726] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.872581  7558 tablet_bootstrap.cc:654] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.868075  7559 raft_consensus.cc:348] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.875429  7559 raft_consensus.cc:374] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.875888  7559 raft_consensus.cc:729] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.869079  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.876868  7559 consensus_queue.cc:260] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.884167  7559 ts_tablet_manager.cc:1428] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260809 23:23:43.881614  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cffdb27ac1684d2bac75f40ba93712ab" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.897053  7566 ts_tablet_manager.cc:1428] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.089s	user 0.005s	sys 0.003s
I20260809 23:23:43.898600  7566 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:43.903100  7581 raft_consensus.cc:482] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.908042  7559 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:43.911518  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.913156  7173 raft_consensus.cc:2454] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
W20260809 23:23:43.913411  7437 leader_election.cc:341] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.914081  7124 leader_election.cc:302] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:43.915210  7789 raft_consensus.cc:3037] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.917658  7581 raft_consensus.cc:504] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.920614  7030 catalog_manager.cc:5495] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:43.922984  7581 leader_election.cc:288] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.924548  7789 raft_consensus.cc:2726] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.932451  7566 tablet_bootstrap.cc:654] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.925539  7437 leader_election.cc:302] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:43.933283  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.934005  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.927431  7558 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:43.934796  7274 raft_consensus.cc:2454] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:43.935415  7558 ts_tablet_manager.cc:1397] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.071s	user 0.025s	sys 0.021s
I20260809 23:23:43.929919  7559 tablet_bootstrap.cc:654] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.925143  7561 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.935835  7579 consensus_queue.cc:1035] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:43.936301  7561 ts_tablet_manager.cc:1397] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.269s	user 0.012s	sys 0.041s
I20260809 23:23:43.934666  7648 raft_consensus.cc:2781] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.936856  7648 raft_consensus.cc:482] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.937239  7648 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.939036  7558 raft_consensus.cc:348] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.939761  7561 raft_consensus.cc:348] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.942592  7558 raft_consensus.cc:374] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.942616  7561 raft_consensus.cc:374] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.942999  7558 raft_consensus.cc:729] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.947748  7558 consensus_queue.cc:260] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.950121  7558 ts_tablet_manager.cc:1428] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260809 23:23:43.946666  7561 raft_consensus.cc:729] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.950939  7561 consensus_queue.cc:260] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.953321  7561 ts_tablet_manager.cc:1428] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.004s
I20260809 23:23:43.954754  7561 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:43.954881  7558 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:43.955528  7820 raft_consensus.cc:482] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.955992  7820 raft_consensus.cc:504] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:43.958079  7820 leader_election.cc:288] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:43.946118  7648 raft_consensus.cc:504] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:43.960649  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:43.961370  7561 tablet_bootstrap.cc:654] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.943827  7338 leader_election.cc:302] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:43.961977  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:43.962348  7581 raft_consensus.cc:2781] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260809 23:23:43.962514  7338 leader_election.cc:341] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.962613  7380 raft_consensus.cc:2454] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:43.964298  7227 leader_election.cc:302] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:43.961444  7648 leader_election.cc:288] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:43.962801  7581 raft_consensus.cc:482] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:43.970768  7581 raft_consensus.cc:3037] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.973754  7561 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:43.974324  7561 ts_tablet_manager.cc:1397] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.005s
I20260809 23:23:43.977190  7561 raft_consensus.cc:348] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.980624  7613 raft_consensus.cc:482] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.985562  7820 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:43.985316  7613 raft_consensus.cc:504] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:43.986043  7820 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:23:43.981765  7227 leader_election.cc:341] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.988583  7613 leader_election.cc:288] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:43.989296  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:43.981367  7579 consensus_queue.cc:1035] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260809 23:23:43.984222  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:43.989326  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:43.991731  7566 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:43.977869  7561 raft_consensus.cc:374] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:43.992228  7490 raft_consensus.cc:2379] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:43.981843  7572 raft_consensus.cc:2781] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:43.992388  7561 raft_consensus.cc:729] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:43.993455  7572 raft_consensus.cc:482] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:23:43.993495  7437 leader_election.cc:341] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:43.994201  7572 raft_consensus.cc:3037] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:43.994876  7820 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:43.980786  7558 tablet_bootstrap.cc:654] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:43.981344  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.001492  7572 raft_consensus.cc:504] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.002967  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.002422  7581 raft_consensus.cc:504] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
W20260809 23:23:44.004308  7228 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.002324  7172 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.005156  7124 leader_election.cc:302] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:44.006518  7613 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.010248  7579 raft_consensus.cc:482] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.010794  7579 raft_consensus.cc:504] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.013025  7579 leader_election.cc:288] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:44.004997  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.017887  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:44.018059  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.019655  7380 raft_consensus.cc:2454] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
W20260809 23:23:44.019781  7227 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.022562  7227 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.021787  7617 consensus_queue.cc:1035] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.023625  7172 raft_consensus.cc:2454] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:44.024034  7732 raft_consensus.cc:2726] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:43.987625  7559 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:44.025686  7559 ts_tablet_manager.cc:1397] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.137s	user 0.012s	sys 0.004s
W20260809 23:23:44.021274  7437 leader_election.cc:341] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.027439  7437 leader_election.cc:302] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.028597  7437 leader_election.cc:302] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:44.029683  7579 raft_consensus.cc:2781] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:44.030035  7579 raft_consensus.cc:482] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.030160  7648 raft_consensus.cc:2781] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.030422  7579 raft_consensus.cc:3037] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.030684  7648 raft_consensus.cc:686] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:44.031298  7581 leader_election.cc:288] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.031697  7648 consensus_queue.cc:237] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.035305  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:44.035900  7572 leader_election.cc:288] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:44.036006  7275 raft_consensus.cc:3037] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.038291  7613 raft_consensus.cc:2726] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.002825  7561 consensus_queue.cc:260] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.040145  7617 consensus_queue.cc:1035] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:44.003664  7566 ts_tablet_manager.cc:1397] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.105s	user 0.013s	sys 0.003s
I20260809 23:23:44.043558  7566 raft_consensus.cc:348] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.044214  7566 raft_consensus.cc:374] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.044505  7566 raft_consensus.cc:729] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.040633  7561 ts_tablet_manager.cc:1428] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.066s	user 0.006s	sys 0.002s
I20260809 23:23:44.047472  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.048280  7561 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:44.053660  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.054842  7566 consensus_queue.cc:260] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.028692  7559 raft_consensus.cc:348] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.060242  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:44.078615  7559 raft_consensus.cc:374] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.079177  7559 raft_consensus.cc:729] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.068332  7763 consensus_queue.cc:1035] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260809 23:23:44.062206  7771 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ec555693f39b444f95a05ba0bd6b37e7)
I20260809 23:23:44.079919  7559 consensus_queue.cc:260] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.080662  7771 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.081539  7380 raft_consensus.cc:3037] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.082691  7559 ts_tablet_manager.cc:1428] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.057s	user 0.008s	sys 0.000s
W20260809 23:23:44.084134  7338 leader_election.cc:341] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.090569  7572 consensus_queue.cc:1035] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.085724  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:44.091552  7273 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:44.094681  7437 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:44.066428  7617 consensus_queue.cc:1035] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260809 23:23:44.097256  7617 raft_consensus.cc:482] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.105719  7617 raft_consensus.cc:504] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.113063  7617 leader_election.cc:288] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.052821  7579 raft_consensus.cc:504] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.115857  7558 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:44.116340  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:44.123965  7559 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:44.061569  7227 leader_election.cc:341] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.129545  7558 ts_tablet_manager.cc:1397] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.175s	user 0.012s	sys 0.006s
I20260809 23:23:44.083986  7771 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.052771  7275 raft_consensus.cc:2454] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:44.136875  7272 raft_consensus.cc:2454] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:44.153600  7579 leader_election.cc:288] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:44.156390  7579 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:44.156899  7579 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election (detected failure of leader ec555693f39b444f95a05ba0bd6b37e7)
I20260809 23:23:44.156695  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:44.157421  7579 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:44.158151  7172 raft_consensus.cc:2379] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 2: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:44.090592  7380 raft_consensus.cc:2454] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:44.163594  7338 leader_election.cc:302] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
W20260809 23:23:44.137439  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 73994.4us
W20260809 23:23:44.138734  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 44172.6us
I20260809 23:23:44.180410  7572 consensus_queue.cc:1035] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:23:44.177230  7851 raft_consensus.cc:2781] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:44.187417  7566 ts_tablet_manager.cc:1428] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.146s	user 0.013s	sys 0.007s
I20260809 23:23:44.204674  7559 tablet_bootstrap.cc:654] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.204486  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.207093  7855 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.207631  7855 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.208648  7030 catalog_manager.cc:5495] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.210320  7855 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.192886  7572 raft_consensus.cc:482] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.211922  7572 raft_consensus.cc:504] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.214376  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.188500  7851 raft_consensus.cc:482] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.215389  7851 raft_consensus.cc:3037] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.216621  7852 raft_consensus.cc:482] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.217164  7852 raft_consensus.cc:504] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.218168  7561 tablet_bootstrap.cc:654] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.219300  7852 leader_election.cc:288] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.187651  7338 leader_election.cc:302] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
W20260809 23:23:44.219393  7339 leader_election.cc:341] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.222016  7613 raft_consensus.cc:482] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.222615  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.222481  7613 raft_consensus.cc:504] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.224465  7613 leader_election.cc:288] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
W20260809 23:23:44.224969  7339 leader_election.cc:341] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.225857  7852 raft_consensus.cc:2781] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.226377  7852 raft_consensus.cc:686] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:44.227393  7566 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:44.227288  7852 consensus_queue.cc:237] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.214774  7572 leader_election.cc:288] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:44.235270  7558 raft_consensus.cc:348] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.239212  7558 raft_consensus.cc:374] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.239507  7558 raft_consensus.cc:729] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.239754  7172 raft_consensus.cc:3037] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.240159  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.240285  7558 consensus_queue.cc:260] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.242112  7558 ts_tablet_manager.cc:1428] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.110s	user 0.005s	sys 0.000s
I20260809 23:23:44.279275  7558 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
W20260809 23:23:44.239766  7437 leader_election.cc:341] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.280735  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:44.280861  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:44.304076  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:44.304659  7864 log.cc:924] Time spent T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260809 23:23:44.307090  7789 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.307606  7789 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.309669  7789 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.311410  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.315304  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.323099  7227 leader_election.cc:302] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.325369  7851 raft_consensus.cc:504] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.328426  7851 leader_election.cc:288] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.329452  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
W20260809 23:23:44.330618  7338 leader_election.cc:341] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.330783  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:44.331116  7338 leader_election.cc:302] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:44.331542  7380 raft_consensus.cc:3037] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.332587  7856 raft_consensus.cc:2726] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.334606  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:44.335461  7339 leader_election.cc:341] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.335899  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0590a10d16ff4d12aa9edfc6985ebede" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.336517  7173 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.337566  7124 leader_election.cc:302] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.339385  7579 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.340659  7490 raft_consensus.cc:2396] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260809 23:23:44.340751  7820 consensus_queue.cc:1035] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:44.343592  7380 raft_consensus.cc:2454] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:44.346103  7579 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.360617  7732 raft_consensus.cc:2781] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 23:23:44.368202  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 45260.9us
W20260809 23:23:44.381438  7227 leader_election.cc:341] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.387739  7561 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:44.388298  7561 ts_tablet_manager.cc:1397] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.340s	user 0.033s	sys 0.028s
I20260809 23:23:44.388213  7613 raft_consensus.cc:2726] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.391891  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:44.392333  7227 leader_election.cc:341] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.392827  7227 leader_election.cc:302] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.393129  7272 raft_consensus.cc:2396] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260809 23:23:44.394371  7869 raft_consensus.cc:2726] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.374264  7732 raft_consensus.cc:686] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:44.399438  7771 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:23:44.396646  7030 catalog_manager.cc:5495] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:44.405855  7732 consensus_queue.cc:237] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.403529  7566 tablet_bootstrap.cc:654] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.410079  7437 leader_election.cc:302] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.414209  7771 raft_consensus.cc:2781] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.415012  7559 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:44.415584  7559 ts_tablet_manager.cc:1397] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.329s	user 0.026s	sys 0.057s
I20260809 23:23:44.415676  7561 raft_consensus.cc:348] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.416435  7561 raft_consensus.cc:374] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.416695  7561 raft_consensus.cc:729] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.417325  7561 consensus_queue.cc:260] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.418522  7613 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.419032  7771 raft_consensus.cc:686] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:44.419229  7613 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.419921  7771 consensus_queue.cc:237] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.421243  7613 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.378710  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:44.430022  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.432129  7820 consensus_queue.cc:1035] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.433264  7124 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:44.435088  7613 raft_consensus.cc:2726] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:23:44.440734  7025 catalog_manager.cc:5495] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.449213  7561 ts_tablet_manager.cc:1428] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.058s	user 0.004s	sys 0.003s
I20260809 23:23:44.449317  7566 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:44.447798  7559 raft_consensus.cc:348] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.449857  7566 ts_tablet_manager.cc:1397] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.223s	user 0.014s	sys 0.000s
I20260809 23:23:44.450045  7559 raft_consensus.cc:374] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.450335  7559 raft_consensus.cc:729] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.427338  7274 raft_consensus.cc:3037] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.451354  7559 consensus_queue.cc:260] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.453831  7561 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:44.457968  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.458666  7376 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:44.460079  7124 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.460813  7561 tablet_bootstrap.cc:654] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.466956  7613 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:44.467388  7613 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.467721  7613 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:44.468086  7559 ts_tablet_manager.cc:1428] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.052s	user 0.005s	sys 0.002s
I20260809 23:23:44.473047  7558 tablet_bootstrap.cc:654] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:44.447278  7437 leader_election.cc:341] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.473191  7559 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:44.463389  7566 raft_consensus.cc:348] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.477383  7566 raft_consensus.cc:374] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.477701  7566 raft_consensus.cc:729] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.479753  7566 consensus_queue.cc:260] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.484823  7613 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.488651  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:44.489241  7376 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:44.489297  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:44.493811  7613 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.514170  7613 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.527729  7566 ts_tablet_manager.cc:1428] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.078s	user 0.006s	sys 0.001s
I20260809 23:23:44.536713  7771 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:44.518988  7613 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.547596  7566 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
W20260809 23:23:44.544365  7437 leader_election.cc:341] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.553066  7559 tablet_bootstrap.cc:654] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.553675  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:44.555296  7126 leader_election.cc:341] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.555954  7613 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.557999  7873 raft_consensus.cc:482] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.558248  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.559016  7377 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:44.558733  7873 raft_consensus.cc:504] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:23:44.560973  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:44.562175  7566 tablet_bootstrap.cc:654] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.564828  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.565743  7490 raft_consensus.cc:2454] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:44.567337  7228 leader_election.cc:302] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:44.567768  7124 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.562482  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.568459  7820 raft_consensus.cc:2781] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:44.568866  7820 raft_consensus.cc:482] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.569227  7820 raft_consensus.cc:3037] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:44.569742  7227 leader_election.cc:341] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.570621  7873 leader_election.cc:288] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.571355  7437 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.576000  7376 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:23:44.577178  7124 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.577730  7820 raft_consensus.cc:504] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.578063  7789 raft_consensus.cc:2781] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:44.578362  7771 raft_consensus.cc:2726] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.578634  7789 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.579082  7789 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.580452  7558 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:44.580973  7558 ts_tablet_manager.cc:1397] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.332s	user 0.019s	sys 0.013s
I20260809 23:23:44.581970  7561 tablet_bootstrap.cc:492] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:44.582437  7561 ts_tablet_manager.cc:1397] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.129s	user 0.023s	sys 0.053s
I20260809 23:23:44.587081  7613 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:44.587294  7561 raft_consensus.cc:348] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.588088  7561 raft_consensus.cc:374] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.588361  7561 raft_consensus.cc:729] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.589056  7561 consensus_queue.cc:260] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.590322  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.590649  7561 ts_tablet_manager.cc:1428] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260809 23:23:44.591236  7172 raft_consensus.cc:2379] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 2: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:44.592072  7561 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:44.592875  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:44.555629  7025 catalog_manager.cc:5495] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.610608  7561 tablet_bootstrap.cc:654] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.615197  7856 raft_consensus.cc:482] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.615695  7856 raft_consensus.cc:504] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.617801  7272 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:44.617902  7856 leader_election.cc:288] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:44.619306  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:44.554118  7274 raft_consensus.cc:2454] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:44.594586  7613 raft_consensus.cc:686] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:44.625499  7613 consensus_queue.cc:237] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.625806  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.628063  7376 raft_consensus.cc:1226] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Rejecting Update request from peer a3cc3ec2dcd449de83f6c88e80d2d6de for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 23:23:44.630102  7487 raft_consensus.cc:3037] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.634559  7558 raft_consensus.cc:348] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.638293  7558 raft_consensus.cc:374] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.638655  7558 raft_consensus.cc:729] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.600883  7820 leader_election.cc:288] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.642292  7558 consensus_queue.cc:260] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.649302  7789 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.653887  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:44.654234  7338 leader_election.cc:302] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.655160  7617 raft_consensus.cc:2781] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.655632  7617 raft_consensus.cc:686] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:44.656553  7617 consensus_queue.cc:237] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.660094  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:44.660866  7171 raft_consensus.cc:2379] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:44.662712  7437 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:44.674825  7172 raft_consensus.cc:1226] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Rejecting Update request from peer a3cc3ec2dcd449de83f6c88e80d2d6de for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260809 23:23:44.665797  7579 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:44.679852  7579 raft_consensus.cc:686] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:44.683171  7338 leader_election.cc:302] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:44.683578  7487 raft_consensus.cc:2454] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:44.684696  7851 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:44.686412  7227 leader_election.cc:341] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.688417  7558 ts_tablet_manager.cc:1428] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.107s	user 0.012s	sys 0.000s
I20260809 23:23:44.690794  7228 leader_election.cc:302] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:44.697254  7869 raft_consensus.cc:2781] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.697752  7869 raft_consensus.cc:686] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:44.698551  7869 consensus_queue.cc:237] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.680756  7579 consensus_queue.cc:237] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.707804  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:44.709851  7377 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.706113  7027 catalog_manager.cc:5495] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:44.711496  7558 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:44.717924  7851 raft_consensus.cc:2726] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.717839  7377 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
W20260809 23:23:44.722453  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 54389.2us
I20260809 23:23:44.736804  7561 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:44.737313  7561 ts_tablet_manager.cc:1397] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.145s	user 0.009s	sys 0.004s
I20260809 23:23:44.738339  7566 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:44.738847  7566 ts_tablet_manager.cc:1397] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.194s	user 0.027s	sys 0.012s
I20260809 23:23:44.729187  7124 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:44.735669  7789 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.752455  7559 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:44.753134  7559 ts_tablet_manager.cc:1397] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.280s	user 0.058s	sys 0.040s
I20260809 23:23:44.755076  7789 raft_consensus.cc:2781] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:44.755498  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:44.758919  7028 catalog_manager.cc:5495] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.760356  7771 consensus_queue.cc:1046] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.763914  7789 raft_consensus.cc:686] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:44.764853  7789 consensus_queue.cc:237] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.760267  7561 raft_consensus.cc:348] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.765936  7559 raft_consensus.cc:348] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.766664  7559 raft_consensus.cc:374] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.767748  7566 raft_consensus.cc:348] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.768760  7566 raft_consensus.cc:374] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.769096  7566 raft_consensus.cc:729] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.772110  7566 consensus_queue.cc:260] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.767779  7025 catalog_manager.cc:5495] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 1 to 2, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 2 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.766884  7559 raft_consensus.cc:729] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.776166  7566 ts_tablet_manager.cc:1428] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20260809 23:23:44.779407  7566 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:44.780063  7558 tablet_bootstrap.cc:654] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.776152  7852 consensus_queue.cc:1035] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 23:23:44.753880  7579 consensus_queue.cc:1035] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260809 23:23:44.780550  7559 consensus_queue.cc:260] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
W20260809 23:23:44.783615  7338 consensus_peers.cc:487] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:44.798002  7566 tablet_bootstrap.cc:654] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.765864  7561 raft_consensus.cc:374] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.800135  7771 consensus_queue.cc:1225] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Peer ff5d35f70eb7411ba26062ea7d36ab15 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260809 23:23:44.801244  7857 consensus_queue.cc:1046] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.801502  7561 raft_consensus.cc:729] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.802474  7561 consensus_queue.cc:260] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.803464  7490 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.805907  7648 raft_consensus.cc:3032] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:23:44.806525  7648 raft_consensus.cc:729] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:44.807605  7648 consensus_queue.cc:260] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.809391  7648 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:44.814917  7566 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:44.815510  7566 ts_tablet_manager.cc:1397] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.004s
I20260809 23:23:44.820019  7566 raft_consensus.cc:348] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.820883  7566 raft_consensus.cc:374] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.821169  7566 raft_consensus.cc:729] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.821918  7566 consensus_queue.cc:260] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.823170  7559 ts_tablet_manager.cc:1428] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.001s
I20260809 23:23:44.824378  7566 ts_tablet_manager.cc:1428] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260809 23:23:44.825430  7566 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:44.827901  7559 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:44.832716  7566 tablet_bootstrap.cc:654] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:44.834790  7025 catalog_manager.cc:5495] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:44.835741  7490 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:44.842609  7561 ts_tablet_manager.cc:1428] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.105s	user 0.010s	sys 0.000s
I20260809 23:23:44.848201  7869 consensus_queue.cc:1035] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:44.857252  7579 consensus_queue.cc:1035] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:44.770527  7027 catalog_manager.cc:5495] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:44.859691  7901 raft_consensus.cc:482] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.867576  7901 raft_consensus.cc:504] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.875509  7901 leader_election.cc:288] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.886941  7561 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
W20260809 23:23:44.895529  7227 consensus_peers.cc:487] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:44.898842  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:44.899801  7274 raft_consensus.cc:2454] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:44.900947  7583 raft_consensus.cc:482] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.901532  7583 raft_consensus.cc:504] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:44.903632  7583 leader_election.cc:288] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.904318  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:44.901012  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:44.905311  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:44.905817  7124 leader_election.cc:341] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:44.908047  7126 leader_election.cc:341] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.908788  7126 leader_election.cc:302] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:44.910339  7789 raft_consensus.cc:2726] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.921811  7437 leader_election.cc:302] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:44.924280  7901 raft_consensus.cc:2781] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:44.925004  7901 raft_consensus.cc:482] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:44.925341  7901 raft_consensus.cc:3037] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.932762  7896 consensus_queue.cc:1035] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260809 23:23:44.937048  7437 leader_election.cc:341] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.937597  7851 raft_consensus.cc:482] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:44.938156  7851 raft_consensus.cc:504] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.941450  7851 leader_election.cc:288] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:44.942555  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:44.942879  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:44.946700  7559 tablet_bootstrap.cc:654] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:44.947463  7339 leader_election.cc:341] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:44.948770  7338 leader_election.cc:341] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:44.941334  7901 raft_consensus.cc:504] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:44.949291  7338 leader_election.cc:302] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:44.950115  7792 raft_consensus.cc:2726] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:44.950891  7901 leader_election.cc:288] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:44.959339  7558 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:44.959980  7558 ts_tablet_manager.cc:1397] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.249s	user 0.035s	sys 0.073s
I20260809 23:23:44.969877  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:44.971513  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:44.972131  7274 raft_consensus.cc:3037] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:44.974388  7558 raft_consensus.cc:348] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.975160  7558 raft_consensus.cc:374] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.975419  7558 raft_consensus.cc:729] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.976058  7558 consensus_queue.cc:260] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.978319  7558 ts_tablet_manager.cc:1428] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260809 23:23:44.982992  7558 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:44.984603  7559 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:44.985118  7559 ts_tablet_manager.cc:1397] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.157s	user 0.016s	sys 0.011s
I20260809 23:23:44.989291  7559 raft_consensus.cc:348] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.990288  7559 raft_consensus.cc:374] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.990600  7559 raft_consensus.cc:729] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.991288  7559 consensus_queue.cc:260] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:44.993391  7566 tablet_bootstrap.cc:492] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:44.993654  7559 ts_tablet_manager.cc:1428] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:23:44.994277  7566 ts_tablet_manager.cc:1397] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.169s	user 0.034s	sys 0.012s
I20260809 23:23:44.995270  7559 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:44.997279  7566 raft_consensus.cc:348] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.998013  7566 raft_consensus.cc:374] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:44.998327  7566 raft_consensus.cc:729] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:44.999040  7566 consensus_queue.cc:260] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:44.990279  7558 tablet_bootstrap.cc:654] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.009480  7792 raft_consensus.cc:482] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.010279  7559 tablet_bootstrap.cc:654] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.010056  7792 raft_consensus.cc:504] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.010898  7274 raft_consensus.cc:2454] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:45.012099  7792 leader_election.cc:288] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.015355  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:45.019505  7338 leader_election.cc:341] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.020031  7561 tablet_bootstrap.cc:654] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.024296  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.025074  7274 raft_consensus.cc:2454] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:45.026221  7437 leader_election.cc:302] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:45.026599  7338 leader_election.cc:302] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:45.027199  7896 raft_consensus.cc:2781] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.027911  7896 raft_consensus.cc:686] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:45.028761  7896 consensus_queue.cc:237] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.029891  7851 raft_consensus.cc:2781] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.030301  7851 raft_consensus.cc:482] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.030627  7851 raft_consensus.cc:3037] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.035610  7558 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.036146  7558 ts_tablet_manager.cc:1397] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.053s	user 0.005s	sys 0.007s
W20260809 23:23:45.040437  7437 leader_election.cc:341] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:45.041579  7437 consensus_peers.cc:487] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:45.041510  7851 raft_consensus.cc:504] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.045992  7559 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:45.046559  7559 ts_tablet_manager.cc:1397] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.010s
I20260809 23:23:45.046896  7901 raft_consensus.cc:482] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.047458  7901 raft_consensus.cc:504] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.050005  7901 leader_election.cc:288] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:45.001255  7566 ts_tablet_manager.cc:1428] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:23:45.052563  7566 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:45.052146  7027 catalog_manager.cc:5495] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:45.050021  7559 raft_consensus.cc:348] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.055002  7559 raft_consensus.cc:374] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.038767  7558 raft_consensus.cc:348] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.056104  7558 raft_consensus.cc:374] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.056391  7558 raft_consensus.cc:729] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.055311  7559 raft_consensus.cc:729] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.057113  7558 consensus_queue.cc:260] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.057240  7559 consensus_queue.cc:260] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.061970  7559 ts_tablet_manager.cc:1428] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260809 23:23:45.073721  7566 tablet_bootstrap.cc:654] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.075991  7558 ts_tablet_manager.cc:1428] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20260809 23:23:45.077406  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.078209  7274 raft_consensus.cc:2379] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 1: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
I20260809 23:23:45.078202  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:45.078576  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c057787ab114bba890199f5b36b37b7" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:45.079177  7272 raft_consensus.cc:3037] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.079264  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7beeffe6914835b84417315f44dd34" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:45.079854  7338 leader_election.cc:341] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.082013  7559 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:45.088707  7558 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:45.088562  7437 leader_election.cc:302] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.089643  7272 raft_consensus.cc:2454] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:45.075590  7851 leader_election.cc:288] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.090865  7338 leader_election.cc:302] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:45.091542  7648 raft_consensus.cc:3037] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.092032  7851 raft_consensus.cc:2781] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.092226  7561 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:45.092592  7851 raft_consensus.cc:686] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:45.092751  7561 ts_tablet_manager.cc:1397] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.231s	user 0.018s	sys 0.000s
I20260809 23:23:45.093627  7851 consensus_queue.cc:237] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.095547  7561 raft_consensus.cc:348] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.096239  7561 raft_consensus.cc:374] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.096494  7561 raft_consensus.cc:729] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.097100  7561 consensus_queue.cc:260] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.102293  7559 tablet_bootstrap.cc:654] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.102896  7648 raft_consensus.cc:2726] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:45.104450  7558 tablet_bootstrap.cc:654] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.104656  7272 raft_consensus.cc:1261] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:45.111914  7852 consensus_queue.cc:1035] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:23:45.115013  7648 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:23:45.112850  7561 ts_tablet_manager.cc:1428] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260809 23:23:45.120662  7561 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:45.122699  7558 tablet_bootstrap.cc:492] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.123386  7558 ts_tablet_manager.cc:1397] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.035s	user 0.014s	sys 0.010s
I20260809 23:23:45.126418  7558 raft_consensus.cc:348] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.131796  7558 raft_consensus.cc:374] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.132061  7558 raft_consensus.cc:729] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.132874  7558 consensus_queue.cc:260] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.135659  7558 ts_tablet_manager.cc:1428] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20260809 23:23:45.136814  7558 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:45.142063  7173 raft_consensus.cc:1261] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:45.142670  7558 tablet_bootstrap.cc:654] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.145643  7648 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.150964  7873 consensus_queue.cc:1035] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:45.152408  7566 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:45.152976  7566 ts_tablet_manager.cc:1397] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.101s	user 0.012s	sys 0.006s
I20260809 23:23:45.155267  7792 consensus_queue.cc:1035] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.157456  7851 raft_consensus.cc:482] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.158344  7851 raft_consensus.cc:504] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.162972  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:45.165031  7873 raft_consensus.cc:482] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.165578  7873 raft_consensus.cc:504] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.168249  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:45.168983  7377 raft_consensus.cc:2454] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:45.169245  7559 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:45.169734  7559 ts_tablet_manager.cc:1397] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.089s	user 0.012s	sys 0.000s
I20260809 23:23:45.170455  7227 leader_election.cc:302] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:45.162426  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9c4f8d91c504cdd8f1fd9dd8f707602" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.171386  7572 raft_consensus.cc:2781] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.171766  7572 raft_consensus.cc:482] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.172183  7338 leader_election.cc:302] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.172278  7572 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.173521  7792 raft_consensus.cc:2726] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:45.174834  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:45.176493  7228 leader_election.cc:341] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.177201  7873 leader_election.cc:288] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:45.180274  7789 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.180783  7789 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.182235  7572 raft_consensus.cc:504] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.185264  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:45.182767  7789 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.166002  7851 leader_election.cc:288] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:45.186138  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.186844  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:45.187541  7377 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.173277  7559 raft_consensus.cc:348] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.188167  7559 raft_consensus.cc:374] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.189541  7561 tablet_bootstrap.cc:654] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:45.189725  7124 leader_election.cc:341] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.186004  7487 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:45.188565  7559 raft_consensus.cc:729] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.194252  7559 consensus_queue.cc:260] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.196138  7566 raft_consensus.cc:348] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.197032  7566 raft_consensus.cc:374] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.199262  7030 catalog_manager.cc:5495] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:45.201284  7486 raft_consensus.cc:1261] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260809 23:23:45.201668  7486 pending_rounds.cc:76] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Aborting all ops after (but not including) 0
I20260809 23:23:45.201893  7486 pending_rounds.cc:97] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260809 23:23:45.202171  7486 raft_consensus.cc:2866] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260809 23:23:45.205646  7126 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.208983  7559 ts_tablet_manager.cc:1428] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260809 23:23:45.210358  7559 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:45.210458  7583 raft_consensus.cc:482] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.211045  7583 raft_consensus.cc:504] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.212559  7572 leader_election.cc:288] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:45.213203  7583 leader_election.cc:288] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:45.212244  7377 raft_consensus.cc:2454] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:45.213922  7583 raft_consensus.cc:2781] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.214505  7583 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.214857  7583 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.215598  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:45.216447  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7102c5cad94641438ac6c66721ce193c" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.218251  7559 tablet_bootstrap.cc:654] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.197340  7566 raft_consensus.cc:729] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.221351  7566 consensus_queue.cc:260] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.225311  7124 leader_election.cc:302] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.231278  7789 raft_consensus.cc:2726] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:45.225400  7227 leader_election.cc:302] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:45.233161  7572 raft_consensus.cc:2781] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.233721  7572 raft_consensus.cc:686] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:45.236267  7572 consensus_queue.cc:237] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.241276  7566 ts_tablet_manager.cc:1428] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.088s	user 0.008s	sys 0.000s
I20260809 23:23:45.224480  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:45.263396  7566 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:45.272332  7789 raft_consensus.cc:482] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.272877  7789 raft_consensus.cc:504] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.274920  7789 leader_election.cc:288] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.280679  7558 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.282166  7377 raft_consensus.cc:3037] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.299201  7558 ts_tablet_manager.cc:1397] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.163s	user 0.016s	sys 0.008s
I20260809 23:23:45.301889  7558 raft_consensus.cc:348] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.302563  7558 raft_consensus.cc:374] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260809 23:23:45.285358  7228 leader_election.cc:341] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.280964  7851 raft_consensus.cc:482] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.304211  7851 raft_consensus.cc:504] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.305080  7648 raft_consensus.cc:482] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.305594  7648 raft_consensus.cc:504] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.307413  7558 raft_consensus.cc:729] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.310583  7558 consensus_queue.cc:260] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.315227  7558 ts_tablet_manager.cc:1428] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260809 23:23:45.308482  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.322229  7272 raft_consensus.cc:2454] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:45.297061  7566 tablet_bootstrap.cc:654] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.306808  7851 leader_election.cc:288] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.293071  7583 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.325181  7583 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.317895  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:45.325368  7338 leader_election.cc:302] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:45.325932  7376 raft_consensus.cc:2454] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:45.326442  7792 raft_consensus.cc:2781] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.326833  7792 raft_consensus.cc:482] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.327342  7792 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.327967  7558 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:45.328379  7437 leader_election.cc:302] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:45.330758  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.331493  7272 raft_consensus.cc:2454] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:45.333371  7901 raft_consensus.cc:2781] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.333791  7901 raft_consensus.cc:482] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.334117  7901 raft_consensus.cc:3037] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.342718  7873 consensus_queue.cc:1035] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:45.307626  7648 leader_election.cc:288] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:45.355892  7558 tablet_bootstrap.cc:654] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.359650  7901 raft_consensus.cc:504] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.362907  7792 raft_consensus.cc:504] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.362401  7561 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:45.364055  7561 ts_tablet_manager.cc:1397] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.244s	user 0.022s	sys 0.045s
I20260809 23:23:45.366370  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:45.368232  7272 raft_consensus.cc:3037] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.368723  7901 leader_election.cc:288] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
W20260809 23:23:45.369403  7126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 163844us
I20260809 23:23:45.371268  7561 raft_consensus.cc:348] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.372489  7561 raft_consensus.cc:374] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.372759  7561 raft_consensus.cc:729] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.373363  7561 consensus_queue.cc:260] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.370633  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ff74f37a38496b8bf542b0df4d74f7" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:45.380395  7377 raft_consensus.cc:3037] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.391552  7561 ts_tablet_manager.cc:1428] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.001s
I20260809 23:23:45.386089  7583 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260809 23:23:45.402369  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:45.403155  7274 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.403453  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:45.404106  7490 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:45.408583  7378 raft_consensus.cc:1261] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260809 23:23:45.406580  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:45.409422  7486 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.411355  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.412750  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:45.413384  7275 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.417338  7934 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.424661  7171 raft_consensus.cc:3037] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.433311  7934 consensus_queue.cc:1035] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260809 23:23:45.394811  7792 leader_election.cc:288] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.441277  7561 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
W20260809 23:23:45.443043  7124 leader_election.cc:341] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.446336  7486 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:45.448236  7559 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:45.452195  7559 ts_tablet_manager.cc:1397] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.242s	user 0.020s	sys 0.005s
I20260809 23:23:45.465198  7559 raft_consensus.cc:348] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.465910  7559 raft_consensus.cc:374] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.466205  7559 raft_consensus.cc:729] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.466430  7274 raft_consensus.cc:2454] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:45.467896  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:45.472651  7126 leader_election.cc:302] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.473109  7275 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:45.474036  7934 raft_consensus.cc:2781] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.474433  7934 raft_consensus.cc:482] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.474838  7934 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.478196  7561 tablet_bootstrap.cc:654] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.474401  7124 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:45.481804  7558 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.482414  7558 ts_tablet_manager.cc:1397] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.155s	user 0.014s	sys 0.004s
I20260809 23:23:45.482481  7941 raft_consensus.cc:2781] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.484840  7941 raft_consensus.cc:686] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:45.482802  7559 consensus_queue.cc:260] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.485823  7941 consensus_queue.cc:237] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.481938  7938 consensus_queue.cc:1035] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260809 23:23:45.488260  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:45.490008  7566 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:45.490476  7566 ts_tablet_manager.cc:1397] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.227s	user 0.017s	sys 0.010s
I20260809 23:23:45.490556  7559 ts_tablet_manager.cc:1428] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.003s
I20260809 23:23:45.492194  7559 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:45.493074  7566 raft_consensus.cc:348] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.493775  7566 raft_consensus.cc:374] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.494057  7566 raft_consensus.cc:729] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.494148  7272 raft_consensus.cc:2454] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:45.494818  7566 consensus_queue.cc:260] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.495589  7437 leader_election.cc:302] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:45.496917  7377 raft_consensus.cc:2454] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:45.497169  7648 raft_consensus.cc:2781] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.497318  7566 ts_tablet_manager.cc:1428] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260809 23:23:45.497903  7648 raft_consensus.cc:686] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:45.498549  7566 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:45.498773  7648 consensus_queue.cc:237] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.502291  7025 catalog_manager.cc:5495] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:45.516842  7338 leader_election.cc:302] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
W20260809 23:23:45.530797  7338 leader_election.cc:341] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.535959  7934 raft_consensus.cc:504] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.537089  7851 raft_consensus.cc:2781] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:45.537662  7851 raft_consensus.cc:686] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:45.538506  7851 consensus_queue.cc:237] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.538985  7934 leader_election.cc:288] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:45.541244  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:45.553524  7028 catalog_manager.cc:5495] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:45.558408  7558 raft_consensus.cc:348] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.559224  7558 raft_consensus.cc:374] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.559540  7558 raft_consensus.cc:729] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.560420  7558 consensus_queue.cc:260] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.562222  7274 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.576123  7561 tablet_bootstrap.cc:492] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:45.576231  7566 tablet_bootstrap.cc:654] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.576701  7561 ts_tablet_manager.cc:1397] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.139s	user 0.010s	sys 0.011s
I20260809 23:23:45.581233  7027 catalog_manager.cc:5495] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:45.582652  7561 raft_consensus.cc:348] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.583428  7561 raft_consensus.cc:374] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.583732  7561 raft_consensus.cc:729] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.584771  7561 consensus_queue.cc:260] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.588143  7559 tablet_bootstrap.cc:654] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:45.590493  7437 consensus_peers.cc:487] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:45.593485  7851 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.594162  7851 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.600927  7558 ts_tablet_manager.cc:1428] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.118s	user 0.009s	sys 0.005s
I20260809 23:23:45.622082  7579 consensus_queue.cc:1035] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 23:23:45.622740  7338 leader_election.cc:341] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.631889  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:45.632834  7561 ts_tablet_manager.cc:1428] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.056s	user 0.005s	sys 0.004s
I20260809 23:23:45.638072  7488 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.651402  7561 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:45.655414  7173 raft_consensus.cc:3037] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.670104  7851 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:45.670452  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:45.672399  7339 leader_election.cc:341] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.677770  7488 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:45.679286  7558 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:45.680346  7028 catalog_manager.cc:5495] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:45.694722  7561 tablet_bootstrap.cc:654] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.697824  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:45.701751  7558 tablet_bootstrap.cc:654] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.702521  7851 consensus_queue.cc:1035] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.704453  7274 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:45.706014  7559 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:45.707508  7559 ts_tablet_manager.cc:1397] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.216s	user 0.007s	sys 0.009s
I20260809 23:23:45.714524  7126 leader_election.cc:302] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:45.716583  7566 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:45.718103  7566 ts_tablet_manager.cc:1397] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.220s	user 0.014s	sys 0.029s
I20260809 23:23:45.720885  7566 raft_consensus.cc:348] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.721680  7566 raft_consensus.cc:374] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.722000  7566 raft_consensus.cc:729] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.722684  7566 consensus_queue.cc:260] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.723479  7934 raft_consensus.cc:2781] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 23:23:45.723830  7338 leader_election.cc:341] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.724155  7934 raft_consensus.cc:686] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:45.724439  7338 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:45.725057  7934 consensus_queue.cc:237] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:45.717021  7559 raft_consensus.cc:348] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.726240  7559 raft_consensus.cc:374] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.726542  7559 raft_consensus.cc:729] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.725322  7851 raft_consensus.cc:2726] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:45.730187  7566 ts_tablet_manager.cc:1428] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260809 23:23:45.737921  7566 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:45.751327  7559 consensus_queue.cc:260] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.758163  7566 tablet_bootstrap.cc:654] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.777446  7559 ts_tablet_manager.cc:1428] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.001s
I20260809 23:23:45.785750  7559 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:45.777611  7583 raft_consensus.cc:482] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.786338  7561 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:45.786733  7583 raft_consensus.cc:504] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.787768  7561 ts_tablet_manager.cc:1397] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.137s	user 0.013s	sys 0.009s
I20260809 23:23:45.789497  7583 leader_election.cc:288] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:45.790035  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:45.786756  7856 consensus_queue.cc:1035] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260809 23:23:45.791831  7124 leader_election.cc:341] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.790501  7561 raft_consensus.cc:348] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.793459  7561 raft_consensus.cc:374] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.793516  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:45.793825  7561 raft_consensus.cc:729] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
W20260809 23:23:45.795012  7124 leader_election.cc:341] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:45.795442  7124 leader_election.cc:302] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:45.796938  7583 raft_consensus.cc:2726] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:45.797875  7561 consensus_queue.cc:260] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.802794  7561 ts_tablet_manager.cc:1428] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260809 23:23:45.821811  7561 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:45.827015  7490 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:45.859138  7873 consensus_queue.cc:1035] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260809 23:23:45.863639  7559 tablet_bootstrap.cc:654] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.843148  7025 catalog_manager.cc:5495] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:45.867669  7561 tablet_bootstrap.cc:654] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:45.868165  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:45.872602  7558 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.873164  7558 ts_tablet_manager.cc:1397] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.194s	user 0.013s	sys 0.060s
I20260809 23:23:45.875895  7558 raft_consensus.cc:348] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.876595  7558 raft_consensus.cc:374] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.876868  7558 raft_consensus.cc:729] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.877595  7558 consensus_queue.cc:260] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.880330  7558 ts_tablet_manager.cc:1428] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260809 23:23:45.882058  7558 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:45.883579  7566 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:45.884150  7566 ts_tablet_manager.cc:1397] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.146s	user 0.012s	sys 0.005s
I20260809 23:23:45.885468  7873 consensus_queue.cc:1035] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.886960  7566 raft_consensus.cc:348] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.889983  7566 raft_consensus.cc:374] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.893282  7566 raft_consensus.cc:729] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.894101  7566 consensus_queue.cc:260] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:45.896291  7566 ts_tablet_manager.cc:1428] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260809 23:23:45.897296  7566 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:45.903280  7566 tablet_bootstrap.cc:654] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.910285  7579 consensus_queue.cc:1035] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.922211  7558 tablet_bootstrap.cc:654] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:45.930884  7559 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:45.931393  7559 ts_tablet_manager.cc:1397] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.146s	user 0.019s	sys 0.005s
I20260809 23:23:45.939925  7559 raft_consensus.cc:348] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.940651  7559 raft_consensus.cc:374] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:45.940945  7559 raft_consensus.cc:729] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:45.941654  7559 consensus_queue.cc:260] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:45.952684  7896 consensus_queue.cc:1035] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.953930  7559 ts_tablet_manager.cc:1428] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260809 23:23:45.955128  7559 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:45.969457  7558 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:45.970135  7558 ts_tablet_manager.cc:1397] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.088s	user 0.008s	sys 0.004s
I20260809 23:23:45.971822  7937 raft_consensus.cc:482] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:45.980739  7937 raft_consensus.cc:504] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:45.984150  7937 leader_election.cc:288] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:45.988008  7852 consensus_queue.cc:1035] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:45.991966  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:45.992676  7171 raft_consensus.cc:2454] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:45.995179  7227 leader_election.cc:302] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:45.996582  7873 raft_consensus.cc:2781] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:45.996935  7873 raft_consensus.cc:482] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:45.997247  7873 raft_consensus.cc:3037] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:46.011673  7338 consensus_peers.cc:487] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:46.011605  7558 raft_consensus.cc:348] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.013815  7558 raft_consensus.cc:374] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.014480  7558 raft_consensus.cc:729] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.015697  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:46.017333  7227 leader_election.cc:341] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.021422  7937 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.021888  7937 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.027591  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:46.030896  7227 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.033332  7937 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.020689  7873 raft_consensus.cc:504] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.036895  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:46.037653  7171 raft_consensus.cc:3037] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.039398  7583 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.040057  7583 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.043650  7852 raft_consensus.cc:482] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.044238  7852 raft_consensus.cc:504] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.045533  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.042795  7583 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.047464  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:46.049352  7338 leader_election.cc:341] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:46.052457  7228 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.053212  7228 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:46.057395  7852 leader_election.cc:288] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.060056  7583 consensus_queue.cc:1035] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:46.060271  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:46.065418  7339 leader_election.cc:341] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.065871  7339 leader_election.cc:302] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:46.067358  7852 raft_consensus.cc:2726] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.067816  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.068311  7789 consensus_queue.cc:1035] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:46.069129  7488 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:46.070499  7126 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:46.075673  7572 raft_consensus.cc:2726] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.076346  7789 raft_consensus.cc:2781] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.077015  7789 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.077515  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.078022  7789 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.078563  7378 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:46.086587  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
W20260809 23:23:46.091071  7227 leader_election.cc:341] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.106484  7561 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:46.107117  7561 ts_tablet_manager.cc:1397] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.286s	user 0.032s	sys 0.042s
I20260809 23:23:46.110044  7873 leader_election.cc:288] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.015535  7558 consensus_queue.cc:260] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.117115  7566 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.117735  7566 ts_tablet_manager.cc:1397] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.221s	user 0.077s	sys 0.102s
I20260809 23:23:46.117988  7558 ts_tablet_manager.cc:1428] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.148s	user 0.008s	sys 0.000s
I20260809 23:23:46.119140  7558 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:46.119956  7171 raft_consensus.cc:2454] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:46.122862  7566 raft_consensus.cc:348] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.123678  7566 raft_consensus.cc:374] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.123945  7566 raft_consensus.cc:729] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.123952  7227 leader_election.cc:302] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:46.124679  7566 consensus_queue.cc:260] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.125775  7937 raft_consensus.cc:2781] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:46.126449  7937 raft_consensus.cc:686] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:46.126626  7566 ts_tablet_manager.cc:1428] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.000s
I20260809 23:23:46.127599  7937 consensus_queue.cc:237] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.115090  7561 raft_consensus.cc:348] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.133565  7561 raft_consensus.cc:374] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.133878  7561 raft_consensus.cc:729] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.134513  7561 consensus_queue.cc:260] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.138314  7561 ts_tablet_manager.cc:1428] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260809 23:23:46.139351  7561 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:46.128192  7566 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.146564  7561 tablet_bootstrap.cc:654] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.159262  7561 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:46.159730  7561 ts_tablet_manager.cc:1397] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.002s
I20260809 23:23:46.162282  7561 raft_consensus.cc:348] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.163059  7561 raft_consensus.cc:374] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.163324  7561 raft_consensus.cc:729] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.168653  7561 consensus_queue.cc:260] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.171139  7561 ts_tablet_manager.cc:1428] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260809 23:23:46.172139  7561 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:46.174633  7566 tablet_bootstrap.cc:654] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.176107  7559 tablet_bootstrap.cc:654] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.178381  7561 tablet_bootstrap.cc:654] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.187649  7792 raft_consensus.cc:482] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.188213  7792 raft_consensus.cc:504] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.191130  7559 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:46.192034  7561 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:46.193328  7792 leader_election.cc:288] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:46.193475  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.197330  7559 ts_tablet_manager.cc:1397] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.242s	user 0.013s	sys 0.001s
I20260809 23:23:46.197672  7561 ts_tablet_manager.cc:1397] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.006s
I20260809 23:23:46.196524  7566 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.193899  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:46.198810  7338 leader_election.cc:341] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.199023  7566 ts_tablet_manager.cc:1397] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.071s	user 0.009s	sys 0.002s
W20260809 23:23:46.199559  7338 leader_election.cc:341] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.200011  7338 leader_election.cc:302] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.200608  7561 raft_consensus.cc:348] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.201129  7792 raft_consensus.cc:2726] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.201407  7561 raft_consensus.cc:374] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.201712  7561 raft_consensus.cc:729] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.202581  7561 consensus_queue.cc:260] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.203022  7566 raft_consensus.cc:348] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.203907  7566 raft_consensus.cc:374] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.204236  7566 raft_consensus.cc:729] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.200001  7559 raft_consensus.cc:348] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.205325  7559 raft_consensus.cc:374] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.205386  7561 ts_tablet_manager.cc:1428] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260809 23:23:46.204950  7566 consensus_queue.cc:260] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.206863  7561 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:46.207060  7566 ts_tablet_manager.cc:1428] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:23:46.207808  7901 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.208479  7566 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.208314  7901 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.210319  7901 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.213066  7561 tablet_bootstrap.cc:654] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.213255  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.213956  7272 raft_consensus.cc:2454] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:46.215335  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.205672  7559 raft_consensus.cc:729] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
W20260809 23:23:46.219583  7437 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.219712  7559 consensus_queue.cc:260] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.220260  7437 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:46.220664  7566 tablet_bootstrap.cc:654] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.221767  7579 raft_consensus.cc:2781] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.222000  7559 ts_tablet_manager.cc:1428] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.024s	user 0.001s	sys 0.003s
I20260809 23:23:46.222149  7579 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.222522  7579 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.223104  7559 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:46.231153  7558 tablet_bootstrap.cc:654] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.234579  7789 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.238845  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:46.239665  7378 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.241806  7559 tablet_bootstrap.cc:654] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.243152  7789 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.252270  7566 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.252947  7566 ts_tablet_manager.cc:1397] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.010s
I20260809 23:23:46.260661  7378 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:46.262255  7124 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:46.264647  7566 raft_consensus.cc:348] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.265592  7566 raft_consensus.cc:374] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.265905  7566 raft_consensus.cc:729] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.266652  7566 consensus_queue.cc:260] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.270460  7789 raft_consensus.cc:2781] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:46.272554  7789 raft_consensus.cc:686] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:46.272985  7566 ts_tablet_manager.cc:1428] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260809 23:23:46.274231  7566 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.273932  7789 consensus_queue.cc:237] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.278425  7558 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:46.279022  7558 ts_tablet_manager.cc:1397] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.160s	user 0.016s	sys 0.015s
I20260809 23:23:46.284117  7559 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:46.288511  7559 ts_tablet_manager.cc:1397] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.066s	user 0.011s	sys 0.013s
I20260809 23:23:46.286629  7558 raft_consensus.cc:348] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.290705  7558 raft_consensus.cc:374] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.291419  7558 raft_consensus.cc:729] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.288609  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:46.286361  7566 tablet_bootstrap.cc:654] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.295112  7490 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.295121  7558 consensus_queue.cc:260] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.292848  7579 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.300663  7579 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.300870  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:46.301258  7579 raft_consensus.cc:482] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.300699  7558 ts_tablet_manager.cc:1428] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260809 23:23:46.301676  7579 raft_consensus.cc:504] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
W20260809 23:23:46.302183  7437 leader_election.cc:341] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.293870  7559 raft_consensus.cc:348] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.303692  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:46.305161  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.305236  7272 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.306022  7171 raft_consensus.cc:2454] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:46.306844  7566 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.306653  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.307371  7566 ts_tablet_manager.cc:1397] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.000s
I20260809 23:23:46.307684  7378 raft_consensus.cc:2454] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:46.307380  7437 leader_election.cc:302] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:46.308962  7896 raft_consensus.cc:2781] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.309324  7896 raft_consensus.cc:482] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.309670  7896 raft_consensus.cc:3037] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.311044  7566 raft_consensus.cc:348] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.312000  7566 raft_consensus.cc:374] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.312332  7566 raft_consensus.cc:729] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.313185  7566 consensus_queue.cc:260] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.313894  7490 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:46.262364  7024 catalog_manager.cc:5495] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:46.316465  7566 ts_tablet_manager.cc:1428] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260809 23:23:46.317608  7566 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.317243  7896 raft_consensus.cc:504] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.303864  7559 raft_consensus.cc:374] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.318498  7559 raft_consensus.cc:729] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.319603  7792 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.319437  7559 consensus_queue.cc:260] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.320050  7792 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.321952  7559 ts_tablet_manager.cc:1428] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.032s	user 0.001s	sys 0.005s
I20260809 23:23:46.322018  7792 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:46.323031  7559 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:46.324101  7566 tablet_bootstrap.cc:654] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.324229  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.303726  7558 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:46.326386  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.329459  7934 consensus_queue.cc:1035] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:46.331768  7851 raft_consensus.cc:482] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.332304  7851 raft_consensus.cc:504] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.333245  7896 leader_election.cc:288] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:46.336517  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.337530  7851 leader_election.cc:288] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.338112  7558 tablet_bootstrap.cc:654] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:46.330603  7338 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:46.338863  7339 leader_election.cc:341] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:46.339325  7338 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.339915  7338 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.342736  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:46.343425  7173 raft_consensus.cc:3037] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.344440  7792 raft_consensus.cc:2726] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.346777  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.347664  7934 consensus_queue.cc:1035] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:46.349488  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e08fe5d08748ba98111cd243575917" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:46.350966  7378 raft_consensus.cc:3037] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:46.363055  7338 leader_election.cc:341] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.363744  7338 leader_election.cc:302] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:46.364753  7851 raft_consensus.cc:2726] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.380261  7559 tablet_bootstrap.cc:654] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.303712  7579 leader_election.cc:288] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
W20260809 23:23:46.396137  7992 log.cc:924] Time spent T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260809 23:23:46.396255  7566 tablet_bootstrap.cc:492] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.396922  7566 ts_tablet_manager.cc:1397] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.080s	user 0.031s	sys 0.043s
I20260809 23:23:46.405474  7173 raft_consensus.cc:2454] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:46.407509  7437 leader_election.cc:302] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:46.382051  7378 raft_consensus.cc:2454] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:46.411154  7896 raft_consensus.cc:2781] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:46.411965  7272 raft_consensus.cc:2454] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:46.412195  7896 raft_consensus.cc:686] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:46.413627  7437 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:46.413374  7896 consensus_queue.cc:237] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.414521  7901 raft_consensus.cc:2781] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:46.415016  7901 raft_consensus.cc:686] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:46.415813  7901 consensus_queue.cc:237] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.419227  7566 raft_consensus.cc:348] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.420080  7566 raft_consensus.cc:374] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.420408  7566 raft_consensus.cc:729] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.421273  7566 consensus_queue.cc:260] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.427137  7566 ts_tablet_manager.cc:1428] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260809 23:23:46.428550  7566 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.436736  7566 tablet_bootstrap.cc:654] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.454034  7561 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:46.454684  7561 ts_tablet_manager.cc:1397] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.248s	user 0.035s	sys 0.038s
I20260809 23:23:46.456243  7851 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.457216  7851 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.460013  7851 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.461941  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.463086  7272 raft_consensus.cc:2379] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:46.480890  7024 catalog_manager.cc:5495] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:46.482684  7937 raft_consensus.cc:482] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.492209  7566 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.491488  7937 raft_consensus.cc:504] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.492870  7566 ts_tablet_manager.cc:1397] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.065s	user 0.016s	sys 0.028s
I20260809 23:23:46.494563  7937 leader_election.cc:288] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
W20260809 23:23:46.487146  7124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 64099.4us
I20260809 23:23:46.495189  7561 raft_consensus.cc:348] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.495507  7566 raft_consensus.cc:348] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.496245  7566 raft_consensus.cc:374] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.496521  7566 raft_consensus.cc:729] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.496603  7789 raft_consensus.cc:482] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.497447  7566 consensus_queue.cc:260] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.499112  7566 ts_tablet_manager.cc:1428] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:23:46.501061  7566 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:46.497319  7789 raft_consensus.cc:504] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.503998  7789 leader_election.cc:288] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.488639  7558 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:46.495910  7561 raft_consensus.cc:374] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.505213  7561 raft_consensus.cc:729] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.505911  7561 consensus_queue.cc:260] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.506960  7566 tablet_bootstrap.cc:654] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.506657  7851 raft_consensus.cc:482] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.507493  7561 ts_tablet_manager.cc:1428] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.052s	user 0.006s	sys 0.000s
I20260809 23:23:46.507467  7851 raft_consensus.cc:504] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.508710  7561 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:46.509407  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.509773  7851 leader_election.cc:288] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:46.506712  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b04e56cb2d4321a0b5e6317136d6f2" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.504916  7558 ts_tablet_manager.cc:1397] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.203s	user 0.010s	sys 0.008s
I20260809 23:23:46.513777  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.514731  7124 leader_election.cc:302] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:46.515925  7789 raft_consensus.cc:2726] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:23:46.516198  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 78483.2us
I20260809 23:23:46.519029  7789 raft_consensus.cc:482] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.519519  7789 raft_consensus.cc:504] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.519771  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.521191  7339 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:46.521651  7789 leader_election.cc:288] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.526916  7272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.529950  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.529978  7559 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:46.531239  7559 ts_tablet_manager.cc:1397] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.208s	user 0.035s	sys 0.026s
I20260809 23:23:46.532328  7124 leader_election.cc:302] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:46.534159  7789 raft_consensus.cc:2726] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.534073  7851 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.535884  7559 raft_consensus.cc:348] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.536384  7561 tablet_bootstrap.cc:654] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.536639  7559 raft_consensus.cc:374] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.536928  7559 raft_consensus.cc:729] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.537693  7559 consensus_queue.cc:260] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.539932  7559 ts_tablet_manager.cc:1428] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:23:46.541260  7559 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:46.561584  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.563580  7174 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.567325  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5b4438b1d84d7198189995b2d2d409" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.568185  7173 raft_consensus.cc:2379] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
W20260809 23:23:46.569334  7227 leader_election.cc:341] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.572077  7338 leader_election.cc:302] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.574599  7792 raft_consensus.cc:3037] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.581682  7558 raft_consensus.cc:348] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.582474  7558 raft_consensus.cc:374] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.582716  7558 raft_consensus.cc:729] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.583359  7558 consensus_queue.cc:260] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.581713  7579 raft_consensus.cc:482] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.584389  7579 raft_consensus.cc:504] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.585335  7558 ts_tablet_manager.cc:1428] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.073s	user 0.005s	sys 0.001s
I20260809 23:23:46.590533  7792 raft_consensus.cc:2726] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.591449  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:46.592144  7275 raft_consensus.cc:2454] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:46.595142  7437 leader_election.cc:302] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:46.596802  8016 raft_consensus.cc:2781] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.597157  8016 raft_consensus.cc:482] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.597448  8016 raft_consensus.cc:3037] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.598601  7851 raft_consensus.cc:2726] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.601760  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:46.618325  7437 leader_election.cc:341] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.623111  7558 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:46.586711  7579 leader_election.cc:288] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:46.660893  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.659689  7572 raft_consensus.cc:482] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.666841  7873 consensus_queue.cc:1035] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:46.665170  7572 raft_consensus.cc:504] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.685544  7572 leader_election.cc:288] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.686612  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.686906  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:46.698681  7227 leader_election.cc:341] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.708351  7028 catalog_manager.cc:5495] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:46.698268  7378 raft_consensus.cc:2454] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:46.721148  7583 consensus_queue.cc:1035] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:46.723755  7227 leader_election.cc:302] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.726045  7572 raft_consensus.cc:2781] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.726547  7572 raft_consensus.cc:482] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.726948  7572 raft_consensus.cc:3037] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.727108  8024 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.732681  8024 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.734450  7561 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:46.735008  7561 ts_tablet_manager.cc:1397] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.227s	user 0.044s	sys 0.020s
I20260809 23:23:46.734738  8024 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:46.737900  7561 raft_consensus.cc:348] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.739480  7559 tablet_bootstrap.cc:654] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.739580  7561 raft_consensus.cc:374] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.740061  7561 raft_consensus.cc:729] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.697811  8016 raft_consensus.cc:504] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.744421  7561 consensus_queue.cc:260] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.760245  7561 ts_tablet_manager.cc:1428] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.002s
I20260809 23:23:46.764448  7566 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:46.764675  7561 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:46.764993  7566 ts_tablet_manager.cc:1397] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.264s	user 0.032s	sys 0.043s
I20260809 23:23:46.769407  7488 raft_consensus.cc:2379] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:46.770447  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.771368  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:46.772881  7558 tablet_bootstrap.cc:654] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.775274  7566 raft_consensus.cc:348] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.776139  7566 raft_consensus.cc:374] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.776377  7566 raft_consensus.cc:729] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.776930  7566 consensus_queue.cc:260] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.779161  7566 ts_tablet_manager.cc:1428] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260809 23:23:46.780207  7566 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
W20260809 23:23:46.780490  7228 leader_election.cc:341] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.781605  7228 leader_election.cc:302] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
W20260809 23:23:46.784080  7227 leader_election.cc:341] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.784617  7227 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.783993  7934 consensus_queue.cc:1035] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:46.794086  7572 raft_consensus.cc:504] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:46.791177  7937 raft_consensus.cc:2726] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.792270  8024 raft_consensus.cc:3037] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.801611  7873 consensus_queue.cc:1035] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:46.808383  7572 leader_election.cc:288] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:46.810113  8016 leader_election.cc:288] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:46.812036  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:46.812533  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:46.818418  7028 catalog_manager.cc:5495] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } }
W20260809 23:23:46.827145  7437 leader_election.cc:341] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.830690  7851 raft_consensus.cc:482] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.831208  7851 raft_consensus.cc:504] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.833237  7851 leader_election.cc:288] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.834981  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a40c34e514b9fbcec1c0000466ecc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.836009  7486 raft_consensus.cc:2454] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:46.838858  7339 leader_election.cc:302] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:46.839985  7851 raft_consensus.cc:2781] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:46.840435  7851 raft_consensus.cc:482] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:46.840823  7851 raft_consensus.cc:3037] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:46.853761  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 60058.6us
I20260809 23:23:46.854122  7561 tablet_bootstrap.cc:654] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.855955  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:46.856629  7377 raft_consensus.cc:3037] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.864662  7559 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:46.865125  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ecdc375af4a473ca3c09c02da2a2594" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:46.867539  7171 raft_consensus.cc:3037] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.867864  7566 tablet_bootstrap.cc:654] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.865223  7559 ts_tablet_manager.cc:1397] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.324s	user 0.011s	sys 0.013s
I20260809 23:23:46.882228  7559 raft_consensus.cc:348] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:46.882982  7559 raft_consensus.cc:374] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:46.883256  7559 raft_consensus.cc:729] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:46.883932  7559 consensus_queue.cc:260] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:23:46.885979  7227 leader_election.cc:341] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.897622  7934 raft_consensus.cc:482] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.898092  7934 raft_consensus.cc:504] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
W20260809 23:23:46.899165  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 76890.2us
I20260809 23:23:46.921308  7559 ts_tablet_manager.cc:1428] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.047s	user 0.005s	sys 0.005s
I20260809 23:23:46.909109  8024 raft_consensus.cc:2726] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:46.899183  7851 raft_consensus.cc:504] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:46.931480  7934 leader_election.cc:288] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.950770  7559 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:46.954190  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.956188  7851 leader_election.cc:288] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:46.957415  7172 raft_consensus.cc:3037] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.959195  7852 raft_consensus.cc:482] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.959697  7852 raft_consensus.cc:504] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.960602  7275 raft_consensus.cc:2454] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:46.961771  7852 leader_election.cc:288] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:46.963200  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbd59811ea94e13b359f7c9db694683" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:46.966890  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a40c34e514b9fbcec1c0000466ecc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:46.967521  7488 raft_consensus.cc:3037] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:46.968356  7559 tablet_bootstrap.cc:654] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:46.970643  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:46.974334  7579 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:46.974818  7579 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:46.976472  7437 leader_election.cc:302] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:46.976684  7579 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:46.979586  7579 raft_consensus.cc:2781] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:46.980149  7579 raft_consensus.cc:686] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:46.981973  7558 tablet_bootstrap.cc:492] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:46.986645  7558 ts_tablet_manager.cc:1397] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.400s	user 0.026s	sys 0.016s
I20260809 23:23:46.981077  7579 consensus_queue.cc:237] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.001861  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.002597  7561 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.003173  7561 ts_tablet_manager.cc:1397] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.239s	user 0.010s	sys 0.008s
I20260809 23:23:47.003296  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.005424  7488 raft_consensus.cc:2454] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
W20260809 23:23:47.008586  7437 leader_election.cc:341] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:46.990180  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a40c34e514b9fbcec1c0000466ecc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.017897  7561 raft_consensus.cc:348] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.018452  7561 raft_consensus.cc:374] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.018671  7561 raft_consensus.cc:729] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.019320  7561 consensus_queue.cc:260] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.011106  7377 raft_consensus.cc:2454] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:47.026491  7566 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.027020  7566 ts_tablet_manager.cc:1397] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.247s	user 0.020s	sys 0.013s
I20260809 23:23:47.024753  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.031509  7558 raft_consensus.cc:348] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.032353  7558 raft_consensus.cc:374] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.032608  7558 raft_consensus.cc:729] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.033579  7558 consensus_queue.cc:260] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:23:47.016304  7339 leader_election.cc:341] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.035876  7558 ts_tablet_manager.cc:1428] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.001s
I20260809 23:23:47.044255  7174 raft_consensus.cc:3037] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.052126  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1a40c34e514b9fbcec1c0000466ecc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:47.052850  7275 raft_consensus.cc:3037] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.060217  7559 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
W20260809 23:23:47.061657  7339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 49684.1us
I20260809 23:23:47.066836  7339 leader_election.cc:302] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:47.068701  8043 raft_consensus.cc:2781] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.070106  8043 raft_consensus.cc:686] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:47.036317  7126 leader_election.cc:302] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:47.071929  7227 leader_election.cc:302] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.073338  7937 raft_consensus.cc:2781] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
W20260809 23:23:47.073613  7338 leader_election.cc:341] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.073853  7937 raft_consensus.cc:686] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:47.074666  7937 consensus_queue.cc:237] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.075276  8043 consensus_queue.cc:237] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.079116  7558 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.079789  7559 ts_tablet_manager.cc:1397] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.136s	user 0.028s	sys 0.001s
I20260809 23:23:47.080163  7566 raft_consensus.cc:348] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.088462  7566 raft_consensus.cc:374] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.088825  7566 raft_consensus.cc:729] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
W20260809 23:23:47.092311  7126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 67224.7us
W20260809 23:23:47.094014  7338 leader_election.cc:341] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.094471  7338 leader_election.cc:302] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.089771  7566 consensus_queue.cc:260] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.096066  7559 raft_consensus.cc:348] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.097129  8035 raft_consensus.cc:2726] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.097085  7561 ts_tablet_manager.cc:1428] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.094s	user 0.007s	sys 0.000s
I20260809 23:23:47.106294  7559 raft_consensus.cc:374] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.107146  7561 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.107198  7559 raft_consensus.cc:729] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.109014  7559 consensus_queue.cc:260] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.104034  7566 ts_tablet_manager.cc:1428] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.077s	user 0.007s	sys 0.000s
I20260809 23:23:47.117061  7559 ts_tablet_manager.cc:1428] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.001s
I20260809 23:23:47.117807  7275 raft_consensus.cc:2454] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:47.135776  7559 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:47.136273  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 98140.1us
W20260809 23:23:47.140542  7437 leader_election.cc:341] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.151917  7437 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.154228  7566 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.160157  8051 consensus_queue.cc:1035] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:47.153848  7901 raft_consensus.cc:2726] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.163527  7558 tablet_bootstrap.cc:654] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.163414  7027 catalog_manager.cc:5495] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.177634  8035 raft_consensus.cc:482] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.178143  8035 raft_consensus.cc:504] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.182953  8035 leader_election.cc:288] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:47.185750  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.186997  7559 tablet_bootstrap.cc:654] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.189534  7025 catalog_manager.cc:5495] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:47.193224  7339 leader_election.cc:341] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.166676  7561 tablet_bootstrap.cc:654] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.169183  7583 raft_consensus.cc:2726] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.212965  8051 consensus_queue.cc:1035] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:47.213447  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:47.225807  7338 leader_election.cc:341] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.229071  7338 leader_election.cc:302] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.229984  7896 consensus_queue.cc:1035] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260809 23:23:47.235674  7583 raft_consensus.cc:482] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.234603  8014 consensus_queue.cc:1035] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260809 23:23:47.245445  7028 catalog_manager.cc:5495] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.244560  7852 raft_consensus.cc:2726] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.250131  7566 tablet_bootstrap.cc:654] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.244192  7583 raft_consensus.cc:504] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.253507  7583 leader_election.cc:288] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.255718  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.256549  7271 raft_consensus.cc:2379] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:47.275738  7561 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.276515  7561 ts_tablet_manager.cc:1397] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.170s	user 0.008s	sys 0.004s
I20260809 23:23:47.281047  7789 raft_consensus.cc:482] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3cc3ec2dcd449de83f6c88e80d2d6de)
I20260809 23:23:47.281635  7789 raft_consensus.cc:504] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.293435  7559 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:47.282038  7561 raft_consensus.cc:348] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.297657  7561 raft_consensus.cc:374] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.297673  7559 ts_tablet_manager.cc:1397] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.162s	user 0.009s	sys 0.012s
W20260809 23:23:47.286451  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:47.298005  7561 raft_consensus.cc:729] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.292059  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.300709  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89212541eac4b91975dbfbe73e23c3a" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.291028  7789 leader_election.cc:288] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.304134  7566 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.304638  7566 ts_tablet_manager.cc:1397] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.151s	user 0.009s	sys 0.004s
I20260809 23:23:47.304687  7558 tablet_bootstrap.cc:492] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.305199  7558 ts_tablet_manager.cc:1397] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.226s	user 0.025s	sys 0.053s
I20260809 23:23:47.308444  7566 raft_consensus.cc:348] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.309301  7566 raft_consensus.cc:374] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.309281  7559 raft_consensus.cc:348] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.309916  7566 raft_consensus.cc:729] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.310158  7559 raft_consensus.cc:374] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.311146  7561 consensus_queue.cc:260] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.310485  7559 raft_consensus.cc:729] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.314819  7566 consensus_queue.cc:260] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.317364  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c7eaaf2fea43f7aae67f0524f08d94" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.317939  7558 raft_consensus.cc:348] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.323053  7558 raft_consensus.cc:374] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.324932  7126 leader_election.cc:302] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:47.326185  7126 leader_election.cc:302] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:47.327075  7583 raft_consensus.cc:3037] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.327939  7938 raft_consensus.cc:2726] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:47.323339  7558 raft_consensus.cc:729] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.330364  7559 consensus_queue.cc:260] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.330982  7558 consensus_queue.cc:260] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.333454  7558 ts_tablet_manager.cc:1428] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260809 23:23:47.335187  7566 ts_tablet_manager.cc:1428] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260809 23:23:47.335248  7558 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.337502  7561 ts_tablet_manager.cc:1428] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.061s	user 0.009s	sys 0.000s
I20260809 23:23:47.338593  7561 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.342983  7559 ts_tablet_manager.cc:1428] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260809 23:23:47.344009  7559 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:47.349802  7561 tablet_bootstrap.cc:654] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.352890  7583 raft_consensus.cc:2726] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.356484  7559 tablet_bootstrap.cc:654] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.359027  7558 tablet_bootstrap.cc:654] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.353507  7566 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.381832  7566 tablet_bootstrap.cc:654] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.383958  7789 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.384486  7789 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.386592  7789 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.391887  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.392658  7490 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:47.394455  7558 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.395730  7558 ts_tablet_manager.cc:1397] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.061s	user 0.011s	sys 0.000s
I20260809 23:23:47.398527  7558 raft_consensus.cc:348] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.399966  7558 raft_consensus.cc:374] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.400216  7558 raft_consensus.cc:729] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.400233  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.401032  7382 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:47.400993  7126 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:47.401899  7558 consensus_queue.cc:260] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.406072  7561 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.407416  7561 ts_tablet_manager.cc:1397] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.069s	user 0.017s	sys 0.000s
I20260809 23:23:47.408581  7901 consensus_queue.cc:1035] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:47.412094  7583 raft_consensus.cc:2781] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.412569  7583 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.428682  7583 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.430547  8051 consensus_queue.cc:1035] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:47.415768  7901 raft_consensus.cc:482] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.435218  7901 raft_consensus.cc:504] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.437335  7901 leader_election.cc:288] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:47.438030  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.418479  7559 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:47.438764  7275 raft_consensus.cc:2454] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:47.438911  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.439544  8024 raft_consensus.cc:482] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.439996  8024 raft_consensus.cc:504] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.439776  7382 raft_consensus.cc:2454] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:47.442183  7437 leader_election.cc:302] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:47.442972  7901 raft_consensus.cc:2781] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.443382  7901 raft_consensus.cc:482] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.443982  7901 raft_consensus.cc:3037] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.444453  8024 leader_election.cc:288] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.419415  7558 ts_tablet_manager.cc:1428] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260809 23:23:47.446194  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.447322  7558 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.439041  7559 ts_tablet_manager.cc:1397] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.095s	user 0.012s	sys 0.006s
I20260809 23:23:47.442975  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.448998  7382 raft_consensus.cc:2454] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:47.450151  7227 leader_election.cc:302] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:47.453845  7558 tablet_bootstrap.cc:654] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.454448  8024 raft_consensus.cc:2781] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.455224  8024 raft_consensus.cc:482] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.455678  8024 raft_consensus.cc:3037] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.412410  7561 raft_consensus.cc:348] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.457173  7561 raft_consensus.cc:374] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.457532  7561 raft_consensus.cc:729] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.458343  7561 consensus_queue.cc:260] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.464473  7901 raft_consensus.cc:504] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.465907  7488 raft_consensus.cc:2379] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:47.466266  7583 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.468834  7558 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.469560  7558 ts_tablet_manager.cc:1397] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.000s
I20260809 23:23:47.471565  7583 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.468837  7566 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.472126  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:47.481817  7382 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.481215  8024 raft_consensus.cc:504] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.483722  7566 ts_tablet_manager.cc:1397] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.136s	user 0.011s	sys 0.004s
I20260809 23:23:47.484050  8024 leader_election.cc:288] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.485248  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:47.485773  7566 raft_consensus.cc:348] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.486073  7490 raft_consensus.cc:2379] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:47.473129  7558 raft_consensus.cc:348] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.473380  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:47.486634  7558 raft_consensus.cc:374] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.486817  7488 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.486994  7558 raft_consensus.cc:729] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.487711  7558 consensus_queue.cc:260] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.488495  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:47.489166  7376 raft_consensus.cc:3037] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.490058  7558 ts_tablet_manager.cc:1428] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260809 23:23:47.474606  7559 raft_consensus.cc:348] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.476089  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:47.491251  7271 raft_consensus.cc:2379] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a3cc3ec2dcd449de83f6c88e80d2d6de in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:47.476364  7579 raft_consensus.cc:482] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.477329  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895d1e78ae154da4a1c299f234d88965" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:47.492067  7579 raft_consensus.cc:504] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.478022  7561 ts_tablet_manager.cc:1428] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.070s	user 0.006s	sys 0.000s
I20260809 23:23:47.493666  7561 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.493698  7437 leader_election.cc:302] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:47.495720  7382 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:47.496505  8051 raft_consensus.cc:2726] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.493921  7579 leader_election.cc:288] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:47.497946  7124 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:47.499476  7583 raft_consensus.cc:2781] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.500190  7583 raft_consensus.cc:686] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:47.486249  7566 raft_consensus.cc:374] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.497833  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.501014  7583 consensus_queue.cc:237] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.501991  7566 raft_consensus.cc:729] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
W20260809 23:23:47.502897  7437 leader_election.cc:341] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.502578  7558 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.472788  7901 leader_election.cc:288] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:47.495173  7559 raft_consensus.cc:374] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.505921  7488 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:47.504395  7566 consensus_queue.cc:260] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.511236  7559 raft_consensus.cc:729] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.516561  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.518045  7382 raft_consensus.cc:2454] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:47.519116  7437 leader_election.cc:302] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.519857  7873 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.520226  7579 raft_consensus.cc:2781] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.520610  7579 raft_consensus.cc:482] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.520364  7873 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.520943  7579 raft_consensus.cc:3037] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.522707  7873 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:47.530099  7558 tablet_bootstrap.cc:654] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.531015  7561 tablet_bootstrap.cc:654] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.533120  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.508229  7376 raft_consensus.cc:2454] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
W20260809 23:23:47.534853  7227 leader_election.cc:341] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.536450  7227 leader_election.cc:302] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:47.512050  7559 consensus_queue.cc:260] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.543996  8037 raft_consensus.cc:2781] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.542965  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.545432  7566 ts_tablet_manager.cc:1428] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.061s	user 0.007s	sys 0.000s
I20260809 23:23:47.546361  8037 raft_consensus.cc:686] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:47.551591  7579 raft_consensus.cc:504] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.556522  8037 consensus_queue.cc:237] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.559005  7559 ts_tablet_manager.cc:1428] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.111s	user 0.007s	sys 0.000s
I20260809 23:23:47.561067  7559 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:47.562386  7579 leader_election.cc:288] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:47.567294  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:47.567315  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:47.567965  7382 raft_consensus.cc:3037] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:47.568715  7228 leader_election.cc:341] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.569182  7228 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.570420  7937 raft_consensus.cc:2726] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:23:47.571830  7437 leader_election.cc:341] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.575675  7566 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.578014  7382 raft_consensus.cc:2454] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:47.567734  7027 catalog_manager.cc:5495] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:47.588542  7937 consensus_queue.cc:1035] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:47.590682  7437 leader_election.cc:302] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.604467  7901 raft_consensus.cc:2781] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.605003  7901 raft_consensus.cc:686] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:47.605826  7901 consensus_queue.cc:237] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.608217  7579 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.608659  7579 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.610212  7579 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.598387  7025 catalog_manager.cc:5495] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.612538  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.614993  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.615520  7172 raft_consensus.cc:3037] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.615717  7271 raft_consensus.cc:2454] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:47.591988  7558 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.617255  7558 ts_tablet_manager.cc:1397] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.115s	user 0.018s	sys 0.004s
I20260809 23:23:47.619843  7558 raft_consensus.cc:348] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.625830  7558 raft_consensus.cc:374] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.627187  7558 raft_consensus.cc:729] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.627549  8035 consensus_queue.cc:1035] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:47.621451  7437 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:47.628494  7561 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.628983  7561 ts_tablet_manager.cc:1397] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.136s	user 0.006s	sys 0.005s
I20260809 23:23:47.629068  8015 raft_consensus.cc:2781] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.629156  7937 consensus_queue.cc:1035] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:47.629622  8015 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.630018  8015 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.631479  7561 raft_consensus.cc:348] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.633320  7561 raft_consensus.cc:374] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.633571  7561 raft_consensus.cc:729] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.634142  7561 consensus_queue.cc:260] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.636265  7561 ts_tablet_manager.cc:1428] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:23:47.637976  7561 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
W20260809 23:23:47.639251  7437 leader_election.cc:341] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.641085  7937 raft_consensus.cc:482] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.627941  7558 consensus_queue.cc:260] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.641566  7937 raft_consensus.cc:504] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.643036  7558 ts_tablet_manager.cc:1428] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260809 23:23:47.643582  7937 leader_election.cc:288] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:47.646731  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.648140  7566 tablet_bootstrap.cc:654] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.648427  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
W20260809 23:23:47.649521  7227 leader_election.cc:341] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.638348  8015 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.652810  8015 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.654546  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:47.655246  7271 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.655510  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:47.656121  7561 tablet_bootstrap.cc:654] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
W20260809 23:23:47.659657  7437 leader_election.cc:341] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:47.650753  7227 leader_election.cc:341] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.664584  7558 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.650763  7559 tablet_bootstrap.cc:654] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.664389  7227 leader_election.cc:302] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.668655  7873 raft_consensus.cc:2726] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.674152  7271 raft_consensus.cc:2454] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:47.677153  7025 catalog_manager.cc:5495] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.678577  7437 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.679370  7901 raft_consensus.cc:2781] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.679855  7901 raft_consensus.cc:686] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:47.679754  8035 consensus_queue.cc:1035] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:47.681264  7901 consensus_queue.cc:237] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.694556  7561 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.697361  7561 ts_tablet_manager.cc:1397] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.001s
I20260809 23:23:47.700482  7566 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.700955  7566 ts_tablet_manager.cc:1397] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.140s	user 0.019s	sys 0.000s
I20260809 23:23:47.702576  7561 raft_consensus.cc:348] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.703337  7561 raft_consensus.cc:374] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.703593  7561 raft_consensus.cc:729] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.704222  7561 consensus_queue.cc:260] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.705790  7558 tablet_bootstrap.cc:654] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.706315  7561 ts_tablet_manager.cc:1428] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.000s
I20260809 23:23:47.717864  7566 raft_consensus.cc:348] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.718608  7566 raft_consensus.cc:374] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.718878  7566 raft_consensus.cc:729] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.719530  7566 consensus_queue.cc:260] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.721138  7566 ts_tablet_manager.cc:1428] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I20260809 23:23:47.722247  7566 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.727890  7566 tablet_bootstrap.cc:654] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.728286  7558 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.729041  7558 ts_tablet_manager.cc:1397] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.065s	user 0.009s	sys 0.003s
I20260809 23:23:47.729982  7559 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:47.732637  7559 ts_tablet_manager.cc:1397] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.173s	user 0.015s	sys 0.014s
I20260809 23:23:47.734560  7558 raft_consensus.cc:348] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.735432  7558 raft_consensus.cc:374] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.735682  7558 raft_consensus.cc:729] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.743729  7558 consensus_queue.cc:260] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.746565  7559 raft_consensus.cc:348] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.747430  7559 raft_consensus.cc:374] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.747692  7559 raft_consensus.cc:729] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.748466  7559 consensus_queue.cc:260] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.750609  7559 ts_tablet_manager.cc:1428] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260809 23:23:47.751562  7559 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:47.757923  7566 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.758447  7566 ts_tablet_manager.cc:1397] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.036s	user 0.021s	sys 0.012s
I20260809 23:23:47.763139  7559 tablet_bootstrap.cc:654] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.765434  7558 ts_tablet_manager.cc:1428] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.002s
I20260809 23:23:47.767572  7561 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.774300  7561 tablet_bootstrap.cc:654] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.778065  7566 raft_consensus.cc:348] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.778801  7566 raft_consensus.cc:374] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.779083  7566 raft_consensus.cc:729] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.779698  7566 consensus_queue.cc:260] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.781252  7566 ts_tablet_manager.cc:1428] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.000s
I20260809 23:23:47.782270  7566 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.789323  7558 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.792989  7566 tablet_bootstrap.cc:654] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.798550  7558 tablet_bootstrap.cc:654] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.801177  7559 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:47.801848  8015 raft_consensus.cc:482] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.802495  8015 raft_consensus.cc:504] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.801941  7024 catalog_manager.cc:5495] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.804523  8015 leader_election.cc:288] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.802520  7559 ts_tablet_manager.cc:1397] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.051s	user 0.006s	sys 0.005s
I20260809 23:23:47.806325  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:47.808482  7437 leader_election.cc:341] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.808794  7559 raft_consensus.cc:348] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.809513  7559 raft_consensus.cc:374] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.809772  7559 raft_consensus.cc:729] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.810395  7559 consensus_queue.cc:260] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.812806  7559 ts_tablet_manager.cc:1428] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260809 23:23:47.813251  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.813766  7559 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:47.813916  7273 raft_consensus.cc:2454] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:47.815682  7437 leader_election.cc:302] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.816912  8015 raft_consensus.cc:2781] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.817260  8015 raft_consensus.cc:482] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.817572  8015 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.820859  7561 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.821364  7561 ts_tablet_manager.cc:1397] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.054s	user 0.013s	sys 0.006s
I20260809 23:23:47.823909  7561 raft_consensus.cc:348] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.824536  7561 raft_consensus.cc:374] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.824803  7561 raft_consensus.cc:729] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.825506  7561 consensus_queue.cc:260] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.829213  8015 raft_consensus.cc:504] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.831219  7559 tablet_bootstrap.cc:654] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.830070  7561 ts_tablet_manager.cc:1428] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260809 23:23:47.832715  7561 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.842453  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
W20260809 23:23:47.846309  7437 leader_election.cc:341] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.848874  8015 leader_election.cc:288] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.849143  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:47.850425  7273 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.854276  7558 tablet_bootstrap.cc:492] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.854918  7558 ts_tablet_manager.cc:1397] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.066s	user 0.010s	sys 0.001s
I20260809 23:23:47.860716  7561 tablet_bootstrap.cc:654] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.862262  7558 raft_consensus.cc:348] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.864240  7558 raft_consensus.cc:374] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.864598  7558 raft_consensus.cc:729] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.868047  7566 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:47.868528  7566 ts_tablet_manager.cc:1397] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.086s	user 0.017s	sys 0.025s
I20260809 23:23:47.865461  7558 consensus_queue.cc:260] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.871368  7558 ts_tablet_manager.cc:1428] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260809 23:23:47.871461  7566 raft_consensus.cc:348] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.872052  7566 raft_consensus.cc:374] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.872153  7559 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:47.872359  7566 raft_consensus.cc:729] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.872835  7559 ts_tablet_manager.cc:1397] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.003s
I20260809 23:23:47.873011  7558 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:47.873178  7566 consensus_queue.cc:260] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.875478  7559 raft_consensus.cc:348] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.876168  7559 raft_consensus.cc:374] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.876493  7559 raft_consensus.cc:729] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.875821  7566 ts_tablet_manager.cc:1428] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:23:47.877394  7559 consensus_queue.cc:260] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.879938  7559 ts_tablet_manager.cc:1428] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:23:47.878479  7561 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:47.881768  7561 ts_tablet_manager.cc:1397] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.049s	user 0.013s	sys 0.003s
I20260809 23:23:47.884187  7558 tablet_bootstrap.cc:654] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.884748  7273 raft_consensus.cc:2454] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:47.888197  7583 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.889518  7437 leader_election.cc:302] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.889658  7583 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.891078  8015 raft_consensus.cc:2781] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:47.891749  8015 raft_consensus.cc:686] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:47.892738  8015 consensus_queue.cc:237] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.893766  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.894418  7376 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 0.
I20260809 23:23:47.895915  7124 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:47.896780  7938 raft_consensus.cc:2781] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.897118  7938 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.897202  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.897511  7938 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.884866  7561 raft_consensus.cc:348] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.886147  7566 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: Bootstrap starting.
I20260809 23:23:47.892100  7583 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:47.901691  7937 raft_consensus.cc:482] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.904083  7583 raft_consensus.cc:482] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.903793  7937 raft_consensus.cc:504] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:47.904583  7583 raft_consensus.cc:504] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:47.906802  7583 leader_election.cc:288] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.907452  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.909495  7937 leader_election.cc:288] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.899595  7561 raft_consensus.cc:374] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.910301  7561 raft_consensus.cc:729] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.903256  7559 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:47.912463  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.913733  7561 consensus_queue.cc:260] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
W20260809 23:23:47.918745  7124 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.920744  7561 ts_tablet_manager.cc:1428] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.000s
I20260809 23:23:47.921931  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:47.922902  7275 raft_consensus.cc:2379] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:47.911093  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:47.914216  7382 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
W20260809 23:23:47.926308  7228 leader_election.cc:341] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.927258  7124 leader_election.cc:302] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:47.927670  8042 consensus_queue.cc:1035] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:47.928251  7566 tablet_bootstrap.cc:654] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.931372  7227 leader_election.cc:302] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:47.933107  7937 raft_consensus.cc:2781] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:47.933588  7937 raft_consensus.cc:482] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:47.934044  7937 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.921988  7561 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:47.938064  7789 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.939127  8042 consensus_queue.cc:1035] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:47.944568  7559 tablet_bootstrap.cc:654] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.945861  7938 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.962841  7561 tablet_bootstrap.cc:654] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:47.967945  8035 raft_consensus.cc:482] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.968447  8035 raft_consensus.cc:504] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.961442  7938 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:47.968783  7027 catalog_manager.cc:5495] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:47.976518  7558 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:47.972795  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:47.977631  7558 ts_tablet_manager.cc:1397] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.105s	user 0.007s	sys 0.018s
I20260809 23:23:47.979847  7938 raft_consensus.cc:482] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:47.980381  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:47.980350  7938 raft_consensus.cc:504] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.981153  7376 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:47.982985  7579 raft_consensus.cc:482] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:23:47.979713  7124 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.983438  7579 raft_consensus.cc:504] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:47.985297  7579 leader_election.cc:288] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:47.985808  7938 leader_election.cc:288] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:47.987169  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.987684  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:47.990163  7437 leader_election.cc:341] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.991093  8035 leader_election.cc:288] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:47.992684  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:47.994014  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:47.993319  7376 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:47.995157  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:47.995611  7338 leader_election.cc:341] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.993352  7382 raft_consensus.cc:2454] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
W20260809 23:23:47.997195  7338 leader_election.cc:341] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:47.996735  7558 raft_consensus.cc:348] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:47.997740  7338 leader_election.cc:302] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:47.997838  7558 raft_consensus.cc:374] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:47.998450  7558 raft_consensus.cc:729] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:47.998989  8035 raft_consensus.cc:2726] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:47.999480  7558 consensus_queue.cc:260] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.006986  8037 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.012641  7437 leader_election.cc:302] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.018882  7579 raft_consensus.cc:2781] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.019424  7579 raft_consensus.cc:482] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:48.019917  7579 raft_consensus.cc:3037] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.029279  7558 ts_tablet_manager.cc:1428] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.051s	user 0.005s	sys 0.002s
I20260809 23:23:48.010910  7789 raft_consensus.cc:2726] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:48.045250  7558 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:48.050176  7566 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: No bootstrap required, opened a new log
I20260809 23:23:48.050868  7566 ts_tablet_manager.cc:1397] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: Time spent bootstrapping tablet: real 0.165s	user 0.006s	sys 0.023s
I20260809 23:23:48.059087  7579 raft_consensus.cc:504] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.018154  8037 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.054908  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f126def92864a43a94113cdd599ee4f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:48.030058  7937 raft_consensus.cc:504] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.069583  7382 raft_consensus.cc:2379] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:48.069456  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:48.070367  7937 leader_election.cc:288] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:48.070369  7171 raft_consensus.cc:2379] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:48.065907  8037 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:48.066738  7566 raft_consensus.cc:348] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.058483  7124 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:48.089617  7561 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:48.090047  7566 raft_consensus.cc:374] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.091286  7566 raft_consensus.cc:729] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.091408  7561 ts_tablet_manager.cc:1397] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.170s	user 0.024s	sys 0.019s
I20260809 23:23:48.091647  7579 leader_election.cc:288] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:48.092070  7566 consensus_queue.cc:260] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.093806  7566 ts_tablet_manager.cc:1428] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: Time spent starting tablet: real 0.043s	user 0.006s	sys 0.000s
W20260809 23:23:48.094801  7124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 46700.3us
I20260809 23:23:48.096372  8042 raft_consensus.cc:2781] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:48.096879  8042 raft_consensus.cc:686] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:23:48.097673  8042 consensus_queue.cc:237] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.102619  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
W20260809 23:23:48.106000  7228 leader_election.cc:341] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.108958  7559 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:48.109541  7559 ts_tablet_manager.cc:1397] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.206s	user 0.015s	sys 0.011s
I20260809 23:23:48.111985  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:48.112685  7377 raft_consensus.cc:3037] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.115104  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:48.115947  7380 raft_consensus.cc:2379] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 1: Already voted for candidate ff5d35f70eb7411ba26062ea7d36ab15 in this term.
I20260809 23:23:48.117676  7124 leader_election.cc:302] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:48.117990  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:48.118615  7380 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.118466  7561 raft_consensus.cc:348] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.118893  7789 raft_consensus.cc:3037] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.119254  7561 raft_consensus.cc:374] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.119565  7561 raft_consensus.cc:729] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.127175  7937 raft_consensus.cc:482] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.127900  7937 raft_consensus.cc:504] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.132807  7561 consensus_queue.cc:260] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.134215  8014 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.134721  8014 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.137545  8014 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.134888  8037 consensus_queue.cc:1035] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260809 23:23:48.139946  7377 raft_consensus.cc:2454] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:48.142231  7380 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:48.143152  7937 leader_election.cc:288] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:48.145277  8035 raft_consensus.cc:482] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.146073  8035 raft_consensus.cc:504] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.148182  8035 leader_election.cc:288] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:48.149214  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:48.156944  7561 ts_tablet_manager.cc:1428] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.001s
I20260809 23:23:48.159232  7227 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.161067  7561 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:48.167414  7561 tablet_bootstrap.cc:654] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.167707  7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:48.159101  7437 leader_election.cc:302] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
W20260809 23:23:48.172741  7338 leader_election.cc:341] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.173352  7579 raft_consensus.cc:2781] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:48.173849  7579 raft_consensus.cc:686] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:48.174564  7579 consensus_queue.cc:237] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.184230  7559 raft_consensus.cc:348] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.185040  7559 raft_consensus.cc:374] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.185729  7559 raft_consensus.cc:729] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
W20260809 23:23:48.185482  7124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 66039.7us
I20260809 23:23:48.186467  7559 consensus_queue.cc:260] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.182649  8037 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.195688  7559 ts_tablet_manager.cc:1428] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.086s	user 0.004s	sys 0.002s
W20260809 23:23:48.199829  7437 leader_election.cc:341] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.194339  7873 consensus_queue.cc:1035] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260809 23:23:48.199390  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:48.221544  7559 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:48.223021  7558 tablet_bootstrap.cc:654] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.218084  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:48.248709  7437 leader_election.cc:341] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.242103  8036 raft_consensus.cc:482] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.254186  7561 tablet_bootstrap.cc:492] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:48.254707  7561 ts_tablet_manager.cc:1397] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.094s	user 0.014s	sys 0.009s
I20260809 23:23:48.243237  7170 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.236545  7490 raft_consensus.cc:2454] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:48.266392  7027 catalog_manager.cc:5495] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:48.279075  7561 raft_consensus.cc:348] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.279958  7561 raft_consensus.cc:374] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.280246  7561 raft_consensus.cc:729] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.282239  7561 consensus_queue.cc:260] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.285516  7561 ts_tablet_manager.cc:1428] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.001s
I20260809 23:23:48.270751  7789 raft_consensus.cc:2726] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:48.292110  7579 consensus_queue.cc:1035] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260809 23:23:48.299834  7339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 112151us
I20260809 23:23:48.303673  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:48.270593  7228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 83885.1us
I20260809 23:23:48.310429  7227 leader_election.cc:302] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:48.258608  8036 raft_consensus.cc:504] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.312080  7559 tablet_bootstrap.cc:654] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.313740  7561 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:48.331534  8036 leader_election.cc:288] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.314313  8134 raft_consensus.cc:2781] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:48.339151  8134 raft_consensus.cc:686] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:48.340000  8134 consensus_queue.cc:237] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.340868  8037 raft_consensus.cc:2726] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:48.343758  8118 consensus_queue.cc:1035] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:48.349112  8014 consensus_queue.cc:1035] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:48.353073  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:48.353765  7271 raft_consensus.cc:2454] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:48.392252  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:48.395589  7338 leader_election.cc:341] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.403035  7579 raft_consensus.cc:482] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.403584  7579 raft_consensus.cc:504] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.414294  7579 leader_election.cc:288] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:48.387784  7228 leader_election.cc:302] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:48.416833  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:48.421073  7558 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:48.423195  7377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:48.424031  7377 raft_consensus.cc:2454] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:48.425231  7558 ts_tablet_manager.cc:1397] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.388s	user 0.032s	sys 0.038s
I20260809 23:23:48.411525  7938 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.428619  7437 leader_election.cc:302] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:48.418166  7030 catalog_manager.cc:5495] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:48.432085  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:48.432767  7271 raft_consensus.cc:2454] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 0.
I20260809 23:23:48.433701  7901 raft_consensus.cc:2781] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.434131  7901 raft_consensus.cc:482] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:48.434453  7901 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:48.434952  7339 leader_election.cc:341] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.426789  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:48.444479  7338 leader_election.cc:302] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.435729  8134 raft_consensus.cc:2781] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.448911  7579 consensus_queue.cc:1035] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:48.435459  7339 leader_election.cc:302] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.455894  7561 tablet_bootstrap.cc:654] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.452368  8134 raft_consensus.cc:482] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:23:48.455578  7437 leader_election.cc:341] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.451401  8035 raft_consensus.cc:2781] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.436352  7938 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.452800  7380 raft_consensus.cc:2454] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:48.459973  8134 raft_consensus.cc:3037] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.474718  7579 consensus_queue.cc:1035] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:48.465451  8035 raft_consensus.cc:482] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:48.483287  7437 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.497390  7938 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.482733  8035 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.503718  7271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:48.504014  8051 raft_consensus.cc:2781] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.504429  8051 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:48.504796  8051 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.482069  8132 raft_consensus.cc:2726] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:48.515097  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:48.522119  7558 raft_consensus.cc:348] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.523365  7558 raft_consensus.cc:374] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.523710  7558 raft_consensus.cc:729] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.524552  7558 consensus_queue.cc:260] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.537479  7559 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:48.538201  7559 ts_tablet_manager.cc:1397] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.333s	user 0.024s	sys 0.013s
I20260809 23:23:48.540959  7559 raft_consensus.cc:348] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.541673  7559 raft_consensus.cc:374] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.541929  7559 raft_consensus.cc:729] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.504432  7271 raft_consensus.cc:2454] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 1.
I20260809 23:23:48.544454  7124 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:48.542697  7559 consensus_queue.cc:260] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.553205  7559 ts_tablet_manager.cc:1428] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260809 23:23:48.554320  7559 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:48.555320  8035 raft_consensus.cc:504] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.561611  8035 leader_election.cc:288] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.537479  7558 ts_tablet_manager.cc:1428] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.112s	user 0.007s	sys 0.001s
I20260809 23:23:48.569973  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:48.570807  7273 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.571039  7558 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: Bootstrap starting.
I20260809 23:23:48.577450  8134 raft_consensus.cc:504] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.588510  8134 leader_election.cc:288] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:48.584064  8051 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.579687  7579 consensus_queue.cc:1035] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:48.594674  7938 consensus_queue.cc:1035] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:48.595392  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:48.622588  7171 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.625694  8042 raft_consensus.cc:2781] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:48.626874  8042 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3cc3ec2dcd449de83f6c88e80d2d6de)
I20260809 23:23:48.627352  8042 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:48.602566  7559 tablet_bootstrap.cc:654] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.602689  7901 raft_consensus.cc:504] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.605173  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:48.607060  8015 raft_consensus.cc:482] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260809 23:23:48.610129  7093 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:23:48.615936  7561 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
W20260809 23:23:48.630297  7339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 195391us
I20260809 23:23:48.650012  7558 tablet_bootstrap.cc:654] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.657642  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e008aea8ad64b2b9fa356cca2300ea1" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:48.658488  7382 raft_consensus.cc:3037] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.635624  7171 raft_consensus.cc:2454] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:48.639854  7486 raft_consensus.cc:3037] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.640465  7561 ts_tablet_manager.cc:1397] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.327s	user 0.018s	sys 0.011s
I20260809 23:23:48.627743  8051 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.672254  7561 raft_consensus.cc:348] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.676525  7561 raft_consensus.cc:374] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.676817  7561 raft_consensus.cc:729] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.677448  7561 consensus_queue.cc:260] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.679625  7561 ts_tablet_manager.cc:1428] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260809 23:23:48.679548  8042 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.686300  8042 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:48.686666  7561 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: Bootstrap starting.
I20260809 23:23:48.679915  7382 raft_consensus.cc:2454] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:48.695493  8123 consensus_queue.cc:1035] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:48.698519  7227 leader_election.cc:302] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:48.699226  8037 raft_consensus.cc:2781] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:48.699754  8037 raft_consensus.cc:686] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:48.700584  8037 consensus_queue.cc:237] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.718704  7558 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: No bootstrap required, opened a new log
I20260809 23:23:48.719272  7558 ts_tablet_manager.cc:1397] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: Time spent bootstrapping tablet: real 0.148s	user 0.015s	sys 0.008s
I20260809 23:23:48.722369  7558 raft_consensus.cc:348] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.640444  8015 raft_consensus.cc:504] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:48.737097  8015 leader_election.cc:288] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:48.641005  7901 leader_election.cc:288] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:48.725625  7561 tablet_bootstrap.cc:654] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:48.673558  7273 raft_consensus.cc:2454] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:48.747116  7338 leader_election.cc:302] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:48.748597  7852 raft_consensus.cc:2781] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:48.749190  7852 raft_consensus.cc:686] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:48.750319  7852 consensus_queue.cc:237] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.751847  7558 raft_consensus.cc:374] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.742462  7024 catalog_manager.cc:5495] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:48.755113  7558 raft_consensus.cc:729] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Initialized, Role: FOLLOWER
W20260809 23:23:48.758772  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 60332.4us
I20260809 23:23:48.759518  7558 consensus_queue.cc:260] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.762362  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:48.758844  7561 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: No bootstrap required, opened a new log
I20260809 23:23:48.825428  7561 ts_tablet_manager.cc:1397] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent bootstrapping tablet: real 0.139s	user 0.011s	sys 0.001s
I20260809 23:23:48.689747  7486 raft_consensus.cc:2454] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:48.833375  7558 ts_tablet_manager.cc:1428] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: Time spent starting tablet: real 0.114s	user 0.007s	sys 0.001s
I20260809 23:23:48.845721  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
W20260809 23:23:48.889315  7228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 48713.8us
W20260809 23:23:48.892009  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 116874us
I20260809 23:23:48.903522  7559 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:48.904121  7559 ts_tablet_manager.cc:1397] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.350s	user 0.027s	sys 0.044s
I20260809 23:23:48.896166  7873 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260809 23:23:48.912084  7873 raft_consensus.cc:482] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.912555  7873 raft_consensus.cc:504] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:48.913152  7561 raft_consensus.cc:348] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.913877  7561 raft_consensus.cc:374] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.914145  7561 raft_consensus.cc:729] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Initialized, Role: FOLLOWER
I20260809 23:23:48.914549  7873 leader_election.cc:288] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:48.916419  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:48.917449  7559 raft_consensus.cc:348] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.918155  7559 raft_consensus.cc:374] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:48.918428  7559 raft_consensus.cc:729] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
W20260809 23:23:48.921873  7227 leader_election.cc:341] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.919137  7559 consensus_queue.cc:260] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:48.918269  7561 consensus_queue.cc:260] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.943248  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:48.944031  7381 raft_consensus.cc:2454] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:48.951229  8137 raft_consensus.cc:482] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:48.951800  8137 raft_consensus.cc:504] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:48.951690  7227 leader_election.cc:302] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:48.954470  7559 ts_tablet_manager.cc:1428] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
I20260809 23:23:48.955615  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d68de750e444f43a513619eac515b03" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:48.955852  7873 raft_consensus.cc:2781] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:48.956230  7873 raft_consensus.cc:482] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:48.956547  7873 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.953789  8137 leader_election.cc:288] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
W20260809 23:23:48.958825  7227 leader_election.cc:341] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:48.959566  7559 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:48.963485  7561 ts_tablet_manager.cc:1428] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: Time spent starting tablet: real 0.138s	user 0.005s	sys 0.003s
I20260809 23:23:48.968657  7126 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:48.971400  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:48.972055  7381 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.975026  8125 raft_consensus.cc:2726] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:23:48.976727  7174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:48.977226  7171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:48.977396  7174 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.982947  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:48.983539  7273 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:48.986577  7381 raft_consensus.cc:2454] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:48.990063  7174 raft_consensus.cc:2454] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:49.001137  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:49.013254  8180 raft_consensus.cc:482] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:49.014099  8180 raft_consensus.cc:504] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.021255  8180 leader_election.cc:288] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:49.034401  7488 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.049206  7273 raft_consensus.cc:2454] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:49.053444  7559 tablet_bootstrap.cc:654] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:49.066507  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:49.066972  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:49.067301  7378 raft_consensus.cc:2454] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 0.
I20260809 23:23:49.076388  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d68de750e444f43a513619eac515b03" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:49.077138  7487 raft_consensus.cc:2454] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 0.
I20260809 23:23:49.094828  7937 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.103051  7030 catalog_manager.cc:5495] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 1 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:49.110419  7228 leader_election.cc:302] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:49.111824  8137 raft_consensus.cc:2781] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:49.112228  8137 raft_consensus.cc:482] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:49.112551  8137 raft_consensus.cc:3037] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.054142  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:49.125435  7437 leader_election.cc:302] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:49.139134  8118 consensus_queue.cc:1035] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260809 23:23:49.135236  7901 raft_consensus.cc:2781] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:49.144448  7901 raft_consensus.cc:686] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:49.134656  7873 raft_consensus.cc:504] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:49.145414  7901 consensus_queue.cc:237] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.152794  7437 leader_election.cc:302] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:49.153839  8180 raft_consensus.cc:2781] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:23:49.154345  8180 raft_consensus.cc:482] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:23:49.154814  8180 raft_consensus.cc:3037] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.162418  8180 raft_consensus.cc:504] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.164260  8180 leader_election.cc:288] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:49.164278  7873 leader_election.cc:288] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:49.165400  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
W20260809 23:23:49.167104  7227 leader_election.cc:341] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:49.167160  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:49.167953  7382 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.169145  7437 leader_election.cc:302] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:49.171362  7896 raft_consensus.cc:2726] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:49.172636  7437 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:49.179324  7896 raft_consensus.cc:2781] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:49.179843  7896 raft_consensus.cc:686] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:49.180729  7896 consensus_queue.cc:237] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.142505  8137 raft_consensus.cc:504] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.190335  7030 catalog_manager.cc:5495] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:49.196084  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d68de750e444f43a513619eac515b03" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:49.196703  7488 raft_consensus.cc:3037] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:23:49.198963  7437 leader_election.cc:341] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:49.200477  7437 consensus_peers.cc:487] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:49.201771  8137 leader_election.cc:288] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:49.202710  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d68de750e444f43a513619eac515b03" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
W20260809 23:23:49.202713  7437 leader_election.cc:341] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:23:49.204207  7227 leader_election.cc:341] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:49.207474  7382 raft_consensus.cc:2454] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:49.208765  7227 leader_election.cc:302] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:49.209949  7937 raft_consensus.cc:2781] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:49.210487  7937 raft_consensus.cc:686] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:49.211373  7937 consensus_queue.cc:237] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:49.209642  8137 consensus_queue.cc:1035] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.221606  7488 raft_consensus.cc:2454] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:49.227310  7228 leader_election.cc:302] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:49.228014  7173 raft_consensus.cc:1226] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Rejecting Update request from peer a3cc3ec2dcd449de83f6c88e80d2d6de for earlier term 1. Current term is 2. Ops: []
I20260809 23:23:49.228467  8037 raft_consensus.cc:2781] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:49.228962  8037 raft_consensus.cc:686] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:49.229751  8037 consensus_queue.cc:237] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.243340  7873 consensus_queue.cc:1035] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.257982  8135 consensus_queue.cc:1035] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.262676  7030 catalog_manager.cc:5495] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:49.265184  7030 catalog_manager.cc:5495] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:49.230367  7559 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:49.275985  7559 ts_tablet_manager.cc:1397] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.317s	user 0.010s	sys 0.017s
I20260809 23:23:49.290086  8135 consensus_queue.cc:1035] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.294234  7380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:49.294310  7173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:49.294997  7380 raft_consensus.cc:3037] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.308130  7380 raft_consensus.cc:2454] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:49.345470  7027 catalog_manager.cc:5495] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:49.362000  7437 leader_election.cc:302] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:49.363154  8180 raft_consensus.cc:2781] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:23:49.363672  8180 raft_consensus.cc:686] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:49.365041  8180 consensus_queue.cc:237] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.380668  7030 catalog_manager.cc:5495] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 0 to 1, leader changed from <none> to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 1 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:49.278497  7559 raft_consensus.cc:348] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:49.382576  7559 raft_consensus.cc:374] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:49.393882  7901 consensus_queue.cc:1046] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.395794  8180 raft_consensus.cc:3032] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:23:49.396190  8180 raft_consensus.cc:729] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:49.397037  8180 consensus_queue.cc:260] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.405390  8180 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:23:49.407330  7437 leader_election.cc:341] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:49.382875  7559 raft_consensus.cc:729] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:49.408418  7030 catalog_manager.cc:5495] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:49.413230  7559 consensus_queue.cc:260] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:49.416399  7559 ts_tablet_manager.cc:1428] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.140s	user 0.005s	sys 0.001s
I20260809 23:23:49.418365  7559 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:49.423823  7559 tablet_bootstrap.cc:654] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:49.456287  7559 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:49.456912  7559 ts_tablet_manager.cc:1397] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.039s	user 0.018s	sys 0.008s
I20260809 23:23:49.460383  7559 raft_consensus.cc:348] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.461179  7559 raft_consensus.cc:374] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:49.461496  7559 raft_consensus.cc:729] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:49.462252  7559 consensus_queue.cc:260] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.464686  7559 ts_tablet_manager.cc:1428] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:23:49.465695  7559 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:49.474751  7559 tablet_bootstrap.cc:654] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:49.489591  7559 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:49.490172  7559 ts_tablet_manager.cc:1397] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.025s	user 0.015s	sys 0.002s
I20260809 23:23:49.492863  7559 raft_consensus.cc:348] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.493628  7559 raft_consensus.cc:374] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:49.493948  7559 raft_consensus.cc:729] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:49.494696  7559 consensus_queue.cc:260] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.496675  7559 ts_tablet_manager.cc:1428] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.002s
I20260809 23:23:49.497566  7559 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:49.502964  7559 tablet_bootstrap.cc:654] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:49.507318  7559 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:49.507694  7559 ts_tablet_manager.cc:1397] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.010s	user 0.009s	sys 0.000s
I20260809 23:23:49.509479  7559 raft_consensus.cc:348] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.509985  7559 raft_consensus.cc:374] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:49.510270  7559 raft_consensus.cc:729] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:49.511003  7559 consensus_queue.cc:260] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.512976  7559 ts_tablet_manager.cc:1428] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:23:49.514447  7559 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:49.515713  7789 raft_consensus.cc:482] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:49.516261  7789 raft_consensus.cc:504] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:49.518384  7789 leader_election.cc:288] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:49.519042  7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:49.522320  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:49.523132  7378 raft_consensus.cc:2379] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:49.523698  7559 tablet_bootstrap.cc:654] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Neither blocks nor log segments found. Creating new log.
I20260809 23:23:49.525064  7124 leader_election.cc:302] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:49.526396  8125 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.548565  8125 raft_consensus.cc:2726] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:49.559985  7559 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: No bootstrap required, opened a new log
I20260809 23:23:49.560937  7559 ts_tablet_manager.cc:1397] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.047s	user 0.020s	sys 0.018s
I20260809 23:23:49.564306  7559 raft_consensus.cc:348] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.564998  7559 raft_consensus.cc:374] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:23:49.565258  7559 raft_consensus.cc:729] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:49.565891  7559 consensus_queue.cc:260] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.567868  7559 ts_tablet_manager.cc:1428] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:23:49.614499  8125 raft_consensus.cc:482] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:49.615182  8125 raft_consensus.cc:504] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.616264  7173 raft_consensus.cc:3037] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.621297  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:49.623510  8125 leader_election.cc:288] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:49.624191  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29afbcdad59b414fa48484b70ff56499" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:49.626235  7124 leader_election.cc:302] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:49.634035  7938 raft_consensus.cc:2726] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:49.651993  7938 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:49.652555  7938 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:49.655155  7938 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:49.656973  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:49.659176  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:49.659912  7275 raft_consensus.cc:2379] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:49.660919  7124 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:23:49.660970  7937 consensus_queue.cc:1035] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.662005  7938 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.695950  8174 consensus_queue.cc:1035] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.696636  8037 consensus_queue.cc:1035] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.698907  7174 raft_consensus.cc:3037] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.711189  8125 raft_consensus.cc:482] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:49.711066  8180 consensus_queue.cc:1035] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.711786  8125 raft_consensus.cc:504] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:49.713863  8125 leader_election.cc:288] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:23:49.715678  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:49.715984  7938 raft_consensus.cc:2726] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:49.717388  8037 consensus_queue.cc:1035] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.719786  8037 consensus_queue.cc:1035] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:23:49.721268  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964bd1390e2d43b9ab381a795bdddedf" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:49.722056  7378 raft_consensus.cc:2379] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff5d35f70eb7411ba26062ea7d36ab15 in current term 1: Already voted for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in this term.
I20260809 23:23:49.727283  7124 leader_election.cc:302] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de
I20260809 23:23:49.729060  8014 consensus_queue.cc:1035] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.728752  8125 raft_consensus.cc:3037] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:23:49.811107  8180 consensus_queue.cc:1035] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.817998  8125 raft_consensus.cc:2726] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:23:49.824054  8180 consensus_queue.cc:1035] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:23:49.849864 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
W20260809 23:23:49.874692  7124 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 104456us
W20260809 23:23:49.887668  7437 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:23:49.988052  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 86380.9us
W20260809 23:23:50.016884  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 64081.5us
W20260809 23:23:50.025593  7437 consensus_peers.cc:487] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.026808  7437 consensus_peers.cc:487] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.029804  7437 consensus_peers.cc:487] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.031302  7437 consensus_peers.cc:487] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.032063  7437 consensus_peers.cc:487] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.040674  7227 consensus_peers.cc:487] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.042520  7437 consensus_peers.cc:487] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.054800  7227 consensus_peers.cc:487] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.074337  7227 consensus_peers.cc:487] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.077615  7437 consensus_peers.cc:487] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.078295  7437 consensus_peers.cc:487] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.112792  7227 consensus_peers.cc:487] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.113313  7338 consensus_peers.cc:487] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.125964  7437 consensus_peers.cc:487] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.139459  7338 consensus_peers.cc:487] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.140161  7338 consensus_peers.cc:487] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.161650  7338 consensus_peers.cc:487] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.162330  7338 consensus_peers.cc:487] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.163534  7338 consensus_peers.cc:487] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.164615  7437 consensus_peers.cc:487] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.170255  7338 consensus_peers.cc:487] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.172264  7338 consensus_peers.cc:487] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.179847  7437 consensus_peers.cc:487] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.185416  7227 consensus_peers.cc:487] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.191681 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:23:50.193065 22524 tablet_replica.cc:330] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
W20260809 23:23:50.193786  7227 consensus_peers.cc:487] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.194499 22524 raft_consensus.cc:2229] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.195179 22524 raft_consensus.cc:2258] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.197268 22524 tablet_replica.cc:330] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.198053 22524 raft_consensus.cc:2229] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.198436 22524 pending_rounds.cc:62] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Trying to abort 1 pending ops.
I20260809 23:23:50.198621 22524 pending_rounds.cc:68] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316757828336472064 op_type: NO_OP noop_request { }
I20260809 23:23:50.198989 22524 raft_consensus.cc:2866] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:23:50.199294 22524 raft_consensus.cc:2258] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.201591 22524 tablet_replica.cc:330] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.202212 22524 raft_consensus.cc:2229] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.202620 22524 raft_consensus.cc:2258] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.205680 22524 tablet_replica.cc:330] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.206447 22524 raft_consensus.cc:2229] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.206990 22524 raft_consensus.cc:2258] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.210244 22524 tablet_replica.cc:330] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.211110 22524 raft_consensus.cc:2229] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.211537 22524 raft_consensus.cc:2258] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.213294 22524 tablet_replica.cc:330] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.213881 22524 raft_consensus.cc:2229] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.214274 22524 raft_consensus.cc:2258] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.215986 22524 tablet_replica.cc:330] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.216527 22524 raft_consensus.cc:2229] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.217319 22524 raft_consensus.cc:2258] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.220033 22524 tablet_replica.cc:330] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.220697 22524 raft_consensus.cc:2229] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.221124 22524 raft_consensus.cc:2258] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.222823 22524 tablet_replica.cc:330] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.223472 22524 raft_consensus.cc:2229] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.223776 22524 pending_rounds.cc:62] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Trying to abort 1 pending ops.
I20260809 23:23:50.223908 22524 pending_rounds.cc:68] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316757822224310272 op_type: NO_OP noop_request { }
I20260809 23:23:50.224143 22524 raft_consensus.cc:2866] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:23:50.224344 22524 raft_consensus.cc:2258] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.226279 22524 tablet_replica.cc:330] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.226905 22524 raft_consensus.cc:2229] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.227744 22524 raft_consensus.cc:2258] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.229589 22524 tablet_replica.cc:330] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.230134 22524 raft_consensus.cc:2229] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.231070 22524 raft_consensus.cc:2258] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.233844 22524 tablet_replica.cc:330] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.234637 22524 raft_consensus.cc:2229] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.235186 22524 raft_consensus.cc:2258] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.237695 22524 tablet_replica.cc:330] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.238482 22524 raft_consensus.cc:2229] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.239012 22524 raft_consensus.cc:2258] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.241111 22524 tablet_replica.cc:330] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.241868 22524 raft_consensus.cc:2229] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.242336 22524 raft_consensus.cc:2258] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.244585 22524 tablet_replica.cc:330] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.245393 22524 raft_consensus.cc:2229] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.245887 22524 raft_consensus.cc:2258] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.249289 22524 tablet_replica.cc:330] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.250126 22524 raft_consensus.cc:2229] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.251375 22524 raft_consensus.cc:2258] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:50.253473  7338 consensus_peers.cc:487] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:50.253926  7227 consensus_peers.cc:487] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.254189 22524 tablet_replica.cc:330] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.255149 22524 raft_consensus.cc:2229] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.256201 22524 raft_consensus.cc:2258] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:50.257876  7227 consensus_peers.cc:487] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.258956 22524 tablet_replica.cc:330] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.259940 22524 raft_consensus.cc:2229] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.261011 22524 raft_consensus.cc:2258] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:50.262295  7437 consensus_peers.cc:487] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.263698 22524 tablet_replica.cc:330] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
W20260809 23:23:50.264387  7437 consensus_peers.cc:487] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.264716 22524 raft_consensus.cc:2229] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.265275 22524 pending_rounds.cc:62] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Trying to abort 1 pending ops.
I20260809 23:23:50.265533 22524 pending_rounds.cc:68] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316757828193095680 op_type: NO_OP noop_request { }
I20260809 23:23:50.265936 22524 raft_consensus.cc:2866] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:23:50.266323 22524 raft_consensus.cc:2258] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.271070 22524 tablet_replica.cc:330] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.272984 22524 raft_consensus.cc:2229] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.273542 22524 raft_consensus.cc:2258] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.276116 22524 tablet_replica.cc:330] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.277006 22524 raft_consensus.cc:2229] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.277561 22524 raft_consensus.cc:2258] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.280077 22524 tablet_replica.cc:330] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.280970 22524 raft_consensus.cc:2229] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.281533 22524 raft_consensus.cc:2258] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.284585 22524 tablet_replica.cc:330] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.285454 22524 raft_consensus.cc:2229] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:50.285569  7227 consensus_peers.cc:487] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.286484 22524 raft_consensus.cc:2258] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.289800 22524 tablet_replica.cc:330] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.291090 22524 raft_consensus.cc:2229] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 23:23:50.291637  7437 consensus_peers.cc:487] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.291725 22524 raft_consensus.cc:2258] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.294771 22524 tablet_replica.cc:330] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.295739 22524 raft_consensus.cc:2229] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.296293 22524 raft_consensus.cc:2258] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.298970 22524 tablet_replica.cc:330] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.300431 22524 raft_consensus.cc:2229] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.301503 22524 raft_consensus.cc:2258] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.304260 22524 tablet_replica.cc:330] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.305276 22524 raft_consensus.cc:2229] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.305838 22524 raft_consensus.cc:2258] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.308619 22524 tablet_replica.cc:330] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.309473 22524 raft_consensus.cc:2229] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.309989 22524 raft_consensus.cc:2258] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.312371 22524 tablet_replica.cc:330] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.313218 22524 raft_consensus.cc:2229] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.313751 22524 raft_consensus.cc:2258] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.317807 22524 tablet_replica.cc:330] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.318486 22524 raft_consensus.cc:2229] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.318825 22524 raft_consensus.cc:2258] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.320621 22524 tablet_replica.cc:330] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.321257 22524 raft_consensus.cc:2229] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.321662 22524 raft_consensus.cc:2258] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.323359 22524 tablet_replica.cc:330] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.324111 22524 raft_consensus.cc:2229] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.325146 22524 raft_consensus.cc:2258] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.327647 22524 tablet_replica.cc:330] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.328145 22524 raft_consensus.cc:2229] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.328830 22524 raft_consensus.cc:2258] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.330660 22524 tablet_replica.cc:330] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.331297 22524 raft_consensus.cc:2229] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.331741 22524 raft_consensus.cc:2258] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.334384 22524 tablet_replica.cc:330] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.335238 22524 raft_consensus.cc:2229] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.335784 22524 raft_consensus.cc:2258] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.341166 22524 tablet_replica.cc:330] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.342033 22524 raft_consensus.cc:2229] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.342636 22524 raft_consensus.cc:2258] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.345206 22524 tablet_replica.cc:330] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.346031 22524 raft_consensus.cc:2229] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.346550 22524 raft_consensus.cc:2258] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.349092 22524 tablet_replica.cc:330] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.349905 22524 raft_consensus.cc:2229] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.350589 22524 raft_consensus.cc:2258] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.352859 22524 tablet_replica.cc:330] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.353637 22524 raft_consensus.cc:2229] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.354260 22524 raft_consensus.cc:2258] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.357049 22524 tablet_replica.cc:330] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.357834 22524 raft_consensus.cc:2229] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.358395 22524 raft_consensus.cc:2258] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.360723 22524 tablet_replica.cc:330] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.361245 22524 raft_consensus.cc:2229] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.361598 22524 pending_rounds.cc:62] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Trying to abort 1 pending ops.
I20260809 23:23:50.361789 22524 pending_rounds.cc:68] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316757828566794240 op_type: NO_OP noop_request { }
I20260809 23:23:50.362072 22524 raft_consensus.cc:2866] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260809 23:23:50.362340 22524 raft_consensus.cc:2258] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.364817 22524 tablet_replica.cc:330] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.365363 22524 raft_consensus.cc:2229] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.365769 22524 raft_consensus.cc:2258] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.367658 22524 tablet_replica.cc:330] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.368317 22524 raft_consensus.cc:2229] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:23:50.369342 22524 raft_consensus.cc:2258] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.372023 22524 tablet_replica.cc:330] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
W20260809 23:23:50.372118  7227 consensus_peers.cc:487] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.372960 22524 raft_consensus.cc:2229] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.373572 22524 raft_consensus.cc:2258] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:23:50.376173 22524 tablet_replica.cc:330] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: stopping tablet replica
I20260809 23:23:50.376919 22524 raft_consensus.cc:2229] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:23:50.377470 22524 raft_consensus.cc:2258] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:23:50.396767  7437 consensus_peers.cc:487] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:50.467764 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
I20260809 23:23:50.542402 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:23:50.556468  8250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:50.571007  8174 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:23:50.571864  8174 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:50.573974  8174 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:50.578184  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:50.585597  7437 leader_election.cc:334] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:50.586038  7437 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:50.588161  8174 raft_consensus.cc:2726] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260809 23:23:50.589584  8251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:23:50.601899  8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:23:50.621645 22524 server_base.cc:1034] running on GCE node
I20260809 23:23:50.624773 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:23:50.625662 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:23:50.626130 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317830626042 us; error 0 us; skew 500 ppm
I20260809 23:23:50.627161 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:23:50.635149 22524 webserver.cc:460] Webserver started at http://127.21.255.1:46231/ using document root <none> and password file <none>
I20260809 23:23:50.635761 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:23:50.635990 22524 fs_manager.cc:362] Using existing metadata directory in first data directory
I20260809 23:23:50.640370 22524 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.000s	sys 0.004s
I20260809 23:23:50.658025  8260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:23:50.659144 22524 fs_manager.cc:725] Time spent opening block manager: real 0.009s	user 0.003s	sys 0.005s
I20260809 23:23:50.659600 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
format_stamp: "Formatted at 2026-08-09 23:23:38 on dist-test-slave-drl0"
I20260809 23:23:50.660339 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:23:50.681614 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:23:50.683142 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:23:50.685330 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:23:50.690042  8269 ts_tablet_manager.cc:542] Loading tablet metadata (0/45 complete)
I20260809 23:23:50.726949  8241 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3cc3ec2dcd449de83f6c88e80d2d6de)
I20260809 23:23:50.727702  8241 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:50.731624  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:50.732189  8241 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:50.732313  7490 raft_consensus.cc:2454] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 2.
W20260809 23:23:50.734890  7227 leader_election.cc:334] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:50.735500  7228 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:50.739408  8240 raft_consensus.cc:2781] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:50.739823  8240 raft_consensus.cc:482] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3cc3ec2dcd449de83f6c88e80d2d6de)
I20260809 23:23:50.740176  8240 raft_consensus.cc:3037] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:50.750841  8240 raft_consensus.cc:504] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:50.753770  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43caf538bbac4e0da20c274fad4d7f1f" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:50.758595  8240 leader_election.cc:288] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
W20260809 23:23:50.759147  7227 leader_election.cc:334] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:50.760967  7490 raft_consensus.cc:2454] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 2.
I20260809 23:23:50.761833  7228 leader_election.cc:302] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:50.762610  8240 raft_consensus.cc:2781] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:50.763304  8240 raft_consensus.cc:686] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:50.764037  8240 consensus_queue.cc:237] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:50.775998  7027 catalog_manager.cc:5495] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:51.022107 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260809 23:23:51.022511 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.334s	user 0.009s	sys 0.002s
I20260809 23:23:51.022917 22524 ts_tablet_manager.cc:594] Registering tablets (0/45 complete)
I20260809 23:23:51.033802  8269 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.049113  8269 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.050030  8269 tablet_bootstrap.cc:492] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.050662  8269 ts_tablet_manager.cc:1397] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260809 23:23:51.053040  8269 raft_consensus.cc:348] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.053789  8269 raft_consensus.cc:729] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.054461  8269 consensus_queue.cc:260] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.060295  8269 ts_tablet_manager.cc:1428] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.009s	user 0.012s	sys 0.000s
I20260809 23:23:51.061081  8269 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.074066  8269 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 23:23:51.075006  8269 tablet_bootstrap.cc:492] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.075522  8269 ts_tablet_manager.cc:1397] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.015s	user 0.013s	sys 0.000s
I20260809 23:23:51.077342  8269 raft_consensus.cc:348] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.079111  8269 raft_consensus.cc:729] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.079710  8269 consensus_queue.cc:260] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.081480  8269 ts_tablet_manager.cc:1428] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260809 23:23:51.082294  8269 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.096652  8269 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.097501  8269 tablet_bootstrap.cc:492] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.098203  8269 ts_tablet_manager.cc:1397] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.004s
I20260809 23:23:51.100616  8269 raft_consensus.cc:348] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.101361  8269 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.102013  8269 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.103888  8269 ts_tablet_manager.cc:1428] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260809 23:23:51.104622  8269 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.118445  8269 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.119249  8269 tablet_bootstrap.cc:492] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.119773  8269 ts_tablet_manager.cc:1397] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20260809 23:23:51.121488  8269 raft_consensus.cc:348] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.122008  8269 raft_consensus.cc:729] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.122516  8269 consensus_queue.cc:260] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.124091  8269 ts_tablet_manager.cc:1428] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.004s
I20260809 23:23:51.124670  8269 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.137475  8269 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.138345  8269 tablet_bootstrap.cc:492] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.138989  8269 ts_tablet_manager.cc:1397] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.002s
I20260809 23:23:51.141043  8269 raft_consensus.cc:348] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.141717  8269 raft_consensus.cc:729] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.142402  8269 consensus_queue.cc:260] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.144220  8269 ts_tablet_manager.cc:1428] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I20260809 23:23:51.144927  8269 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.146340  7490 raft_consensus.cc:1261] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:51.147918  8240 consensus_queue.cc:1035] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:23:51.163774  8269 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.168049  8269 tablet_bootstrap.cc:492] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.168781  8269 ts_tablet_manager.cc:1397] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.024s	user 0.017s	sys 0.000s
I20260809 23:23:51.171605  8269 raft_consensus.cc:348] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.172475  8269 raft_consensus.cc:729] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.173348  8269 consensus_queue.cc:260] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.176294  8269 ts_tablet_manager.cc:1428] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:23:51.177016  8269 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:51.186901  7227 consensus_peers.cc:487] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:51.199579  8269 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.201016  8269 tablet_bootstrap.cc:492] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.201697  8269 ts_tablet_manager.cc:1397] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.000s
I20260809 23:23:51.203812  8269 raft_consensus.cc:348] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:51.204488  8269 raft_consensus.cc:729] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.205106  8269 consensus_queue.cc:260] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:23:51.206717  8269 ts_tablet_manager.cc:1428] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260809 23:23:51.207379  8269 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.218344  8269 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.219214  8269 tablet_bootstrap.cc:492] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.219722  8269 ts_tablet_manager.cc:1397] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.012s	user 0.008s	sys 0.004s
I20260809 23:23:51.221350  8269 raft_consensus.cc:348] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.221858  8269 raft_consensus.cc:729] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.222435  8269 consensus_queue.cc:260] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.224088  8269 ts_tablet_manager.cc:1428] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260809 23:23:51.224762  8269 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.247627  8269 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 23:23:51.248675  8269 tablet_bootstrap.cc:492] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.248935  8241 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.249598  8269 ts_tablet_manager.cc:1397] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.003s
I20260809 23:23:51.249500  8241 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.251811  8241 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.253623  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.252314  8269 raft_consensus.cc:348] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.254552  8269 raft_consensus.cc:729] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.255414  8269 consensus_queue.cc:260] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:23:51.257238  7227 leader_election.cc:334] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:51.257689  8269 ts_tablet_manager.cc:1428] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.001s
I20260809 23:23:51.257664  7227 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.258598  8269 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.258855  8241 raft_consensus.cc:2726] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.263702  8135 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.264122  8135 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.265586  8135 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.266626  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
W20260809 23:23:51.272109  7338 leader_election.cc:334] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:51.272601  7338 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.273530  8135 raft_consensus.cc:2726] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:23:51.289688  8269 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.291442  8269 tablet_bootstrap.cc:492] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.292932  8269 ts_tablet_manager.cc:1397] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.034s	user 0.027s	sys 0.004s
I20260809 23:23:51.295843  8269 raft_consensus.cc:348] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.296628  8269 raft_consensus.cc:729] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.297407  8269 consensus_queue.cc:260] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.299513  8269 ts_tablet_manager.cc:1428] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:23:51.300326  8269 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.311416 22524 ts_tablet_manager.cc:610] Registered 45 tablets
I20260809 23:23:51.311909 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.289s	user 0.260s	sys 0.003s
I20260809 23:23:51.321137  8269 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.322093  8269 tablet_bootstrap.cc:492] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.322773  8269 ts_tablet_manager.cc:1397] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.004s
I20260809 23:23:51.325747  8269 raft_consensus.cc:348] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.326728  8269 raft_consensus.cc:729] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.327570  8269 consensus_queue.cc:260] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.330314  8269 ts_tablet_manager.cc:1428] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260809 23:23:51.331406  8269 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.350281  8269 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.351485  8269 tablet_bootstrap.cc:492] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.352268  8269 ts_tablet_manager.cc:1397] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.021s	user 0.015s	sys 0.004s
I20260809 23:23:51.355368  8269 raft_consensus.cc:348] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.356230  8269 raft_consensus.cc:729] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.356976  8269 consensus_queue.cc:260] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.359483  8269 ts_tablet_manager.cc:1428] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260809 23:23:51.360450  8269 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.379879  8269 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.381021  8269 tablet_bootstrap.cc:492] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.381055  8240 raft_consensus.cc:482] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.382139  8269 ts_tablet_manager.cc:1397] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.022s	user 0.021s	sys 0.000s
I20260809 23:23:51.381852  8240 raft_consensus.cc:504] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.384033  8240 leader_election.cc:288] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.386324  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.385666  8269 raft_consensus.cc:348] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.388769  8269 raft_consensus.cc:729] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
W20260809 23:23:51.388978  7227 leader_election.cc:334] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:23:51.389428  7227 leader_election.cc:302] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.390090  8240 raft_consensus.cc:2726] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.389704  8269 consensus_queue.cc:260] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.392695  8269 ts_tablet_manager.cc:1428] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.007s
I20260809 23:23:51.393666  8269 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.396306 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:37549
I20260809 23:23:51.396435  8373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:37549 every 8 connection(s)
I20260809 23:23:51.398766  8240 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.399322  8240 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.405123  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.406780  8240 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:51.416749  8269 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.417922  8269 tablet_bootstrap.cc:492] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.418556  8269 ts_tablet_manager.cc:1397] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.006s
I20260809 23:23:51.428627  8269 raft_consensus.cc:348] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.429327  8269 raft_consensus.cc:729] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.430112  8269 consensus_queue.cc:260] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.432947  8269 ts_tablet_manager.cc:1428] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260809 23:23:51.434068  8269 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.437021  8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:51.445664  7227 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.446122  7227 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.447352  8240 raft_consensus.cc:2726] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.453084  8014 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.453982  8014 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.458195  8269 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.459304  8269 tablet_bootstrap.cc:492] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.459544  8014 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.460122  8269 ts_tablet_manager.cc:1397] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.026s	user 0.020s	sys 0.000s
I20260809 23:23:51.460857  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.461583  7378 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.462909  8374 heartbeater.cc:347] Connected to a master server at 127.21.255.62:37409
I20260809 23:23:51.463166  7437 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:51.463384  8374 heartbeater.cc:464] Registering TS with master...
I20260809 23:23:51.463363  8269 raft_consensus.cc:348] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.464033  8385 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260809 23:23:51.464416  8269 raft_consensus.cc:729] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.464545  8374 heartbeater.cc:511] Master 127.21.255.62:37409 requested a full tablet report, sending...
I20260809 23:23:51.464499  8385 raft_consensus.cc:482] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.465202  8385 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:23:51.465147  8269 consensus_queue.cc:260] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.468161  8269 ts_tablet_manager.cc:1428] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:23:51.469000  8269 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.473348  8385 raft_consensus.cc:504] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.475492  8385 leader_election.cc:288] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.477212  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:51.477814  7378 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:23:51.483363  7378 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 3.
I20260809 23:23:51.484402  7437 leader_election.cc:302] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:51.484839  7025 ts_manager.cc:194] Re-registered known tserver with Master: ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.485198  8269 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.487736  8385 raft_consensus.cc:2781] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:23:51.488286  8385 raft_consensus.cc:686] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:51.489989  8269 tablet_bootstrap.cc:492] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.489252  8385 consensus_queue.cc:237] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.491888  8269 ts_tablet_manager.cc:1397] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.007s
I20260809 23:23:51.494259  8269 raft_consensus.cc:348] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.494913  8269 raft_consensus.cc:729] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.499361  8269 consensus_queue.cc:260] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.501860  8269 ts_tablet_manager.cc:1428] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260809 23:23:51.504101  8269 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.505666  8385 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.506304  8385 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.508898  8385 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.509835  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:51.510572  7273 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:51.511648  7437 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.512272  8385 raft_consensus.cc:2781] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.512600  8385 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.512975  8385 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.519045  8385 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.520391  8269 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.521072  8385 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.521443  8269 tablet_bootstrap.cc:492] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.522178  8269 ts_tablet_manager.cc:1397] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.000s
I20260809 23:23:51.524155  7027 catalog_manager.cc:5495] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 2 to 3, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 3 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:51.525002  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:51.525626  7273 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.526360  8269 raft_consensus.cc:348] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.527364  8269 raft_consensus.cc:729] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.528023  8269 consensus_queue.cc:260] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.530113  8269 ts_tablet_manager.cc:1428] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.004s
I20260809 23:23:51.530865  8269 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.532994  7273 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.539567  8240 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.540122  8240 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.540892  7437 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.542230  8240 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.543027  8385 raft_consensus.cc:2781] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.543574  8385 raft_consensus.cc:686] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:51.544421  8385 consensus_queue.cc:237] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.548774  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.553855  8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:51.555939  7227 leader_election.cc:341] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.556571  7227 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.559370  7025 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44462
I20260809 23:23:51.560577  8240 raft_consensus.cc:2726] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.566841  8374 heartbeater.cc:503] Master 127.21.255.62:37409 was elected leader, sending a full tablet report...
I20260809 23:23:51.586647  8269 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.586632  7027 catalog_manager.cc:5495] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 2 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:51.587770  8269 tablet_bootstrap.cc:492] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.588505  8269 ts_tablet_manager.cc:1397] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.058s	user 0.022s	sys 0.016s
I20260809 23:23:51.592657  8269 raft_consensus.cc:348] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.593376  8269 raft_consensus.cc:729] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.594115  8269 consensus_queue.cc:260] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.596717  8269 ts_tablet_manager.cc:1428] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:23:51.597473  8269 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.611863  8149 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.612809  8269 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.612533  8149 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.613901  8269 tablet_bootstrap.cc:492] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.614858  8149 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.614850  8269 ts_tablet_manager.cc:1397] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.008s
I20260809 23:23:51.616808  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.617610  7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:51.619029  8269 raft_consensus.cc:348] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.619764  8269 raft_consensus.cc:729] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
W20260809 23:23:51.620528  7338 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.620427  8269 consensus_queue.cc:260] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.621026  7338 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.623270  8269 ts_tablet_manager.cc:1428] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:23:51.623289  8149 raft_consensus.cc:2726] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.624284  8269 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.637936  8385 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.638713  8385 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.639721  8269 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.639978  8014 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.640789  8269 tablet_bootstrap.cc:492] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.640584  8014 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.641583  8269 ts_tablet_manager.cc:1397] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20260809 23:23:51.642105  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.642999  8014 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.643659  8245 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.649986  8245 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.639001  8118 raft_consensus.cc:482] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.644423  8149 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.651346  8118 raft_consensus.cc:504] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.651499  8149 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.652204  8245 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.653472  8118 leader_election.cc:288] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.653699  8149 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.645200  8385 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.645241  7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:51.663264  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:51.664024  7275 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:51.663904  7270 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:51.646169  8135 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.644222  8269 raft_consensus.cc:348] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.665096  8269 raft_consensus.cc:729] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.665339  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.665843  8269 consensus_queue.cc:260] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.666240  7437 leader_election.cc:302] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.667153  8385 raft_consensus.cc:2781] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.667505  8385 raft_consensus.cc:482] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.667838  8385 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.668102  8269 ts_tablet_manager.cc:1428] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260809 23:23:51.669070  8269 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.665122  8135 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.673223  8135 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.654376  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.659855  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
I20260809 23:23:51.675065  7273 raft_consensus.cc:2454] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
W20260809 23:23:51.675943  7338 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.677084  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.679728  7437 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.680899  8014 raft_consensus.cc:2781] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.681250  8014 raft_consensus.cc:482] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.681612  8014 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.681694  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:51.684252  7338 leader_election.cc:341] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.685168  7338 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.687266  8244 raft_consensus.cc:2781] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.687713  8244 raft_consensus.cc:482] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.688215  8244 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.692476  8014 raft_consensus.cc:504] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.687816  8385 raft_consensus.cc:504] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.696478  8385 leader_election.cc:288] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.697427  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:51.698362  7274 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.698912  8014 leader_election.cc:288] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.699772  8173 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.700326  8173 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.702430  8173 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:51.702855  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:51.704774  7227 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.707190  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:51.707967  7273 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.709232  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.709895  7381 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:51.710582  8014 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.711339  8404 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.711141  8014 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.711823  8404 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.713117  8014 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.713843  8404 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.715850  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.717008  8338 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:51.720371  7338 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:51.720428  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.721302  7381 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
I20260809 23:23:51.721439  8404 raft_consensus.cc:2781] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.721830  8404 raft_consensus.cc:482] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.722229  8404 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.725576  7274 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.719805  7339 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.726791  8135 raft_consensus.cc:2726] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.729187  7273 raft_consensus.cc:2454] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.729393  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.730041  7488 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:51.730182  8404 raft_consensus.cc:504] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.722257  7437 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:51.731539  7437 leader_election.cc:302] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.731813  8014 raft_consensus.cc:2781] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.731386  7227 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.732290  8385 raft_consensus.cc:2781] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.732811  8385 raft_consensus.cc:686] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:51.733500  8173 raft_consensus.cc:2781] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.733945  8173 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.733665  8385 consensus_queue.cc:237] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.734340  8173 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.736778  8269 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 23:23:51.737843  8269 tablet_bootstrap.cc:492] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.732316  8014 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.738729  8269 ts_tablet_manager.cc:1397] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.070s	user 0.025s	sys 0.009s
I20260809 23:23:51.738767  8014 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.741441  8269 raft_consensus.cc:348] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.742343  8269 raft_consensus.cc:729] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:51.743160  8269 consensus_queue.cc:260] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.732400  7437 leader_election.cc:302] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.745432  8269 ts_tablet_manager.cc:1428] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:23:51.746625  8401 raft_consensus.cc:2781] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.747241  8401 raft_consensus.cc:686] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:51.747846  8135 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.748303  8149 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.748096  8401 consensus_queue.cc:237] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:51.748389  8135 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.749008  8269 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:51.748838  8149 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.750594  8135 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.750856  8149 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.753309  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.754849  8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.755564  8340 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:51.756721  8338 raft_consensus.cc:2454] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:51.762084  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" is_pre_election: true
I20260809 23:23:51.762753  7488 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 1.
I20260809 23:23:51.722115  8244 raft_consensus.cc:504] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.767166  8244 leader_election.cc:288] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:23:51.769831  7339 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:51.771059  8244 raft_consensus.cc:2781] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.771474  8244 raft_consensus.cc:482] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.771955  8244 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.746691  8173 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.792346  8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.794174  8338 raft_consensus.cc:3037] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.797324  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:51.797948  7275 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.795045  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa78046012472d8e5472478adbe1b9" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.795210  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:51.799300  7488 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.799784  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.800827  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.801762  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d542b5134ed04d51b8f1e946e99c69a5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
W20260809 23:23:51.804936  7437 leader_election.cc:341] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.780367  8404 leader_election.cc:288] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.796145  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7" is_pre_election: true
W20260809 23:23:51.806135  7437 leader_election.cc:341] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.806339  8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.806139  7488 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
W20260809 23:23:51.807267  7437 leader_election.cc:341] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.808905  7339 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:51.809520  7338 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:51.793077  8328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.807288  8340 raft_consensus.cc:3037] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.812423  7275 raft_consensus.cc:2454] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:23:51.813401  8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a1d03c208b4696910c2b7dfe1469ab" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.814043  8135 raft_consensus.cc:2781] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:51.819200  7338 leader_election.cc:302] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:23:51.814400  8404 raft_consensus.cc:2781] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.815109  8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.802665  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.822633  7025 catalog_manager.cc:5495] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 2 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:51.822108  8149 raft_consensus.cc:2781] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.819063  8135 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.825856  8135 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.826603  8014 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.828683  8014 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.829321  8244 raft_consensus.cc:504] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.831722  8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f841b906f634c05afe5a33637229c8f" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.832927  8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.833511  8330 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.834313  8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.821224  8331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9505b9c3f0342279113f757c317a868" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.835453  8331 raft_consensus.cc:3037] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:23:51.837800  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:51.838436  7381 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.838861  8402 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:51.839401  8402 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.841504  8402 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:51.849623  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa" is_pre_election: true
I20260809 23:23:51.850334  7027 catalog_manager.cc:5495] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 2 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:51.848137  8333 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 1.
W20260809 23:23:51.859448  7227 leader_election.cc:341] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.872556  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:23:51.873413  7488 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.875648  7437 leader_election.cc:302] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:51.848385  8173 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:51.896690  7488 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:23:51.877913  8118 raft_consensus.cc:2726] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:23:51.880334  8404 raft_consensus.cc:686] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:51.900332  7339 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
W20260809 23:23:51.902161  7437 leader_election.cc:341] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:51.880344  8149 raft_consensus.cc:686] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:51.881299  8328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" is_pre_election: true
I20260809 23:23:51.915271  7437 leader_election.cc:302] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:51.919636  8338 raft_consensus.cc:2454] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.919344  8404 consensus_queue.cc:237] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.894901  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:51.921640  7376 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.894666  8244 leader_election.cc:288] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:23:51.927932  8245 raft_consensus.cc:2741] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260809 23:23:51.946264  8415 raft_consensus.cc:2781] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:51.946597  7381 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 2.
I20260809 23:23:51.946902  8415 raft_consensus.cc:686] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:23:51.948098  8415 consensus_queue.cc:237] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:51.927541  8328 raft_consensus.cc:2454] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 1.
I20260809 23:23:51.967846  8269 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:51.968926  8269 tablet_bootstrap.cc:492] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:51.969653  8269 ts_tablet_manager.cc:1397] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.221s	user 0.013s	sys 0.036s
I20260809 23:23:51.952518  8149 consensus_queue.cc:237] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.976473  8330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0e3fe783b214e9ba0376a444a7306ce" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:51.977144  8330 raft_consensus.cc:3037] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:51.969208  8135 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:51.982184  7378 raft_consensus.cc:2379] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 2: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
W20260809 23:23:52.014645  7338 leader_election.cc:341] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:52.026022  7227 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:23:51.981941  7376 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 2.
I20260809 23:23:52.037660  7227 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:52.056375  8420 raft_consensus.cc:2781] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 23:23:52.056913  8420 raft_consensus.cc:482] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:23:52.057252  8420 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:52.062398  8328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:52.067374  8423 raft_consensus.cc:2781] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:52.067961  8423 raft_consensus.cc:686] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:52.069073  8423 consensus_queue.cc:237] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.074069  8135 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
W20260809 23:23:52.092111  7437 leader_election.cc:341] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:52.102525  7025 catalog_manager.cc:5495] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:52.110076  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 72521.4us
I20260809 23:23:52.110235  7437 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:23:52.112573  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:23:52.112594  8014 raft_consensus.cc:2781] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:52.113575  8014 raft_consensus.cc:686] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Leader. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:23:52.117199  8331 raft_consensus.cc:2454] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3cc3ec2dcd449de83f6c88e80d2d6de in term 3.
I20260809 23:23:52.117381  8269 raft_consensus.cc:348] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.118131  8269 raft_consensus.cc:729] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.118789  8269 consensus_queue.cc:260] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.134681  8269 ts_tablet_manager.cc:1428] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.165s	user 0.010s	sys 0.000s
I20260809 23:23:52.136098  7275 raft_consensus.cc:1261] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.114552  8014 consensus_queue.cc:237] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.147567  8269 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:52.151532  7437 leader_election.cc:341] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:23:52.185474  7376 raft_consensus.cc:1261] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:23:52.180446  8330 raft_consensus.cc:2454] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:23:52.188338  8340 raft_consensus.cc:2454] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:23:52.188488  8427 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:23:52.215312  7228 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 79774us
I20260809 23:23:52.227568  8340 raft_consensus.cc:1261] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260809 23:23:52.229202  8118 consensus_queue.cc:1035] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:23:52.226809  8335 raft_consensus.cc:1261] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:23:52.230607  7437 consensus_peers.cc:487] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:52.225077  8420 raft_consensus.cc:504] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
W20260809 23:23:52.235142  7437 consensus_peers.cc:487] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:23:52.237941  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 103186us
I20260809 23:23:52.240510  8447 consensus_queue.cc:1035] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:23:52.241509  7273 raft_consensus.cc:2379] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 081930af26fd4f7fab29988df3e8b9aa in current term 2: Already voted for candidate ec555693f39b444f95a05ba0bd6b37e7 in this term.
I20260809 23:23:52.241709  8420 leader_election.cc:288] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Requested vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:23:52.246177  8245 consensus_queue.cc:1035] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:23:52.257086  8335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:52.257828  8335 raft_consensus.cc:3037] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:52.258736  8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
I20260809 23:23:52.283679  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f4e2e6c1834f09905dc17072c08a53" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:23:52.284616  7381 raft_consensus.cc:2379] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ec555693f39b444f95a05ba0bd6b37e7 in current term 2: Already voted for candidate 081930af26fd4f7fab29988df3e8b9aa in this term.
I20260809 23:23:52.301455  7024 catalog_manager.cc:5495] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260809 23:23:52.304112  7024 catalog_manager.cc:5495] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:52.306393  7030 catalog_manager.cc:5495] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4). New cstate: current_term: 2 leader_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:52.314116  7338 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:23:52.316143  8407 raft_consensus.cc:2726] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260809 23:23:52.319970  7270 raft_consensus.cc:1261] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.321471  7274 raft_consensus.cc:1261] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.324113  7027 catalog_manager.cc:5495] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:23:52.329419  8335 raft_consensus.cc:2454] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 2.
I20260809 23:23:52.331691  7227 leader_election.cc:302] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:23:52.332979  8444 raft_consensus.cc:2781] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:23:52.333781  8444 raft_consensus.cc:686] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:23:52.334594  8444 consensus_queue.cc:237] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.340523  8443 consensus_queue.cc:1035] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:23:52.341071  8448 consensus_queue.cc:1035] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:23:52.362169  7338 consensus_peers.cc:487] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.377625  8269 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.384212  8269 tablet_bootstrap.cc:492] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.386898  8269 ts_tablet_manager.cc:1397] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.241s	user 0.017s	sys 0.010s
I20260809 23:23:52.389575  8269 raft_consensus.cc:348] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.390295  8269 raft_consensus.cc:729] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.398454  8269 consensus_queue.cc:260] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.401345  8269 ts_tablet_manager.cc:1428] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.003s
I20260809 23:23:52.402069  8269 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.449637  8335 raft_consensus.cc:1261] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.457002  8149 consensus_queue.cc:1035] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260809 23:23:52.497208  7488 raft_consensus.cc:1261] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:23:52.498282  7338 consensus_peers.cc:487] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:52.517208  8149 consensus_queue.cc:1035] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260809 23:23:52.532979  8331 raft_consensus.cc:1261] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.550159  8149 consensus_queue.cc:1035] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:23:52.563010  8463 log.cc:924] Time spent T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: Append to log took a long time: real 0.122s	user 0.004s	sys 0.000s
I20260809 23:23:52.571312  7490 raft_consensus.cc:1261] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.574342  7025 catalog_manager.cc:5495] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 1 to 2, leader changed from ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1) to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
W20260809 23:23:52.575928  7227 consensus_peers.cc:487] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:52.591192  7275 raft_consensus.cc:1261] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.596316  8244 consensus_queue.cc:1035] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:23:52.601238  7338 consensus_peers.cc:487] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.602180  8269 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.579175  8149 consensus_queue.cc:1035] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260809 23:23:52.605633  8269 tablet_bootstrap.cc:492] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.606509  8269 ts_tablet_manager.cc:1397] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.205s	user 0.036s	sys 0.025s
I20260809 23:23:52.611847  8269 raft_consensus.cc:348] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.612723  8269 raft_consensus.cc:729] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.616266  8269 consensus_queue.cc:260] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.617357  8269 ts_tablet_manager.cc:1428] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.003s
I20260809 23:23:52.617944  8269 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.658324  7382 raft_consensus.cc:1261] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.662830  8439 consensus_queue.cc:1035] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:23:52.683652  8472 log.cc:924] Time spent T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
W20260809 23:23:52.704303  7227 consensus_peers.cc:487] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:52.721674  7437 consensus_peers.cc:487] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:23:52.722347  8269 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.723734  8269 tablet_bootstrap.cc:492] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.724467  8269 ts_tablet_manager.cc:1397] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.107s	user 0.021s	sys 0.014s
I20260809 23:23:52.726863  8269 raft_consensus.cc:348] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.727643  8269 raft_consensus.cc:729] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.728448  8269 consensus_queue.cc:260] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.730697  8269 ts_tablet_manager.cc:1428] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:23:52.731544  8269 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.736615  7380 raft_consensus.cc:1261] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer a3cc3ec2dcd449de83f6c88e80d2d6de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.738185  8428 consensus_queue.cc:1035] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:23:52.775698  8269 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.776648  8269 tablet_bootstrap.cc:492] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.777347  8269 ts_tablet_manager.cc:1397] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.046s	user 0.021s	sys 0.015s
W20260809 23:23:52.779387  7338 consensus_peers.cc:487] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.781229  8269 raft_consensus.cc:348] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.781967  8269 raft_consensus.cc:729] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.782595  8269 consensus_queue.cc:260] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.787786  8269 ts_tablet_manager.cc:1428] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260809 23:23:52.788566  8269 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.803097  8269 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.804152  8269 tablet_bootstrap.cc:492] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.804812  8269 ts_tablet_manager.cc:1397] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.016s	user 0.011s	sys 0.004s
I20260809 23:23:52.807121  8269 raft_consensus.cc:348] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.807703  8269 raft_consensus.cc:729] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.808336  8269 consensus_queue.cc:260] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.810304  8269 ts_tablet_manager.cc:1428] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260809 23:23:52.813115  7382 raft_consensus.cc:1261] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:23:52.814483  8269 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.814430  8421 consensus_queue.cc:1035] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:23:52.830045  8269 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260809 23:23:52.831620  8269 tablet_bootstrap.cc:492] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.832443  8269 ts_tablet_manager.cc:1397] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.006s
I20260809 23:23:52.834841  8269 raft_consensus.cc:348] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.835827  8269 raft_consensus.cc:729] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.836560  8269 consensus_queue.cc:260] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.838717  8269 ts_tablet_manager.cc:1428] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I20260809 23:23:52.840579  8335 raft_consensus.cc:1261] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:23:52.847353  7227 consensus_peers.cc:487] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.847735  8402 consensus_queue.cc:1035] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:23:52.856473  7227 consensus_peers.cc:487] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.863262  8269 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:52.867691  7338 consensus_peers.cc:487] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260809 23:23:52.884681  7437 consensus_peers.cc:487] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.889539  8269 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.890362  8269 tablet_bootstrap.cc:492] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.891016  8269 ts_tablet_manager.cc:1397] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.007s
I20260809 23:23:52.893246  8269 raft_consensus.cc:348] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.893931  8269 raft_consensus.cc:729] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.894714  8269 consensus_queue.cc:260] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.896827  8269 ts_tablet_manager.cc:1428] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:23:52.897567  8269 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.914232  8269 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.915227  8269 tablet_bootstrap.cc:492] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.915781  8269 ts_tablet_manager.cc:1397] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260809 23:23:52.917340  8269 raft_consensus.cc:348] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.917855  8269 raft_consensus.cc:729] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.918397  8269 consensus_queue.cc:260] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.919931  8269 ts_tablet_manager.cc:1428] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.001s
I20260809 23:23:52.921046  8269 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.938416  8269 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.939402  8269 tablet_bootstrap.cc:492] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.940073  8269 ts_tablet_manager.cc:1397] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.004s
W20260809 23:23:52.941231  7437 consensus_peers.cc:487] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.946218  8269 raft_consensus.cc:348] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.947388  8269 raft_consensus.cc:729] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.947889  8269 consensus_queue.cc:260] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:52.949966  8269 ts_tablet_manager.cc:1428] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260809 23:23:52.950742  8269 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:52.955835  7227 consensus_peers.cc:487] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:52.963997  8269 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.964929  8269 tablet_bootstrap.cc:492] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.965530  8269 ts_tablet_manager.cc:1397] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.000s
I20260809 23:23:52.967546  8269 raft_consensus.cc:348] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.968201  8269 raft_consensus.cc:729] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.968864  8269 consensus_queue.cc:260] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:52.970523  8269 ts_tablet_manager.cc:1428] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:23:52.971261  8269 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:52.988198  8269 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:52.989225  8269 tablet_bootstrap.cc:492] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:52.989948  8269 ts_tablet_manager.cc:1397] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.003s
I20260809 23:23:52.993296  8269 raft_consensus.cc:348] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.994391  8269 raft_consensus.cc:729] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:52.995240  8269 consensus_queue.cc:260] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:52.997552  8269 ts_tablet_manager.cc:1428] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:23:52.998558  8269 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.014523  8269 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.015481  8269 tablet_bootstrap.cc:492] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
W20260809 23:23:53.104795  7437 consensus_peers.cc:487] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:53.121366  8335 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:53.132649  8269 ts_tablet_manager.cc:1397] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.134s	user 0.017s	sys 0.000s
I20260809 23:23:53.182662  8269 raft_consensus.cc:348] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.191994  8269 raft_consensus.cc:729] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.192741  8269 consensus_queue.cc:260] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.197113  8269 ts_tablet_manager.cc:1428] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260809 23:23:53.199018  8269 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
W20260809 23:23:53.305279  7437 consensus_peers.cc:487] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260809 23:23:53.339041  8330 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:53.382822  8269 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.384184  8269 tablet_bootstrap.cc:492] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.385159  8269 ts_tablet_manager.cc:1397] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.187s	user 0.025s	sys 0.033s
I20260809 23:23:53.387667  8269 raft_consensus.cc:348] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.388336  8269 raft_consensus.cc:729] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.388964  8269 consensus_queue.cc:260] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.391458  8269 ts_tablet_manager.cc:1428] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:23:53.392675  8269 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.409775  8269 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.410624  8269 tablet_bootstrap.cc:492] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.411224  8269 ts_tablet_manager.cc:1397] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.004s
I20260809 23:23:53.413640  8269 raft_consensus.cc:348] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.414296  8269 raft_consensus.cc:729] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.415055  8269 consensus_queue.cc:260] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.417428  8269 ts_tablet_manager.cc:1428] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:23:53.418641  8269 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.433772  8269 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.434894  8269 tablet_bootstrap.cc:492] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.435536  8269 ts_tablet_manager.cc:1397] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.000s
I20260809 23:23:53.437919  8269 raft_consensus.cc:348] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.438514  8269 raft_consensus.cc:729] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.439162  8269 consensus_queue.cc:260] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.441308  8269 ts_tablet_manager.cc:1428] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:23:53.443015  8269 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.456050  8269 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.456838  8269 tablet_bootstrap.cc:492] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.457378  8269 ts_tablet_manager.cc:1397] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260809 23:23:53.459571  8269 raft_consensus.cc:348] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.460178  8269 raft_consensus.cc:729] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.460807  8269 consensus_queue.cc:260] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.463187  8269 ts_tablet_manager.cc:1428] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:23:53.464581  8269 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.479146  8269 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.480080  8269 tablet_bootstrap.cc:492] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.480715  8269 ts_tablet_manager.cc:1397] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.016s	user 0.014s	sys 0.000s
I20260809 23:23:53.482587  8269 raft_consensus.cc:348] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.483165  8269 raft_consensus.cc:729] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.483678  8269 consensus_queue.cc:260] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.485378  8269 ts_tablet_manager.cc:1428] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.004s	user 0.004s	sys 0.000s
I20260809 23:23:53.486531  8269 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.502151  8269 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.503113  8269 tablet_bootstrap.cc:492] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.503808  8269 ts_tablet_manager.cc:1397] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.000s
I20260809 23:23:53.506292  8269 raft_consensus.cc:348] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.507023  8269 raft_consensus.cc:729] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.507674  8269 consensus_queue.cc:260] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.509757  8269 ts_tablet_manager.cc:1428] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:23:53.510869  8269 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.525381  8269 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.526218  8269 tablet_bootstrap.cc:492] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.526830  8269 ts_tablet_manager.cc:1397] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.000s
I20260809 23:23:53.529384  8269 raft_consensus.cc:348] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.530062  8269 raft_consensus.cc:729] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.530732  8269 consensus_queue.cc:260] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.532312  8269 ts_tablet_manager.cc:1428] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:23:53.534010  8269 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.556499  8269 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.557507  8269 tablet_bootstrap.cc:492] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.558176  8269 ts_tablet_manager.cc:1397] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.003s
I20260809 23:23:53.560623  8269 raft_consensus.cc:348] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.561348  8269 raft_consensus.cc:729] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.561961  8269 consensus_queue.cc:260] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:23:53.563608  8269 ts_tablet_manager.cc:1428] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260809 23:23:53.564759  8269 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.580672  8269 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.581617  8269 tablet_bootstrap.cc:492] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.582275  8269 ts_tablet_manager.cc:1397] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.004s
I20260809 23:23:53.584683  8269 raft_consensus.cc:348] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.585341  8269 raft_consensus.cc:729] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.586050  8269 consensus_queue.cc:260] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.590049  8269 ts_tablet_manager.cc:1428] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260809 23:23:53.591426  8269 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.604902  8269 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.605669  8269 tablet_bootstrap.cc:492] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.606257  8269 ts_tablet_manager.cc:1397] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.001s
I20260809 23:23:53.608696  8269 raft_consensus.cc:348] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.609403  8269 raft_consensus.cc:729] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.610040  8269 consensus_queue.cc:260] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:23:53.612562  8269 ts_tablet_manager.cc:1428] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.002s
I20260809 23:23:53.613524  8269 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap starting.
I20260809 23:23:53.623684  8330 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:53.651721  8335 raft_consensus.cc:3037] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:23:53.676738  8269 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 23:23:53.677799  8269 tablet_bootstrap.cc:492] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Bootstrap complete.
I20260809 23:23:53.678470  8269 ts_tablet_manager.cc:1397] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent bootstrapping tablet: real 0.065s	user 0.026s	sys 0.032s
I20260809 23:23:53.680979  8269 raft_consensus.cc:348] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.681671  8269 raft_consensus.cc:729] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Initialized, Role: FOLLOWER
I20260809 23:23:53.682349  8269 consensus_queue.cc:260] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:23:53.684719  8269 ts_tablet_manager.cc:1428] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260809 23:23:53.702625  8337 raft_consensus.cc:3037] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
W20260809 23:23:59.512334  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 348831us
W20260809 23:23:59.702677  7409 debug-util.cc:398] Leaking SignalData structure 0x7b08000f2220 after lost signal to thread 22536
W20260809 23:23:59.704105  7409 debug-util.cc:398] Leaking SignalData structure 0x7b08003b0e80 after lost signal to thread 7079
W20260809 23:23:59.705699  7409 debug-util.cc:398] Leaking SignalData structure 0x7b080012c9e0 after lost signal to thread 7311
W20260809 23:23:59.707263  7409 debug-util.cc:398] Leaking SignalData structure 0x7b08002ed260 after lost signal to thread 7414
W20260809 23:23:59.708983  7409 debug-util.cc:398] Leaking SignalData structure 0x7b080028f760 after lost signal to thread 7525
W20260809 23:23:59.710444  7409 debug-util.cc:398] Leaking SignalData structure 0x7b08002ea400 after lost signal to thread 8373
W20260809 23:24:00.240830  7073 debug-util.cc:398] Leaking SignalData structure 0x7b080035f260 after lost signal to thread 22536
W20260809 23:24:00.241959  7073 debug-util.cc:398] Leaking SignalData structure 0x7b0800049560 after lost signal to thread 7079
W20260809 23:24:00.243248  7073 debug-util.cc:398] Leaking SignalData structure 0x7b08000f83e0 after lost signal to thread 7311
W20260809 23:24:00.244905  7073 debug-util.cc:398] Leaking SignalData structure 0x7b08003c4a80 after lost signal to thread 7414
W20260809 23:24:00.246070  7073 debug-util.cc:398] Leaking SignalData structure 0x7b0800222b00 after lost signal to thread 7525
W20260809 23:24:00.247316  7073 debug-util.cc:398] Leaking SignalData structure 0x7b08002c50c0 after lost signal to thread 8373
I20260809 23:24:02.054522 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:02.325062  7487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:02.326184  7487 raft_consensus.cc:595] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:02.331146  7487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4468dee129594149bc011deeaef72185"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:02.331722  7487 raft_consensus.cc:595] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:02.334285  7487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "615321f1b8064c90892728123be05c59"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:02.334872  7487 raft_consensus.cc:595] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:02.338171  7487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:02.338759  7487 raft_consensus.cc:595] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:24:02.350991  8573 raft_consensus.cc:979] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.351493  8573 raft_consensus.cc:1067] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.353255  8338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "615321f1b8064c90892728123be05c59"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:02.353828  8338 raft_consensus.cc:482] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.354234  8338 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.370261  8338 raft_consensus.cc:504] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.374133  8338 leader_election.cc:288] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:02.387930  8428 raft_consensus.cc:979] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:02.388365  8428 raft_consensus.cc:1067] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:02.392481  7381 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:41172
I20260809 23:24:02.393030  7381 raft_consensus.cc:482] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.393378  7381 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.458564  7381 raft_consensus.cc:504] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:02.461975  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:02.462244  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ded669d4bcc4a7a81c44ad0b578c899" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:02.462718  7487 raft_consensus.cc:3032] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:02.463129  7487 raft_consensus.cc:729] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:02.464156  7487 consensus_queue.cc:260] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:02.465559  7487 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.473932  7487 raft_consensus.cc:2454] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:02.475950  7273 raft_consensus.cc:3037] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.487039  7381 leader_election.cc:288] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:02.488545  7339 leader_election.cc:302] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:24:02.489549  8570 raft_consensus.cc:2781] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:02.503476  7274 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:44526
I20260809 23:24:02.504153  7274 raft_consensus.cc:595] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:02.506920  7487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:02.507653  7487 raft_consensus.cc:595] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:02.508682  8570 raft_consensus.cc:686] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:02.510561 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:24:02.512858 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:02.513809 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:02.515010  7273 raft_consensus.cc:2454] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:02.516919  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:02.517530  7381 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.518559  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "615321f1b8064c90892728123be05c59" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:02.519170  7487 raft_consensus.cc:3032] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:02.519426  7487 raft_consensus.cc:729] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:02.520135  7487 consensus_queue.cc:260] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.521399  7487 raft_consensus.cc:3037] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.528839  7487 raft_consensus.cc:2454] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:02.530478  8263 leader_election.cc:302] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:02.532380  7381 raft_consensus.cc:2454] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:02.537878  8595 raft_consensus.cc:2781] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:02.533215  8570 consensus_queue.cc:237] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
W20260809 23:24:02.542608  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 50470.4us
I20260809 23:24:02.576362  8595 raft_consensus.cc:686] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:02.577772  8595 consensus_queue.cc:237] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.588289  8573 raft_consensus.cc:979] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.588801  8428 raft_consensus.cc:1067] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.590574  8338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4468dee129594149bc011deeaef72185"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:02.591254  8338 raft_consensus.cc:482] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.591586  8338 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.593935  7027 catalog_manager.cc:5495] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:02.601442  8338 raft_consensus.cc:504] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.644425  8567 raft_consensus.cc:979] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.644995  8567 raft_consensus.cc:1067] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.655782  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:02.656351  8330 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.656679  8330 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:02.660318  8338 leader_election.cc:288] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:02.660553  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:02.661170  7273 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.672286  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4468dee129594149bc011deeaef72185" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:02.673002  7487 raft_consensus.cc:3032] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:02.673276  7487 raft_consensus.cc:729] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:02.674331  8567 raft_consensus.cc:979] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.674005  7487 consensus_queue.cc:260] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.674834  8567 raft_consensus.cc:1067] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.678270  8578 raft_consensus.cc:979] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.678462  7487 raft_consensus.cc:3037] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.678802  8578 raft_consensus.cc:1067] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:02.682333  8338 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:33472
I20260809 23:24:02.682914  8338 raft_consensus.cc:482] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.683279  8338 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.688198  7487 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:02.692179  8340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:02.692972  8340 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:02.693408  8340 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:02.703966  8338 raft_consensus.cc:504] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:02.717609  8338 leader_election.cc:288] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:02.724855  7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:02.725636  7274 raft_consensus.cc:3032] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:02.725996  7274 raft_consensus.cc:729] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:02.734174  7274 consensus_queue.cc:260] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:02.736083  7274 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.790145  8263 leader_election.cc:302] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:02.791153  8595 raft_consensus.cc:2781] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:02.791657  8595 raft_consensus.cc:686] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:02.792634  8595 consensus_queue.cc:237] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.877174  8330 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:02.884758  8340 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:02.916440  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:02.917042  7275 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:02.944235  7270 raft_consensus.cc:1261] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:02.945494  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec734a4c9690454f8c3986ae773f6e9f" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:02.946146  7490 raft_consensus.cc:3032] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:02.946412  7490 raft_consensus.cc:729] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:02.947039  8575 consensus_queue.cc:1035] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:02.947109  7490 consensus_queue.cc:260] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:02.948454  7490 raft_consensus.cc:3037] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:02.956697  7273 raft_consensus.cc:2454] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:02.958379  7490 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:02.963395  8340 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:02.902846  7274 raft_consensus.cc:2454] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:02.969084  8263 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:02.908459  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb2091b41c5341ebb239c60326eb8a22" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:02.973879  7487 raft_consensus.cc:3037] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:02.956200  7275 raft_consensus.cc:2454] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:02.979961  8262 leader_election.cc:302] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:02.984047  8330 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:02.981778  8595 raft_consensus.cc:2781] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:02.991290  7381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:02.993237  8596 raft_consensus.cc:2781] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:02.994709  7381 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:03.019093  7487 raft_consensus.cc:2454] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:03.027009  7381 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
W20260809 23:24:03.032641  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 62587.1us
I20260809 23:24:03.032090  8595 raft_consensus.cc:686] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:03.044354  8596 raft_consensus.cc:686] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:03.056495  8262 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:03.082376  7486 raft_consensus.cc:1261] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.087551  8610 raft_consensus.cc:2781] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:03.088289  8610 raft_consensus.cc:686] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:03.088573  7487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207ede1fd2664d3a871bbb0a5b854a64" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:03.089294  7487 raft_consensus.cc:3032] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:03.089783  7487 raft_consensus.cc:729] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:03.092103  7487 consensus_queue.cc:260] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:03.095954  7030 catalog_manager.cc:5495] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:03.099244  7487 raft_consensus.cc:3037] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:03.104671  8596 consensus_queue.cc:237] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:03.107811  7486 raft_consensus.cc:1261] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.089665  8610 consensus_queue.cc:237] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:03.120350  8577 consensus_queue.cc:1035] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:03.119820  7487 raft_consensus.cc:2454] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:03.125897  8605 consensus_queue.cc:1035] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:03.115734  8595 consensus_queue.cc:237] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:03.222069  7381 raft_consensus.cc:1261] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260809 23:24:03.317734  8262 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 59282.4us
I20260809 23:24:03.322225  8605 consensus_queue.cc:1035] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260809 23:24:03.337705  7490 raft_consensus.cc:1261] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.376564  8610 consensus_queue.cc:1035] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:03.400882  7270 raft_consensus.cc:1261] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.443328  8605 consensus_queue.cc:1035] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260809 23:24:03.464874  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 81832.3us
I20260809 23:24:03.498747  7024 catalog_manager.cc:5495] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 2 to 3, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:03.501339  7024 catalog_manager.cc:5495] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 2 to 3, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:03.544601  7024 catalog_manager.cc:5495] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:03.546788  7024 catalog_manager.cc:5495] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:03.607540  7488 raft_consensus.cc:1261] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:03.621846  8596 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:24:03.653074 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:03.667213  7378 raft_consensus.cc:1261] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:03.669430  8596 consensus_queue.cc:1035] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:03.675112  7270 raft_consensus.cc:1261] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.680876  8605 consensus_queue.cc:1035] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:03.685359  7487 raft_consensus.cc:1261] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:03.690382  8605 consensus_queue.cc:1035] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:03.696416  7275 raft_consensus.cc:1261] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:03.699752  8605 consensus_queue.cc:1035] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:03.719198  7490 raft_consensus.cc:1261] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:03.720854  8596 consensus_queue.cc:1035] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260809 23:24:03.870074  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 56330.4us
I20260809 23:24:04.087615  7486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:04.088585  7486 raft_consensus.cc:595] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:04.095427  7486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f7f029122e441ad9d3194829e439141"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:04.096114  7486 raft_consensus.cc:595] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:04.101580  8625 raft_consensus.cc:979] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.102090  8625 raft_consensus.cc:1067] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.102118  7486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:04.102890  7486 raft_consensus.cc:595] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:04.104347  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f7f029122e441ad9d3194829e439141"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:04.104984  8330 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.105357  8330 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.121107  7486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b64607aa57f542cda93ba367af662006"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:04.121961  7486 raft_consensus.cc:595] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:04.125380  7275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
mode: GRACEFUL
new_leader_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:44526
I20260809 23:24:04.126040  7275 raft_consensus.cc:595] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Received request to transfer leadership to TS 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:24:04.154417  8330 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.167806  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:04.168576  7382 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.185528  8567 raft_consensus.cc:979] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.186128  8567 raft_consensus.cc:1067] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.187359  7380 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f23623f6580149f19787b8924d48ff24"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:60202
I20260809 23:24:04.187846  8334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b64607aa57f542cda93ba367af662006"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:04.187906  7382 raft_consensus.cc:2454] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.188619  8334 raft_consensus.cc:482] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.189049  8334 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.189580  8262 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:04.192550  8605 raft_consensus.cc:2781] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:04.193235  8605 raft_consensus.cc:686] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:04.195655  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7f029122e441ad9d3194829e439141" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:04.196280  7490 raft_consensus.cc:3032] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:04.196627  7490 raft_consensus.cc:729] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:04.197479  7490 consensus_queue.cc:260] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.198702  7490 raft_consensus.cc:3037] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.187470  8330 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:04.188046  7380 raft_consensus.cc:595] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:04.194588  8605 consensus_queue.cc:237] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.211118 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260809 23:24:04.200636  8334 raft_consensus.cc:504] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.213179 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:04.213973 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:04.215245  8334 leader_election.cc:288] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:04.216363  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:04.217087  7382 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.223785  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b64607aa57f542cda93ba367af662006" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:04.224740  7486 raft_consensus.cc:3032] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:04.232227  7382 raft_consensus.cc:2454] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.243042  8262 leader_election.cc:302] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:04.244056  8596 raft_consensus.cc:2781] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:04.246745  8596 raft_consensus.cc:686] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:04.253768  8596 consensus_queue.cc:237] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.225843  7486 raft_consensus.cc:729] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:04.262713  7490 raft_consensus.cc:2454] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.266379  7486 consensus_queue.cc:260] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:04.268095  7486 raft_consensus.cc:3037] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.282886  7024 catalog_manager.cc:5495] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:04.350092  7486 raft_consensus.cc:2454] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.356431  8428 raft_consensus.cc:979] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.358151  8428 raft_consensus.cc:1067] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.360368  8334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:04.361552  8334 raft_consensus.cc:482] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.362023  8334 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.379779  8428 raft_consensus.cc:979] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.380399  8428 raft_consensus.cc:1067] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.380816  8334 raft_consensus.cc:504] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.382741  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:04.399230  8330 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.399773  8330 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:04.398265  7024 catalog_manager.cc:5495] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:04.411204  7490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:04.411957  7490 raft_consensus.cc:3032] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:04.412263  7490 raft_consensus.cc:729] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:04.413069  7490 consensus_queue.cc:260] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.414350  7490 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.423837  8334 leader_election.cc:288] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:04.424190  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46a6ff8a5b7e4c73ae66ec2560453733" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:04.429247  7273 raft_consensus.cc:3037] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.431802  7490 raft_consensus.cc:2454] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.433172  8263 leader_election.cc:302] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:04.434100  8605 raft_consensus.cc:2781] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:04.434566  8605 raft_consensus.cc:686] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:04.438519  8605 consensus_queue.cc:237] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.440593  8330 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.452586  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:04.452751  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a2f292a694ebbab1424f68bfe3dc5" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:04.453198  7275 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:04.453333  7486 raft_consensus.cc:3032] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:04.453718  7486 raft_consensus.cc:729] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:04.454551  7486 consensus_queue.cc:260] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.455757  7486 raft_consensus.cc:3037] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:04.459115  7273 raft_consensus.cc:2454] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.462064  7275 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:04.463380  8262 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:04.465024  8596 raft_consensus.cc:2781] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:04.465524  8596 raft_consensus.cc:686] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:04.465646  7486 raft_consensus.cc:2454] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:04.466547  8596 consensus_queue.cc:237] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:04.467433  8330 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:04.490334  7024 catalog_manager.cc:5495] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:04.525755  8570 raft_consensus.cc:979] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.526268  8611 raft_consensus.cc:1067] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:04.530553  8334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f23623f6580149f19787b8924d48ff24"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:33478
I20260809 23:24:04.531255  8334 raft_consensus.cc:482] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.531704  8334 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.543817  8334 raft_consensus.cc:504] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:04.546676  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:04.547377  7275 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.550334  8602 raft_consensus.cc:979] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: : Instructing follower 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:04.550942  8334 leader_election.cc:288] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:24:04.551090  8580 raft_consensus.cc:1067] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Signalling peer 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:04.551975  7382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f23623f6580149f19787b8924d48ff24" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:04.552572  7382 raft_consensus.cc:3032] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:04.552865  7382 raft_consensus.cc:729] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:04.553329  7380 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:41154
I20260809 23:24:04.553893  7380 raft_consensus.cc:482] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:04.553691  7382 consensus_queue.cc:260] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:04.555228  7382 raft_consensus.cc:3037] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.554260  7380 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.570802  7024 catalog_manager.cc:5495] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 2 to 3, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:04.584863  7380 raft_consensus.cc:504] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:04.605418  7382 raft_consensus.cc:2454] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.606781  8262 leader_election.cc:302] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:04.607869  8605 raft_consensus.cc:2781] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:04.608404  8605 raft_consensus.cc:686] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:04.608711  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:04.609405  7273 raft_consensus.cc:3032] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:04.609784  7273 raft_consensus.cc:729] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:04.609488  8605 consensus_queue.cc:237] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:04.610620  7273 consensus_queue.cc:260] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:04.611938  7273 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.619943  7273 raft_consensus.cc:2454] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:04.620996  7338 leader_election.cc:302] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:24:04.621856  8611 raft_consensus.cc:2781] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:04.622165  7380 leader_election.cc:288] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:04.622360  8611 raft_consensus.cc:686] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:04.623641  7275 raft_consensus.cc:2454] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:04.623725  8611 consensus_queue.cc:237] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:04.627199  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d87722cb5bdb4aa797b10938f4e1a640" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:04.627810  7486 raft_consensus.cc:3037] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:04.637110  7486 raft_consensus.cc:2454] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:04.644479  7382 raft_consensus.cc:1261] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:04.645882  8596 consensus_queue.cc:1035] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:04.654592  7382 raft_consensus.cc:1261] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:04.656118  8596 consensus_queue.cc:1035] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:04.675092  7024 catalog_manager.cc:5495] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:04.701771  7486 raft_consensus.cc:1261] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:04.703732  8647 consensus_queue.cc:1035] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:04.716740  7490 raft_consensus.cc:1261] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:04.756119  7027 catalog_manager.cc:5495] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:04.729298  8671 consensus_queue.cc:1035] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260809 23:24:04.895037  7273 raft_consensus.cc:1261] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:04.896737  8671 consensus_queue.cc:1035] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:24:04.954255  7488 raft_consensus.cc:1261] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:04.957508  8668 consensus_queue.cc:1035] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:04.997447  7275 raft_consensus.cc:1261] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:04.998950  8668 consensus_queue.cc:1035] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:05.042392  7378 raft_consensus.cc:1261] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.050992  8605 consensus_queue.cc:1035] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:05.065028  7275 raft_consensus.cc:1261] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.066313  7486 raft_consensus.cc:1261] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.067118  8647 consensus_queue.cc:1035] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:05.077689  8673 consensus_queue.cc:1035] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:05.074587  7275 raft_consensus.cc:1261] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.095258  8593 consensus_queue.cc:1035] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:05.176939  7486 raft_consensus.cc:1261] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.202389  8611 consensus_queue.cc:1035] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260809 23:24:05.210534  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 63350.7us
I20260809 23:24:05.306005 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:05.406281  7378 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:60202
I20260809 23:24:05.407006  7378 raft_consensus.cc:595] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:05.411550  7486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80"
dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:34014
I20260809 23:24:05.412542  7486 raft_consensus.cc:595] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:05.419128  7273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
mode: GRACEFUL
new_leader_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:44526
I20260809 23:24:05.419787  7273 raft_consensus.cc:595] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Received request to transfer leadership to TS 081930af26fd4f7fab29988df3e8b9aa
I20260809 23:24:05.426251  7275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:44526
I20260809 23:24:05.428709  7275 raft_consensus.cc:595] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:05.438246  8593 raft_consensus.cc:979] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.438829  8593 raft_consensus.cc:1067] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.440739  8334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:33478
I20260809 23:24:05.441383  8334 raft_consensus.cc:482] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:05.441843  8334 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.468783  8334 raft_consensus.cc:504] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.472982  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:05.473551  7275 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.483098 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260809 23:24:05.485473 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:05.486296 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:05.545176  7275 raft_consensus.cc:2454] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:05.546774  8262 leader_election.cc:302] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:05.547837  8673 raft_consensus.cc:2781] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:05.548556  8673 raft_consensus.cc:686] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:05.549511  8673 consensus_queue.cc:237] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.596561  7030 catalog_manager.cc:5495] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:05.613734  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3066e3b731424266bfd6155e1dddc90b" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:05.625751  7378 raft_consensus.cc:3032] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:05.626046  7378 raft_consensus.cc:729] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:05.621155  8602 raft_consensus.cc:979] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.627599  8602 raft_consensus.cc:1067] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.626863  7378 consensus_queue.cc:260] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.630584  7378 raft_consensus.cc:3037] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.642832  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:33472
I20260809 23:24:05.643888  8330 raft_consensus.cc:482] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:05.649328  8330 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:05.612073  8334 leader_election.cc:288] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:24:05.689574  7378 raft_consensus.cc:2454] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:05.729547  8330 raft_consensus.cc:504] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.743678  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:05.744410  7275 raft_consensus.cc:3032] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:05.744763  7275 raft_consensus.cc:729] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:05.745569  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d22e37cd9d24996abca51e457d761cc" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:05.746697  7376 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:05.746737  8330 leader_election.cc:288] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:24:05.750648  7275 consensus_queue.cc:260] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.757397  7275 raft_consensus.cc:3037] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:05.764596  8573 raft_consensus.cc:979] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.765177  8696 raft_consensus.cc:1067] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:05.766971  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:32942
I20260809 23:24:05.767472  8330 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:05.767813  8330 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.772855  7376 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:05.777020  8330 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:05.780645  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:05.780704  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7f251a895a462abb71b8668cf0aa80" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:05.781313  7486 raft_consensus.cc:3032] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:05.781452  7273 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.781685  7486 raft_consensus.cc:729] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3cc3ec2dcd449de83f6c88e80d2d6de, State: Running, Role: LEADER
I20260809 23:24:05.783581  7486 consensus_queue.cc:260] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:05.784760  7486 raft_consensus.cc:3037] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:05.787840  8262 leader_election.cc:302] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:05.788837  7275 raft_consensus.cc:2454] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 3.
I20260809 23:24:05.788863  8673 raft_consensus.cc:2781] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:05.791535  8673 raft_consensus.cc:686] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:05.792560  8673 consensus_queue.cc:237] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:05.794008  7273 raft_consensus.cc:2454] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:05.797169  8262 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:05.798195  8647 raft_consensus.cc:2781] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:05.798677  8647 raft_consensus.cc:686] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:05.799585  8647 consensus_queue.cc:237] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:05.794459  8330 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:05.822739  7486 raft_consensus.cc:2454] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:05.833518  7030 catalog_manager.cc:5495] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 2 to 3, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 3 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:05.858093  7027 catalog_manager.cc:5495] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:05.984042  7275 raft_consensus.cc:1261] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:05.985761  8647 consensus_queue.cc:1035] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:05.994308  8580 raft_consensus.cc:979] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: : Instructing follower 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:05.994833  8602 raft_consensus.cc:1067] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Signalling peer 081930af26fd4f7fab29988df3e8b9aa to start an election
I20260809 23:24:05.996469  7378 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
 from {username='slave'} at 127.0.0.1:41154
I20260809 23:24:05.997046  7378 raft_consensus.cc:482] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:05.997366  7378 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:06.006959  7378 raft_consensus.cc:504] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:06.012405  7273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:06.013024  7273 raft_consensus.cc:3032] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:06.013343  7273 raft_consensus.cc:729] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:06.014375  7273 consensus_queue.cc:260] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:06.016716  7273 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:06.024726  7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "081930af26fd4f7fab29988df3e8b9aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:06.025460  7488 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:06.077428  7273 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:06.078233  7378 leader_election.cc:288] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:06.077762  7488 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 081930af26fd4f7fab29988df3e8b9aa in term 2.
I20260809 23:24:06.081761  7339 leader_election.cc:302] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 
I20260809 23:24:06.083505  8697 raft_consensus.cc:2781] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:06.083961  8697 raft_consensus.cc:686] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Leader. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:06.084873  8697 consensus_queue.cc:237] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:06.088410  7376 raft_consensus.cc:1261] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:06.098906  8647 consensus_queue.cc:1035] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
W20260809 23:24:06.118752  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:37549, user_credentials={real_user=slave}} blocked reactor thread for 72806.4us
I20260809 23:24:06.162006  7027 catalog_manager.cc:5495] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa reported cstate change: term changed from 1 to 2, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3). New cstate: current_term: 2 leader_uuid: "081930af26fd4f7fab29988df3e8b9aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:06.239102  7275 raft_consensus.cc:1261] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:06.240518  8673 consensus_queue.cc:1035] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:06.283685  7486 raft_consensus.cc:1261] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:06.294147  8605 consensus_queue.cc:1035] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:06.295475  7378 raft_consensus.cc:1261] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:06.335793  8720 consensus_queue.cc:1035] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:24:06.379932  7275 raft_consensus.cc:1261] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:06.381502  8719 consensus_queue.cc:1035] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:24:06.574753  7273 raft_consensus.cc:1261] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:06.576367  8714 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:06.576586 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:06.602442  7486 raft_consensus.cc:1261] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Refusing update from remote peer 081930af26fd4f7fab29988df3e8b9aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:06.603899  8593 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:06.666685  7376 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68"
dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
mode: GRACEFUL
new_leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
 from {username='slave'} at 127.0.0.1:60202
I20260809 23:24:06.667596  7376 raft_consensus.cc:595] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Received request to transfer leadership to TS ec555693f39b444f95a05ba0bd6b37e7
I20260809 23:24:06.669044 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260809 23:24:06.673311 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:06.674976 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:24:06.793406  7338 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 63288.4us
W20260809 23:24:06.799722  8731 log.cc:924] Time spent T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260809 23:24:07.026921  8734 raft_consensus.cc:979] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: : Instructing follower ec555693f39b444f95a05ba0bd6b37e7 to start an election
I20260809 23:24:07.027674  8593 raft_consensus.cc:1067] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Signalling peer ec555693f39b444f95a05ba0bd6b37e7 to start an election
I20260809 23:24:07.029637  7273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
 from {username='slave'} at 127.0.0.1:40650
I20260809 23:24:07.030191  7273 raft_consensus.cc:482] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:07.030550  7273 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:07.039654  7273 raft_consensus.cc:504] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:07.042994  7378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:07.043267  7486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58137ea1624f4b1db96f7a6b9ada7f68" candidate_uuid: "ec555693f39b444f95a05ba0bd6b37e7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de"
I20260809 23:24:07.043599  7378 raft_consensus.cc:3032] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:07.043864  7486 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:07.043953  7378 raft_consensus.cc:729] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 081930af26fd4f7fab29988df3e8b9aa, State: Running, Role: LEADER
I20260809 23:24:07.044726  7378 consensus_queue.cc:260] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:07.045853  7378 raft_consensus.cc:3037] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:07.062003  7378 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 3.
I20260809 23:24:07.064298  7227 leader_election.cc:302] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 081930af26fd4f7fab29988df3e8b9aa, ec555693f39b444f95a05ba0bd6b37e7; no voters: 
I20260809 23:24:07.065099  8580 raft_consensus.cc:2781] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:07.065603  8580 raft_consensus.cc:686] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 LEADER]: Becoming Leader. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:07.066406  8580 consensus_queue.cc:237] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } }
I20260809 23:24:07.063362  7273 leader_election.cc:288] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [CANDIDATE]: Term 3 election: Requested vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), a3cc3ec2dcd449de83f6c88e80d2d6de (127.21.255.4:42323)
I20260809 23:24:07.072103  7486 raft_consensus.cc:2454] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec555693f39b444f95a05ba0bd6b37e7 in term 3.
I20260809 23:24:07.082532  7027 catalog_manager.cc:5495] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 reported cstate change: term changed from 2 to 3, leader changed from 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3) to ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2). New cstate: current_term: 3 leader_uuid: "ec555693f39b444f95a05ba0bd6b37e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } health_report { overall_health: UNKNOWN } } }
W20260809 23:24:07.195534  8263 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.4:42323, user_credentials={real_user=slave}} blocked reactor thread for 56366.6us
W20260809 23:24:07.258383  7437 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.2:42321, user_credentials={real_user=slave}} blocked reactor thread for 46078.3us
I20260809 23:24:07.499444  7486 raft_consensus.cc:1261] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:07.500931  8602 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:07.529985  7378 raft_consensus.cc:1261] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Refusing update from remote peer ec555693f39b444f95a05ba0bd6b37e7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:07.536993  8711 consensus_queue.cc:1035] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260809 23:24:07.587301  7227 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:46099, user_credentials={real_user=slave}} blocked reactor thread for 55852.2us
I20260809 23:24:07.821136 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:07.928826  7275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb"
dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
mode: GRACEFUL
new_leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:44526
I20260809 23:24:07.931288  7275 raft_consensus.cc:595] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Received request to transfer leadership to TS ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:07.935750 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260809 23:24:07.937980 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:07.938591 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:08.205933  8747 raft_consensus.cc:979] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: : Instructing follower ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:08.206449  8747 raft_consensus.cc:1067] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Signalling peer ff5d35f70eb7411ba26062ea7d36ab15 to start an election
I20260809 23:24:08.208619  8330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb"
dest_uuid: "ff5d35f70eb7411ba26062ea7d36ab15"
 from {username='slave'} at 127.0.0.1:33472
I20260809 23:24:08.209149  8330 raft_consensus.cc:482] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:08.209434  8330 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:08.222338  8330 raft_consensus.cc:504] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:08.224924  7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec555693f39b444f95a05ba0bd6b37e7"
I20260809 23:24:08.225471  7275 raft_consensus.cc:3032] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:08.225742  7275 raft_consensus.cc:729] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec555693f39b444f95a05ba0bd6b37e7, State: Running, Role: LEADER
I20260809 23:24:08.225613  7376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67a2e2de3a9d45a8a776ce5d568059bb" candidate_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "081930af26fd4f7fab29988df3e8b9aa"
I20260809 23:24:08.226426  7376 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:08.226622  7275 consensus_queue.cc:260] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:08.227890  7275 raft_consensus.cc:3037] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:08.233474  8330 leader_election.cc:288] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Requested vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099)
I20260809 23:24:08.237970  7275 raft_consensus.cc:2454] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:08.239646  8262 leader_election.cc:302] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15; no voters: 
I20260809 23:24:08.241227  8720 raft_consensus.cc:2781] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:08.241716  7376 raft_consensus.cc:2454] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5d35f70eb7411ba26062ea7d36ab15 in term 2.
I20260809 23:24:08.242532  8720 raft_consensus.cc:686] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Becoming Leader. State: Replica: ff5d35f70eb7411ba26062ea7d36ab15, State: Running, Role: LEADER
I20260809 23:24:08.243546  8720 consensus_queue.cc:237] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } }
I20260809 23:24:08.256634  7027 catalog_manager.cc:5495] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 reported cstate change: term changed from 1 to 2, leader changed from ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2) to ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:08.713464  7376 raft_consensus.cc:1261] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:08.715072  8720 consensus_queue.cc:1035] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:08.736460  7275 raft_consensus.cc:1261] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Refusing update from remote peer ff5d35f70eb7411ba26062ea7d36ab15: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:08.738902  8760 consensus_queue.cc:1035] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:09.049347 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:09.090922 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:09.092172 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:09.092640 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:09.131934 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:09.181115 22524 tablet_server.cc:178] TabletServer@127.21.255.1:37549 shutting down...
W20260809 23:24:09.187973  7437 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 90 similar messages]
W20260809 23:24:09.219852  7437 consensus_peers.cc:487] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.229653  7227 consensus_peers.cc:487] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.241336  7227 consensus_peers.cc:487] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.248133  7338 consensus_peers.cc:487] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.265590  7437 consensus_peers.cc:487] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.266448  7437 consensus_peers.cc:487] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.268641  7338 consensus_peers.cc:487] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.276541  7338 consensus_peers.cc:487] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.278632 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:09.279884 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.280848 22524 raft_consensus.cc:2229] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.281508 22524 raft_consensus.cc:2258] T 28c7eaaf2fea43f7aae67f0524f08d94 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.284561  7338 consensus_peers.cc:487] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.284814 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.285823 22524 raft_consensus.cc:2229] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.286345 22524 raft_consensus.cc:2258] T 10b54da326434a3398b667f5d2955e1c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.289215  7227 consensus_peers.cc:487] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.290159 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.291134 22524 raft_consensus.cc:2229] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.291733 22524 raft_consensus.cc:2258] T 29afbcdad59b414fa48484b70ff56499 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.293901  7338 consensus_peers.cc:487] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.294657 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.295518 22524 raft_consensus.cc:2229] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.295969 22524 raft_consensus.cc:2258] T 0400132440374a1c88fa8fb7d7c5a6b4 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.298363 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.299181 22524 raft_consensus.cc:2229] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.300042 22524 raft_consensus.cc:2258] T 356a2f292a694ebbab1424f68bfe3dc5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.302383 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.303166 22524 raft_consensus.cc:2229] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.303570 22524 raft_consensus.cc:2258] T cffdb27ac1684d2bac75f40ba93712ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.306658 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.307762 22524 raft_consensus.cc:2229] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
W20260809 23:24:09.307865  7437 consensus_peers.cc:487] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.308912 22524 raft_consensus.cc:2258] T 3f7f029122e441ad9d3194829e439141 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.311776 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.312942 22524 raft_consensus.cc:2229] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.313519 22524 raft_consensus.cc:2258] T c18336f4a3644c798d5f8bb715908fd0 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.316500 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.317127 22524 raft_consensus.cc:2229] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.317531 22524 raft_consensus.cc:2258] T 327e1d8704e64831b8ebda9aaf3710b5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.319494 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.320094 22524 raft_consensus.cc:2229] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.320585 22524 raft_consensus.cc:2258] T 964bd1390e2d43b9ab381a795bdddedf P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.323315 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.323904 22524 raft_consensus.cc:2229] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.324791 22524 raft_consensus.cc:2258] T f23623f6580149f19787b8924d48ff24 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.326820 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.327590 22524 raft_consensus.cc:2229] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.328378 22524 raft_consensus.cc:2258] T 46a6ff8a5b7e4c73ae66ec2560453733 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.330423 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.331336 22524 raft_consensus.cc:2229] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.331774 22524 raft_consensus.cc:2258] T 3c057787ab114bba890199f5b36b37b7 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.334870 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.335557 22524 raft_consensus.cc:2229] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.336455 22524 raft_consensus.cc:2258] T 4468dee129594149bc011deeaef72185 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.338765 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.339438 22524 raft_consensus.cc:2229] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.339862 22524 raft_consensus.cc:2258] T 65b04e56cb2d4321a0b5e6317136d6f2 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.342572 22524 tablet_replica.cc:330] stopping tablet replica
W20260809 23:24:09.343047  7437 consensus_peers.cc:487] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.343374 22524 raft_consensus.cc:2229] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.343917 22524 raft_consensus.cc:2258] T e89212541eac4b91975dbfbe73e23c3a P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.346873 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.347630 22524 raft_consensus.cc:2229] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.348124 22524 raft_consensus.cc:2258] T b3e08fe5d08748ba98111cd243575917 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.351073 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.351770 22524 raft_consensus.cc:2229] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.352236 22524 raft_consensus.cc:2258] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.354506 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.355124 22524 raft_consensus.cc:2229] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.355649 22524 raft_consensus.cc:2258] T a0fa78046012472d8e5472478adbe1b9 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.357806 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.358412 22524 raft_consensus.cc:2229] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.359254 22524 raft_consensus.cc:2258] T 67a2e2de3a9d45a8a776ce5d568059bb P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.361941 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.362589 22524 raft_consensus.cc:2229] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.363130 22524 raft_consensus.cc:2258] T a9505b9c3f0342279113f757c317a868 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.365593 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.366310 22524 raft_consensus.cc:2229] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.367209 22524 raft_consensus.cc:2258] T 8d22e37cd9d24996abca51e457d761cc P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.369812 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.370462 22524 raft_consensus.cc:2229] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.371071 22524 raft_consensus.cc:2258] T 43caf538bbac4e0da20c274fad4d7f1f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.373874 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.374459 22524 raft_consensus.cc:2229] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.375082 22524 raft_consensus.cc:2258] T 4f841b906f634c05afe5a33637229c8f P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.377318 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.377903 22524 raft_consensus.cc:2229] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.378306 22524 raft_consensus.cc:2258] T 2e5b4438b1d84d7198189995b2d2d409 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.380775 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.381548 22524 raft_consensus.cc:2229] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.382067 22524 raft_consensus.cc:2258] T 7102c5cad94641438ac6c66721ce193c P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.384786 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.385370 22524 raft_consensus.cc:2229] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.385798 22524 raft_consensus.cc:2258] T 03a1d03c208b4696910c2b7dfe1469ab P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.388518 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.389485 22524 raft_consensus.cc:2229] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.390062 22524 raft_consensus.cc:2258] T c0e3fe783b214e9ba0376a444a7306ce P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.390225  7227 consensus_peers.cc:487] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.392886 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.393760 22524 raft_consensus.cc:2229] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.394820 22524 raft_consensus.cc:2258] T b64607aa57f542cda93ba367af662006 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.395220  7227 consensus_peers.cc:487] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.396147  7227 consensus_peers.cc:487] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.397846 22524 tablet_replica.cc:330] stopping tablet replica
W20260809 23:24:09.398662  7437 consensus_peers.cc:487] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.398765 22524 raft_consensus.cc:2229] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.399441 22524 raft_consensus.cc:2258] T 0590a10d16ff4d12aa9edfc6985ebede P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.401736 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.402261 22524 raft_consensus.cc:2229] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.403106 22524 raft_consensus.cc:2258] T bb2091b41c5341ebb239c60326eb8a22 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.405194 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.405752 22524 raft_consensus.cc:2229] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.406121 22524 raft_consensus.cc:2258] T edbd59811ea94e13b359f7c9db694683 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.408862 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.409631 22524 raft_consensus.cc:2229] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.410677 22524 raft_consensus.cc:2258] T 207ede1fd2664d3a871bbb0a5b854a64 P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.412420  7338 consensus_peers.cc:487] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.413250 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.414124 22524 raft_consensus.cc:2229] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.415083 22524 raft_consensus.cc:2258] T 3066e3b731424266bfd6155e1dddc90b P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.417654 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.418385 22524 raft_consensus.cc:2229] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.419360 22524 raft_consensus.cc:2258] T 4e7f251a895a462abb71b8668cf0aa80 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.421650 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.422173 22524 raft_consensus.cc:2229] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.422583 22524 raft_consensus.cc:2258] T 73788985062f47c397a6b02115c97a1e P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.424630 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.425177 22524 raft_consensus.cc:2229] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.425611 22524 raft_consensus.cc:2258] T bf9f1e8974a54b8b8c1d3a790e28a058 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.428081 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.429412 22524 raft_consensus.cc:2229] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.430032 22524 raft_consensus.cc:2258] T 7d68de750e444f43a513619eac515b03 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.432752 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.433892 22524 raft_consensus.cc:2229] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.434706 22524 raft_consensus.cc:2258] T 1f13e532e7304ece940fbe319522775f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.437641 22524 tablet_replica.cc:330] stopping tablet replica
W20260809 23:24:09.438301  7437 consensus_peers.cc:487] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.438380 22524 raft_consensus.cc:2229] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.440629 22524 raft_consensus.cc:2258] T 615321f1b8064c90892728123be05c59 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.442910 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.443543 22524 raft_consensus.cc:2229] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.444463 22524 raft_consensus.cc:2258] T ec734a4c9690454f8c3986ae773f6e9f P ff5d35f70eb7411ba26062ea7d36ab15 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.448036 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.449025 22524 raft_consensus.cc:2229] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.449695 22524 raft_consensus.cc:2258] T 2ecdc375af4a473ca3c09c02da2a2594 P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.453162 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.453755 22524 raft_consensus.cc:2229] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.454229 22524 raft_consensus.cc:2258] T d542b5134ed04d51b8f1e946e99c69a5 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.456547 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.457063 22524 raft_consensus.cc:2229] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.457484 22524 raft_consensus.cc:2258] T 3f126def92864a43a94113cdd599ee4f P ff5d35f70eb7411ba26062ea7d36ab15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.459717 22524 tablet_replica.cc:330] stopping tablet replica
I20260809 23:24:09.460186 22524 raft_consensus.cc:2229] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.460618 22524 raft_consensus.cc:2258] T 02f4e2e6c1834f09905dc17072c08a53 P ff5d35f70eb7411ba26062ea7d36ab15 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.468428  7227 consensus_peers.cc:487] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.479935  7437 consensus_peers.cc:487] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.485293  7338 consensus_peers.cc:487] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.488108  7437 consensus_peers.cc:487] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.526006  7227 consensus_peers.cc:487] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.557328 22524 tablet_server.cc:194] TabletServer@127.21.255.1:37549 shutdown complete.
W20260809 23:24:09.586113  7227 consensus_peers.cc:487] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.614071  7338 consensus_peers.cc:487] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.621223  7338 consensus_peers.cc:487] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.632496  7437 consensus_peers.cc:487] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.654620  7437 consensus_peers.cc:487] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.667335 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
W20260809 23:24:09.676254  7227 consensus_peers.cc:487] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.678527  7338 consensus_peers.cc:487] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Couldn't send request to peer ff5d35f70eb7411ba26062ea7d36ab15. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.681445  7437 consensus_peers.cc:487] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.715885  7437 consensus_peers.cc:487] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:09.736192  7338 consensus_peers.cc:487] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.739907 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:09.740870 22524 tablet_replica.cc:330] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.742080 22524 raft_consensus.cc:2229] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.742703 22524 raft_consensus.cc:2258] T 4e7f251a895a462abb71b8668cf0aa80 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.745376 22524 tablet_replica.cc:330] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.746712 22524 raft_consensus.cc:2229] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.747633 22524 raft_consensus.cc:2258] T 7a1a40c34e514b9fbcec1c0000466ecc P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.750758 22524 tablet_replica.cc:330] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.751585 22524 raft_consensus.cc:2229] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.752629 22524 raft_consensus.cc:2258] T 58137ea1624f4b1db96f7a6b9ada7f68 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.755295 22524 tablet_replica.cc:330] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.756160 22524 raft_consensus.cc:2229] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.757395 22524 raft_consensus.cc:2258] T 2ecdc375af4a473ca3c09c02da2a2594 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.757632  7437 consensus_peers.cc:487] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.761150 22524 tablet_replica.cc:330] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.762283 22524 raft_consensus.cc:2229] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.762972 22524 raft_consensus.cc:2258] T e89212541eac4b91975dbfbe73e23c3a P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.766270 22524 tablet_replica.cc:330] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.767223 22524 raft_consensus.cc:2229] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.767818 22524 raft_consensus.cc:2258] T 67a2e2de3a9d45a8a776ce5d568059bb P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.771791  7437 consensus_peers.cc:487] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.771824 22524 tablet_replica.cc:330] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.772972 22524 raft_consensus.cc:2229] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.773663 22524 raft_consensus.cc:2258] T a0fa78046012472d8e5472478adbe1b9 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.776907 22524 tablet_replica.cc:330] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
W20260809 23:24:09.777597  7338 consensus_peers.cc:487] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.777910 22524 raft_consensus.cc:2229] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.779115 22524 raft_consensus.cc:2258] T 7d68de750e444f43a513619eac515b03 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.780030  7338 consensus_peers.cc:487] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.781919 22524 tablet_replica.cc:330] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.782729 22524 raft_consensus.cc:2229] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.783209 22524 raft_consensus.cc:2258] T 64ff74f37a38496b8bf542b0df4d74f7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.785737 22524 tablet_replica.cc:330] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.786553 22524 raft_consensus.cc:2229] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.787147 22524 raft_consensus.cc:2258] T 3066e3b731424266bfd6155e1dddc90b P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.789695 22524 tablet_replica.cc:330] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.790388 22524 raft_consensus.cc:2229] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260809 23:24:09.790663  7338 consensus_peers.cc:487] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.791019 22524 raft_consensus.cc:2258] T 356a2f292a694ebbab1424f68bfe3dc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.793404 22524 tablet_replica.cc:330] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.794094 22524 raft_consensus.cc:2229] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.794592 22524 raft_consensus.cc:2258] T d87722cb5bdb4aa797b10938f4e1a640 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.797156 22524 tablet_replica.cc:330] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.797917 22524 raft_consensus.cc:2229] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.798408 22524 raft_consensus.cc:2258] T f23623f6580149f19787b8924d48ff24 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.801054 22524 tablet_replica.cc:330] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.801832 22524 raft_consensus.cc:2229] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.802899 22524 raft_consensus.cc:2258] T c9c4f8d91c504cdd8f1fd9dd8f707602 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.805648 22524 tablet_replica.cc:330] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.806460 22524 raft_consensus.cc:2229] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.807274 22524 raft_consensus.cc:2258] T 46a6ff8a5b7e4c73ae66ec2560453733 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.809867 22524 tablet_replica.cc:330] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.810678 22524 raft_consensus.cc:2229] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.811231 22524 raft_consensus.cc:2258] T 3c057787ab114bba890199f5b36b37b7 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.815126 22524 tablet_replica.cc:330] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.815685 22524 raft_consensus.cc:2229] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.816144 22524 raft_consensus.cc:2258] T 3f7beeffe6914835b84417315f44dd34 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.818821 22524 tablet_replica.cc:330] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.819593 22524 raft_consensus.cc:2229] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.820109 22524 raft_consensus.cc:2258] T 8ded669d4bcc4a7a81c44ad0b578c899 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.822858 22524 tablet_replica.cc:330] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.823649 22524 raft_consensus.cc:2229] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.824133 22524 raft_consensus.cc:2258] T bb2091b41c5341ebb239c60326eb8a22 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.827263 22524 tablet_replica.cc:330] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.828200 22524 raft_consensus.cc:2229] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.828768 22524 raft_consensus.cc:2258] T 10b54da326434a3398b667f5d2955e1c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.831884 22524 tablet_replica.cc:330] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.832988 22524 raft_consensus.cc:2229] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.833491 22524 raft_consensus.cc:2258] T e279157f9a0a4fb696c9bc01ae212b96 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.837201 22524 tablet_replica.cc:330] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.837978 22524 raft_consensus.cc:2229] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.838522 22524 raft_consensus.cc:2258] T 7102c5cad94641438ac6c66721ce193c P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.841274 22524 tablet_replica.cc:330] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.841970 22524 raft_consensus.cc:2229] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.842509 22524 raft_consensus.cc:2258] T cffdb27ac1684d2bac75f40ba93712ab P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.844502 22524 tablet_replica.cc:330] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.845022 22524 raft_consensus.cc:2229] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.845463 22524 raft_consensus.cc:2258] T 4468dee129594149bc011deeaef72185 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.847607 22524 tablet_replica.cc:330] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.848119 22524 raft_consensus.cc:2229] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.848852 22524 raft_consensus.cc:2258] T 65b04e56cb2d4321a0b5e6317136d6f2 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.850768 22524 tablet_replica.cc:330] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.851472 22524 raft_consensus.cc:2229] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.852452 22524 raft_consensus.cc:2258] T 2e008aea8ad64b2b9fa356cca2300ea1 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.854858 22524 tablet_replica.cc:330] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.855918 22524 raft_consensus.cc:2229] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.857884 22524 raft_consensus.cc:2258] T 0400132440374a1c88fa8fb7d7c5a6b4 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.860524 22524 tablet_replica.cc:330] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.861151 22524 raft_consensus.cc:2229] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.861603 22524 raft_consensus.cc:2258] T f9f47812f9fe462abdd14952f3aeb72b P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.863631 22524 tablet_replica.cc:330] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.864185 22524 raft_consensus.cc:2229] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.864985 22524 raft_consensus.cc:2258] T 1f13e532e7304ece940fbe319522775f P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.867063 22524 tablet_replica.cc:330] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.867574 22524 raft_consensus.cc:2229] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.868853 22524 raft_consensus.cc:2258] T 02f4e2e6c1834f09905dc17072c08a53 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.872181 22524 tablet_replica.cc:330] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.872946 22524 raft_consensus.cc:2229] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.874158 22524 raft_consensus.cc:2258] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.876708 22524 tablet_replica.cc:330] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.877388 22524 raft_consensus.cc:2229] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260809 23:24:09.877772  7338 consensus_peers.cc:487] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.877949 22524 raft_consensus.cc:2258] T 03a1d03c208b4696910c2b7dfe1469ab P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.880574 22524 tablet_replica.cc:330] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.881371 22524 raft_consensus.cc:2229] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.881880 22524 raft_consensus.cc:2258] T 28c7eaaf2fea43f7aae67f0524f08d94 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.884368 22524 tablet_replica.cc:330] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
W20260809 23:24:09.884591  7338 consensus_peers.cc:487] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.885010 22524 raft_consensus.cc:2229] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.885499 22524 raft_consensus.cc:2258] T 5ad1578784c042f68a1e8d865c5583aa P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.887753 22524 tablet_replica.cc:330] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.888518 22524 raft_consensus.cc:2229] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.889099 22524 raft_consensus.cc:2258] T d542b5134ed04d51b8f1e946e99c69a5 P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.892117 22524 tablet_replica.cc:330] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.892877 22524 raft_consensus.cc:2229] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.893393 22524 raft_consensus.cc:2258] T 1632dcfee20a4fc0a39347412a4dcc42 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.895020  7437 consensus_peers.cc:487] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.896199 22524 tablet_replica.cc:330] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.896798 22524 raft_consensus.cc:2229] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.897274 22524 raft_consensus.cc:2258] T 8d22e37cd9d24996abca51e457d761cc P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.900005 22524 tablet_replica.cc:330] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.900687 22524 raft_consensus.cc:2229] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.901836 22524 raft_consensus.cc:2258] T 43caf538bbac4e0da20c274fad4d7f1f P ec555693f39b444f95a05ba0bd6b37e7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.904662 22524 tablet_replica.cc:330] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
W20260809 23:24:09.905674  7338 consensus_peers.cc:487] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.905851 22524 raft_consensus.cc:2229] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.907603 22524 raft_consensus.cc:2258] T 2e5b4438b1d84d7198189995b2d2d409 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.908681  7437 consensus_peers.cc:487] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.911149 22524 tablet_replica.cc:330] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.911860 22524 raft_consensus.cc:2229] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.912969 22524 raft_consensus.cc:2258] T ce32c0f77b0c4572a2e2bf042e987cc5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.916172 22524 tablet_replica.cc:330] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
W20260809 23:24:09.916664  7437 consensus_peers.cc:487] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.916932 22524 raft_consensus.cc:2229] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.918043 22524 raft_consensus.cc:2258] T c18336f4a3644c798d5f8bb715908fd0 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:09.919085  7437 consensus_peers.cc:487] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:09.920889 22524 tablet_replica.cc:330] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.921887 22524 raft_consensus.cc:2229] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.922538 22524 raft_consensus.cc:2258] T ec734a4c9690454f8c3986ae773f6e9f P ec555693f39b444f95a05ba0bd6b37e7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.925724 22524 tablet_replica.cc:330] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.926496 22524 raft_consensus.cc:2229] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:09.927610 22524 raft_consensus.cc:2258] T 895d1e78ae154da4a1c299f234d88965 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:09.931372 22524 tablet_replica.cc:330] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7: stopping tablet replica
I20260809 23:24:09.932096 22524 raft_consensus.cc:2229] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:09.932777 22524 raft_consensus.cc:2258] T 327e1d8704e64831b8ebda9aaf3710b5 P ec555693f39b444f95a05ba0bd6b37e7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.013233  7338 consensus_peers.cc:487] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.023768 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
W20260809 23:24:10.109938  7437 consensus_peers.cc:487] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:10.114207  7338 consensus_peers.cc:487] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.125321 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
W20260809 23:24:10.135447  7437 consensus_peers.cc:487] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:10.161425  7338 consensus_peers.cc:487] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa -> Peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Couldn't send request to peer ec555693f39b444f95a05ba0bd6b37e7. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.199853 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:10.201833 22524 tablet_replica.cc:330] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.203116 22524 raft_consensus.cc:2229] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.203725 22524 raft_consensus.cc:2258] T 964bd1390e2d43b9ab381a795bdddedf P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.206291 22524 tablet_replica.cc:330] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.207059 22524 raft_consensus.cc:2229] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.207476 22524 raft_consensus.cc:2258] T 29afbcdad59b414fa48484b70ff56499 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.210139 22524 tablet_replica.cc:330] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.211011 22524 raft_consensus.cc:2229] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.211555 22524 raft_consensus.cc:2258] T 67a2e2de3a9d45a8a776ce5d568059bb P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.214061 22524 tablet_replica.cc:330] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.215014 22524 raft_consensus.cc:2229] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.215528 22524 raft_consensus.cc:2258] T 3f126def92864a43a94113cdd599ee4f P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.217816 22524 tablet_replica.cc:330] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.218430 22524 raft_consensus.cc:2229] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.219300 22524 raft_consensus.cc:2258] T 7a1a40c34e514b9fbcec1c0000466ecc P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.222477 22524 tablet_replica.cc:330] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.223408 22524 raft_consensus.cc:2229] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.224658 22524 raft_consensus.cc:2258] T 8ded669d4bcc4a7a81c44ad0b578c899 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.227560 22524 tablet_replica.cc:330] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.228255 22524 raft_consensus.cc:2229] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.228713 22524 raft_consensus.cc:2258] T b3e08fe5d08748ba98111cd243575917 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.230683 22524 tablet_replica.cc:330] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.231271 22524 raft_consensus.cc:2229] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.231694 22524 raft_consensus.cc:2258] T 58137ea1624f4b1db96f7a6b9ada7f68 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.233572 22524 tablet_replica.cc:330] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.234086 22524 raft_consensus.cc:2229] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.234594 22524 raft_consensus.cc:2258] T 2ecdc375af4a473ca3c09c02da2a2594 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.236475 22524 tablet_replica.cc:330] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.237012 22524 raft_consensus.cc:2229] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.237679 22524 raft_consensus.cc:2258] T 207ede1fd2664d3a871bbb0a5b854a64 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.240154 22524 tablet_replica.cc:330] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.240747 22524 raft_consensus.cc:2229] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.241195 22524 raft_consensus.cc:2258] T 3066e3b731424266bfd6155e1dddc90b P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.244129 22524 tablet_replica.cc:330] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.244966 22524 raft_consensus.cc:2229] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.245556 22524 raft_consensus.cc:2258] T f23623f6580149f19787b8924d48ff24 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.248209 22524 tablet_replica.cc:330] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.248875 22524 raft_consensus.cc:2229] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.249370 22524 raft_consensus.cc:2258] T c9c4f8d91c504cdd8f1fd9dd8f707602 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.251513 22524 tablet_replica.cc:330] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.252067 22524 raft_consensus.cc:2229] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.252858 22524 raft_consensus.cc:2258] T d87722cb5bdb4aa797b10938f4e1a640 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.254618 22524 tablet_replica.cc:330] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.255216 22524 raft_consensus.cc:2229] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.255985 22524 raft_consensus.cc:2258] T 3c057787ab114bba890199f5b36b37b7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.257773 22524 tablet_replica.cc:330] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.258706 22524 raft_consensus.cc:2229] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.259189 22524 raft_consensus.cc:2258] T 615321f1b8064c90892728123be05c59 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.261010 22524 tablet_replica.cc:330] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.261552 22524 raft_consensus.cc:2229] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.262825 22524 raft_consensus.cc:2258] T 3f7beeffe6914835b84417315f44dd34 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.265519 22524 tablet_replica.cc:330] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.266254 22524 raft_consensus.cc:2229] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.267083 22524 raft_consensus.cc:2258] T bf9f1e8974a54b8b8c1d3a790e28a058 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.269297 22524 tablet_replica.cc:330] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.270133 22524 raft_consensus.cc:2229] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.270664 22524 raft_consensus.cc:2258] T 64ff74f37a38496b8bf542b0df4d74f7 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.273326 22524 tablet_replica.cc:330] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.274096 22524 raft_consensus.cc:2229] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.274617 22524 raft_consensus.cc:2258] T 895d1e78ae154da4a1c299f234d88965 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.276648 22524 tablet_replica.cc:330] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.277698 22524 raft_consensus.cc:2229] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.278892 22524 raft_consensus.cc:2258] T 327e1d8704e64831b8ebda9aaf3710b5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.281991 22524 tablet_replica.cc:330] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.282783 22524 raft_consensus.cc:2229] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.283365 22524 raft_consensus.cc:2258] T e279157f9a0a4fb696c9bc01ae212b96 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.286767 22524 tablet_replica.cc:330] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.287504 22524 raft_consensus.cc:2229] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.288415 22524 raft_consensus.cc:2258] T 7102c5cad94641438ac6c66721ce193c P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.290803 22524 tablet_replica.cc:330] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.291455 22524 raft_consensus.cc:2229] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.292307 22524 raft_consensus.cc:2258] T 03a1d03c208b4696910c2b7dfe1469ab P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.296074 22524 tablet_replica.cc:330] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.297045 22524 raft_consensus.cc:2229] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.297998 22524 raft_consensus.cc:2258] T c0e3fe783b214e9ba0376a444a7306ce P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.299750  7437 consensus_peers.cc:487] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.301056 22524 tablet_replica.cc:330] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.301790 22524 raft_consensus.cc:2229] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.302605 22524 raft_consensus.cc:2258] T cffdb27ac1684d2bac75f40ba93712ab P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.305015 22524 tablet_replica.cc:330] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.305759 22524 raft_consensus.cc:2229] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.306293 22524 raft_consensus.cc:2258] T edbd59811ea94e13b359f7c9db694683 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.309370 22524 tablet_replica.cc:330] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.310189 22524 raft_consensus.cc:2229] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.310731 22524 raft_consensus.cc:2258] T b64607aa57f542cda93ba367af662006 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.313395 22524 tablet_replica.cc:330] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.313997 22524 raft_consensus.cc:2229] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.315021 22524 raft_consensus.cc:2258] T 0590a10d16ff4d12aa9edfc6985ebede P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.317505 22524 tablet_replica.cc:330] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.318723 22524 raft_consensus.cc:2229] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.319370 22524 raft_consensus.cc:2258] T 0f6db5f7d4fa4aa3973cc96d9c5fbdec P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.322046 22524 tablet_replica.cc:330] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.322683 22524 raft_consensus.cc:2229] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.323746 22524 raft_consensus.cc:2258] T 1632dcfee20a4fc0a39347412a4dcc42 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.326229 22524 tablet_replica.cc:330] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.327114 22524 raft_consensus.cc:2229] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.327682 22524 raft_consensus.cc:2258] T a9505b9c3f0342279113f757c317a868 P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.329741 22524 tablet_replica.cc:330] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.330472 22524 raft_consensus.cc:2229] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.331044 22524 raft_consensus.cc:2258] T 3f7f029122e441ad9d3194829e439141 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.333305 22524 tablet_replica.cc:330] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.334095 22524 raft_consensus.cc:2229] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.334657 22524 raft_consensus.cc:2258] T 5ad1578784c042f68a1e8d865c5583aa P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.337760 22524 tablet_replica.cc:330] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.338531 22524 raft_consensus.cc:2229] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.339134 22524 raft_consensus.cc:2258] T 2e008aea8ad64b2b9fa356cca2300ea1 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.342670 22524 tablet_replica.cc:330] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.343504 22524 raft_consensus.cc:2229] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.344115 22524 raft_consensus.cc:2258] T f9f47812f9fe462abdd14952f3aeb72b P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.345294  7437 consensus_peers.cc:487] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.346872 22524 tablet_replica.cc:330] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.347784 22524 raft_consensus.cc:2229] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.348371 22524 raft_consensus.cc:2258] T 0400132440374a1c88fa8fb7d7c5a6b4 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.351244 22524 tablet_replica.cc:330] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.351963 22524 raft_consensus.cc:2229] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.352479 22524 raft_consensus.cc:2258] T 2e5b4438b1d84d7198189995b2d2d409 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.354997 22524 tablet_replica.cc:330] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.355553 22524 raft_consensus.cc:2229] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.355993 22524 raft_consensus.cc:2258] T 8d22e37cd9d24996abca51e457d761cc P 081930af26fd4f7fab29988df3e8b9aa [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.358409 22524 tablet_replica.cc:330] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.359462 22524 raft_consensus.cc:2229] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.360630 22524 raft_consensus.cc:2258] T 4f841b906f634c05afe5a33637229c8f P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.364233 22524 tablet_replica.cc:330] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.365141 22524 raft_consensus.cc:2229] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.365667 22524 raft_consensus.cc:2258] T ce32c0f77b0c4572a2e2bf042e987cc5 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.368716 22524 tablet_replica.cc:330] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.369349 22524 raft_consensus.cc:2229] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.370306 22524 raft_consensus.cc:2258] T 73788985062f47c397a6b02115c97a1e P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.370563  7437 consensus_peers.cc:487] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.372452 22524 tablet_replica.cc:330] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.373047 22524 raft_consensus.cc:2229] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.373454 22524 raft_consensus.cc:2258] T 65b04e56cb2d4321a0b5e6317136d6f2 P 081930af26fd4f7fab29988df3e8b9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.375391 22524 tablet_replica.cc:330] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa: stopping tablet replica
I20260809 23:24:10.375797 22524 raft_consensus.cc:2229] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.376166 22524 raft_consensus.cc:2258] T 02f4e2e6c1834f09905dc17072c08a53 P 081930af26fd4f7fab29988df3e8b9aa [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.396800  7437 consensus_peers.cc:487] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.417202  8696 raft_consensus.cc:482] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.417840  8696 raft_consensus.cc:504] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:10.419787  8696 leader_election.cc:288] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
W20260809 23:24:10.423487  7437 leader_election.cc:334] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260809 23:24:10.423985  7437 leader_election.cc:334] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:24:10.424294  7437 leader_election.cc:302] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.424916  8696 raft_consensus.cc:2726] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:24:10.492843 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:24:10.497526  8696 raft_consensus.cc:482] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.498840  8696 raft_consensus.cc:504] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
W20260809 23:24:10.500123  7437 consensus_peers.cc:487] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.500692  8696 leader_election.cc:288] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
W20260809 23:24:10.504918  7437 leader_election.cc:334] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
W20260809 23:24:10.505847  7437 leader_election.cc:334] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111)
I20260809 23:24:10.506170  7437 leader_election.cc:302] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.506747  8696 raft_consensus.cc:2726] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260809 23:24:10.508342  7437 consensus_peers.cc:487] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:10.525710  7437 consensus_peers.cc:487] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.528937  8696 raft_consensus.cc:482] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.529441  8696 raft_consensus.cc:504] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:10.531617  8696 leader_election.cc:288] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
W20260809 23:24:10.535780  7437 leader_election.cc:334] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
W20260809 23:24:10.536512  7437 leader_election.cc:334] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111)
I20260809 23:24:10.536918  7437 leader_election.cc:302] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.537811  8696 raft_consensus.cc:2726] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:24:10.583202 22524 tablet_server.cc:178] TabletServer@127.21.255.4:0 shutting down...
I20260809 23:24:10.589138  8696 raft_consensus.cc:482] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.589767  8696 raft_consensus.cc:504] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:10.592031  8696 leader_election.cc:288] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
W20260809 23:24:10.600857  7437 leader_election.cc:334] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
W20260809 23:24:10.605072  7437 leader_election.cc:334] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111)
I20260809 23:24:10.605549  7437 leader_election.cc:302] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.606348  8696 raft_consensus.cc:2726] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:24:10.634569 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:10.635807 22524 tablet_replica.cc:330] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.636857 22524 raft_consensus.cc:2229] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.638033 22524 raft_consensus.cc:2258] T 964bd1390e2d43b9ab381a795bdddedf P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.641014 22524 tablet_replica.cc:330] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.641721 22524 raft_consensus.cc:2229] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.642225 22524 raft_consensus.cc:2258] T 58137ea1624f4b1db96f7a6b9ada7f68 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.644735 22524 tablet_replica.cc:330] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.645712 22524 raft_consensus.cc:2229] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.647238 22524 raft_consensus.cc:2258] T 29afbcdad59b414fa48484b70ff56499 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.647169  8794 raft_consensus.cc:482] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.648623  8794 raft_consensus.cc:504] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "081930af26fd4f7fab29988df3e8b9aa" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 46099 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:10.650979 22524 tablet_replica.cc:330] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.651381  8794 leader_election.cc:288] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:24:10.651942 22524 raft_consensus.cc:2229] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.652529 22524 raft_consensus.cc:2258] T 46a6ff8a5b7e4c73ae66ec2560453733 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.655198 22524 tablet_replica.cc:330] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.655990 22524 raft_consensus.cc:2229] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
W20260809 23:24:10.655988  7437 leader_election.cc:334] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111)
I20260809 23:24:10.656759 22524 raft_consensus.cc:2258] T 4e7f251a895a462abb71b8668cf0aa80 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.657406  7437 leader_election.cc:334] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:24:10.657881  7437 leader_election.cc:302] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: 081930af26fd4f7fab29988df3e8b9aa, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.658772  8794 raft_consensus.cc:2726] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:24:10.660288 22524 tablet_replica.cc:330] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.661186 22524 raft_consensus.cc:2229] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.661723 22524 raft_consensus.cc:2258] T 895d1e78ae154da4a1c299f234d88965 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.662786  7437 consensus_peers.cc:487] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.664392 22524 tablet_replica.cc:330] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.665062 22524 raft_consensus.cc:2229] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.665503 22524 raft_consensus.cc:2258] T 207ede1fd2664d3a871bbb0a5b854a64 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.667091  7437 consensus_peers.cc:487] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de -> Peer 081930af26fd4f7fab29988df3e8b9aa (127.21.255.3:46099): Couldn't send request to peer 081930af26fd4f7fab29988df3e8b9aa. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.3:46099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:10.667840 22524 tablet_replica.cc:330] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.668520 22524 raft_consensus.cc:2229] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.668987 22524 raft_consensus.cc:2258] T 7a1a40c34e514b9fbcec1c0000466ecc P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.671238 22524 tablet_replica.cc:330] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.671792 22524 raft_consensus.cc:2229] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.672252 22524 raft_consensus.cc:2258] T 8ded669d4bcc4a7a81c44ad0b578c899 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.674266 22524 tablet_replica.cc:330] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.674845 22524 raft_consensus.cc:2229] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.675740 22524 raft_consensus.cc:2258] T b3e08fe5d08748ba98111cd243575917 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.675894  8794 raft_consensus.cc:482] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.676440  8794 raft_consensus.cc:504] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } } peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } }
I20260809 23:24:10.677934 22524 tablet_replica.cc:330] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.678135  8794 leader_election.cc:288] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549), ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321)
I20260809 23:24:10.678830 22524 raft_consensus.cc:2229] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.679986 22524 raft_consensus.cc:2258] T e89212541eac4b91975dbfbe73e23c3a P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.681334  7437 leader_election.cc:334] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
W20260809 23:24:10.682032  7437 leader_election.cc:334] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111)
I20260809 23:24:10.682395  7437 leader_election.cc:302] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.682996  8794 raft_consensus.cc:2726] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260809 23:24:10.683495 22524 tablet_replica.cc:330] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.684136 22524 raft_consensus.cc:2229] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.684937 22524 raft_consensus.cc:2258] T a0fa78046012472d8e5472478adbe1b9 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.687152 22524 tablet_replica.cc:330] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.687705 22524 raft_consensus.cc:2229] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.688172 22524 raft_consensus.cc:2258] T c0e3fe783b214e9ba0376a444a7306ce P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.690237 22524 tablet_replica.cc:330] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.690807 22524 raft_consensus.cc:2229] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.691273 22524 raft_consensus.cc:2258] T 3f7beeffe6914835b84417315f44dd34 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.692989 22524 tablet_replica.cc:330] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.693560 22524 raft_consensus.cc:2229] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.694334 22524 raft_consensus.cc:2258] T 5ad1578784c042f68a1e8d865c5583aa P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.696221 22524 tablet_replica.cc:330] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.696735 22524 raft_consensus.cc:2229] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.697590 22524 raft_consensus.cc:2258] T 28c7eaaf2fea43f7aae67f0524f08d94 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.699581 22524 tablet_replica.cc:330] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.700137 22524 raft_consensus.cc:2229] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.700551 22524 raft_consensus.cc:2258] T d87722cb5bdb4aa797b10938f4e1a640 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.702476 22524 tablet_replica.cc:330] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.703070 22524 raft_consensus.cc:2229] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.703486 22524 raft_consensus.cc:2258] T 356a2f292a694ebbab1424f68bfe3dc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.705812 22524 tablet_replica.cc:330] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.706405 22524 raft_consensus.cc:2229] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.706804 22524 raft_consensus.cc:2258] T 1f13e532e7304ece940fbe319522775f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.709249 22524 tablet_replica.cc:330] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.710090 22524 raft_consensus.cc:2229] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.710657 22524 raft_consensus.cc:2258] T 615321f1b8064c90892728123be05c59 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.713150 22524 tablet_replica.cc:330] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.713923 22524 raft_consensus.cc:2229] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.714825 22524 raft_consensus.cc:2258] T 10b54da326434a3398b667f5d2955e1c P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.715204  8794 raft_consensus.cc:482] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ff5d35f70eb7411ba26062ea7d36ab15)
I20260809 23:24:10.715778  8794 raft_consensus.cc:504] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec555693f39b444f95a05ba0bd6b37e7" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 42321 } } peers { permanent_uuid: "a3cc3ec2dcd449de83f6c88e80d2d6de" member_type: VOTER last_known_addr { host: "127.21.255.4" port: 42323 } } peers { permanent_uuid: "ff5d35f70eb7411ba26062ea7d36ab15" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 37549 } }
I20260809 23:24:10.717023 22524 tablet_replica.cc:330] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.717571  8794 leader_election.cc:288] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321), ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549)
I20260809 23:24:10.718011 22524 raft_consensus.cc:2229] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.719003 22524 raft_consensus.cc:2258] T bb2091b41c5341ebb239c60326eb8a22 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:10.721542  7437 leader_election.cc:334] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ec555693f39b444f95a05ba0bd6b37e7 (127.21.255.2:42321): Network error: Client connection negotiation failed: client connection to 127.21.255.2:42321: connect: Connection refused (error 111)
W20260809 23:24:10.722363  7437 leader_election.cc:334] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ff5d35f70eb7411ba26062ea7d36ab15 (127.21.255.1:37549): Network error: Client connection negotiation failed: client connection to 127.21.255.1:37549: connect: Connection refused (error 111)
I20260809 23:24:10.722630 22524 tablet_replica.cc:330] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.722805  7437 leader_election.cc:302] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3cc3ec2dcd449de83f6c88e80d2d6de; no voters: ec555693f39b444f95a05ba0bd6b37e7, ff5d35f70eb7411ba26062ea7d36ab15
I20260809 23:24:10.723615 22524 raft_consensus.cc:2229] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.723711  8794 raft_consensus.cc:2726] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:24:10.724591 22524 raft_consensus.cc:2258] T edbd59811ea94e13b359f7c9db694683 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.727087 22524 tablet_replica.cc:330] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.727662 22524 raft_consensus.cc:2229] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.728078 22524 raft_consensus.cc:2258] T b64607aa57f542cda93ba367af662006 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.730083 22524 tablet_replica.cc:330] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.730672 22524 raft_consensus.cc:2229] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.731215 22524 raft_consensus.cc:2258] T 0590a10d16ff4d12aa9edfc6985ebede P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.733714 22524 tablet_replica.cc:330] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.734422 22524 raft_consensus.cc:2229] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.734833 22524 raft_consensus.cc:2258] T 2e008aea8ad64b2b9fa356cca2300ea1 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.737267 22524 tablet_replica.cc:330] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.737847 22524 raft_consensus.cc:2229] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.738694 22524 raft_consensus.cc:2258] T f9f47812f9fe462abdd14952f3aeb72b P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.740731 22524 tablet_replica.cc:330] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.741379 22524 raft_consensus.cc:2229] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.742524 22524 raft_consensus.cc:2258] T a9505b9c3f0342279113f757c317a868 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.745232 22524 tablet_replica.cc:330] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.745977 22524 raft_consensus.cc:2229] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.746470 22524 raft_consensus.cc:2258] T 1632dcfee20a4fc0a39347412a4dcc42 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.748636 22524 tablet_replica.cc:330] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.749190 22524 raft_consensus.cc:2229] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.750010 22524 raft_consensus.cc:2258] T 3f126def92864a43a94113cdd599ee4f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.752100 22524 tablet_replica.cc:330] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.752636 22524 raft_consensus.cc:2229] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.753424 22524 raft_consensus.cc:2258] T d542b5134ed04d51b8f1e946e99c69a5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.755371 22524 tablet_replica.cc:330] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.756166 22524 raft_consensus.cc:2229] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.756805 22524 raft_consensus.cc:2258] T 4468dee129594149bc011deeaef72185 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.759199 22524 tablet_replica.cc:330] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.759917 22524 raft_consensus.cc:2229] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.760391 22524 raft_consensus.cc:2258] T c18336f4a3644c798d5f8bb715908fd0 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.762377 22524 tablet_replica.cc:330] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.762907 22524 raft_consensus.cc:2229] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.763403 22524 raft_consensus.cc:2258] T 3f7f029122e441ad9d3194829e439141 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.765144 22524 tablet_replica.cc:330] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.765617 22524 raft_consensus.cc:2229] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.765971 22524 raft_consensus.cc:2258] T ec734a4c9690454f8c3986ae773f6e9f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.767967 22524 tablet_replica.cc:330] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.768536 22524 raft_consensus.cc:2229] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.768903 22524 raft_consensus.cc:2258] T c9c4f8d91c504cdd8f1fd9dd8f707602 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.770658 22524 tablet_replica.cc:330] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.771162 22524 raft_consensus.cc:2229] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.771554 22524 raft_consensus.cc:2258] T 43caf538bbac4e0da20c274fad4d7f1f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.773450 22524 tablet_replica.cc:330] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.773913 22524 raft_consensus.cc:2229] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.774449 22524 raft_consensus.cc:2258] T 4f841b906f634c05afe5a33637229c8f P a3cc3ec2dcd449de83f6c88e80d2d6de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.776736 22524 tablet_replica.cc:330] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.777287 22524 raft_consensus.cc:2229] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.777696 22524 raft_consensus.cc:2258] T ce32c0f77b0c4572a2e2bf042e987cc5 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.779513 22524 tablet_replica.cc:330] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.780006 22524 raft_consensus.cc:2229] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.780426 22524 raft_consensus.cc:2258] T 7d68de750e444f43a513619eac515b03 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.782573 22524 tablet_replica.cc:330] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.783128 22524 raft_consensus.cc:2229] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.784181 22524 raft_consensus.cc:2258] T 64ff74f37a38496b8bf542b0df4d74f7 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.786545 22524 tablet_replica.cc:330] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.787125 22524 raft_consensus.cc:2229] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.787521 22524 raft_consensus.cc:2258] T bf9f1e8974a54b8b8c1d3a790e28a058 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.789453 22524 tablet_replica.cc:330] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.789932 22524 raft_consensus.cc:2229] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.790719 22524 raft_consensus.cc:2258] T e279157f9a0a4fb696c9bc01ae212b96 P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.792593 22524 tablet_replica.cc:330] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de: stopping tablet replica
I20260809 23:24:10.793046 22524 raft_consensus.cc:2229] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:10.793500 22524 raft_consensus.cc:2258] T 73788985062f47c397a6b02115c97a1e P a3cc3ec2dcd449de83f6c88e80d2d6de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.867327 22524 tablet_server.cc:194] TabletServer@127.21.255.4:0 shutdown complete.
I20260809 23:24:10.937501 22524 master.cc:537] Master@127.21.255.62:37409 shutting down...
I20260809 23:24:10.973541 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:10.974246 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:10.974586 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P de4b75404640471e847c3177bd586290: stopping tablet replica
I20260809 23:24:11.022701 22524 master.cc:552] Master@127.21.255.62:37409 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (33136 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:471: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[  SKIPPED ] LeaderRebalancerTest.TestMaintenanceMode (8 ms)
[----------] 5 tests from LeaderRebalancerTest (466194 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260809 23:24:11.138204 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:41225
I20260809 23:24:11.139555 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:11.146823  8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.147574  8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.148020  8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:11.148761 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:11.150278 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:11.150509 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:11.150663 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317851150649 us; error 0 us; skew 500 ppm
I20260809 23:24:11.151259 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:11.157728 22524 webserver.cc:460] Webserver started at http://127.21.255.62:35471/ using document root <none> and password file <none>
I20260809 23:24:11.158317 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:11.158531 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:11.158856 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:11.160142 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "020a52ca66bf4a679112f70554a4411e"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.165268 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260809 23:24:11.169356  8810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.170616 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260809 23:24:11.171077 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "020a52ca66bf4a679112f70554a4411e"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.171403 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:11.183811 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:11.185014 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:11.228729 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:41225
I20260809 23:24:11.228896  8880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:41225 every 8 connection(s)
I20260809 23:24:11.232717  8882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.243491  8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: Bootstrap starting.
I20260809 23:24:11.248332  8882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:11.252846  8882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: No bootstrap required, opened a new log
I20260809 23:24:11.254897  8882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER }
I20260809 23:24:11.255286  8882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:11.255533  8882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020a52ca66bf4a679112f70554a4411e, State: Initialized, Role: FOLLOWER
I20260809 23:24:11.256217  8882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER }
I20260809 23:24:11.256660  8882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:24:11.256896  8882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:24:11.257165  8882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:11.261844  8882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER }
I20260809 23:24:11.262360  8882 leader_election.cc:302] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 020a52ca66bf4a679112f70554a4411e; no voters: 
I20260809 23:24:11.263531  8882 leader_election.cc:288] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:24:11.263985  8886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:11.265414  8886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 1 LEADER]: Becoming Leader. State: Replica: 020a52ca66bf4a679112f70554a4411e, State: Running, Role: LEADER
I20260809 23:24:11.266003  8886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER }
I20260809 23:24:11.266552  8882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:24:11.271905  8890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 020a52ca66bf4a679112f70554a4411e. Latest consensus state: current_term: 1 leader_uuid: "020a52ca66bf4a679112f70554a4411e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER } }
I20260809 23:24:11.272898  8890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [sys.catalog]: This master's current role is: LEADER
I20260809 23:24:11.273522  8887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "020a52ca66bf4a679112f70554a4411e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "020a52ca66bf4a679112f70554a4411e" member_type: VOTER } }
I20260809 23:24:11.275120  8887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [sys.catalog]: This master's current role is: LEADER
I20260809 23:24:11.282889  8901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:24:11.284137 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:24:11.288435  8901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:24:11.298218  8901 catalog_manager.cc:1347] Generated new cluster ID: c9248841a3624f6c9d80fd5aa7a06b39
I20260809 23:24:11.298537  8901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:24:11.321916  8901 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:24:11.323776  8901 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:24:11.341890  8901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: Generated new TSK 0
I20260809 23:24:11.342557  8901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:24:11.351444 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:11.358916  8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.360175  8917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.363372  8921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:11.363632 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:11.365046 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:11.365269 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:11.365448 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317851365432 us; error 0 us; skew 500 ppm
I20260809 23:24:11.366046 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:11.369496 22524 webserver.cc:460] Webserver started at http://127.21.255.1:45669/ using document root <none> and password file <none>
I20260809 23:24:11.370152 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:11.370386 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:11.370687 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:11.372161 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "f6964fd5f558447d99495c9e69df90c1"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.377955 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260809 23:24:11.382220  8927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.383275 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:24:11.383611 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "f6964fd5f558447d99495c9e69df90c1"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.383960 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:11.402351 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:11.403653 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:11.405241 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:11.407712 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:11.407923 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.408298 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:11.408555 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.461721 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:32919
I20260809 23:24:11.461850  9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:32919 every 8 connection(s)
I20260809 23:24:11.466629 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:11.475351  9016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.477659  9017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.480680  9019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:11.483021 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:11.484498  9011 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41225
I20260809 23:24:11.485095  9011 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:11.485769 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:11.486153 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:11.486482  9011 heartbeater.cc:511] Master 127.21.255.62:41225 requested a full tablet report, sending...
I20260809 23:24:11.486626 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317851486605 us; error 0 us; skew 500 ppm
I20260809 23:24:11.487545 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:11.489283  8834 ts_manager.cc:194] Registered new tserver with Master: f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919)
I20260809 23:24:11.490482 22524 webserver.cc:460] Webserver started at http://127.21.255.2:36519/ using document root <none> and password file <none>
I20260809 23:24:11.491113 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:11.491330 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:11.491600 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:11.491801  8834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36464
I20260809 23:24:11.493126 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "19446675307f4e709bb1c63b9c529e54"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.498173 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260809 23:24:11.502692  9027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.503705 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:24:11.504069 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "19446675307f4e709bb1c63b9c529e54"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.504391 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:11.525210 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:11.526376 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:11.527902 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:11.530241 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:11.530437 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.530741 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:11.530884 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.581269 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:44705
I20260809 23:24:11.581435  9115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:44705 every 8 connection(s)
I20260809 23:24:11.587236 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:11.596737  9119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:11.598981  9120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:11.601961 22524 server_base.cc:1034] running on GCE node
W20260809 23:24:11.603260  9122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:11.604828 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:11.605152 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:11.605381 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317851605360 us; error 0 us; skew 500 ppm
I20260809 23:24:11.606230 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:11.606530  9116 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41225
I20260809 23:24:11.607091  9116 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:11.608261  9116 heartbeater.cc:511] Master 127.21.255.62:41225 requested a full tablet report, sending...
I20260809 23:24:11.609687 22524 webserver.cc:460] Webserver started at http://127.21.255.3:34393/ using document root <none> and password file <none>
I20260809 23:24:11.610392 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:11.610562 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:11.610849 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:11.611405  8834 ts_manager.cc:194] Registered new tserver with Master: 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:11.612053 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.613440  8834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36470
I20260809 23:24:11.617617 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20260809 23:24:11.621841  9129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.623108 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:24:11.623483 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
format_stamp: "Formatted at 2026-08-09 23:24:11 on dist-test-slave-drl0"
I20260809 23:24:11.623782 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:11.647982 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:11.649158 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:11.650732 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:11.653009 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:11.653184 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.653481 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:11.653630 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:11.704638 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:33703
I20260809 23:24:11.704917  9211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:33703 every 8 connection(s)
I20260809 23:24:11.723786  9212 heartbeater.cc:347] Connected to a master server at 127.21.255.62:41225
I20260809 23:24:11.724460  9212 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:11.725610  9212 heartbeater.cc:511] Master 127.21.255.62:41225 requested a full tablet report, sending...
I20260809 23:24:11.727864  8834 ts_manager.cc:194] Registered new tserver with Master: 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:11.728969 22524 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020304798s
I20260809 23:24:11.729465  8834 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36480
I20260809 23:24:11.731741 22524 test_util.cc:268] Using random seed: -1996347639
I20260809 23:24:11.762247  8834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36496:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:24:11.765122  8834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:24:11.851908  8966 tablet_service.cc:1467] Processing CreateTablet for tablet 92a11e232e63467d81e6958e7d1335dc (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:11.852391  8964 tablet_service.cc:1467] Processing CreateTablet for tablet afdc11e9bcc643a9ad27e9ec2ee34cff (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:11.853451  8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a11e232e63467d81e6958e7d1335dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.854264  8965 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab99868842422583b64aa796efc47f (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:11.854660  8963 tablet_service.cc:1467] Processing CreateTablet for tablet 65773d68820e48a1a6ba7970141ed22c (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:11.855640  8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab99868842422583b64aa796efc47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.856258  9163 tablet_service.cc:1467] Processing CreateTablet for tablet 65773d68820e48a1a6ba7970141ed22c (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:11.857024  8962 tablet_service.cc:1467] Processing CreateTablet for tablet 745c983bc5294452934b5f19db51906e (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:11.857828  9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65773d68820e48a1a6ba7970141ed22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.858613  8962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c983bc5294452934b5f19db51906e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.860108  8961 tablet_service.cc:1467] Processing CreateTablet for tablet a5a1382dea7240848a159b9a6b6e82ab (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:11.860286  9165 tablet_service.cc:1467] Processing CreateTablet for tablet afdc11e9bcc643a9ad27e9ec2ee34cff (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:11.861636  9165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc11e9bcc643a9ad27e9ec2ee34cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.862568  8960 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc5a93704de4fafad83971c539f2427 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:11.864151  9166 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab99868842422583b64aa796efc47f (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:11.864203  9162 tablet_service.cc:1467] Processing CreateTablet for tablet 745c983bc5294452934b5f19db51906e (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:11.866214  9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab99868842422583b64aa796efc47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.867007  9169 tablet_service.cc:1467] Processing CreateTablet for tablet 92a11e232e63467d81e6958e7d1335dc (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:11.867645  9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c983bc5294452934b5f19db51906e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.869715  8959 tablet_service.cc:1467] Processing CreateTablet for tablet 58d56d427cdd455c997502702b5c752b (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:11.872195  8957 tablet_service.cc:1467] Processing CreateTablet for tablet b062a667501b46ceae2ac26b45ee1149 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:11.873931  8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d56d427cdd455c997502702b5c752b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.874593  8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc5a93704de4fafad83971c539f2427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.875444  9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a11e232e63467d81e6958e7d1335dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.879428  9161 tablet_service.cc:1467] Processing CreateTablet for tablet a5a1382dea7240848a159b9a6b6e82ab (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:11.875480  8963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65773d68820e48a1a6ba7970141ed22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.883709  9157 tablet_service.cc:1467] Processing CreateTablet for tablet b062a667501b46ceae2ac26b45ee1149 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:11.886533  9158 tablet_service.cc:1467] Processing CreateTablet for tablet 58d56d427cdd455c997502702b5c752b (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:11.888870  9160 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc5a93704de4fafad83971c539f2427 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:11.891157  9161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a1382dea7240848a159b9a6b6e82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.895354  9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc5a93704de4fafad83971c539f2427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.899658  9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b062a667501b46ceae2ac26b45ee1149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.903499  9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d56d427cdd455c997502702b5c752b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.912901  8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b062a667501b46ceae2ac26b45ee1149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.915359  8964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc11e9bcc643a9ad27e9ec2ee34cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.921411  8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a1382dea7240848a159b9a6b6e82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.947207  9067 tablet_service.cc:1467] Processing CreateTablet for tablet 92a11e232e63467d81e6958e7d1335dc (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:11.948954  9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a11e232e63467d81e6958e7d1335dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.952195  9066 tablet_service.cc:1467] Processing CreateTablet for tablet 4eab99868842422583b64aa796efc47f (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:11.953598  9066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eab99868842422583b64aa796efc47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.956244  9065 tablet_service.cc:1467] Processing CreateTablet for tablet afdc11e9bcc643a9ad27e9ec2ee34cff (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:11.957648  9065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afdc11e9bcc643a9ad27e9ec2ee34cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.960378  9063 tablet_service.cc:1467] Processing CreateTablet for tablet 65773d68820e48a1a6ba7970141ed22c (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:11.962401  9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65773d68820e48a1a6ba7970141ed22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.966216  9061 tablet_service.cc:1467] Processing CreateTablet for tablet a5a1382dea7240848a159b9a6b6e82ab (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:11.967840  9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5a1382dea7240848a159b9a6b6e82ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.974236  9062 tablet_service.cc:1467] Processing CreateTablet for tablet 745c983bc5294452934b5f19db51906e (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:11.975761  9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c983bc5294452934b5f19db51906e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.981580  9060 tablet_service.cc:1467] Processing CreateTablet for tablet 4bc5a93704de4fafad83971c539f2427 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:11.982999  9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bc5a93704de4fafad83971c539f2427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.987753  9058 tablet_service.cc:1467] Processing CreateTablet for tablet b062a667501b46ceae2ac26b45ee1149 (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:11.989243  9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b062a667501b46ceae2ac26b45ee1149. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:11.999728  9059 tablet_service.cc:1467] Processing CreateTablet for tablet 58d56d427cdd455c997502702b5c752b (DEFAULT_TABLE table=test-workload [id=52050b38b50547efbd1ae02cb1d53b34]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:12.001663  9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d56d427cdd455c997502702b5c752b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:12.011444  9230 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.027547  9229 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.043113  9230 tablet_bootstrap.cc:654] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.100054  9229 tablet_bootstrap.cc:654] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.106820  9233 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.108580  9230 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.109200  9230 ts_tablet_manager.cc:1397] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.098s	user 0.023s	sys 0.015s
I20260809 23:24:12.112677  9230 raft_consensus.cc:348] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.115139  9230 raft_consensus.cc:374] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.115460  9230 raft_consensus.cc:729] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.116629  9233 tablet_bootstrap.cc:654] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.116901  9230 consensus_queue.cc:260] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.117576  9229 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.117960  9235 raft_consensus.cc:482] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.118171  9229 ts_tablet_manager.cc:1397] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.091s	user 0.016s	sys 0.000s
I20260809 23:24:12.118603  9235 raft_consensus.cc:504] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.121619  9229 raft_consensus.cc:348] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.122509  9229 raft_consensus.cc:374] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.122831  9229 raft_consensus.cc:729] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.123780  9229 consensus_queue.cc:260] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.126911  9230 ts_tablet_manager.cc:1428] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.017s	user 0.002s	sys 0.012s
I20260809 23:24:12.128005  9212 heartbeater.cc:503] Master 127.21.255.62:41225 was elected leader, sending a full tablet report...
I20260809 23:24:12.128261  9230 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.133375  9235 leader_election.cc:288] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.135154  9230 tablet_bootstrap.cc:654] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.144495  9233 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.145141  9233 ts_tablet_manager.cc:1397] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.039s	user 0.021s	sys 0.005s
I20260809 23:24:12.149880  9233 raft_consensus.cc:348] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.150911  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.151401  9233 raft_consensus.cc:374] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.151878  8978 raft_consensus.cc:2454] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.151913  9233 raft_consensus.cc:729] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.153465  9134 leader_election.cc:302] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.154479  9235 raft_consensus.cc:2781] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.153461  9233 consensus_queue.cc:260] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.154888  9235 raft_consensus.cc:482] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.155370  9235 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.160799  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
W20260809 23:24:12.163002  9132 leader_election.cc:341] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.163475  9235 raft_consensus.cc:504] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.165360  9235 leader_election.cc:288] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.166990  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
W20260809 23:24:12.168526  9132 leader_election.cc:341] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.169479  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.170161  8978 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.171664  9233 ts_tablet_manager.cc:1428] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.026s	user 0.012s	sys 0.008s
I20260809 23:24:12.174319  9116 heartbeater.cc:503] Master 127.21.255.62:41225 was elected leader, sending a full tablet report...
I20260809 23:24:12.175529  9011 heartbeater.cc:503] Master 127.21.255.62:41225 was elected leader, sending a full tablet report...
I20260809 23:24:12.179899  9229 ts_tablet_manager.cc:1428] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.061s	user 0.010s	sys 0.000s
I20260809 23:24:12.180742  8978 raft_consensus.cc:2454] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:12.181206  9229 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.187142  9134 leader_election.cc:302] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:12.187474  9233 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.191241  9237 raft_consensus.cc:2781] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.191797  9237 raft_consensus.cc:686] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:12.192780  9237 consensus_queue.cc:237] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.196928  9233 tablet_bootstrap.cc:654] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.203927  9229 tablet_bootstrap.cc:654] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.208042  9230 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.208606  9230 ts_tablet_manager.cc:1397] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.081s	user 0.041s	sys 0.037s
W20260809 23:24:12.209607  9213 tablet.cc:2369] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 23:24:12.210371  9233 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.210894  9233 ts_tablet_manager.cc:1397] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.025s	user 0.017s	sys 0.005s
I20260809 23:24:12.211615  9230 raft_consensus.cc:348] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.212427  9230 raft_consensus.cc:374] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.212713  9230 raft_consensus.cc:729] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.213554  9230 consensus_queue.cc:260] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.213773  9233 raft_consensus.cc:348] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.214478  9233 raft_consensus.cc:374] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.214790  9233 raft_consensus.cc:729] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.215735  9233 consensus_queue.cc:260] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.216852  9230 ts_tablet_manager.cc:1428] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.003s
I20260809 23:24:12.217864  9230 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.218181  9233 ts_tablet_manager.cc:1428] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:24:12.219197  9233 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.224282  9230 tablet_bootstrap.cc:654] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.225407  9233 tablet_bootstrap.cc:654] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.225435  8834 catalog_manager.cc:5495] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.230005  9229 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.230670  9229 ts_tablet_manager.cc:1397] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.050s	user 0.004s	sys 0.026s
I20260809 23:24:12.237643  9229 raft_consensus.cc:348] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.238919  9229 raft_consensus.cc:374] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.239329  9229 raft_consensus.cc:729] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.240391  9229 consensus_queue.cc:260] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.243758  9229 ts_tablet_manager.cc:1428] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.002s
I20260809 23:24:12.244805  9229 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.248234  9233 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.249459  9233 ts_tablet_manager.cc:1397] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.011s
I20260809 23:24:12.249931  9230 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.250483  9230 ts_tablet_manager.cc:1397] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.006s
I20260809 23:24:12.251296  9229 tablet_bootstrap.cc:654] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.259298  9233 raft_consensus.cc:348] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.260372  9233 raft_consensus.cc:374] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.260054  9230 raft_consensus.cc:348] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.260741  9233 raft_consensus.cc:729] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.261144  9230 raft_consensus.cc:374] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.261631  9230 raft_consensus.cc:729] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.261740  9233 consensus_queue.cc:260] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.262882  9230 consensus_queue.cc:260] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.264503  9233 ts_tablet_manager.cc:1428] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.015s	user 0.000s	sys 0.007s
I20260809 23:24:12.265622  9233 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.266263  9230 ts_tablet_manager.cc:1428] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.008s
I20260809 23:24:12.267556  9230 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.269723  9229 tablet_bootstrap.cc:492] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.270256  9229 ts_tablet_manager.cc:1397] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.004s
I20260809 23:24:12.273576  9233 tablet_bootstrap.cc:654] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.273268  9229 raft_consensus.cc:348] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.276700  9229 raft_consensus.cc:374] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.277077  9229 raft_consensus.cc:729] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.273757  9230 tablet_bootstrap.cc:654] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.278137  9229 consensus_queue.cc:260] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.280627  9229 ts_tablet_manager.cc:1428] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.006s
I20260809 23:24:12.281733  9229 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
W20260809 23:24:12.283211  8900 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:12.290591  9229 tablet_bootstrap.cc:654] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.306497  9233 tablet_bootstrap.cc:492] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.307233  9233 ts_tablet_manager.cc:1397] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.015s
I20260809 23:24:12.308171  9230 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.308955  9230 ts_tablet_manager.cc:1397] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.042s	user 0.019s	sys 0.016s
I20260809 23:24:12.309418  9233 raft_consensus.cc:348] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.310052  9233 raft_consensus.cc:374] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.310309  9233 raft_consensus.cc:729] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.310889  9233 consensus_queue.cc:260] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.312716  9233 ts_tablet_manager.cc:1428] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260809 23:24:12.313504  9230 raft_consensus.cc:348] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.314188  9233 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.314304  9230 raft_consensus.cc:374] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.315012  9230 raft_consensus.cc:729] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.315398  9229 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.315932  9229 ts_tablet_manager.cc:1397] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.034s	user 0.009s	sys 0.021s
I20260809 23:24:12.315857  9230 consensus_queue.cc:260] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.318918  9229 raft_consensus.cc:348] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.319671  9229 raft_consensus.cc:374] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.319947  9229 raft_consensus.cc:729] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.320631  9229 consensus_queue.cc:260] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.321179  9230 ts_tablet_manager.cc:1428] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260809 23:24:12.321856  9233 tablet_bootstrap.cc:654] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.322218  9230 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.322816  9229 ts_tablet_manager.cc:1428] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260809 23:24:12.324223  9229 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.329541  9230 tablet_bootstrap.cc:654] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.330154  9229 tablet_bootstrap.cc:654] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.345278  9233 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.345846  9233 ts_tablet_manager.cc:1397] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.032s	user 0.017s	sys 0.012s
I20260809 23:24:12.349354  9229 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.349239  9233 raft_consensus.cc:348] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.350118  9229 ts_tablet_manager.cc:1397] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.004s
I20260809 23:24:12.350560  9233 raft_consensus.cc:374] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.351650  9233 raft_consensus.cc:729] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.352612  9233 consensus_queue.cc:260] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.354732  9229 raft_consensus.cc:348] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.355393  9230 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.355460  9233 ts_tablet_manager.cc:1428] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.004s
I20260809 23:24:12.355726  9229 raft_consensus.cc:374] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.356031  9230 ts_tablet_manager.cc:1397] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.034s	user 0.026s	sys 0.004s
I20260809 23:24:12.356102  9229 raft_consensus.cc:729] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.356716  9233 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.357128  9229 consensus_queue.cc:260] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.359475  9229 ts_tablet_manager.cc:1428] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.002s
I20260809 23:24:12.359375  9230 raft_consensus.cc:348] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.360177  9230 raft_consensus.cc:374] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.360491  9229 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.360458  9230 raft_consensus.cc:729] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.361305  9230 consensus_queue.cc:260] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.363686  9233 tablet_bootstrap.cc:654] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.363857  9230 ts_tablet_manager.cc:1428] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260809 23:24:12.364948  9230 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.367107  9229 tablet_bootstrap.cc:654] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.371421  9230 tablet_bootstrap.cc:654] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.379379  9229 tablet_bootstrap.cc:492] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.380018  9229 ts_tablet_manager.cc:1397] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260809 23:24:12.383599  9229 raft_consensus.cc:348] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.384596  9229 raft_consensus.cc:374] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.384934  9229 raft_consensus.cc:729] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.387276  9229 consensus_queue.cc:260] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.388882  9230 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.389441  9230 ts_tablet_manager.cc:1397] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.008s
I20260809 23:24:12.389698  9229 ts_tablet_manager.cc:1428] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.004s
I20260809 23:24:12.390360  9246 raft_consensus.cc:482] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.390786  9229 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.391086  9246 raft_consensus.cc:504] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.392637  9230 raft_consensus.cc:348] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.393392  9230 raft_consensus.cc:374] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.393672  9230 raft_consensus.cc:729] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.394563  9230 consensus_queue.cc:260] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.397130  9230 ts_tablet_manager.cc:1428] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260809 23:24:12.397368  9229 tablet_bootstrap.cc:654] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.398134  9230 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.400295  9237 raft_consensus.cc:482] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.400969  9237 raft_consensus.cc:504] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.403611  9242 raft_consensus.cc:482] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.404265  9242 raft_consensus.cc:504] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.406291  9230 tablet_bootstrap.cc:654] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.407637  9246 leader_election.cc:288] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.410862  9242 leader_election.cc:288] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.415632  9237 leader_election.cc:288] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.435986  9246 raft_consensus.cc:482] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.436576  9246 raft_consensus.cc:504] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.438855  9246 leader_election.cc:288] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.441238  9233 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.441840  9233 ts_tablet_manager.cc:1397] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.085s	user 0.027s	sys 0.051s
I20260809 23:24:12.442827  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.444959  9083 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.445128  9233 raft_consensus.cc:348] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.445919  9233 raft_consensus.cc:374] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.446209  9233 raft_consensus.cc:729] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.446689  9132 leader_election.cc:302] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:12.447032  9233 consensus_queue.cc:260] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.447793  9237 raft_consensus.cc:2781] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.448153  9237 raft_consensus.cc:482] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.448474  9237 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.452903  9233 ts_tablet_manager.cc:1428] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:24:12.453939  9233 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.459998  9233 tablet_bootstrap.cc:654] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.464004  9242 raft_consensus.cc:482] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.464519  9242 raft_consensus.cc:504] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.466576  9242 leader_election.cc:288] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.467351  9237 raft_consensus.cc:504] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.478988  9229 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.479707  9229 ts_tablet_manager.cc:1397] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.089s	user 0.041s	sys 0.036s
I20260809 23:24:12.483223  9229 raft_consensus.cc:348] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.484197  9229 raft_consensus.cc:374] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.484550  9229 raft_consensus.cc:729] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.485939  9233 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.485481  9229 consensus_queue.cc:260] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.486581  9233 ts_tablet_manager.cc:1397] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.007s
I20260809 23:24:12.489164  9229 ts_tablet_manager.cc:1428] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:24:12.489779  9233 raft_consensus.cc:348] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.490581  9233 raft_consensus.cc:374] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.490871  9233 raft_consensus.cc:729] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.491287  9229 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.491782  9233 consensus_queue.cc:260] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.496379  9237 leader_election.cc:288] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.497406  9230 tablet_bootstrap.cc:492] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.497915  9230 ts_tablet_manager.cc:1397] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.100s	user 0.018s	sys 0.024s
I20260809 23:24:12.501106  9233 ts_tablet_manager.cc:1428] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.014s	user 0.001s	sys 0.005s
I20260809 23:24:12.504683  9230 raft_consensus.cc:348] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.505487  9233 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.508060  9237 raft_consensus.cc:482] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.508688  9242 raft_consensus.cc:482] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.511474  9242 raft_consensus.cc:504] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.511994  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.512812  9083 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.513845  9242 leader_election.cc:288] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.510890  9229 tablet_bootstrap.cc:654] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.519883  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.520249  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.514768  9233 tablet_bootstrap.cc:654] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.521087  9183 raft_consensus.cc:2379] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 19446675307f4e709bb1c63b9c529e54 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
W20260809 23:24:12.522097  9030 leader_election.cc:341] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.508657  9237 raft_consensus.cc:504] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.522456  9083 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:12.523682  9132 leader_election.cc:302] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:12.524766  9235 raft_consensus.cc:2781] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.525681  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.526552  8978 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.526813  8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.527602  8976 raft_consensus.cc:2454] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.528172  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.528746  8975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.528961  8977 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.529639  8975 raft_consensus.cc:2454] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:12.530439  9134 leader_election.cc:302] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.530771  9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.531731  9080 raft_consensus.cc:2379] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:12.532481  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.532660  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.532807  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.533914  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
W20260809 23:24:12.534060  9132 leader_election.cc:341] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.505494  9230 raft_consensus.cc:374] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.534814  9237 leader_election.cc:288] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.535272  9230 raft_consensus.cc:729] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.536489  9230 consensus_queue.cc:260] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.537177  9032 leader_election.cc:302] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.539120  9246 raft_consensus.cc:2781] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.538676  9032 leader_election.cc:302] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54; no voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.539376  9230 ts_tablet_manager.cc:1428] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.041s	user 0.010s	sys 0.000s
I20260809 23:24:12.539532  9246 raft_consensus.cc:482] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.541083  9246 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.533205  9083 raft_consensus.cc:2454] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.542294  9230 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.544622  9229 tablet_bootstrap.cc:492] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.545172  9229 ts_tablet_manager.cc:1397] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.054s	user 0.011s	sys 0.003s
I20260809 23:24:12.548143  9229 raft_consensus.cc:348] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.548918  9229 raft_consensus.cc:374] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.549284  9229 raft_consensus.cc:729] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.550002  9229 consensus_queue.cc:260] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.552518  9246 raft_consensus.cc:504] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.557267  9237 raft_consensus.cc:2781] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.559546  9246 leader_election.cc:288] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.560111  9246 raft_consensus.cc:2726] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260809 23:24:12.537456  8930 leader_election.cc:341] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.561175  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.561852  8930 leader_election.cc:302] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.557706  9237 raft_consensus.cc:482] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:24:12.562474  9030 leader_election.cc:341] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:12.562922  9237 raft_consensus.cc:3037] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.564376  9230 tablet_bootstrap.cc:654] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.567265  8977 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:12.568032  9242 raft_consensus.cc:2781] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.568647  9242 raft_consensus.cc:482] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.569304  9242 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.582790  9291 raft_consensus.cc:482] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.583557  9291 raft_consensus.cc:504] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.585100  9230 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.586117  9230 ts_tablet_manager.cc:1397] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.044s	user 0.020s	sys 0.000s
I20260809 23:24:12.586779  9235 raft_consensus.cc:686] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:12.587682  9286 raft_consensus.cc:482] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.588225  9286 raft_consensus.cc:504] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.588049  9235 consensus_queue.cc:237] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.591259  9230 raft_consensus.cc:348] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.594280  9230 raft_consensus.cc:374] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.594990  9230 raft_consensus.cc:729] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.597831  9286 leader_election.cc:288] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.598475  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.599352  9184 raft_consensus.cc:2454] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.618808  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.619652  8978 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:12.622654  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.623224  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.623908  9183 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:12.622232  9230 consensus_queue.cc:260] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.627444  9230 ts_tablet_manager.cc:1428] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.040s	user 0.015s	sys 0.005s
I20260809 23:24:12.627590  9032 leader_election.cc:302] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.629500  9286 raft_consensus.cc:2781] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.629314  9242 raft_consensus.cc:504] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.629952  9286 raft_consensus.cc:482] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260809 23:24:12.628302  9134 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:32919, user_credentials={real_user=slave}} blocked reactor thread for 97947.4us
I20260809 23:24:12.630447  9286 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.633068  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.634225  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.623389  8977 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.639793  9286 raft_consensus.cc:504] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.630048  9230 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:12.643488  9291 leader_election.cc:288] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.646204  9242 leader_election.cc:288] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.648104  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.648167  9237 raft_consensus.cc:504] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.648276  9233 tablet_bootstrap.cc:492] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.649166  9083 raft_consensus.cc:2379] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 19446675307f4e709bb1c63b9c529e54 in this term.
I20260809 23:24:12.649447  9183 raft_consensus.cc:2379] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:12.650029  9233 ts_tablet_manager.cc:1397] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.145s	user 0.032s	sys 0.043s
I20260809 23:24:12.652905  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.653554  9083 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.654574  9233 raft_consensus.cc:348] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.655754  9233 raft_consensus.cc:374] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.656147  9233 raft_consensus.cc:729] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.657199  9233 consensus_queue.cc:260] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.658623  8834 catalog_manager.cc:5495] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:12.644169  9291 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.661767  9083 raft_consensus.cc:2454] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:12.661931  9291 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.663996  9291 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.663931  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.664595  8978 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.666450  8930 leader_election.cc:302] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.668931  9230 tablet_bootstrap.cc:654] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.671075  8930 leader_election.cc:302] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.671111  9235 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.672909  9235 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.673885  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.675621  9184 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.676527  8978 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:12.677767  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.678458  9183 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.678505  8977 raft_consensus.cc:2454] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:12.680680  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.688975  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.682436  9287 raft_consensus.cc:2726] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.689519  9182 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.683246  9229 ts_tablet_manager.cc:1428] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.138s	user 0.045s	sys 0.032s
I20260809 23:24:12.689878  9078 raft_consensus.cc:3037] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.683681  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.690595  9083 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.685397  9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.694053  9229 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:12.694509  9181 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.686794  8930 leader_election.cc:302] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.696153  9032 leader_election.cc:302] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.697310  9246 raft_consensus.cc:2781] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.686832  9284 consensus_queue.cc:1035] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:12.687342  9235 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.681282  9286 leader_election.cc:288] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.687016  9242 raft_consensus.cc:3037] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.711127  9287 raft_consensus.cc:2781] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.711809  9032 leader_election.cc:302] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.712855  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b062a667501b46ceae2ac26b45ee1149" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.687922  9237 leader_election.cc:288] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.688982  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.714951  9078 raft_consensus.cc:2454] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:12.715555  9077 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.716104  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.716823  8978 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.718142  9296 raft_consensus.cc:2781] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.718649  9296 raft_consensus.cc:686] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:12.720323  9246 raft_consensus.cc:686] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:12.720678  9296 consensus_queue.cc:237] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.721603  9246 consensus_queue.cc:237] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.723376  8930 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.707245  9229 tablet_bootstrap.cc:654] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.725667  9291 raft_consensus.cc:2781] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.726197  9291 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.726528  9291 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.727161  9237 raft_consensus.cc:482] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.727674  9237 raft_consensus.cc:504] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.686829  9233 ts_tablet_manager.cc:1428] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260809 23:24:12.729465  9233 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:12.732210  9132 leader_election.cc:302] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:12.725634  9242 raft_consensus.cc:2726] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.732878  9287 raft_consensus.cc:686] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:12.733989  9182 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:12.735567  9284 raft_consensus.cc:2781] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.738627  9284 raft_consensus.cc:686] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:12.739480  9132 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:12.739780  9284 consensus_queue.cc:237] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.741926  9282 raft_consensus.cc:2781] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.744577  9282 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.745074  9282 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.730427  9237 leader_election.cc:288] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.734202  9287 consensus_queue.cc:237] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.742533  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.748368  9077 raft_consensus.cc:2454] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.752610  9184 raft_consensus.cc:2454] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:12.756096  9229 tablet_bootstrap.cc:492] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:12.756675  9229 ts_tablet_manager.cc:1397] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.063s	user 0.021s	sys 0.009s
I20260809 23:24:12.757607  8977 raft_consensus.cc:2454] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:12.759675  9229 raft_consensus.cc:348] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.760504  9229 raft_consensus.cc:374] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.760812  9229 raft_consensus.cc:729] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.761654  9229 consensus_queue.cc:260] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.761134  8832 catalog_manager.cc:5495] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.763976  9291 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.767287  9287 raft_consensus.cc:482] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.767802  9287 raft_consensus.cc:504] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.770402  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.771075  8978 raft_consensus.cc:2454] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:12.772611  9134 leader_election.cc:302] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.773525  9284 raft_consensus.cc:2781] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.773895  9284 raft_consensus.cc:482] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.775677  9284 raft_consensus.cc:3037] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.777920  9229 ts_tablet_manager.cc:1428] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260809 23:24:12.742758  9233 tablet_bootstrap.cc:654] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:12.780104  9246 raft_consensus.cc:482] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.780616  9246 raft_consensus.cc:504] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.782642  9230 tablet_bootstrap.cc:492] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:12.783187  9230 ts_tablet_manager.cc:1397] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.153s	user 0.027s	sys 0.012s
I20260809 23:24:12.783963  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.784662  9184 raft_consensus.cc:2454] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:12.786193  9246 leader_election.cc:288] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.785745  9230 raft_consensus.cc:348] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.786829  9230 raft_consensus.cc:374] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.787196  9230 raft_consensus.cc:729] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.788901  9230 consensus_queue.cc:260] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.790743  8832 catalog_manager.cc:5495] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:12.811007  9233 tablet_bootstrap.cc:492] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:12.797263  9284 raft_consensus.cc:504] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.811573  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.814682  9233 ts_tablet_manager.cc:1397] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.000s
I20260809 23:24:12.810395  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.798281  8833 catalog_manager.cc:5495] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.815148  9078 raft_consensus.cc:2454] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.815497  9077 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.816702  8930 leader_election.cc:302] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.809572  9291 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.817695  9301 raft_consensus.cc:2781] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.817986  9291 raft_consensus.cc:482] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.818157  9301 raft_consensus.cc:482] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.818578  9301 raft_consensus.cc:3037] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.818517  9291 raft_consensus.cc:504] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.820672  9291 leader_election.cc:288] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
W20260809 23:24:12.798689  9132 consensus_peers.cc:487] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:12.800259  9230 ts_tablet_manager.cc:1428] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260809 23:24:12.825158  8930 leader_election.cc:302] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:12.799674  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.826360  8978 raft_consensus.cc:2454] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:12.792486  9282 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.827131  9301 raft_consensus.cc:504] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.828101  9030 leader_election.cc:302] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:12.828931  9282 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.829022  9246 raft_consensus.cc:2781] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.829524  9246 raft_consensus.cc:482] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.830034  9246 raft_consensus.cc:3037] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.832289  9287 leader_election.cc:288] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.834316  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.835172  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.835319  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.835832  9077 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:12.836205  9183 raft_consensus.cc:2379] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:12.833009  9287 raft_consensus.cc:2741] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 23:24:12.837146  9083 raft_consensus.cc:2379] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d0a6b0fd63d4091a84bb1215772b32d in current term 1: Already voted for candidate f6964fd5f558447d99495c9e69df90c1 in this term.
I20260809 23:24:12.839736  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.840229  9301 leader_election.cc:288] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.817632  9233 raft_consensus.cc:348] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.841537  9233 raft_consensus.cc:374] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:12.841952  9233 raft_consensus.cc:729] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:12.842810  9233 consensus_queue.cc:260] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.843104  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.843766  9182 raft_consensus.cc:2454] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:12.846262  9233 ts_tablet_manager.cc:1428] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.000s
I20260809 23:24:12.848943  8930 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.850790  9301 raft_consensus.cc:2781] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.851051  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.852166  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.852775  9083 raft_consensus.cc:3037] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.856137  9184 raft_consensus.cc:2379] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:12.856679  8930 leader_election.cc:302] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:12.859172  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.861505  9284 leader_election.cc:288] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.862480  9291 raft_consensus.cc:2781] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:12.863214  9291 raft_consensus.cc:482] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:12.864881  9291 raft_consensus.cc:3037] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.864769  9246 raft_consensus.cc:504] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.866140  9078 raft_consensus.cc:2379] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 19446675307f4e709bb1c63b9c529e54 in this term.
I20260809 23:24:12.868780  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.871371  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afdc11e9bcc643a9ad27e9ec2ee34cff" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.872218  8977 raft_consensus.cc:2379] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d0a6b0fd63d4091a84bb1215772b32d in current term 1: Already voted for candidate f6964fd5f558447d99495c9e69df90c1 in this term.
I20260809 23:24:12.873344  9291 raft_consensus.cc:504] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.874539  9246 leader_election.cc:288] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.875567  9291 leader_election.cc:288] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.851306  9301 raft_consensus.cc:686] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:12.878517  9301 consensus_queue.cc:237] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.881083  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:12.882032  9077 raft_consensus.cc:2379] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 19446675307f4e709bb1c63b9c529e54 in this term.
I20260809 23:24:12.881057  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:12.883708  8977 raft_consensus.cc:2379] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 19446675307f4e709bb1c63b9c529e54 in current term 1: Already voted for candidate f6964fd5f558447d99495c9e69df90c1 in this term.
I20260809 23:24:12.883019  8834 catalog_manager.cc:5495] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.884198  9132 leader_election.cc:302] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.884271  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.884600  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:12.885022  9182 raft_consensus.cc:3037] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.886878  9284 raft_consensus.cc:2726] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.888404  8930 leader_election.cc:302] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.889513  9291 raft_consensus.cc:2726] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.889731  9083 raft_consensus.cc:2454] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:12.890985  8930 leader_election.cc:302] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:12.891705  9291 raft_consensus.cc:2781] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.892320  9291 raft_consensus.cc:686] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:12.893639  9291 consensus_queue.cc:237] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.896453  9182 raft_consensus.cc:2454] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:12.897683  9030 leader_election.cc:302] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.899269  9246 raft_consensus.cc:2781] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:12.900004  9246 raft_consensus.cc:686] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:12.900993  9246 consensus_queue.cc:237] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.914597  9134 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.916061  9284 raft_consensus.cc:2726] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.916826  8832 catalog_manager.cc:5495] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.919369  9284 raft_consensus.cc:482] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.919878  9284 raft_consensus.cc:504] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:12.920895  8833 catalog_manager.cc:5495] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.923136  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:12.923588  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.924422  8978 raft_consensus.cc:2379] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d0a6b0fd63d4091a84bb1215772b32d in current term 1: Already voted for candidate 19446675307f4e709bb1c63b9c529e54 in this term.
I20260809 23:24:12.926045  9134 leader_election.cc:302] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.926834  9282 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:12.926916  9284 leader_election.cc:288] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:12.935503  9282 raft_consensus.cc:2726] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:12.951556  8833 catalog_manager.cc:5495] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:12.955765  9246 raft_consensus.cc:482] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:12.956297  9246 raft_consensus.cc:504] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:12.958010  9246 leader_election.cc:288] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:12.958882  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:12.959718  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:12.961342  9030 leader_election.cc:302] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54; no voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:12.962093  9246 raft_consensus.cc:2726] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:13.010353  9282 consensus_queue.cc:1035] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.024653  9235 consensus_queue.cc:1035] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.047015  9182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:44888
I20260809 23:24:13.047736  9182 raft_consensus.cc:595] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:13.065486  9083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eab99868842422583b64aa796efc47f"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:36676
I20260809 23:24:13.066201  9083 raft_consensus.cc:595] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:13.075953  9083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:36676
I20260809 23:24:13.076534  9083 raft_consensus.cc:595] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:13.082003  9182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "745c983bc5294452934b5f19db51906e"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
mode: GRACEFUL
new_leader_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:44888
I20260809 23:24:13.082549  9182 raft_consensus.cc:595] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Received request to transfer leadership to TS 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:13.104385  8978 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
mode: GRACEFUL
new_leader_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:60484
I20260809 23:24:13.105057  8978 raft_consensus.cc:595] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Received request to transfer leadership to TS 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:13.111593  8978 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58d56d427cdd455c997502702b5c752b"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
mode: GRACEFUL
new_leader_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:60484
I20260809 23:24:13.112349  8978 raft_consensus.cc:595] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Received request to transfer leadership to TS 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:13.141600  9237 consensus_queue.cc:1035] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.155364  9237 consensus_queue.cc:1035] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.214306  9246 consensus_queue.cc:1035] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.228253  9329 consensus_queue.cc:1035] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.237035  9246 raft_consensus.cc:979] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.237884  9329 raft_consensus.cc:1067] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.239686  8978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60470
I20260809 23:24:13.240511  8978 raft_consensus.cc:482] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.240921  8978 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.247319  8978 raft_consensus.cc:504] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.249111  8978 leader_election.cc:288] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:13.249933  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:13.250488  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.250746  9083 raft_consensus.cc:3032] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.251139  9182 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.251235  9083 raft_consensus.cc:729] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:13.252182  9083 consensus_queue.cc:260] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.253567  9083 raft_consensus.cc:3037] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.258383  9301 consensus_queue.cc:1035] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.260356  9083 raft_consensus.cc:2454] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:13.261511  8930 leader_election.cc:302] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:13.262569  9301 raft_consensus.cc:2781] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.262715  9182 raft_consensus.cc:2454] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:13.265889  9301 raft_consensus.cc:686] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:13.267380  9301 consensus_queue.cc:237] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.272068  9329 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.274102  9291 raft_consensus.cc:979] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: : Instructing follower 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.274624  9291 raft_consensus.cc:1067] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Signalling peer 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.277104  9237 raft_consensus.cc:979] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.277572  9237 raft_consensus.cc:1067] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.277871  9077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:36666
I20260809 23:24:13.278478  9077 raft_consensus.cc:482] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.278847  9077 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.282334  8978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60458
I20260809 23:24:13.281371  8832 catalog_manager.cc:5495] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.282975  8978 raft_consensus.cc:482] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.283404  8978 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.284078  9301 consensus_queue.cc:1035] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.288415  9246 raft_consensus.cc:979] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.288993  9296 raft_consensus.cc:1067] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:13.294018  9077 raft_consensus.cc:504] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.296838  9077 leader_election.cc:288] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:13.297812  9083 raft_consensus.cc:1226] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Rejecting Update request from peer f6964fd5f558447d99495c9e69df90c1 for earlier term 1. Current term is 2. Ops: []
I20260809 23:24:13.302901  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.301069  9296 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.315980  8977 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eab99868842422583b64aa796efc47f"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60470
I20260809 23:24:13.317392  8977 raft_consensus.cc:482] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.318022  8977 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.318642  8978 raft_consensus.cc:504] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:13.320865  8978 leader_election.cc:288] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:13.321789  9083 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:13.325356  9301 consensus_queue.cc:1046] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.326321  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:13.327278  9242 raft_consensus.cc:3032] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.327744  9242 raft_consensus.cc:729] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:13.328641  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a11e232e63467d81e6958e7d1335dc" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.329298  9183 raft_consensus.cc:3032] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.328927  9242 consensus_queue.cc:260] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.329681  9183 raft_consensus.cc:729] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:13.331106  9183 consensus_queue.cc:260] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:13.333220  9183 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.335405  9242 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.339954  8977 raft_consensus.cc:504] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.341316  9183 raft_consensus.cc:2454] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:13.343032  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bc5a93704de4fafad83971c539f2427" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:13.342599  8977 leader_election.cc:288] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:13.344091  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:13.344762  9077 raft_consensus.cc:3032] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.345111  9077 raft_consensus.cc:729] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:13.345911  9077 consensus_queue.cc:260] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.347661  9077 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.355855  8930 leader_election.cc:302] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:13.356777  9242 raft_consensus.cc:2781] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.357257  9242 raft_consensus.cc:686] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:13.358296  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.358441  9242 consensus_queue.cc:237] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } }
I20260809 23:24:13.359253  9182 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.360472  8978 raft_consensus.cc:2454] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:13.364519  9077 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:13.366768  9182 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:13.368095  9032 leader_election.cc:302] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:13.369290  9296 raft_consensus.cc:2781] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.369024  8834 catalog_manager.cc:5495] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.369837  9296 raft_consensus.cc:686] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:13.370620  9296 consensus_queue.cc:237] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.371490  8930 leader_election.cc:302] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:13.372509  9242 raft_consensus.cc:2781] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.373303  9242 raft_consensus.cc:686] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:13.373528  9301 consensus_queue.cc:1035] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.374349  9242 consensus_queue.cc:237] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.381102  8833 catalog_manager.cc:5495] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 1 to 2, leader changed from f6964fd5f558447d99495c9e69df90c1 (127.21.255.1) to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.386068  8834 catalog_manager.cc:5495] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.393028  9287 consensus_queue.cc:1035] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.398018  9296 consensus_queue.cc:1035] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.419121  9286 consensus_queue.cc:1035] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:13.506865  9242 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:24:13.513614  9235 raft_consensus.cc:979] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: : Instructing follower 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.514442  9235 raft_consensus.cc:1067] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Signalling peer 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.518466  9083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "745c983bc5294452934b5f19db51906e"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:36660
I20260809 23:24:13.519732  9083 raft_consensus.cc:482] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.521250  9083 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.521166  9242 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:13.537322  9083 raft_consensus.cc:504] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.541287  9083 leader_election.cc:288] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:13.542192  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.543380  9183 raft_consensus.cc:3032] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.543743  9183 raft_consensus.cc:729] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:13.545256  9287 raft_consensus.cc:979] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: : Instructing follower 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.545001  9183 consensus_queue.cc:260] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.545830  9287 raft_consensus.cc:1067] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Signalling peer 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:13.547545  9183 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.552202  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c983bc5294452934b5f19db51906e" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:13.553130  8978 raft_consensus.cc:3037] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.558175  9077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58d56d427cdd455c997502702b5c752b"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:36666
I20260809 23:24:13.558992  9077 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:13.559429  9077 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.562520  9183 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:13.564284  9030 leader_election.cc:302] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:13.565393  9286 raft_consensus.cc:2781] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.565862  9286 raft_consensus.cc:686] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:13.567533  9286 consensus_queue.cc:237] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.568995  8978 raft_consensus.cc:2454] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:13.572628  9077 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.575148  9077 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:13.575861  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:13.576149  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:13.576874  9183 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.576521  8978 raft_consensus.cc:3032] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:13.577404  8978 raft_consensus.cc:729] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:13.576496  8832 catalog_manager.cc:5495] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 1 to 2, leader changed from 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3) to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.579298  8978 consensus_queue.cc:260] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.581303  8978 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.584098  9183 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:13.585814  9030 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:13.586948  9286 raft_consensus.cc:2781] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:13.587530  9286 raft_consensus.cc:686] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:13.588788  9286 consensus_queue.cc:237] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:13.591818  8978 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:13.597967  8832 catalog_manager.cc:5495] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 1 to 2, leader changed from f6964fd5f558447d99495c9e69df90c1 (127.21.255.1) to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:13.746805  9183 raft_consensus.cc:1261] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.748466  9287 consensus_queue.cc:1035] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:13.760066  9077 raft_consensus.cc:1261] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.762573  9287 consensus_queue.cc:1035] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:13.773242  9077 raft_consensus.cc:1261] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.775317  9291 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:13.788131  9183 raft_consensus.cc:1261] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.789516  9287 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:13.840801  9183 raft_consensus.cc:3037] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.847996  9183 raft_consensus.cc:1261] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.849444  9286 consensus_queue.cc:1035] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:13.863597  8978 raft_consensus.cc:1261] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.865293  9286 consensus_queue.cc:1035] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:13.873636  9077 raft_consensus.cc:3037] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:13.881456  9077 raft_consensus.cc:1261] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.882876  9287 consensus_queue.cc:1035] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:13.894708  9183 raft_consensus.cc:1261] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:13.896356  9242 consensus_queue.cc:1035] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:14.014011  9183 raft_consensus.cc:1261] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:14.015406  9246 consensus_queue.cc:1035] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:14.024837  8978 raft_consensus.cc:1261] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:14.026494  9296 consensus_queue.cc:1035] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:14.155580  9183 raft_consensus.cc:1261] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:14.157047  9296 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:14.167445  8978 raft_consensus.cc:1261] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:14.169358  9246 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:16.117913 22524 test_util.cc:268] Using random seed: -1991961572
I20260809 23:24:16.161536  8834 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50694:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:24:16.165216  8834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:24:16.228696  8961 tablet_service.cc:1467] Processing CreateTablet for tablet 3a98bf8f3830479b87e617c0103af4c8 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:16.229220  9067 tablet_service.cc:1467] Processing CreateTablet for tablet 3a98bf8f3830479b87e617c0103af4c8 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:16.230180  9163 tablet_service.cc:1467] Processing CreateTablet for tablet 3a98bf8f3830479b87e617c0103af4c8 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:16.230808  8961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a98bf8f3830479b87e617c0103af4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.230957  9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a98bf8f3830479b87e617c0103af4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.231765  9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a98bf8f3830479b87e617c0103af4c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.239598  8959 tablet_service.cc:1467] Processing CreateTablet for tablet 7f00e6101a1647a4acd59b961be1c4ef (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:16.240141  8964 tablet_service.cc:1467] Processing CreateTablet for tablet 01ea2a0690c3432cb878ca2e5baec036 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:16.241530  8959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f00e6101a1647a4acd59b961be1c4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.242985  8963 tablet_service.cc:1467] Processing CreateTablet for tablet 91c5267ebb5645269e7259a7bc69e1f9 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:16.244452  8957 tablet_service.cc:1467] Processing CreateTablet for tablet 85ee34544fb94c6ba00a4b152119564a (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:16.244462  8963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c5267ebb5645269e7259a7bc69e1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.249953  8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ee34544fb94c6ba00a4b152119564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.249801  9059 tablet_service.cc:1467] Processing CreateTablet for tablet 85ee34544fb94c6ba00a4b152119564a (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:16.251886  9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ee34544fb94c6ba00a4b152119564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.251581  9058 tablet_service.cc:1467] Processing CreateTablet for tablet 01ea2a0690c3432cb878ca2e5baec036 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:16.253026  9058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ea2a0690c3432cb878ca2e5baec036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.249811  9061 tablet_service.cc:1467] Processing CreateTablet for tablet 7f00e6101a1647a4acd59b961be1c4ef (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:16.254146  9061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f00e6101a1647a4acd59b961be1c4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.255798  9065 tablet_service.cc:1467] Processing CreateTablet for tablet 91c5267ebb5645269e7259a7bc69e1f9 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:16.257622  9065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c5267ebb5645269e7259a7bc69e1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.258889  9161 tablet_service.cc:1467] Processing CreateTablet for tablet 01ea2a0690c3432cb878ca2e5baec036 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:16.263767  9161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ea2a0690c3432cb878ca2e5baec036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.264853  9158 tablet_service.cc:1467] Processing CreateTablet for tablet 85ee34544fb94c6ba00a4b152119564a (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:16.266503  9158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85ee34544fb94c6ba00a4b152119564a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.267186  9169 tablet_service.cc:1467] Processing CreateTablet for tablet e50bb7ef52c74bb499ef218f4aeaf550 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:16.268451  9165 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6cb7380e164263b4d0085648b14066 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:16.269704  9062 tablet_service.cc:1467] Processing CreateTablet for tablet e50bb7ef52c74bb499ef218f4aeaf550 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:16.271109  9066 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6cb7380e164263b4d0085648b14066 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:16.259228  9160 tablet_service.cc:1467] Processing CreateTablet for tablet 91c5267ebb5645269e7259a7bc69e1f9 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:16.259321  9157 tablet_service.cc:1467] Processing CreateTablet for tablet 7f00e6101a1647a4acd59b961be1c4ef (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:16.273977  9060 tablet_service.cc:1467] Processing CreateTablet for tablet c4b3fee9702a41c5b46e3d07db21f6ae (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:16.275980  9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50bb7ef52c74bb499ef218f4aeaf550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.276281  9063 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbc56ce55a49a7b1607889482c1dc4 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:16.277809  9063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbc56ce55a49a7b1607889482c1dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.282524  8962 tablet_service.cc:1467] Processing CreateTablet for tablet e50bb7ef52c74bb499ef218f4aeaf550 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:16.284992  9157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f00e6101a1647a4acd59b961be1c4ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.285769  8966 tablet_service.cc:1467] Processing CreateTablet for tablet c4b3fee9702a41c5b46e3d07db21f6ae (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:16.286295  8965 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbc56ce55a49a7b1607889482c1dc4 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:16.287135  9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50bb7ef52c74bb499ef218f4aeaf550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.288663  9162 tablet_service.cc:1467] Processing CreateTablet for tablet c4b3fee9702a41c5b46e3d07db21f6ae (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:16.260937  8964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ea2a0690c3432cb878ca2e5baec036. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.291952  9166 tablet_service.cc:1467] Processing CreateTablet for tablet 82bbc56ce55a49a7b1607889482c1dc4 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:16.293152  8962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50bb7ef52c74bb499ef218f4aeaf550. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.293690  9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b3fee9702a41c5b46e3d07db21f6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.297113  9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c5267ebb5645269e7259a7bc69e1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.299499  9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b3fee9702a41c5b46e3d07db21f6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.282363  8960 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6cb7380e164263b4d0085648b14066 (DEFAULT_TABLE table=soft_deleted_table [id=a6bcefd342854512a2c8401530168839]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:16.303539  9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbc56ce55a49a7b1607889482c1dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.307382  9165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6cb7380e164263b4d0085648b14066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.315663  8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b3fee9702a41c5b46e3d07db21f6ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.323279  8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82bbc56ce55a49a7b1607889482c1dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.331483  9066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6cb7380e164263b4d0085648b14066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:16.335922  8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6cb7380e164263b4d0085648b14066. 1 dirs total, 0 dirs full, 0 dirs failed
W20260809 23:24:16.360252  8900 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:16.451929  9364 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.482048  9364 tablet_bootstrap.cc:654] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.483958  9365 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.491900  9368 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
W20260809 23:24:16.499224  9032 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:32919, user_credentials={real_user=slave}} blocked reactor thread for 117636us
I20260809 23:24:16.514918  9365 tablet_bootstrap.cc:654] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.518599  9368 tablet_bootstrap.cc:654] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.525259  9365 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.525794  9365 ts_tablet_manager.cc:1397] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.042s	user 0.012s	sys 0.000s
I20260809 23:24:16.528777  9365 raft_consensus.cc:348] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.529403  9365 raft_consensus.cc:374] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.529615  9365 raft_consensus.cc:729] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.530251  9365 consensus_queue.cc:260] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.533164  9365 ts_tablet_manager.cc:1428] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260809 23:24:16.534626  9365 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.537158  9368 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.537767  9368 ts_tablet_manager.cc:1397] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.046s	user 0.006s	sys 0.015s
I20260809 23:24:16.537853  9364 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.538507  9364 ts_tablet_manager.cc:1397] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.087s	user 0.011s	sys 0.010s
I20260809 23:24:16.539600  9284 raft_consensus.cc:482] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.540088  9284 raft_consensus.cc:504] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.540845  9368 raft_consensus.cc:348] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.541640  9368 raft_consensus.cc:374] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.541949  9368 raft_consensus.cc:729] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.542551  9364 raft_consensus.cc:348] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.542687  9368 consensus_queue.cc:260] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.544945  9364 raft_consensus.cc:374] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.545233  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:16.546274  9364 raft_consensus.cc:729] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.546836  9368 ts_tablet_manager.cc:1428] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.005s
W20260809 23:24:16.547688  9134 leader_election.cc:341] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.548151  9368 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.547572  9364 consensus_queue.cc:260] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.550545  9364 ts_tablet_manager.cc:1428] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260809 23:24:16.551844  9364 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.555229  9368 tablet_bootstrap.cc:654] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.542322  9284 leader_election.cc:288] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:16.543792  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.558265  9364 tablet_bootstrap.cc:654] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.558080  9077 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:16.560225  9132 leader_election.cc:302] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:16.563898  9284 raft_consensus.cc:2781] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.564401  9284 raft_consensus.cc:482] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.564733  9284 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.567309  9368 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.567827  9368 ts_tablet_manager.cc:1397] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.005s
I20260809 23:24:16.570901  9368 raft_consensus.cc:348] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.571614  9368 raft_consensus.cc:374] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.571914  9368 raft_consensus.cc:729] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.572855  9368 consensus_queue.cc:260] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.575294  9368 ts_tablet_manager.cc:1428] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260809 23:24:16.577308  9365 tablet_bootstrap.cc:654] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.579149  9364 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.579733  9364 ts_tablet_manager.cc:1397] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.010s
I20260809 23:24:16.581705  9284 raft_consensus.cc:504] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.583037  9364 raft_consensus.cc:348] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.583071  9368 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.584086  9364 raft_consensus.cc:374] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.584623  9364 raft_consensus.cc:729] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.585394  9364 consensus_queue.cc:260] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.588166  9364 ts_tablet_manager.cc:1428] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260809 23:24:16.589195  9364 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.592094  9286 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.592603  9286 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.592962  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:16.593667  9077 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.594673  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:16.595538  9364 tablet_bootstrap.cc:654] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
W20260809 23:24:16.596465  9134 leader_election.cc:341] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.597471  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
W20260809 23:24:16.599319  9032 leader_election.cc:341] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.594709  9286 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.604534  9365 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.604960  9284 leader_election.cc:288] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:16.605206  9365 ts_tablet_manager.cc:1397] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.071s	user 0.006s	sys 0.007s
I20260809 23:24:16.606552  9368 tablet_bootstrap.cc:654] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.606876  9077 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:16.608745  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.609750  9365 raft_consensus.cc:348] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.610914  9365 raft_consensus.cc:374] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.611406  9365 raft_consensus.cc:729] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
W20260809 23:24:16.612232  9030 leader_election.cc:341] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.612743  9030 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54; no voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:16.612558  9365 consensus_queue.cc:260] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.614392  9286 raft_consensus.cc:2726] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:16.615046  9132 leader_election.cc:302] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:16.617893  9365 ts_tablet_manager.cc:1428] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260809 23:24:16.618729  9376 raft_consensus.cc:2781] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:16.619149  9364 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.619262  9376 raft_consensus.cc:686] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:16.619977  9364 ts_tablet_manager.cc:1397] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.005s
I20260809 23:24:16.621753  9365 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.624702  9364 raft_consensus.cc:348] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.625538  9364 raft_consensus.cc:374] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.620455  9376 consensus_queue.cc:237] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.625869  9364 raft_consensus.cc:729] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.626045  9368 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.631943  9368 ts_tablet_manager.cc:1397] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.049s	user 0.011s	sys 0.008s
I20260809 23:24:16.629624  9365 tablet_bootstrap.cc:654] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.634868  9368 raft_consensus.cc:348] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.635550  9368 raft_consensus.cc:374] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.635802  9368 raft_consensus.cc:729] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.636848  9368 consensus_queue.cc:260] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.639957  9368 ts_tablet_manager.cc:1428] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:16.640712  9364 consensus_queue.cc:260] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.641639  9368 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.649776  9368 tablet_bootstrap.cc:654] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.653637  9364 ts_tablet_manager.cc:1428] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.001s
I20260809 23:24:16.661410  9291 raft_consensus.cc:482] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.661923  9291 raft_consensus.cc:504] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.664042  9368 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.664634  9368 ts_tablet_manager.cc:1397] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.003s
I20260809 23:24:16.664712  9291 leader_election.cc:288] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.666231  9364 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.668803  9368 raft_consensus.cc:348] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.669715  9368 raft_consensus.cc:374] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.670101  9368 raft_consensus.cc:729] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.670907  9368 consensus_queue.cc:260] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.672291  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.673002  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.649566  8834 catalog_manager.cc:5495] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
W20260809 23:24:16.674129  8930 leader_election.cc:341] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.681630  9364 tablet_bootstrap.cc:654] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
W20260809 23:24:16.683812  8930 leader_election.cc:341] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.684427  8930 leader_election.cc:302] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:16.685030  9368 ts_tablet_manager.cc:1428] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.006s
I20260809 23:24:16.685566  9386 raft_consensus.cc:2726] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:16.686405  9368 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.694146  9368 tablet_bootstrap.cc:654] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.704650  9364 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.704519  9365 tablet_bootstrap.cc:492] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.705569  9364 ts_tablet_manager.cc:1397] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.040s	user 0.008s	sys 0.012s
I20260809 23:24:16.705681  9365 ts_tablet_manager.cc:1397] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.084s	user 0.023s	sys 0.046s
I20260809 23:24:16.708580  9368 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.708369  9364 raft_consensus.cc:348] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.709134  9364 raft_consensus.cc:374] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.709159  9368 ts_tablet_manager.cc:1397] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.010s
I20260809 23:24:16.709106  9365 raft_consensus.cc:348] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.709815  9364 raft_consensus.cc:729] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.710136  9365 raft_consensus.cc:374] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.710542  9365 raft_consensus.cc:729] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.710644  9364 consensus_queue.cc:260] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.712975  9368 raft_consensus.cc:348] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.713658  9364 ts_tablet_manager.cc:1428] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:16.711287  9365 consensus_queue.cc:260] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.713750  9368 raft_consensus.cc:374] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.714726  9364 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.715842  9365 ts_tablet_manager.cc:1428] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260809 23:24:16.716882  9365 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.714186  9368 raft_consensus.cc:729] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.721549  9364 tablet_bootstrap.cc:654] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.721978  9368 consensus_queue.cc:260] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.724875  9368 ts_tablet_manager.cc:1428] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260809 23:24:16.726394  9368 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.726807  9365 tablet_bootstrap.cc:654] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.733870  9376 raft_consensus.cc:482] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.734656  9368 tablet_bootstrap.cc:654] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.734419  9376 raft_consensus.cc:504] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.736693  9376 leader_election.cc:288] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:16.738150  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:16.740823  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
W20260809 23:24:16.742439  9134 leader_election.cc:341] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:24:16.743875  9132 leader_election.cc:341] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.744372  9132 leader_election.cc:302] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:16.745199  9376 raft_consensus.cc:2726] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:16.747438  9368 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.748185  9368 ts_tablet_manager.cc:1397] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.010s
I20260809 23:24:16.752631  9368 raft_consensus.cc:348] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.753640  9368 raft_consensus.cc:374] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.754012  9368 raft_consensus.cc:729] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.755018  9368 consensus_queue.cc:260] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.761132  9368 ts_tablet_manager.cc:1428] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260809 23:24:16.762529  9368 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.768560  9386 raft_consensus.cc:482] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.769060  9386 raft_consensus.cc:504] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.771114  9386 leader_election.cc:288] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.771522  9368 tablet_bootstrap.cc:654] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.773351  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.773373  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
W20260809 23:24:16.776054  8930 leader_election.cc:341] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
W20260809 23:24:16.776850  8930 leader_election.cc:341] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.777269  8930 leader_election.cc:302] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:16.778529  9286 raft_consensus.cc:482] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.779049  9286 raft_consensus.cc:504] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.781162  9286 leader_election.cc:288] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.782758  9364 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.782917  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ee34544fb94c6ba00a4b152119564a" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.783183  9368 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:16.783346  9364 ts_tablet_manager.cc:1397] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.069s	user 0.024s	sys 0.031s
I20260809 23:24:16.782917  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ee34544fb94c6ba00a4b152119564a" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:16.783960  9368 ts_tablet_manager.cc:1397] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.007s
I20260809 23:24:16.784354  8978 raft_consensus.cc:2454] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:16.783082  9386 raft_consensus.cc:2726] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:16.785418  9032 leader_election.cc:302] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
W20260809 23:24:16.786132  9030 leader_election.cc:341] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:16.786350  9286 raft_consensus.cc:2781] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.786841  9286 raft_consensus.cc:482] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.787235  9286 raft_consensus.cc:3037] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.787359  9368 raft_consensus.cc:348] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.788200  9368 raft_consensus.cc:374] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.788506  9368 raft_consensus.cc:729] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.789057  9365 tablet_bootstrap.cc:492] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.788384  9364 raft_consensus.cc:348] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.789705  9365 ts_tablet_manager.cc:1397] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.073s	user 0.017s	sys 0.021s
I20260809 23:24:16.789479  9368 consensus_queue.cc:260] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.790171  9364 raft_consensus.cc:374] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.790671  9364 raft_consensus.cc:729] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.791523  9364 consensus_queue.cc:260] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.792239  9368 ts_tablet_manager.cc:1428] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:16.793556  9368 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:16.795457  9286 raft_consensus.cc:504] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.796516  9364 ts_tablet_manager.cc:1428] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260809 23:24:16.797653  9364 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.798451  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ee34544fb94c6ba00a4b152119564a" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:16.799206  8978 raft_consensus.cc:3037] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.805732  9368 tablet_bootstrap.cc:654] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.806002  9364 tablet_bootstrap.cc:654] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.809804  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ee34544fb94c6ba00a4b152119564a" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:16.813166  8978 raft_consensus.cc:2454] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:16.815008  9032 leader_election.cc:302] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:16.816375  9366 raft_consensus.cc:2781] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:16.817972  9376 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.818858  9376 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.822057  9365 raft_consensus.cc:348] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.823040  9376 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:16.826041  9386 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.826875  9364 tablet_bootstrap.cc:492] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.826571  9386 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.827616  9364 ts_tablet_manager.cc:1397] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.030s	user 0.006s	sys 0.013s
I20260809 23:24:16.828801  9386 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.829641  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.830457  9077 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:16.830637  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.830919  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.831319  9083 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:16.831403  9365 raft_consensus.cc:374] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.831627  9183 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:16.831928  9365 raft_consensus.cc:729] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.827136  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:16.832885  8930 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:16.833205  9132 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:16.832715  9365 consensus_queue.cc:260] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.833568  8977 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:16.834458  9284 raft_consensus.cc:2781] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.834900  9284 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.835300  9284 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.835551  9365 ts_tablet_manager.cc:1428] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.045s	user 0.004s	sys 0.004s
I20260809 23:24:16.834681  9291 raft_consensus.cc:2781] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.836030  9291 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.836338  9291 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.836602  9365 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.837709  9182 raft_consensus.cc:3037] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.840547  9386 raft_consensus.cc:482] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.841183  9386 raft_consensus.cc:504] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.843967  9364 raft_consensus.cc:348] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.844753  9364 raft_consensus.cc:374] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.845037  9364 raft_consensus.cc:729] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.851256  9365 tablet_bootstrap.cc:654] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.852233  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:16.852422  9284 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.854403  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.853410  9291 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.854390  9364 consensus_queue.cc:260] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.855396  9183 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:16.856017  9083 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 0.
I20260809 23:24:16.856673  8930 leader_election.cc:302] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:16.856943  9291 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.857633  9291 raft_consensus.cc:2781] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.858376  9291 raft_consensus.cc:482] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.859552  9291 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.860756  9284 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:16.862867  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:16.866173  8977 raft_consensus.cc:2379] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d0a6b0fd63d4091a84bb1215772b32d in current term 1: Already voted for candidate f6964fd5f558447d99495c9e69df90c1 in this term.
I20260809 23:24:16.866984  9365 tablet_bootstrap.cc:492] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.864074  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:16.867614  9365 ts_tablet_manager.cc:1397] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.031s	user 0.012s	sys 0.012s
I20260809 23:24:16.868022  9083 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.862731  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:16.872001  9183 raft_consensus.cc:2379] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f6964fd5f558447d99495c9e69df90c1 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:16.871723  9365 raft_consensus.cc:348] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.872962  9365 raft_consensus.cc:374] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.874579  9364 ts_tablet_manager.cc:1428] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.047s	user 0.003s	sys 0.008s
I20260809 23:24:16.875609  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:16.875568  9365 raft_consensus.cc:729] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.876999  9364 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.877074  9365 consensus_queue.cc:260] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.880358  9365 ts_tablet_manager.cc:1428] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260809 23:24:16.879837  9291 raft_consensus.cc:504] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.881659  9365 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.883160  8930 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:16.888224  9291 leader_election.cc:288] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.888808  9291 raft_consensus.cc:2726] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:16.890529  9364 tablet_bootstrap.cc:654] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.892449  9407 raft_consensus.cc:482] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:16.892905  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:16.892958  9407 raft_consensus.cc:504] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.893934  9184 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.896036  9407 leader_election.cc:288] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.899765  9365 tablet_bootstrap.cc:654] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.904640  9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:16.905612  9181 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:16.907997  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:16.908774  9077 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.916033  9386 leader_election.cc:288] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.910666  9030 leader_election.cc:302] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:16.923054  9407 raft_consensus.cc:2781] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:16.923547  9407 raft_consensus.cc:482] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:16.923929  9407 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.902285  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:16.931149  9364 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:16.931783  9364 ts_tablet_manager.cc:1397] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.055s	user 0.016s	sys 0.022s
I20260809 23:24:16.934904  9364 raft_consensus.cc:348] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.935676  9364 raft_consensus.cc:374] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.935967  9364 raft_consensus.cc:729] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.936993  9364 consensus_queue.cc:260] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.938360  9407 raft_consensus.cc:504] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.939496  9364 ts_tablet_manager.cc:1428] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260809 23:24:16.939663  9286 leader_election.cc:288] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.940677  9364 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:16.934043  8977 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:16.942898  9407 leader_election.cc:288] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:16.943608  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:16.944635  9183 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.948280  9364 tablet_bootstrap.cc:654] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.946714  9365 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:16.955176  9365 ts_tablet_manager.cc:1397] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.074s	user 0.029s	sys 0.009s
I20260809 23:24:16.963567  9365 raft_consensus.cc:348] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.964895  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:16.965941  8977 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:16.965889  9365 raft_consensus.cc:374] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:16.966780  9365 raft_consensus.cc:729] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:16.968204  9365 consensus_queue.cc:260] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:16.971590  9365 ts_tablet_manager.cc:1428] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.000s
I20260809 23:24:16.973026  9365 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:16.984443  9365 tablet_bootstrap.cc:654] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.994239  9182 raft_consensus.cc:2454] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:16.999029  9365 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:17.000458  9365 ts_tablet_manager.cc:1397] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.011s
I20260809 23:24:17.005426  9365 raft_consensus.cc:348] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.008150  9365 raft_consensus.cc:374] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.008594  9365 raft_consensus.cc:729] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.009603  9365 consensus_queue.cc:260] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.013288  9365 ts_tablet_manager.cc:1428] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260809 23:24:17.014662  9365 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:17.019272  9083 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:17.021584  9132 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.023228  9284 raft_consensus.cc:2781] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.023870  9284 raft_consensus.cc:686] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:17.025914  9365 tablet_bootstrap.cc:654] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:17.027494  9380 raft_consensus.cc:482] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:17.028188  9380 raft_consensus.cc:504] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.031797  9380 leader_election.cc:288] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:17.025405  9284 consensus_queue.cc:237] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.032195  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:17.033031  8978 raft_consensus.cc:2454] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 0.
I20260809 23:24:17.034267  9134 leader_election.cc:302] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.037334  9380 raft_consensus.cc:2781] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:17.037895  9380 raft_consensus.cc:482] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:17.038889  9380 raft_consensus.cc:3037] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:17.048712  9134 consensus_peers.cc:487] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:17.049816  9365 tablet_bootstrap.cc:492] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:17.049996  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:17.050567  9365 ts_tablet_manager.cc:1397] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.004s
W20260809 23:24:17.052440  9132 leader_election.cc:341] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:17.054522  9365 raft_consensus.cc:348] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.055598  9365 raft_consensus.cc:374] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.055927  9365 raft_consensus.cc:729] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.056389  9366 raft_consensus.cc:686] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.057171  9365 consensus_queue.cc:260] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.060695  9365 ts_tablet_manager.cc:1428] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260809 23:24:17.061940  9365 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: Bootstrap starting.
I20260809 23:24:17.057761  9366 consensus_queue.cc:237] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.069032  9365 tablet_bootstrap.cc:654] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:16.972952  9184 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:17.071377  8930 leader_election.cc:302] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.072418  9406 raft_consensus.cc:2781] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.072997  9406 raft_consensus.cc:686] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:17.073879  9406 consensus_queue.cc:237] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.075122  9077 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 1.
I20260809 23:24:17.082860  9365 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: No bootstrap required, opened a new log
I20260809 23:24:17.083516  9365 ts_tablet_manager.cc:1397] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.003s
I20260809 23:24:17.089155  9183 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.091998  8834 catalog_manager.cc:5495] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 1 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.094251  9030 leader_election.cc:302] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:17.095499  9366 raft_consensus.cc:2781] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.096280  9366 raft_consensus.cc:686] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.096226  9380 raft_consensus.cc:504] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.098352  9380 leader_election.cc:288] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:17.099542  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:17.100260  8978 raft_consensus.cc:3037] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.102849  9368 tablet_bootstrap.cc:492] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:17.103304  9364 tablet_bootstrap.cc:492] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:17.103518  9368 ts_tablet_manager.cc:1397] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.310s	user 0.090s	sys 0.117s
I20260809 23:24:17.104238  9364 ts_tablet_manager.cc:1397] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.164s	user 0.052s	sys 0.083s
I20260809 23:24:17.105495  8833 catalog_manager.cc:5495] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.107702  9368 raft_consensus.cc:348] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.108441  9368 raft_consensus.cc:374] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.108458  9364 raft_consensus.cc:348] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.109809  9364 raft_consensus.cc:374] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.110159  9364 raft_consensus.cc:729] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.111433  8977 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.111264  9364 consensus_queue.cc:260] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.108752  9368 raft_consensus.cc:729] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.112365  9405 raft_consensus.cc:482] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:17.112959  9405 raft_consensus.cc:504] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.113155  9368 consensus_queue.cc:260] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.115783  9368 ts_tablet_manager.cc:1428] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260809 23:24:17.116694  9368 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: Bootstrap starting.
I20260809 23:24:17.118044  9405 leader_election.cc:288] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.119581  9365 raft_consensus.cc:348] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.120401  9365 raft_consensus.cc:374] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.120699  9365 raft_consensus.cc:729] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.120824  9364 ts_tablet_manager.cc:1428] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.003s
I20260809 23:24:17.121990  9364 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Bootstrap starting.
I20260809 23:24:17.121575  9365 consensus_queue.cc:260] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.124326  9368 tablet_bootstrap.cc:654] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:17.125627  9365 ts_tablet_manager.cc:1428] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: Time spent starting tablet: real 0.042s	user 0.006s	sys 0.000s
I20260809 23:24:17.127028  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:17.127776  8977 raft_consensus.cc:2454] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.131306  9032 leader_election.cc:302] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.132370  9405 raft_consensus.cc:2781] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:17.132813  9405 raft_consensus.cc:482] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:17.133138  9405 raft_consensus.cc:3037] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.135506  9364 tablet_bootstrap.cc:654] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:17.108526  9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:17.145778  9368 tablet_bootstrap.cc:492] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: No bootstrap required, opened a new log
I20260809 23:24:17.146284  9368 ts_tablet_manager.cc:1397] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.007s
W20260809 23:24:17.146474  9132 leader_election.cc:341] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:17.149076  9368 raft_consensus.cc:348] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.151706  9368 raft_consensus.cc:374] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.152087  9368 raft_consensus.cc:729] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.152338  9407 raft_consensus.cc:482] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:17.152838  9407 raft_consensus.cc:504] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.152822  9368 consensus_queue.cc:260] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.174806  9368 ts_tablet_manager.cc:1428] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: Time spent starting tablet: real 0.028s	user 0.013s	sys 0.013s
I20260809 23:24:17.174674  9380 consensus_queue.cc:1035] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:17.178064  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b3fee9702a41c5b46e3d07db21f6ae" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:17.178722  9182 raft_consensus.cc:2454] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.178767  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b3fee9702a41c5b46e3d07db21f6ae" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:17.180051  8977 raft_consensus.cc:2454] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.181910  9032 leader_election.cc:302] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.183089  9286 raft_consensus.cc:2781] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:17.183450  9286 raft_consensus.cc:482] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:17.183761  9286 raft_consensus.cc:3037] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.184345  9407 leader_election.cc:288] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.097584  9366 consensus_queue.cc:237] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.186403  8832 catalog_manager.cc:5495] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:17.191998  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:17.191776  9286 raft_consensus.cc:504] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.192797  9183 raft_consensus.cc:2454] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.194494  8978 raft_consensus.cc:2454] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 1.
I20260809 23:24:17.194962  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b3fee9702a41c5b46e3d07db21f6ae" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:17.195752  8977 raft_consensus.cc:3037] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.199620  9286 leader_election.cc:288] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.203459  8977 raft_consensus.cc:2454] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.204684  9032 leader_election.cc:302] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.207718  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4b3fee9702a41c5b46e3d07db21f6ae" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.208245  9182 raft_consensus.cc:3037] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.211773  9134 leader_election.cc:302] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:17.213038  9380 raft_consensus.cc:2781] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.213517  9380 raft_consensus.cc:686] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:17.214339  9380 consensus_queue.cc:237] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.215152  9182 raft_consensus.cc:2454] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.215162  9286 raft_consensus.cc:2781] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.216697  9286 raft_consensus.cc:686] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.217829  9286 consensus_queue.cc:237] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.219545  9405 raft_consensus.cc:504] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.228219  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:17.229173  8978 raft_consensus.cc:3037] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.228380  9405 leader_election.cc:288] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.234246  9364 tablet_bootstrap.cc:492] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: No bootstrap required, opened a new log
I20260809 23:24:17.234912  9364 ts_tablet_manager.cc:1397] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Time spent bootstrapping tablet: real 0.113s	user 0.035s	sys 0.065s
I20260809 23:24:17.241868  8978 raft_consensus.cc:2454] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.236025  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6cb7380e164263b4d0085648b14066" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.242782  9182 raft_consensus.cc:3037] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.242388  9364 raft_consensus.cc:348] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.243278  9364 raft_consensus.cc:374] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:17.243235  9032 leader_election.cc:302] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.243733  9364 raft_consensus.cc:729] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Initialized, Role: FOLLOWER
I20260809 23:24:17.227309  8833 catalog_manager.cc:5495] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 1 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:17.244525  9366 raft_consensus.cc:2781] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.244457  9364 consensus_queue.cc:260] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.245282  9366 raft_consensus.cc:686] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.246763  9364 ts_tablet_manager.cc:1428] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:24:17.246351  9366 consensus_queue.cc:237] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.244165  8834 catalog_manager.cc:5495] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.254103  9182 raft_consensus.cc:2454] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.274390  8834 catalog_manager.cc:5495] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.275760  8834 catalog_manager.cc:5495] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.303149  9286 raft_consensus.cc:482] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:17.303692  9286 raft_consensus.cc:504] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.305784  9286 leader_election.cc:288] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.306457  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:17.306926  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82bbc56ce55a49a7b1607889482c1dc4" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:17.307289  8978 raft_consensus.cc:2379] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 19446675307f4e709bb1c63b9c529e54 in current term 1: Already voted for candidate 3d0a6b0fd63d4091a84bb1215772b32d in this term.
I20260809 23:24:17.308357  9032 leader_election.cc:302] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54; no voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.309027  9286 raft_consensus.cc:3037] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.314483  9286 raft_consensus.cc:2726] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:17.324977  9286 raft_consensus.cc:482] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:17.325460  9286 raft_consensus.cc:504] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.327473  9286 leader_election.cc:288] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.328119  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:17.328692  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:17.328877  8978 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.329380  9182 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 0.
I20260809 23:24:17.329946  9032 leader_election.cc:302] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.331053  9286 raft_consensus.cc:2781] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:17.331534  9286 raft_consensus.cc:482] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:17.331794  9286 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.338892  9286 raft_consensus.cc:504] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.340634  9286 leader_election.cc:288] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.341789  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.342499  9182 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.342852  8978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:17.343456  8978 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.349668  9182 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.349877  8978 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 1.
I20260809 23:24:17.350802  9030 leader_election.cc:302] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:17.351599  9286 raft_consensus.cc:2781] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:17.352029  9286 raft_consensus.cc:686] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.352836  9286 consensus_queue.cc:237] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.361058  8834 catalog_manager.cc:5495] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
W20260809 23:24:17.364799  8900 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:17.486575  9286 consensus_queue.cc:1035] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.491632  9376 consensus_queue.cc:1035] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:17.502612  9407 consensus_queue.cc:1035] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.505262  9182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:44888
I20260809 23:24:17.506560  9182 raft_consensus.cc:595] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.512480  9380 consensus_queue.cc:1035] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.518618  9083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:36676
I20260809 23:24:17.519335  9083 raft_consensus.cc:595] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.528285  9182 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:44888
I20260809 23:24:17.528997  9182 raft_consensus.cc:595] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.533588  9380 raft_consensus.cc:979] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.534030  9380 raft_consensus.cc:1067] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.535918  8977 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60458
I20260809 23:24:17.536473  9083 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:36676
I20260809 23:24:17.536466  8977 raft_consensus.cc:482] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:17.537164  8977 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.537163  9083 raft_consensus.cc:595] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Received request to transfer leadership to TS f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:17.544350  8977 raft_consensus.cc:504] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.547012  8977 leader_election.cc:288] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.548520  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.548723  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c5267ebb5645269e7259a7bc69e1f9" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:17.549201  9182 raft_consensus.cc:3032] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:17.549273  9083 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.549582  9182 raft_consensus.cc:729] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:17.550446  9182 consensus_queue.cc:260] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.551936  9182 raft_consensus.cc:3037] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.555178  8977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
mode: GRACEFUL
new_leader_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:60484
I20260809 23:24:17.555840  8977 raft_consensus.cc:595] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Received request to transfer leadership to TS 19446675307f4e709bb1c63b9c529e54
I20260809 23:24:17.559069  9083 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.560326  8930 leader_election.cc:302] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.561075  9406 raft_consensus.cc:2781] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:17.561537  9406 raft_consensus.cc:686] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:17.562391  9406 consensus_queue.cc:237] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.563089  9182 raft_consensus.cc:2454] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.569258  8833 catalog_manager.cc:5495] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.578284  9286 consensus_queue.cc:1035] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.589560  9405 consensus_queue.cc:1035] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.599270  9406 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.601775  9407 raft_consensus.cc:979] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.602288  9405 raft_consensus.cc:1067] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.603864  8977 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60470
I20260809 23:24:17.604518  8977 raft_consensus.cc:482] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:17.604909  8977 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.609033  9386 raft_consensus.cc:979] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: : Instructing follower 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:17.609575  9386 raft_consensus.cc:1067] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Signalling peer 19446675307f4e709bb1c63b9c529e54 to start an election
I20260809 23:24:17.613663  9078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
 from {username='slave'} at 127.0.0.1:36666
I20260809 23:24:17.614543  9078 raft_consensus.cc:482] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:17.614583  8978 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:17.615146  9078 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.614777  9386 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:17.622712  9078 raft_consensus.cc:504] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.613186  8977 raft_consensus.cc:504] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.625924  8977 leader_election.cc:288] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.627471  9183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.628336  9183 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.634682  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ea2a0690c3432cb878ca2e5baec036" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:17.635416  9083 raft_consensus.cc:3032] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:17.635721  9083 raft_consensus.cc:729] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.636689  9083 consensus_queue.cc:260] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.641196  9183 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.641819  9184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.642343  8930 leader_election.cc:302] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.643989  9406 raft_consensus.cc:2781] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:17.644724  9406 raft_consensus.cc:686] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:17.642922  9078 leader_election.cc:288] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.643213  9083 raft_consensus.cc:3037] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.645638  9406 consensus_queue.cc:237] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.651439  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "19446675307f4e709bb1c63b9c529e54" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:17.652107  8977 raft_consensus.cc:3032] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:17.652470  8977 raft_consensus.cc:729] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:17.652719  9386 raft_consensus.cc:482] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d0a6b0fd63d4091a84bb1215772b32d)
I20260809 23:24:17.653698  9386 raft_consensus.cc:504] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.655263  9083 raft_consensus.cc:2454] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.661372  9078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:17.661566  9386 leader_election.cc:288] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.662319  9376 raft_consensus.cc:979] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.662093  8977 consensus_queue.cc:260] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.662796  9376 raft_consensus.cc:1067] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.664302  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:17.664665  8978 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60458
I20260809 23:24:17.665479  8977 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.665186  8978 raft_consensus.cc:482] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:17.665694  8930 leader_election.cc:302] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:17.666250  8978 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.668972  8832 catalog_manager.cc:5495] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.671897  8978 raft_consensus.cc:504] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.672964  9406 raft_consensus.cc:2726] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 23:24:17.673836  8978 leader_election.cc:288] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.675297  9405 consensus_queue.cc:1035] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.675717  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:17.676326  9083 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.677376  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.677989  9182 raft_consensus.cc:3032] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:17.678296  9182 raft_consensus.cc:729] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:17.679114  9182 consensus_queue.cc:260] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.680485  9182 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.676306  8977 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19446675307f4e709bb1c63b9c529e54 in term 2.
I20260809 23:24:17.683319  9032 leader_election.cc:302] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:17.684199  9405 raft_consensus.cc:2781] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:17.685596  9405 raft_consensus.cc:686] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Leader. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.686357  9083 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.686595  9405 consensus_queue.cc:237] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.687960  8930 leader_election.cc:302] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:17.688446  9182 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:17.688943  9406 raft_consensus.cc:2781] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:17.689422  9406 raft_consensus.cc:686] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:17.690301  9406 consensus_queue.cc:237] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.697073  9286 consensus_queue.cc:1035] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.698093  9407 consensus_queue.cc:1035] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:17.708209  9376 consensus_queue.cc:1035] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.708611  8832 catalog_manager.cc:5495] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.709538  8833 catalog_manager.cc:5495] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 reported cstate change: term changed from 1 to 2, leader changed from f6964fd5f558447d99495c9e69df90c1 (127.21.255.1) to 19446675307f4e709bb1c63b9c529e54 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "19446675307f4e709bb1c63b9c529e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:17.717435  9407 consensus_queue.cc:1035] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.727272  9380 consensus_queue.cc:1035] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.965250  9407 consensus_queue.cc:1035] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.972074  9286 raft_consensus.cc:979] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: : Instructing follower f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.972699  9410 raft_consensus.cc:1067] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Signalling peer f6964fd5f558447d99495c9e69df90c1 to start an election
I20260809 23:24:17.972781  9286 consensus_queue.cc:1035] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:17.976382  8975 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
 from {username='slave'} at 127.0.0.1:60470
I20260809 23:24:17.979239  8975 raft_consensus.cc:482] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:17.979759  8975 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.988397  8975 raft_consensus.cc:504] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.990375  8975 leader_election.cc:288] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Requested vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:17.991178  9083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:17.991822  9083 raft_consensus.cc:3032] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:17.992259  9083 raft_consensus.cc:729] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:17.993240  9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f00e6101a1647a4acd59b961be1c4ef" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
I20260809 23:24:17.993180  9083 consensus_queue.cc:260] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:17.994041  9182 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:17.994455  9083 raft_consensus.cc:3037] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:18.001480  9182 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:18.001593  9083 raft_consensus.cc:2454] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6964fd5f558447d99495c9e69df90c1 in term 2.
I20260809 23:24:18.002477  8930 leader_election.cc:302] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:18.003259  9406 raft_consensus.cc:2781] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:18.003759  9406 raft_consensus.cc:686] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Leader. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:18.004550  9406 consensus_queue.cc:237] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:18.012936  8834 catalog_manager.cc:5495] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 reported cstate change: term changed from 1 to 2, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to f6964fd5f558447d99495c9e69df90c1 (127.21.255.1). New cstate: current_term: 2 leader_uuid: "f6964fd5f558447d99495c9e69df90c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:18.035660  9083 raft_consensus.cc:1261] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.037003  9406 consensus_queue.cc:1035] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.044692  9182 raft_consensus.cc:1261] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.047493  9451 consensus_queue.cc:1035] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:18.059267  9083 raft_consensus.cc:1261] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.060554  9386 consensus_queue.cc:1035] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.068356  9182 raft_consensus.cc:1261] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.070595  9451 consensus_queue.cc:1035] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.148181  9183 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:18.155752  9183 raft_consensus.cc:1261] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.157186  9407 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.165592  8975 raft_consensus.cc:1261] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Refusing update from remote peer 19446675307f4e709bb1c63b9c529e54: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.167514  9405 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.212237  9183 raft_consensus.cc:1261] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.214311  9451 consensus_queue.cc:1035] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.226011  9083 raft_consensus.cc:1261] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.227488  9451 consensus_queue.cc:1035] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:18.472978  9183 raft_consensus.cc:1261] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.474710  9386 consensus_queue.cc:1035] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:18.482795  9083 raft_consensus.cc:1261] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Refusing update from remote peer f6964fd5f558447d99495c9e69df90c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:18.484745  9406 consensus_queue.cc:1035] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:20.566843  8834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50694:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260809 23:24:20.589120  8834 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50694:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260809 23:24:20.634268 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:21.005889  9544 tablet.cc:1722] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.009073  9546 tablet.cc:1722] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.054426  9555 tablet.cc:1722] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260809 23:24:21.075950  9117 tablet_mm_ops.cc:140] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:c4b3fee9702a41c5b46e3d07db21f6ae)
W20260809 23:24:21.088547  9117 tablet_mm_ops.cc:305] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:c4b3fee9702a41c5b46e3d07db21f6ae)
W20260809 23:24:21.090840  9117 tablet_mm_ops.cc:221] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:c4b3fee9702a41c5b46e3d07db21f6ae)
W20260809 23:24:21.103047  9512 log.cc:924] Time spent T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
W20260809 23:24:21.104455  9511 log.cc:924] Time spent T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260809 23:24:21.107681  9535 log.cc:924] Time spent T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Append to log took a long time: real 0.120s	user 0.003s	sys 0.000s
I20260809 23:24:21.087594  9547 tablet.cc:1722] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.037962  9551 tablet.cc:1722] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.143824  9568 tablet.cc:1722] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.149559  9546 tablet.cc:1722] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.132544  9558 tablet.cc:1722] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.119153  9552 tablet.cc:1722] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.170809  8976 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
mode: GRACEFUL
new_leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:40516
I20260809 23:24:21.171566  8976 raft_consensus.cc:595] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Received request to transfer leadership to TS 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:21.170123  9566 tablet.cc:1722] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.127381  9562 tablet.cc:1722] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.195412  9569 tablet.cc:1722] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.263945  9566 tablet.cc:1722] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.267637  9578 tablet.cc:1722] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.272892  9547 tablet.cc:1722] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.281399  9551 tablet.cc:1722] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.283362  9562 tablet.cc:1722] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.285523  9558 tablet.cc:1722] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.297897  9562 tablet.cc:1722] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.301481  9569 tablet.cc:1722] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.304925  9544 tablet.cc:1722] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.304970  9551 tablet.cc:1722] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.310660  9546 tablet.cc:1722] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.314280  9558 tablet.cc:1722] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.323287  9562 tablet.cc:1722] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.326104  9551 tablet.cc:1722] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.337666  9568 tablet.cc:1722] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:21.352558  9074 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:41030
I20260809 23:24:21.353328  9074 raft_consensus.cc:595] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Received request to transfer leadership to TS 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:21.354808 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260809 23:24:21.355507 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:21.366211  8977 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4eab99868842422583b64aa796efc47f"
dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
mode: GRACEFUL
new_leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:40516
I20260809 23:24:21.367028  8977 raft_consensus.cc:595] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Received request to transfer leadership to TS 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:21.373421  9074 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58d56d427cdd455c997502702b5c752b"
dest_uuid: "19446675307f4e709bb1c63b9c529e54"
mode: GRACEFUL
new_leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:41030
I20260809 23:24:21.374106  9074 raft_consensus.cc:595] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Received request to transfer leadership to TS 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:21.377475 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:24:21.378121 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:21.401084  8832 catalog_manager.cc:6040] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: soft_deleted_table [id=a6bcefd342854512a2c8401530168839] alter complete (version 1)
I20260809 23:24:21.498243  9451 raft_consensus.cc:979] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: : Instructing follower 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.498728  9451 raft_consensus.cc:1067] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Signalling peer 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.500478  9177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:44884
I20260809 23:24:21.501073  9177 raft_consensus.cc:482] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:21.501415  9177 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.511389  9177 raft_consensus.cc:504] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.513298  9177 leader_election.cc:288] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:21.513973  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:21.514562  8977 raft_consensus.cc:3032] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:21.514865  8977 raft_consensus.cc:729] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:21.515166  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a98bf8f3830479b87e617c0103af4c8" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:21.515635  8977 consensus_queue.cc:260] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.515956  9074 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.516978  8977 raft_consensus.cc:3037] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.523985  9074 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.525597  9132 leader_election.cc:302] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:21.526599  9471 raft_consensus.cc:2781] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:21.527529  9471 raft_consensus.cc:686] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:21.527269  8977 raft_consensus.cc:2454] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.528746  9471 consensus_queue.cc:237] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.539183  8834 catalog_manager.cc:5495] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 2 to 3, leader changed from f6964fd5f558447d99495c9e69df90c1 (127.21.255.1) to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 3 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:21.646504  9451 raft_consensus.cc:979] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: : Instructing follower 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.647369  9513 raft_consensus.cc:1067] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Signalling peer 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.649439  9177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4eab99868842422583b64aa796efc47f"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:44884
I20260809 23:24:21.650094  9177 raft_consensus.cc:482] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:21.650460  9177 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.658509  9177 raft_consensus.cc:504] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.660320  9177 leader_election.cc:288] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:21.661072  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:21.661806  8977 raft_consensus.cc:3032] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:21.662181  8977 raft_consensus.cc:729] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f6964fd5f558447d99495c9e69df90c1, State: Running, Role: LEADER
I20260809 23:24:21.662748  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eab99868842422583b64aa796efc47f" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:21.663774  9074 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.663300  8977 consensus_queue.cc:260] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.664841  8977 raft_consensus.cc:3037] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.672660  9074 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.673998  9132 leader_election.cc:302] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 
I20260809 23:24:21.674141  8977 raft_consensus.cc:2454] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.675370  9577 raft_consensus.cc:2781] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:21.675930  9577 raft_consensus.cc:686] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:21.676964  9577 consensus_queue.cc:237] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.684263  8834 catalog_manager.cc:5495] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 2 to 3, leader changed from f6964fd5f558447d99495c9e69df90c1 (127.21.255.1) to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 3 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:21.714439  9472 raft_consensus.cc:979] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: : Instructing follower 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.715159  9475 raft_consensus.cc:1067] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Signalling peer 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.716773  9177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:44870
I20260809 23:24:21.717290  9177 raft_consensus.cc:482] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:21.717595  9177 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.723074  9177 raft_consensus.cc:504] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.724843  9177 leader_election.cc:288] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:21.725685  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:21.725943  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50bb7ef52c74bb499ef218f4aeaf550" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:21.726408  8977 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.726557  9074 raft_consensus.cc:3032] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:21.726961  9074 raft_consensus.cc:729] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:21.727831  9074 consensus_queue.cc:260] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.729657  9074 raft_consensus.cc:3037] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.735824  8977 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.737254  9134 leader_election.cc:302] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:21.738329  9577 raft_consensus.cc:2781] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:21.738868  9577 raft_consensus.cc:686] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:21.738822  9074 raft_consensus.cc:2454] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.739933  9577 consensus_queue.cc:237] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.751077  8834 catalog_manager.cc:5495] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 2 to 3, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 3 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:21.782663  9472 raft_consensus.cc:979] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: : Instructing follower 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.783205  9475 raft_consensus.cc:1067] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Signalling peer 3d0a6b0fd63d4091a84bb1215772b32d to start an election
I20260809 23:24:21.784672  9177 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58d56d427cdd455c997502702b5c752b"
dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d"
 from {username='slave'} at 127.0.0.1:44870
I20260809 23:24:21.785259  9177 raft_consensus.cc:482] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:21.785740  9177 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.791813  9177 raft_consensus.cc:504] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.793310  9177 leader_election.cc:288] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Requested vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:21.794121  8977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f6964fd5f558447d99495c9e69df90c1"
I20260809 23:24:21.794304  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58d56d427cdd455c997502702b5c752b" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "19446675307f4e709bb1c63b9c529e54"
I20260809 23:24:21.794818  8977 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.794919  9074 raft_consensus.cc:3032] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:21.795408  9074 raft_consensus.cc:729] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 19446675307f4e709bb1c63b9c529e54, State: Running, Role: LEADER
I20260809 23:24:21.796096  9074 consensus_queue.cc:260] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.797127  9074 raft_consensus.cc:3037] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:21.800789  8977 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.802198  9134 leader_election.cc:302] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d, f6964fd5f558447d99495c9e69df90c1; no voters: 
I20260809 23:24:21.803033  9577 raft_consensus.cc:2781] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:21.803459  9074 raft_consensus.cc:2454] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0a6b0fd63d4091a84bb1215772b32d in term 3.
I20260809 23:24:21.803493  9577 raft_consensus.cc:686] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Becoming Leader. State: Replica: 3d0a6b0fd63d4091a84bb1215772b32d, State: Running, Role: LEADER
I20260809 23:24:21.804665  9577 consensus_queue.cc:237] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:21.812711  8834 catalog_manager.cc:5495] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d reported cstate change: term changed from 2 to 3, leader changed from 19446675307f4e709bb1c63b9c529e54 (127.21.255.2) to 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3). New cstate: current_term: 3 leader_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:22.133404  9080 raft_consensus.cc:1261] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:24:22.135385  9577 consensus_queue.cc:1035] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:24:22.146673  8977 raft_consensus.cc:1261] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:24:22.148259  9591 consensus_queue.cc:1035] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:24:22.150359  8977 raft_consensus.cc:1261] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:24:22.151911  9577 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:24:22.168917  9080 raft_consensus.cc:1261] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260809 23:24:22.172223  9471 consensus_queue.cc:1035] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260809 23:24:22.174392  9080 raft_consensus.cc:1261] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:22.177304  9591 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:22.192489  8976 raft_consensus.cc:1261] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:22.194401  9598 consensus_queue.cc:1035] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260809 23:24:22.287572  8976 raft_consensus.cc:1261] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:22.289402  9074 raft_consensus.cc:1261] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Refusing update from remote peer 3d0a6b0fd63d4091a84bb1215772b32d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:22.289307  9471 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:22.291035  9471 consensus_queue.cc:1035] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:22.379673 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:22.388248 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:22.388857 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:22.393720 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:22.394309 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:23.395313 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:23.402227 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:23.402757 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:23.408694 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:23.409214 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:24.410696 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:24.418839 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:24.419354 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:24.426220 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:24.426823 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:25.428090 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:25.433977 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:25.434446 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:25.439826 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:25.440359 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:26.441735 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:26.446671 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:26.447120 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:26.451833 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:26.452196 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:27.453282 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:27.460964 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:27.461565 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:27.467559 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:27.468269 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:28.469455 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:28.475405 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:28.475903 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:28.481590 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:28.482195 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:29.536829 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:30.234539 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:30.235743 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:30.245829 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:30.247450 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:30.379967  9627 raft_consensus.cc:482] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 19446675307f4e709bb1c63b9c529e54)
I20260809 23:24:30.380882  9627 raft_consensus.cc:504] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:30.383921  9627 leader_election.cc:288] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705), 3d0a6b0fd63d4091a84bb1215772b32d (127.21.255.3:33703)
I20260809 23:24:30.406733  9177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" is_pre_election: true
I20260809 23:24:30.408960  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5a1382dea7240848a159b9a6b6e82ab" candidate_uuid: "f6964fd5f558447d99495c9e69df90c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:30.416460  8930 leader_election.cc:302] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6964fd5f558447d99495c9e69df90c1; no voters: 19446675307f4e709bb1c63b9c529e54, 3d0a6b0fd63d4091a84bb1215772b32d
I20260809 23:24:30.417177  9627 raft_consensus.cc:2726] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260809 23:24:30.464736  8877 debug-util.cc:398] Leaking SignalData structure 0x7b080002f300 after lost signal to thread 22536
W20260809 23:24:30.465950  8877 debug-util.cc:398] Leaking SignalData structure 0x7b0800034fe0 after lost signal to thread 8880
W20260809 23:24:30.467124  8877 debug-util.cc:398] Leaking SignalData structure 0x7b08001db860 after lost signal to thread 9009
W20260809 23:24:30.468200  8877 debug-util.cc:398] Leaking SignalData structure 0x7b0800211520 after lost signal to thread 9115
W20260809 23:24:30.469262  8877 debug-util.cc:398] Leaking SignalData structure 0x7b080015d420 after lost signal to thread 9211
I20260809 23:24:30.938215  9640 raft_consensus.cc:482] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f6964fd5f558447d99495c9e69df90c1)
I20260809 23:24:30.939359  9640 raft_consensus.cc:504] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6964fd5f558447d99495c9e69df90c1" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 32919 } } peers { permanent_uuid: "19446675307f4e709bb1c63b9c529e54" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 44705 } } peers { permanent_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 33703 } }
I20260809 23:24:30.943750  9640 leader_election.cc:288] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919), 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705)
I20260809 23:24:30.944509  8976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f6964fd5f558447d99495c9e69df90c1" is_pre_election: true
I20260809 23:24:30.945016  9074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65773d68820e48a1a6ba7970141ed22c" candidate_uuid: "3d0a6b0fd63d4091a84bb1215772b32d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "19446675307f4e709bb1c63b9c529e54" is_pre_election: true
I20260809 23:24:30.948186  9132 leader_election.cc:302] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d0a6b0fd63d4091a84bb1215772b32d; no voters: 19446675307f4e709bb1c63b9c529e54, f6964fd5f558447d99495c9e69df90c1
I20260809 23:24:30.953217  9640 raft_consensus.cc:2726] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260809 23:24:31.248472 22524 auto_leader_rebalancer.cc:140] leader rebalance for table soft_deleted_table
I20260809 23:24:31.254417 22524 auto_leader_rebalancer.cc:388] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260809 23:24:31.255023 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:31.260596 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:31.261008 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:32.332449 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
W20260809 23:24:32.365489  9032 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 131 similar messages]
W20260809 23:24:32.386721  9134 consensus_peers.cc:487] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:32.391993  9032 consensus_peers.cc:487] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.401651 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:32.405658 22524 tablet_replica.cc:330] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.407645 22524 raft_consensus.cc:2229] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.408797 22524 raft_consensus.cc:2258] T e50bb7ef52c74bb499ef218f4aeaf550 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.413256 22524 tablet_replica.cc:330] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.414281 22524 raft_consensus.cc:2229] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.415890 22524 raft_consensus.cc:2258] T 01ea2a0690c3432cb878ca2e5baec036 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.419962 22524 tablet_replica.cc:330] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.421217 22524 raft_consensus.cc:2229] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.422226 22524 raft_consensus.cc:2258] T 6b6cb7380e164263b4d0085648b14066 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.426478 22524 tablet_replica.cc:330] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.427528 22524 raft_consensus.cc:2229] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.428946 22524 raft_consensus.cc:2258] T 7f00e6101a1647a4acd59b961be1c4ef P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.432626 22524 tablet_replica.cc:330] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.433614 22524 raft_consensus.cc:2229] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.434379 22524 raft_consensus.cc:2258] T 3a98bf8f3830479b87e617c0103af4c8 P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.438506 22524 tablet_replica.cc:330] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.439347 22524 raft_consensus.cc:2229] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.440094 22524 raft_consensus.cc:2258] T 58d56d427cdd455c997502702b5c752b P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.443605 22524 tablet_replica.cc:330] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.444478 22524 raft_consensus.cc:2229] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.445464 22524 raft_consensus.cc:2258] T c4b3fee9702a41c5b46e3d07db21f6ae P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.449944 22524 tablet_replica.cc:330] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.451064 22524 raft_consensus.cc:2229] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.452543 22524 raft_consensus.cc:2258] T afdc11e9bcc643a9ad27e9ec2ee34cff P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.456247 22524 tablet_replica.cc:330] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.457728 22524 raft_consensus.cc:2229] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.459335 22524 raft_consensus.cc:2258] T 65773d68820e48a1a6ba7970141ed22c P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.463858 22524 tablet_replica.cc:330] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.464694 22524 raft_consensus.cc:2229] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.465301 22524 raft_consensus.cc:2258] T b062a667501b46ceae2ac26b45ee1149 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.468832 22524 tablet_replica.cc:330] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.469624 22524 raft_consensus.cc:2229] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.470368 22524 raft_consensus.cc:2258] T 82bbc56ce55a49a7b1607889482c1dc4 P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.475137 22524 tablet_replica.cc:330] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.476236 22524 raft_consensus.cc:2229] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.477944 22524 raft_consensus.cc:2258] T 91c5267ebb5645269e7259a7bc69e1f9 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.481961 22524 tablet_replica.cc:330] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
W20260809 23:24:32.481998  9032 consensus_peers.cc:487] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.482887 22524 raft_consensus.cc:2229] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.483757 22524 raft_consensus.cc:2258] T 745c983bc5294452934b5f19db51906e P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.487712 22524 tablet_replica.cc:330] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.488533 22524 raft_consensus.cc:2229] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.489342 22524 raft_consensus.cc:2258] T 85ee34544fb94c6ba00a4b152119564a P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.493069 22524 tablet_replica.cc:330] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.493902 22524 raft_consensus.cc:2229] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.494601 22524 raft_consensus.cc:2258] T 4bc5a93704de4fafad83971c539f2427 P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.498162 22524 tablet_replica.cc:330] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.499044 22524 raft_consensus.cc:2229] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.499607 22524 raft_consensus.cc:2258] T 4eab99868842422583b64aa796efc47f P f6964fd5f558447d99495c9e69df90c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.503684 22524 tablet_replica.cc:330] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.504451 22524 raft_consensus.cc:2229] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.505018 22524 raft_consensus.cc:2258] T a5a1382dea7240848a159b9a6b6e82ab P f6964fd5f558447d99495c9e69df90c1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:32.505175  9032 consensus_peers.cc:487] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.508244 22524 tablet_replica.cc:330] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1: stopping tablet replica
I20260809 23:24:32.508908 22524 raft_consensus.cc:2229] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.509977 22524 raft_consensus.cc:2258] T 92a11e232e63467d81e6958e7d1335dc P f6964fd5f558447d99495c9e69df90c1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:32.546162  9134 consensus_peers.cc:487] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:32.562249  9134 consensus_peers.cc:487] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:32919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:32.563841  9134 consensus_peers.cc:487] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:32919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.573662 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
I20260809 23:24:32.629858 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
W20260809 23:24:32.667037  9132 consensus_peers.cc:487] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:32.667954  9132 consensus_peers.cc:487] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:32.683208  9032 consensus_peers.cc:487] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:32919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.688313 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:32.689262 22524 tablet_replica.cc:330] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.690378 22524 raft_consensus.cc:2229] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.691481 22524 raft_consensus.cc:2258] T 6b6cb7380e164263b4d0085648b14066 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.694747 22524 tablet_replica.cc:330] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.695492 22524 raft_consensus.cc:2229] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.696326 22524 raft_consensus.cc:2258] T 7f00e6101a1647a4acd59b961be1c4ef P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:32.698794  9132 consensus_peers.cc:487] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.699674 22524 tablet_replica.cc:330] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.700470 22524 raft_consensus.cc:2229] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.701123 22524 raft_consensus.cc:2258] T e50bb7ef52c74bb499ef218f4aeaf550 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.703709 22524 tablet_replica.cc:330] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.704363 22524 raft_consensus.cc:2229] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.705004 22524 raft_consensus.cc:2258] T 01ea2a0690c3432cb878ca2e5baec036 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.707473 22524 tablet_replica.cc:330] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.708128 22524 raft_consensus.cc:2229] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.708662 22524 raft_consensus.cc:2258] T 58d56d427cdd455c997502702b5c752b P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.711354 22524 tablet_replica.cc:330] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.712113 22524 raft_consensus.cc:2229] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.712774 22524 raft_consensus.cc:2258] T 3a98bf8f3830479b87e617c0103af4c8 P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.715824 22524 tablet_replica.cc:330] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.716570 22524 raft_consensus.cc:2229] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.717669 22524 raft_consensus.cc:2258] T c4b3fee9702a41c5b46e3d07db21f6ae P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:32.718884  9132 consensus_peers.cc:487] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.720960 22524 tablet_replica.cc:330] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.721632 22524 raft_consensus.cc:2229] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.722193 22524 raft_consensus.cc:2258] T 65773d68820e48a1a6ba7970141ed22c P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.725589 22524 tablet_replica.cc:330] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.726413 22524 raft_consensus.cc:2229] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260809 23:24:32.726890  9134 consensus_peers.cc:487] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:32919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.727069 22524 raft_consensus.cc:2258] T afdc11e9bcc643a9ad27e9ec2ee34cff P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.730283 22524 tablet_replica.cc:330] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.731137 22524 raft_consensus.cc:2229] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.731683 22524 raft_consensus.cc:2258] T b062a667501b46ceae2ac26b45ee1149 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.734453 22524 tablet_replica.cc:330] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.735152 22524 raft_consensus.cc:2229] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.735862 22524 raft_consensus.cc:2258] T 92a11e232e63467d81e6958e7d1335dc P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.739943 22524 tablet_replica.cc:330] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.740717 22524 raft_consensus.cc:2229] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.741688 22524 raft_consensus.cc:2258] T a5a1382dea7240848a159b9a6b6e82ab P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.744537 22524 tablet_replica.cc:330] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.745074 22524 raft_consensus.cc:2229] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.745643 22524 raft_consensus.cc:2258] T 4eab99868842422583b64aa796efc47f P 19446675307f4e709bb1c63b9c529e54 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.747888 22524 tablet_replica.cc:330] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.748428 22524 raft_consensus.cc:2229] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.748979 22524 raft_consensus.cc:2258] T 82bbc56ce55a49a7b1607889482c1dc4 P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.751477 22524 tablet_replica.cc:330] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.752004 22524 raft_consensus.cc:2229] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.752938 22524 raft_consensus.cc:2258] T 85ee34544fb94c6ba00a4b152119564a P 19446675307f4e709bb1c63b9c529e54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.755506 22524 tablet_replica.cc:330] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.756549 22524 raft_consensus.cc:2229] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.757701 22524 raft_consensus.cc:2258] T 745c983bc5294452934b5f19db51906e P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.760608 22524 tablet_replica.cc:330] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.761127 22524 raft_consensus.cc:2229] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.761655 22524 raft_consensus.cc:2258] T 91c5267ebb5645269e7259a7bc69e1f9 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.763906 22524 tablet_replica.cc:330] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54: stopping tablet replica
I20260809 23:24:32.764370 22524 raft_consensus.cc:2229] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.765287 22524 raft_consensus.cc:2258] T 4bc5a93704de4fafad83971c539f2427 P 19446675307f4e709bb1c63b9c529e54 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:32.779677  9134 consensus_peers.cc:487] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer f6964fd5f558447d99495c9e69df90c1 (127.21.255.1:32919): Couldn't send request to peer f6964fd5f558447d99495c9e69df90c1. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:32919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.820974 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
I20260809 23:24:32.886574 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
W20260809 23:24:32.937492  9132 consensus_peers.cc:487] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:44705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.945964 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:32.946810 22524 tablet_replica.cc:330] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.948062 22524 raft_consensus.cc:2229] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.949007 22524 raft_consensus.cc:2258] T 6b6cb7380e164263b4d0085648b14066 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.952026 22524 tablet_replica.cc:330] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.952973 22524 raft_consensus.cc:2229] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.953717 22524 raft_consensus.cc:2258] T 7f00e6101a1647a4acd59b961be1c4ef P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.957742 22524 tablet_replica.cc:330] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.958868 22524 raft_consensus.cc:2229] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.959688 22524 raft_consensus.cc:2258] T 01ea2a0690c3432cb878ca2e5baec036 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.962844 22524 tablet_replica.cc:330] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.963673 22524 raft_consensus.cc:2229] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Raft consensus shutting down.
W20260809 23:24:32.964308  9132 consensus_peers.cc:487] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d -> Peer 19446675307f4e709bb1c63b9c529e54 (127.21.255.2:44705): Couldn't send request to peer 19446675307f4e709bb1c63b9c529e54. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:44705: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:32.964777 22524 raft_consensus.cc:2258] T e50bb7ef52c74bb499ef218f4aeaf550 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.967514 22524 tablet_replica.cc:330] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.968159 22524 raft_consensus.cc:2229] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.969064 22524 raft_consensus.cc:2258] T 58d56d427cdd455c997502702b5c752b P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.971495 22524 tablet_replica.cc:330] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.972191 22524 raft_consensus.cc:2229] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.973611 22524 raft_consensus.cc:2258] T 3a98bf8f3830479b87e617c0103af4c8 P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.977303 22524 tablet_replica.cc:330] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.978160 22524 raft_consensus.cc:2229] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.978963 22524 raft_consensus.cc:2258] T 4bc5a93704de4fafad83971c539f2427 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.981520 22524 tablet_replica.cc:330] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.982056 22524 raft_consensus.cc:2229] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.982583 22524 raft_consensus.cc:2258] T c4b3fee9702a41c5b46e3d07db21f6ae P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.984965 22524 tablet_replica.cc:330] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.985532 22524 raft_consensus.cc:2229] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.986039 22524 raft_consensus.cc:2258] T afdc11e9bcc643a9ad27e9ec2ee34cff P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.988283 22524 tablet_replica.cc:330] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.988787 22524 raft_consensus.cc:2229] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:32.989784 22524 raft_consensus.cc:2258] T b062a667501b46ceae2ac26b45ee1149 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.993742 22524 tablet_replica.cc:330] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.994976 22524 raft_consensus.cc:2229] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:32.995616 22524 raft_consensus.cc:2258] T 65773d68820e48a1a6ba7970141ed22c P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:32.998807 22524 tablet_replica.cc:330] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:32.999500 22524 raft_consensus.cc:2229] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:33.000419 22524 raft_consensus.cc:2258] T 4eab99868842422583b64aa796efc47f P 3d0a6b0fd63d4091a84bb1215772b32d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.002624 22524 tablet_replica.cc:330] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.003230 22524 raft_consensus.cc:2229] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:33.003708 22524 raft_consensus.cc:2258] T a5a1382dea7240848a159b9a6b6e82ab P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.006052 22524 tablet_replica.cc:330] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.006614 22524 raft_consensus.cc:2229] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:33.007197 22524 raft_consensus.cc:2258] T 92a11e232e63467d81e6958e7d1335dc P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.009963 22524 tablet_replica.cc:330] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.010589 22524 raft_consensus.cc:2229] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:33.011711 22524 raft_consensus.cc:2258] T 82bbc56ce55a49a7b1607889482c1dc4 P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.014580 22524 tablet_replica.cc:330] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.015609 22524 raft_consensus.cc:2229] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:33.016134 22524 raft_consensus.cc:2258] T 85ee34544fb94c6ba00a4b152119564a P 3d0a6b0fd63d4091a84bb1215772b32d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.019747 22524 tablet_replica.cc:330] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.020637 22524 raft_consensus.cc:2229] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:33.021430 22524 raft_consensus.cc:2258] T 91c5267ebb5645269e7259a7bc69e1f9 P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.024753 22524 tablet_replica.cc:330] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d: stopping tablet replica
I20260809 23:24:33.025269 22524 raft_consensus.cc:2229] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:33.025802 22524 raft_consensus.cc:2258] T 745c983bc5294452934b5f19db51906e P 3d0a6b0fd63d4091a84bb1215772b32d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.095592 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:24:33.144282 22524 master.cc:537] Master@127.21.255.62:41225 shutting down...
I20260809 23:24:33.177753 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:33.178836 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:33.179360 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P 020a52ca66bf4a679112f70554a4411e: stopping tablet replica
I20260809 23:24:33.981689 22524 master.cc:552] Master@127.21.255.62:41225 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (22929 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260809 23:24:34.070468 22524 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.21.255.62:44395
I20260809 23:24:34.072055 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:34.079126  9659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.079558  9657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.080853  9661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:34.080964 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:34.082474 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:34.082674 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:34.082882 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317874082868 us; error 0 us; skew 500 ppm
I20260809 23:24:34.083487 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:34.086478 22524 webserver.cc:460] Webserver started at http://127.21.255.62:37827/ using document root <none> and password file <none>
I20260809 23:24:34.087150 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:34.087327 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:34.087702 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:34.089118 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/master-0-root/instance:
uuid: "f6d2ca9e8184414a8d9f866f81943aeb"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.093930 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20260809 23:24:34.097806  9668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.098765 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260809 23:24:34.099098 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/master-0-root
uuid: "f6d2ca9e8184414a8d9f866f81943aeb"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.099407 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:34.118765 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:34.120433 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:34.170010 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.62:44395
I20260809 23:24:34.170147  9740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.62:44395 every 8 connection(s)
I20260809 23:24:34.174413  9741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.193332  9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: Bootstrap starting.
I20260809 23:24:34.199303  9741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:34.204094  9741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: No bootstrap required, opened a new log
I20260809 23:24:34.206346  9741 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER }
I20260809 23:24:34.206866  9741 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:34.207250  9741 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2ca9e8184414a8d9f866f81943aeb, State: Initialized, Role: FOLLOWER
I20260809 23:24:34.207998  9741 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER }
I20260809 23:24:34.208448  9741 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 23:24:34.208747  9741 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 23:24:34.209038  9741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:34.214098  9741 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER }
I20260809 23:24:34.214641  9741 leader_election.cc:302] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d2ca9e8184414a8d9f866f81943aeb; no voters: 
I20260809 23:24:34.215900  9741 leader_election.cc:288] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260809 23:24:34.216413  9745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:34.218003  9745 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 1 LEADER]: Becoming Leader. State: Replica: f6d2ca9e8184414a8d9f866f81943aeb, State: Running, Role: LEADER
I20260809 23:24:34.218904  9745 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER }
I20260809 23:24:34.219087  9741 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260809 23:24:34.226703  9746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER } }
I20260809 23:24:34.227442  9748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d2ca9e8184414a8d9f866f81943aeb. Latest consensus state: current_term: 1 leader_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d2ca9e8184414a8d9f866f81943aeb" member_type: VOTER } }
I20260809 23:24:34.227639  9746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [sys.catalog]: This master's current role is: LEADER
I20260809 23:24:34.228312  9748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [sys.catalog]: This master's current role is: LEADER
I20260809 23:24:34.230265  9761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 23:24:34.235309  9761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 23:24:34.237811 22524 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260809 23:24:34.244765  9761 catalog_manager.cc:1347] Generated new cluster ID: 840fb3cd3e234af2b9917ff329ab087d
I20260809 23:24:34.245054  9761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 23:24:34.284718  9761 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 23:24:34.287388  9761 catalog_manager.cc:1504] Loading token signing keys...
I20260809 23:24:34.304009  9761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: Generated new TSK 0
I20260809 23:24:34.304726  9761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 23:24:34.368938 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:34.377827  9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.378033  9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.379647  9775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:34.380697 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:34.381779 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:34.381960 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:34.382150 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317874382138 us; error 0 us; skew 500 ppm
I20260809 23:24:34.382803 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:34.385525 22524 webserver.cc:460] Webserver started at http://127.21.255.1:33645/ using document root <none> and password file <none>
I20260809 23:24:34.386055 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:34.386221 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:34.386544 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:34.388007 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-0-root/instance:
uuid: "1da1957992c74e29b5f09aeb64fd88fd"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.393600 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260809 23:24:34.398027  9782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.399161 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:24:34.399581 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-0-root
uuid: "1da1957992c74e29b5f09aeb64fd88fd"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.400043 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:34.433357 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:34.434540 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:34.436189 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:34.438880 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:34.439131 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.439420 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:34.439581 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.490921 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.1:43405
I20260809 23:24:34.491259  9871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.1:43405 every 8 connection(s)
I20260809 23:24:34.496605 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:34.505941  9876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.508152  9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.511758  9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:34.511852 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:34.513338 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260809 23:24:34.513537 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:34.513756 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317874513726 us; error 0 us; skew 500 ppm
I20260809 23:24:34.514376 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:34.516816  9872 heartbeater.cc:347] Connected to a master server at 127.21.255.62:44395
I20260809 23:24:34.517239  9872 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:34.518123 22524 webserver.cc:460] Webserver started at http://127.21.255.2:45601/ using document root <none> and password file <none>
I20260809 23:24:34.518206  9872 heartbeater.cc:511] Master 127.21.255.62:44395 requested a full tablet report, sending...
I20260809 23:24:34.518795 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:34.519073 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:34.519381 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:34.521384 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-1-root/instance:
uuid: "2df9f9c44dc3408c8d475c31e28684b4"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.521468  9691 ts_manager.cc:194] Registered new tserver with Master: 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405)
I20260809 23:24:34.524425  9691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35378
I20260809 23:24:34.528142 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:24:34.532606  9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.533646 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260809 23:24:34.533960 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-1-root
uuid: "2df9f9c44dc3408c8d475c31e28684b4"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.534322 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:34.562618 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:34.564159 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:34.565956 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:34.568506 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:34.568771 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.569043 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:34.569235 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.621162 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.2:33409
I20260809 23:24:34.621272  9975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.2:33409 every 8 connection(s)
I20260809 23:24:34.626801 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 23:24:34.637805  9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.638568  9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 23:24:34.642518  9986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 23:24:34.646804 22524 server_base.cc:1034] running on GCE node
I20260809 23:24:34.648447 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260809 23:24:34.648545  9977 heartbeater.cc:347] Connected to a master server at 127.21.255.62:44395
W20260809 23:24:34.648890 22524 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260809 23:24:34.649199 22524 hybrid_clock.cc:648] HybridClock initialized: now 1786317874649178 us; error 0 us; skew 500 ppm
I20260809 23:24:34.649327  9977 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:34.650151 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 23:24:34.650249  9977 heartbeater.cc:511] Master 127.21.255.62:44395 requested a full tablet report, sending...
I20260809 23:24:34.653596  9691 ts_manager.cc:194] Registered new tserver with Master: 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:34.654351 22524 webserver.cc:460] Webserver started at http://127.21.255.3:33959/ using document root <none> and password file <none>
I20260809 23:24:34.654978 22524 fs_manager.cc:359] Metadata directory not provided
I20260809 23:24:34.655187 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 23:24:34.655488 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 23:24:34.655727  9691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35386
I20260809 23:24:34.657312 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-2-root/instance:
uuid: "8a2478dec1844a5481141d44f3136f26"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.662472 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260809 23:24:34.666512  9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.667693 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260809 23:24:34.668026 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-2-root
uuid: "8a2478dec1844a5481141d44f3136f26"
format_stamp: "Formatted at 2026-08-09 23:24:34 on dist-test-slave-drl0"
I20260809 23:24:34.668359 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskSGCZ7H/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1786317384654715-22524-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 23:24:34.683777 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips  26 Jan 2017
I20260809 23:24:34.684995 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 23:24:34.686740 22524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 23:24:34.689117 22524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 23:24:34.689342 22524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.689580 22524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 23:24:34.689749 22524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260809 23:24:34.748555 22524 rpc_server.cc:307] RPC server started. Bound to: 127.21.255.3:40291
I20260809 23:24:34.748684 10079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.21.255.3:40291 every 8 connection(s)
I20260809 23:24:34.767524 10080 heartbeater.cc:347] Connected to a master server at 127.21.255.62:44395
I20260809 23:24:34.768114 10080 heartbeater.cc:464] Registering TS with master...
I20260809 23:24:34.769152 10080 heartbeater.cc:511] Master 127.21.255.62:44395 requested a full tablet report, sending...
I20260809 23:24:34.771976  9691 ts_manager.cc:194] Registered new tserver with Master: 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:34.772670 22524 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.020565241s
I20260809 23:24:34.774296  9691 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35402
I20260809 23:24:34.776197 22524 test_util.cc:268] Using random seed: -1973303182
I20260809 23:24:34.806634  9691 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35406:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:24:34.809731  9691 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:24:34.913151  9818 tablet_service.cc:1467] Processing CreateTablet for tablet 940f2aa4051d42d6aad243f3024117d6 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:34.914979  9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940f2aa4051d42d6aad243f3024117d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.915681  9817 tablet_service.cc:1467] Processing CreateTablet for tablet 30c044125e014a6caf1be9b452d64b76 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:34.916083 10028 tablet_service.cc:1467] Processing CreateTablet for tablet 940f2aa4051d42d6aad243f3024117d6 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:34.916874  9817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c044125e014a6caf1be9b452d64b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.918403 10028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940f2aa4051d42d6aad243f3024117d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.919854  9816 tablet_service.cc:1467] Processing CreateTablet for tablet 18ac43ba34c54f97877eee12223f43ce (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:34.921104  9816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ac43ba34c54f97877eee12223f43ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.921878  9814 tablet_service.cc:1467] Processing CreateTablet for tablet 348dba4300304e79a88b264279c3358e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:34.924170 10026 tablet_service.cc:1467] Processing CreateTablet for tablet 18ac43ba34c54f97877eee12223f43ce (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:34.925585 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ac43ba34c54f97877eee12223f43ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.925397 10025 tablet_service.cc:1467] Processing CreateTablet for tablet 0a48b130b22647e7b52f5f9f777896e3 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:34.927713  9813 tablet_service.cc:1467] Processing CreateTablet for tablet be3d5a988cf147e19133130c67fa914e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:34.928316  9815 tablet_service.cc:1467] Processing CreateTablet for tablet 0a48b130b22647e7b52f5f9f777896e3 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:34.931133  9812 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd6557129314bd59fd84cdeed3211bf (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:34.929373  9813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3d5a988cf147e19133130c67fa914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.935237  9924 tablet_service.cc:1467] Processing CreateTablet for tablet 30c044125e014a6caf1be9b452d64b76 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:34.936323  9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a48b130b22647e7b52f5f9f777896e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.936854  9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c044125e014a6caf1be9b452d64b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.926780 10025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a48b130b22647e7b52f5f9f777896e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.942827  9922 tablet_service.cc:1467] Processing CreateTablet for tablet 18ac43ba34c54f97877eee12223f43ce (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:34.943575  9814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348dba4300304e79a88b264279c3358e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.944332  9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ac43ba34c54f97877eee12223f43ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.944963  9810 tablet_service.cc:1467] Processing CreateTablet for tablet b609b47481a74aa58f03382178c4adad (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:34.946355  9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b609b47481a74aa58f03382178c4adad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.927369  9925 tablet_service.cc:1467] Processing CreateTablet for tablet 940f2aa4051d42d6aad243f3024117d6 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:34.950043  9925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940f2aa4051d42d6aad243f3024117d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.951678  9809 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6a3113d7b540a18727995d759a4ada (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:34.952046  9812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd6557129314bd59fd84cdeed3211bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.932765 10023 tablet_service.cc:1467] Processing CreateTablet for tablet 348dba4300304e79a88b264279c3358e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:34.955416  9809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6a3113d7b540a18727995d759a4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.954483 10022 tablet_service.cc:1467] Processing CreateTablet for tablet be3d5a988cf147e19133130c67fa914e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:34.957283 10021 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd6557129314bd59fd84cdeed3211bf (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:34.960558 10020 tablet_service.cc:1467] Processing CreateTablet for tablet b609b47481a74aa58f03382178c4adad (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:34.920331 10027 tablet_service.cc:1467] Processing CreateTablet for tablet 30c044125e014a6caf1be9b452d64b76 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:34.969901 10019 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6a3113d7b540a18727995d759a4ada (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:34.972527  9920 tablet_service.cc:1467] Processing CreateTablet for tablet 0a48b130b22647e7b52f5f9f777896e3 (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:34.974040  9920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a48b130b22647e7b52f5f9f777896e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.978736  9919 tablet_service.cc:1467] Processing CreateTablet for tablet 348dba4300304e79a88b264279c3358e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:34.980677  9919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348dba4300304e79a88b264279c3358e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.988731  9917 tablet_service.cc:1467] Processing CreateTablet for tablet be3d5a988cf147e19133130c67fa914e (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:34.990262  9917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3d5a988cf147e19133130c67fa914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:34.993386  9916 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd6557129314bd59fd84cdeed3211bf (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:35.002220  9915 tablet_service.cc:1467] Processing CreateTablet for tablet b609b47481a74aa58f03382178c4adad (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:35.005645  9913 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6a3113d7b540a18727995d759a4ada (DEFAULT_TABLE table=test-workload [id=c136110ae14743c1a05c1187099ef78a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:34.955904 10023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348dba4300304e79a88b264279c3358e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.015432 10021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd6557129314bd59fd84cdeed3211bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.021638 10022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be3d5a988cf147e19133130c67fa914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.006297  9916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd6557129314bd59fd84cdeed3211bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.035004 10019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6a3113d7b540a18727995d759a4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.045137 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b609b47481a74aa58f03382178c4adad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.047677 10027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c044125e014a6caf1be9b452d64b76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.068413  9913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6a3113d7b540a18727995d759a4ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.127524  9915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b609b47481a74aa58f03382178c4adad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:35.142369 10101 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.158707 10101 tablet_bootstrap.cc:654] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.212363 10102 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.232836 10103 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
W20260809 23:24:35.233999  9759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:35.246397 10101 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.247035 10101 ts_tablet_manager.cc:1397] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.105s	user 0.028s	sys 0.042s
I20260809 23:24:35.248281 10103 tablet_bootstrap.cc:654] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.250135 10101 raft_consensus.cc:348] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.251015 10101 raft_consensus.cc:374] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.251317 10101 raft_consensus.cc:729] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.252182 10101 consensus_queue.cc:260] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.261582 10103 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.262169 10103 ts_tablet_manager.cc:1397] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.003s
I20260809 23:24:35.265317 10102 tablet_bootstrap.cc:654] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.267925  9872 heartbeater.cc:503] Master 127.21.255.62:44395 was elected leader, sending a full tablet report...
I20260809 23:24:35.265218 10103 raft_consensus.cc:348] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.268715 10103 raft_consensus.cc:374] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.269070 10103 raft_consensus.cc:729] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.269845 10103 consensus_queue.cc:260] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.272848 10080 heartbeater.cc:503] Master 127.21.255.62:44395 was elected leader, sending a full tablet report...
I20260809 23:24:35.273124 10101 ts_tablet_manager.cc:1428] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260809 23:24:35.273288 10103 ts_tablet_manager.cc:1428] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:24:35.274236 10103 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.275532 10102 tablet_bootstrap.cc:492] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.276031 10102 ts_tablet_manager.cc:1397] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.064s	user 0.008s	sys 0.004s
I20260809 23:24:35.274261 10101 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.281008 10103 tablet_bootstrap.cc:654] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.284783 10101 tablet_bootstrap.cc:654] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.285647 10102 raft_consensus.cc:348] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.286526 10102 raft_consensus.cc:374] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.286842 10102 raft_consensus.cc:729] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.287899 10102 consensus_queue.cc:260] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.293231 10102 ts_tablet_manager.cc:1428] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.001s
I20260809 23:24:35.295356  9977 heartbeater.cc:503] Master 127.21.255.62:44395 was elected leader, sending a full tablet report...
I20260809 23:24:35.297533 10103 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.298252 10103 ts_tablet_manager.cc:1397] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.008s
I20260809 23:24:35.302301 10103 raft_consensus.cc:348] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.303457 10103 raft_consensus.cc:374] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.306788 10103 raft_consensus.cc:729] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.309470 10102 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.310091 10103 consensus_queue.cc:260] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.314415 10103 ts_tablet_manager.cc:1428] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.002s
I20260809 23:24:35.315701 10103 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.319257 10101 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.319849 10102 tablet_bootstrap.cc:654] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.319895 10101 ts_tablet_manager.cc:1397] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.024s
I20260809 23:24:35.322784 10103 tablet_bootstrap.cc:654] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.323334 10101 raft_consensus.cc:348] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.324205 10101 raft_consensus.cc:374] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.324553 10101 raft_consensus.cc:729] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.325510 10101 consensus_queue.cc:260] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.329821 10101 ts_tablet_manager.cc:1428] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260809 23:24:35.330886 10101 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.337924 10101 tablet_bootstrap.cc:654] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.339916 10102 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.340544 10102 ts_tablet_manager.cc:1397] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.031s	user 0.016s	sys 0.011s
I20260809 23:24:35.342384 10103 tablet_bootstrap.cc:492] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.342911 10103 ts_tablet_manager.cc:1397] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.027s	user 0.007s	sys 0.007s
I20260809 23:24:35.343307 10101 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.343750 10101 ts_tablet_manager.cc:1397] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.003s
I20260809 23:24:35.343585 10102 raft_consensus.cc:348] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.344472 10102 raft_consensus.cc:374] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.345297 10102 raft_consensus.cc:729] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.346063 10102 consensus_queue.cc:260] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.346382 10103 raft_consensus.cc:348] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.347143 10103 raft_consensus.cc:374] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.346725 10101 raft_consensus.cc:348] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.347470 10103 raft_consensus.cc:729] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.347607 10101 raft_consensus.cc:374] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.347975 10101 raft_consensus.cc:729] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.348737 10102 ts_tablet_manager.cc:1428] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260809 23:24:35.348294 10103 consensus_queue.cc:260] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.349043 10101 consensus_queue.cc:260] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.351104 10103 ts_tablet_manager.cc:1428] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260809 23:24:35.351707 10101 ts_tablet_manager.cc:1428] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:35.352241 10102 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.353276 10101 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.359274 10102 tablet_bootstrap.cc:654] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.360468 10101 tablet_bootstrap.cc:654] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.352245 10103 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.368093 10101 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.368683 10101 ts_tablet_manager.cc:1397] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.001s
I20260809 23:24:35.371376 10103 tablet_bootstrap.cc:654] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.371500 10101 raft_consensus.cc:348] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.372417 10101 raft_consensus.cc:374] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.372717 10101 raft_consensus.cc:729] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.373561 10101 consensus_queue.cc:260] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.375917 10101 ts_tablet_manager.cc:1428] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260809 23:24:35.377022 10102 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.377987 10102 ts_tablet_manager.cc:1397] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.002s
I20260809 23:24:35.378129 10101 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.381958 10102 raft_consensus.cc:348] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.382831 10102 raft_consensus.cc:374] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.383219 10102 raft_consensus.cc:729] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.384263 10103 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.384127 10102 consensus_queue.cc:260] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.384805 10103 ts_tablet_manager.cc:1397] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.003s
I20260809 23:24:35.385567 10101 tablet_bootstrap.cc:654] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.386745 10102 ts_tablet_manager.cc:1428] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:35.387988 10102 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.391381 10103 raft_consensus.cc:348] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.392199 10103 raft_consensus.cc:374] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.395042 10102 tablet_bootstrap.cc:654] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.395411 10103 raft_consensus.cc:729] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.396322 10103 consensus_queue.cc:260] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.403272 10103 ts_tablet_manager.cc:1428] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260809 23:24:35.403877 10101 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.404217 10102 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.404435 10101 ts_tablet_manager.cc:1397] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.009s
I20260809 23:24:35.404495 10103 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.404918 10102 ts_tablet_manager.cc:1397] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.003s
I20260809 23:24:35.407668 10101 raft_consensus.cc:348] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.408456 10101 raft_consensus.cc:374] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.408763 10101 raft_consensus.cc:729] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.408915 10102 raft_consensus.cc:348] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.409904 10102 raft_consensus.cc:374] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.409595 10101 consensus_queue.cc:260] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.413197 10103 tablet_bootstrap.cc:654] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.410265 10102 raft_consensus.cc:729] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.413268 10101 ts_tablet_manager.cc:1428] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260809 23:24:35.418361 10101 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.417770 10102 consensus_queue.cc:260] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.417809 10108 raft_consensus.cc:482] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.420121 10108 raft_consensus.cc:504] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.430949 10101 tablet_bootstrap.cc:654] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.432421 10103 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.433095 10103 ts_tablet_manager.cc:1397] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.029s	user 0.016s	sys 0.001s
I20260809 23:24:35.437397 10103 raft_consensus.cc:348] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.437937 10108 leader_election.cc:288] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.438271 10103 raft_consensus.cc:374] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.438638 10103 raft_consensus.cc:729] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.439642 10103 consensus_queue.cc:260] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.440948 10101 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.441494 10101 ts_tablet_manager.cc:1397] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.000s
I20260809 23:24:35.442623 10103 ts_tablet_manager.cc:1428] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260809 23:24:35.444507 10103 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.444180 10101 raft_consensus.cc:348] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.446520 10101 raft_consensus.cc:374] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.446863 10101 raft_consensus.cc:729] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.447669 10101 consensus_queue.cc:260] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.449958 10101 ts_tablet_manager.cc:1428] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260809 23:24:35.453522 10103 tablet_bootstrap.cc:654] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.456933 10101 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.453555 10102 ts_tablet_manager.cc:1428] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.000s
I20260809 23:24:35.464515 10110 raft_consensus.cc:482] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.464759 10108 raft_consensus.cc:482] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.465116 10110 raft_consensus.cc:504] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.465500 10108 raft_consensus.cc:504] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.466405 10101 tablet_bootstrap.cc:654] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.472445 10108 leader_election.cc:288] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.474668 10110 leader_election.cc:288] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.479180 10108 raft_consensus.cc:482] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.480029 10102 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.481000 10108 raft_consensus.cc:504] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.485204 10108 leader_election.cc:288] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.487613 10102 tablet_bootstrap.cc:654] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.491420 10103 tablet_bootstrap.cc:492] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.492005 10103 ts_tablet_manager.cc:1397] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.048s	user 0.026s	sys 0.012s
I20260809 23:24:35.495862 10103 raft_consensus.cc:348] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.496930 10103 raft_consensus.cc:374] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.497291 10103 raft_consensus.cc:729] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.498198 10103 consensus_queue.cc:260] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.516408 10103 ts_tablet_manager.cc:1428] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260809 23:24:35.517824 10103 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.525806 10102 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.526402 10102 ts_tablet_manager.cc:1397] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.047s	user 0.018s	sys 0.022s
I20260809 23:24:35.526500 10103 tablet_bootstrap.cc:654] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.529605 10102 raft_consensus.cc:348] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.530627 10102 raft_consensus.cc:374] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.531167 10102 raft_consensus.cc:729] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.532301 10102 consensus_queue.cc:260] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.544945 10102 ts_tablet_manager.cc:1428] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260809 23:24:35.546353 10102 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.555886  9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.556828  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.559818 10102 tablet_bootstrap.cc:654] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.559989  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.560885  9937 raft_consensus.cc:2454] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.561363 10103 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.561913 10103 ts_tablet_manager.cc:1397] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.044s	user 0.011s	sys 0.025s
W20260809 23:24:35.562887  9786 leader_election.cc:341] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.565686 10103 raft_consensus.cc:348] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.571732 10103 raft_consensus.cc:374] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260809 23:24:35.571995  9786 leader_election.cc:341] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.572065 10103 raft_consensus.cc:729] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.569183 10102 tablet_bootstrap.cc:492] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.573673 10102 ts_tablet_manager.cc:1397] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.028s	user 0.006s	sys 0.011s
I20260809 23:24:35.573472 10103 consensus_queue.cc:260] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.575291 10108 raft_consensus.cc:482] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.575955 10108 raft_consensus.cc:504] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.576967 10102 raft_consensus.cc:348] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.578017 10102 raft_consensus.cc:374] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.578316 10102 raft_consensus.cc:729] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.579088 10102 consensus_queue.cc:260] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.579486 10108 leader_election.cc:288] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.581089 10103 ts_tablet_manager.cc:1428] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.004s
I20260809 23:24:35.581566 10102 ts_tablet_manager.cc:1428] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260809 23:24:35.582190 10103 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.582590 10102 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.589344 10102 tablet_bootstrap.cc:654] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.589380  9786 leader_election.cc:302] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:35.591472  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940f2aa4051d42d6aad243f3024117d6" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.592156  9937 raft_consensus.cc:2454] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.592478 10110 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.593343 10108 raft_consensus.cc:2781] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.593142 10110 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.593851 10108 raft_consensus.cc:482] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.594264 10108 raft_consensus.cc:3037] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.595723 10110 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.596398 10110 raft_consensus.cc:482] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.596931 10110 raft_consensus.cc:504] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.603207 10110 leader_election.cc:288] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.603557 10102 tablet_bootstrap.cc:492] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.610127 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.610554 10102 ts_tablet_manager.cc:1397] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.028s	user 0.009s	sys 0.006s
I20260809 23:24:35.611019 10042 raft_consensus.cc:2454] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.612389 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.612988 10042 raft_consensus.cc:2454] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.613199 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.613873 10041 raft_consensus.cc:2454] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.613673 10102 raft_consensus.cc:348] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.614528 10102 raft_consensus.cc:374] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.606037 10103 tablet_bootstrap.cc:654] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.608647 10124 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.614868 10102 raft_consensus.cc:729] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.609416 10101 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.615370  9785 leader_election.cc:302] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.615762 10101 ts_tablet_manager.cc:1397] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.159s	user 0.047s	sys 0.095s
I20260809 23:24:35.618355 10124 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.618873 10101 raft_consensus.cc:348] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.619693 10101 raft_consensus.cc:374] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.620000 10101 raft_consensus.cc:729] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.620602 10124 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.621117 10124 raft_consensus.cc:2781] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.620728 10101 consensus_queue.cc:260] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.621493 10124 raft_consensus.cc:482] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.621870 10124 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.608629  9786 leader_election.cc:302] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:35.623238 10101 ts_tablet_manager.cc:1428] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260809 23:24:35.614773 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940f2aa4051d42d6aad243f3024117d6" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.623842 10040 raft_consensus.cc:2454] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.624272 10101 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:35.630523 10102 consensus_queue.cc:260] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.606164 10108 raft_consensus.cc:504] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.633587 10102 ts_tablet_manager.cc:1428] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260809 23:24:35.635922 10102 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.637214 10124 raft_consensus.cc:504] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.659900 10101 tablet_bootstrap.cc:654] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.652225  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.661595  9935 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:35.661849 10124 leader_election.cc:288] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.655411  9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.662855  9933 raft_consensus.cc:3037] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.663916  9931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.656458  9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.656857  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.650918 10103 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.665719  9833 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:35.666113 10103 ts_tablet_manager.cc:1397] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.084s	user 0.006s	sys 0.008s
I20260809 23:24:35.657714  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.667285 10101 tablet_bootstrap.cc:492] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.667729  9832 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:35.667910 10101 ts_tablet_manager.cc:1397] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.044s	user 0.020s	sys 0.015s
W20260809 23:24:35.669041  9786 leader_election.cc:341] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.669158 10103 raft_consensus.cc:348] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.648520  9940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.669921 10103 raft_consensus.cc:374] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.670265 10103 raft_consensus.cc:729] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.671283  9994 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.671224 10103 consensus_queue.cc:260] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.671681  9933 raft_consensus.cc:2454] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.671499 10101 raft_consensus.cc:348] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.672698  9994 leader_election.cc:302] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.641269 10144 raft_consensus.cc:2781] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.673399 10144 raft_consensus.cc:482] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.673781 10144 raft_consensus.cc:3037] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.674377 10143 raft_consensus.cc:2781] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.674820 10143 raft_consensus.cc:482] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.675169 10143 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.645766 10108 leader_election.cc:288] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.665714  9834 raft_consensus.cc:2379] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a2478dec1844a5481141d44f3136f26 in current term 1: Already voted for candidate 1da1957992c74e29b5f09aeb64fd88fd in this term.
I20260809 23:24:35.644761 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.643955  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.649778  9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.659900 10102 tablet_bootstrap.cc:654] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.677425  9937 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.644932  9785 leader_election.cc:302] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.679098  9786 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:35.679100 10108 raft_consensus.cc:2781] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.679889 10108 raft_consensus.cc:482] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.672772 10110 raft_consensus.cc:2781] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.680200 10124 raft_consensus.cc:2781] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.680379 10110 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.681067 10124 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.681320 10110 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.681910 10124 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.684989 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c044125e014a6caf1be9b452d64b76" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.685675 10041 raft_consensus.cc:3037] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.680229 10108 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.687541 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.689724 10042 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.687350 10113 raft_consensus.cc:482] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.691494 10144 raft_consensus.cc:504] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.691636 10113 raft_consensus.cc:504] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.695427  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940f2aa4051d42d6aad243f3024117d6" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.696189  9937 raft_consensus.cc:3037] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.677773  9994 leader_election.cc:302] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.698129 10113 leader_election.cc:288] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.707741  9786 leader_election.cc:302] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:35.714365 10041 raft_consensus.cc:2454] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.685258 10103 ts_tablet_manager.cc:1428] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.003s
I20260809 23:24:35.718554 10103 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:35.672786 10101 raft_consensus.cc:374] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.720158 10101 raft_consensus.cc:729] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.721953 10101 consensus_queue.cc:260] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.723423 10150 raft_consensus.cc:2781] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.706231  9937 raft_consensus.cc:2454] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.726828 10042 raft_consensus.cc:2454] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.730867 10110 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.732537 10124 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.737423 10155 raft_consensus.cc:2726] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:35.734738  9786 leader_election.cc:302] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
W20260809 23:24:35.738432  9994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.1:43405, user_credentials={real_user=slave}} blocked reactor thread for 60793.1us
I20260809 23:24:35.733580 10150 raft_consensus.cc:686] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:35.733404 10143 raft_consensus.cc:504] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.740490 10150 consensus_queue.cc:237] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.744127 10143 leader_election.cc:288] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.744179 10159 raft_consensus.cc:2781] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.744748 10159 raft_consensus.cc:686] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:35.731988 10108 raft_consensus.cc:504] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.736066  9785 leader_election.cc:302] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.735023 10040 raft_consensus.cc:2379] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1da1957992c74e29b5f09aeb64fd88fd in current term 1: Already voted for candidate 8a2478dec1844a5481141d44f3136f26 in this term.
I20260809 23:24:35.746423  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.748268 10150 raft_consensus.cc:2781] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.749290 10150 raft_consensus.cc:686] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:35.750428 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.747411  9937 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.751161 10041 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.747416 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.757393  9933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.749471 10103 tablet_bootstrap.cc:654] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.740430 10124 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.757539 10110 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.758661 10159 consensus_queue.cc:237] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.760982 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940f2aa4051d42d6aad243f3024117d6" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.763056 10040 raft_consensus.cc:3037] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.764566 10144 leader_election.cc:288] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.758023  9933 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.761401 10108 leader_election.cc:288] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.767939 10041 raft_consensus.cc:2454] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.769153  9785 leader_election.cc:302] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.770227 10108 raft_consensus.cc:2781] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.770752 10108 raft_consensus.cc:686] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:35.758679 10150 consensus_queue.cc:237] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.772871 10110 raft_consensus.cc:482] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.773836 10110 raft_consensus.cc:504] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.777796  9933 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:35.778163  9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.779183  9995 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.780082 10155 raft_consensus.cc:2781] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.780589 10155 raft_consensus.cc:686] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
W20260809 23:24:35.781337  9786 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.21.255.2:33409, user_credentials={real_user=slave}} blocked reactor thread for 46705.2us
I20260809 23:24:35.781699 10155 consensus_queue.cc:237] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.782287 10101 ts_tablet_manager.cc:1428] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.114s	user 0.016s	sys 0.003s
I20260809 23:24:35.783833 10101 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
W20260809 23:24:35.780615  9995 leader_election.cc:341] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.779209 10110 leader_election.cc:288] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.779423 10108 consensus_queue.cc:237] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.789551  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.790218  9935 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.787794  9936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.795535  9936 raft_consensus.cc:2379] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1da1957992c74e29b5f09aeb64fd88fd in current term 1: Already voted for candidate 8a2478dec1844a5481141d44f3136f26 in this term.
I20260809 23:24:35.799057  9937 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:35.799839  9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:35.803483  9834 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.803870  9935 raft_consensus.cc:2454] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.803998  9688 catalog_manager.cc:5495] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:35.801216  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:35.807796  9832 raft_consensus.cc:2379] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a2478dec1844a5481141d44f3136f26 in current term 1: Already voted for candidate 1da1957992c74e29b5f09aeb64fd88fd in this term.
I20260809 23:24:35.811019 10144 raft_consensus.cc:482] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.811553 10144 raft_consensus.cc:504] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.815573 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.801527  9786 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd; no voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:35.802352  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.816466 10041 raft_consensus.cc:2454] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.816885  9833 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:35.818513  9785 leader_election.cc:302] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.819880 10156 raft_consensus.cc:2726] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:35.820592 10156 raft_consensus.cc:2781] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.820935 10156 raft_consensus.cc:482] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.821244 10156 raft_consensus.cc:3037] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.824138 10113 raft_consensus.cc:482] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.824656 10113 raft_consensus.cc:504] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.825649  9994 leader_election.cc:302] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.826594 10155 raft_consensus.cc:2781] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.826712 10113 leader_election.cc:288] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.827767 10113 raft_consensus.cc:482] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:35.828276 10113 raft_consensus.cc:504] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.827020 10155 raft_consensus.cc:482] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.788300 10040 raft_consensus.cc:2454] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 1.
I20260809 23:24:35.830011 10144 leader_election.cc:288] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.826846  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:35.832160  9935 raft_consensus.cc:2454] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 0.
I20260809 23:24:35.833470 10155 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.829022  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.839064  9995 leader_election.cc:302] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.837126  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.840626 10102 tablet_bootstrap.cc:492] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.840701 10101 tablet_bootstrap.cc:654] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.843888  9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:35.835673  9834 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:35.845235 10110 raft_consensus.cc:2781] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.850556 10102 ts_tablet_manager.cc:1397] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.215s	user 0.070s	sys 0.084s
I20260809 23:24:35.846237 10113 leader_election.cc:288] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
W20260809 23:24:35.858902  9890 leader_election.cc:341] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.852550 10110 raft_consensus.cc:686] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:35.865077 10103 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:35.864634 10110 consensus_queue.cc:237] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.865761 10103 ts_tablet_manager.cc:1397] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.147s	user 0.018s	sys 0.000s
I20260809 23:24:35.880118 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.881040 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.881735 10041 raft_consensus.cc:2454] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:35.882711 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:35.884328  9890 leader_election.cc:302] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:35.887199 10113 raft_consensus.cc:2781] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:35.888082 10113 raft_consensus.cc:482] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:35.888417 10113 raft_consensus.cc:3037] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:35.885346  9890 leader_election.cc:341] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.889091  9890 leader_election.cc:302] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:35.881099 10040 raft_consensus.cc:2379] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2df9f9c44dc3408c8d475c31e28684b4 in current term 1: Already voted for candidate 1da1957992c74e29b5f09aeb64fd88fd in this term.
I20260809 23:24:35.889861 10155 raft_consensus.cc:504] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.890225 10172 raft_consensus.cc:2726] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:35.892026 10155 leader_election.cc:288] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:35.893501  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.894846  9890 leader_election.cc:302] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:35.895328  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:35.895961  9833 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.896278 10172 raft_consensus.cc:2726] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:35.897179 10113 raft_consensus.cc:504] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.898631 10156 raft_consensus.cc:504] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
W20260809 23:24:35.899662  9995 leader_election.cc:341] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:35.899956  9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:35.900815  9832 raft_consensus.cc:2379] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2df9f9c44dc3408c8d475c31e28684b4 in current term 1: Already voted for candidate 1da1957992c74e29b5f09aeb64fd88fd in this term.
I20260809 23:24:35.903388 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.904161 10113 leader_election.cc:288] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.904009  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:35.908540  9937 raft_consensus.cc:2379] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1da1957992c74e29b5f09aeb64fd88fd in current term 1: Already voted for candidate 2df9f9c44dc3408c8d475c31e28684b4 in this term.
I20260809 23:24:35.908689 10156 leader_election.cc:288] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:35.904021 10040 raft_consensus.cc:3037] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:35.887066 10103 raft_consensus.cc:348] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.913393 10103 raft_consensus.cc:374] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.913623 10103 raft_consensus.cc:729] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.914700 10103 consensus_queue.cc:260] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.916096 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be3d5a988cf147e19133130c67fa914e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:35.916869 10103 ts_tablet_manager.cc:1428] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.000s
I20260809 23:24:35.918021 10101 tablet_bootstrap.cc:492] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:35.918605 10101 ts_tablet_manager.cc:1397] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.135s	user 0.036s	sys 0.007s
I20260809 23:24:35.918807 10102 raft_consensus.cc:348] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.919596 10102 raft_consensus.cc:374] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.919930 10102 raft_consensus.cc:729] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.918035  9785 leader_election.cc:302] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd; no voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:35.921154 10156 raft_consensus.cc:2726] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:35.920681 10102 consensus_queue.cc:260] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.924535 10102 ts_tablet_manager.cc:1428] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.069s	user 0.007s	sys 0.000s
I20260809 23:24:35.925755 10101 raft_consensus.cc:348] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.926294 10102 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:35.926528 10101 raft_consensus.cc:374] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.920076  9833 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:35.926801 10101 raft_consensus.cc:729] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.927523 10101 consensus_queue.cc:260] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.929205 10040 raft_consensus.cc:2454] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:35.929479 10101 ts_tablet_manager.cc:1428] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:24:35.929571  9994 leader_election.cc:302] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:35.936324  9890 leader_election.cc:302] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:35.934345  9690 catalog_manager.cc:5495] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:35.937587 10113 raft_consensus.cc:2781] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.938062 10113 raft_consensus.cc:686] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:35.938700 10102 tablet_bootstrap.cc:654] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:35.939261 10113 consensus_queue.cc:237] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.942325  9689 catalog_manager.cc:5495] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 1 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:35.943629 10155 raft_consensus.cc:2781] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:35.948028 10155 raft_consensus.cc:686] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:35.948271 10102 tablet_bootstrap.cc:492] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:35.949040 10102 ts_tablet_manager.cc:1397] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.005s
I20260809 23:24:35.949796 10155 consensus_queue.cc:237] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.952014  9689 catalog_manager.cc:5495] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 1 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:35.954104  9689 catalog_manager.cc:5495] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 1 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:35.955722  9689 catalog_manager.cc:5495] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 1 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:35.956247 10102 raft_consensus.cc:348] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.957221 10102 raft_consensus.cc:374] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:35.957528 10102 raft_consensus.cc:729] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:35.958335 10102 consensus_queue.cc:260] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:35.960780 10102 ts_tablet_manager.cc:1428] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260809 23:24:35.965602  9688 catalog_manager.cc:5495] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:35.970541  9690 catalog_manager.cc:5495] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.127420 10156 consensus_queue.cc:1035] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.144909 10156 consensus_queue.cc:1035] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.182675 10155 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.191459 10159 consensus_queue.cc:1035] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.195832 10110 raft_consensus.cc:482] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:36.196451 10110 raft_consensus.cc:504] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.199755 10191 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:36.201849  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:36.202382 10110 leader_election.cc:288] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:36.202580  9833 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:36.204288  9994 leader_election.cc:302] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:36.205134 10110 raft_consensus.cc:2781] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:36.205487 10110 raft_consensus.cc:482] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:36.205823 10110 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:36.206306  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:36.206812 10159 consensus_queue.cc:1035] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.209223 10156 consensus_queue.cc:1035] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.214128 10113 raft_consensus.cc:482] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:36.217063 10113 raft_consensus.cc:504] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.221766  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:36.222692 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:36.223207 10113 leader_election.cc:288] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.222689  9833 raft_consensus.cc:2379] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2df9f9c44dc3408c8d475c31e28684b4 in current term 1: Already voted for candidate 8a2478dec1844a5481141d44f3136f26 in this term.
I20260809 23:24:36.215234  9937 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:36.227034  9890 leader_election.cc:302] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:36.228601 10113 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:36.241402  9759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:36.244529 10110 raft_consensus.cc:504] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.246449 10113 raft_consensus.cc:2726] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:36.247387  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:36.249665  9832 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:36.247504 10110 leader_election.cc:288] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:36.248167  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:36.250702  9937 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:36.255314  9933 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:36.259034  9937 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:36.260270  9995 leader_election.cc:302] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:36.261163 10191 raft_consensus.cc:2781] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:36.261667 10191 raft_consensus.cc:686] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.262589 10191 consensus_queue.cc:237] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.264206 10144 consensus_queue.cc:1035] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.266721  9832 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:36.275350  9688 catalog_manager.cc:5495] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:36.287972 10144 consensus_queue.cc:1035] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.299628 10108 consensus_queue.cc:1035] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.334102 10113 consensus_queue.cc:1035] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.347579 10113 consensus_queue.cc:1035] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.386811 10042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:49402
I20260809 23:24:36.387526 10042 raft_consensus.cc:595] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:36.393831 10042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:49402
I20260809 23:24:36.394349 10042 raft_consensus.cc:595] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:36.411124  9832 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "348dba4300304e79a88b264279c3358e"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
mode: GRACEFUL
new_leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:47254
I20260809 23:24:36.411682  9832 raft_consensus.cc:595] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Received request to transfer leadership to TS 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:36.421377 10042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
mode: GRACEFUL
new_leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:49402
I20260809 23:24:36.421931 10042 raft_consensus.cc:595] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Received request to transfer leadership to TS 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:36.425895 10191 consensus_queue.cc:1035] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:36.426864 10110 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:36.429719 10042 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b609b47481a74aa58f03382178c4adad"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
mode: GRACEFUL
new_leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:49402
I20260809 23:24:36.430331 10042 raft_consensus.cc:595] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Received request to transfer leadership to TS 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:36.437974  9832 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
mode: GRACEFUL
new_leader_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:47254
I20260809 23:24:36.439052  9832 raft_consensus.cc:595] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Received request to transfer leadership to TS 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:36.439754 10143 raft_consensus.cc:979] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: : Instructing follower 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.440177 10143 raft_consensus.cc:1067] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Signalling peer 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.445302  9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b609b47481a74aa58f03382178c4adad"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:57872
I20260809 23:24:36.446650  9935 raft_consensus.cc:482] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.447151  9935 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.448670 10155 consensus_queue.cc:1035] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:36.451534 10155 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.462095  9935 raft_consensus.cc:504] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.465392  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:36.466140  9833 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.468369 10155 raft_consensus.cc:979] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: : Instructing follower 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.468932 10155 raft_consensus.cc:1067] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Signalling peer 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.468412  9935 leader_election.cc:288] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.471074  9935 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:57872
I20260809 23:24:36.471364 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:36.471707  9935 raft_consensus.cc:482] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.472005 10042 raft_consensus.cc:3032] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.472261  9935 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.472354 10042 raft_consensus.cc:729] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.473727 10042 consensus_queue.cc:260] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.474624  9833 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.475219 10042 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.476341  9890 leader_election.cc:302] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:36.477340 10113 raft_consensus.cc:2781] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.477818 10113 raft_consensus.cc:686] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:36.478564 10113 consensus_queue.cc:237] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.480894  9935 raft_consensus.cc:504] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.482980  9935 leader_election.cc:288] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.484061 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:36.484225 10042 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.484773 10040 raft_consensus.cc:3032] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.485126 10040 raft_consensus.cc:729] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.485363  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd6557129314bd59fd84cdeed3211bf" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:36.486116  9833 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.485914 10040 consensus_queue.cc:260] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.487435 10040 raft_consensus.cc:3037] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.493373  9833 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.494918  9890 leader_election.cc:302] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:36.495194  9688 catalog_manager.cc:5495] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 1 to 2, leader changed from 8a2478dec1844a5481141d44f3136f26 (127.21.255.3) to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.495981 10203 raft_consensus.cc:2781] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.496084 10040 raft_consensus.cc:2454] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.496542 10203 raft_consensus.cc:686] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:36.497555 10203 consensus_queue.cc:237] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.509131  9688 catalog_manager.cc:5495] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 1 to 2, leader changed from 8a2478dec1844a5481141d44f3136f26 (127.21.255.3) to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.597160 10155 raft_consensus.cc:979] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:36.597748 10110 raft_consensus.cc:1067] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:36.599359  9833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47232
I20260809 23:24:36.599928  9833 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.600266  9833 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.606006  9833 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.607647  9833 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.608708  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:36.609357  9935 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.608754 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:36.610024 10040 raft_consensus.cc:3032] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.610291 10040 raft_consensus.cc:729] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.611217 10040 consensus_queue.cc:260] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.612627 10040 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.615603  9935 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:36.616963  9786 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:36.617909 10144 raft_consensus.cc:2781] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.617991 10040 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:36.618712 10144 raft_consensus.cc:686] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:36.619915 10144 consensus_queue.cc:237] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.626797  9688 catalog_manager.cc:5495] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 8a2478dec1844a5481141d44f3136f26 (127.21.255.3) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.672071 10144 raft_consensus.cc:979] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: : Instructing follower 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.672602 10159 raft_consensus.cc:1067] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Signalling peer 2df9f9c44dc3408c8d475c31e28684b4 to start an election
I20260809 23:24:36.675074  9937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "348dba4300304e79a88b264279c3358e"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
 from {username='slave'} at 127.0.0.1:57862
I20260809 23:24:36.675695  9937 raft_consensus.cc:482] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.676064  9937 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.677237 10144 raft_consensus.cc:979] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: : Instructing follower 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:36.677827 10159 raft_consensus.cc:1067] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Signalling peer 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:36.680310 10040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:49384
I20260809 23:24:36.680951 10040 raft_consensus.cc:482] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.681272 10040 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.685971  9937 raft_consensus.cc:504] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.687655  9937 leader_election.cc:288] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.688608  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:36.688820 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348dba4300304e79a88b264279c3358e" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:36.689215 10040 raft_consensus.cc:504] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.689386 10042 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.689307  9833 raft_consensus.cc:3032] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.690083  9833 raft_consensus.cc:729] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:36.690907  9833 consensus_queue.cc:260] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.691896  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:36.692281  9833 raft_consensus.cc:3037] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.692400 10040 leader_election.cc:288] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 2 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:36.692416  9832 raft_consensus.cc:3032] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.693030  9832 raft_consensus.cc:729] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:36.693570  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a6a3113d7b540a18727995d759a4ada" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:36.694293  9937 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.693801  9832 consensus_queue.cc:260] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.695286  9832 raft_consensus.cc:3037] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.698086 10042 raft_consensus.cc:2454] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.699498  9890 leader_election.cc:302] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:36.700076  9833 raft_consensus.cc:2454] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 2.
I20260809 23:24:36.700399 10203 raft_consensus.cc:2781] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.700984 10203 raft_consensus.cc:686] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:36.701390  9937 raft_consensus.cc:2454] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 2.
I20260809 23:24:36.701985 10203 consensus_queue.cc:237] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.702409  9995 leader_election.cc:302] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:36.703317 10110 raft_consensus.cc:2781] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.703807 10110 raft_consensus.cc:686] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 2 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.704604 10110 consensus_queue.cc:237] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.710057  9832 raft_consensus.cc:2454] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 2.
I20260809 23:24:36.710332  9688 catalog_manager.cc:5495] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 1 to 2, leader changed from 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1) to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 2 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.712284  9689 catalog_manager.cc:5495] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 1 to 2, leader changed from 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1) to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 2 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:36.783305 10110 consensus_queue.cc:1035] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.794039 10110 consensus_queue.cc:1035] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:36.802454 10191 raft_consensus.cc:979] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:36.803390 10110 raft_consensus.cc:1067] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:36.805348  9832 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47232
I20260809 23:24:36.805963  9832 raft_consensus.cc:482] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:36.806274  9832 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.813102  9832 raft_consensus.cc:504] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.814939  9832 leader_election.cc:288] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:36.815928  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:36.816401 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ac43ba34c54f97877eee12223f43ce" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:36.816737  9937 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.817037 10042 raft_consensus.cc:3032] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:36.817490 10042 raft_consensus.cc:729] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:36.818436 10042 consensus_queue.cc:260] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.819926 10042 raft_consensus.cc:3037] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:36.824878  9937 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:36.825400 10042 raft_consensus.cc:2454] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:36.826308  9786 leader_election.cc:302] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:36.827452 10159 raft_consensus.cc:2781] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:36.827929 10159 raft_consensus.cc:686] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:36.828596 10159 consensus_queue.cc:237] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:36.835436  9689 catalog_manager.cc:5495] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 8a2478dec1844a5481141d44f3136f26 (127.21.255.3) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:36.965982 10042 raft_consensus.cc:1261] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:36.967308 10218 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:36.976217  9832 raft_consensus.cc:1261] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:36.979030 10218 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.071981 10042 raft_consensus.cc:1261] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.073706 10218 consensus_queue.cc:1035] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.083719  9832 raft_consensus.cc:1261] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.085101 10113 consensus_queue.cc:1035] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:37.121557  9832 raft_consensus.cc:1261] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.123230 10110 consensus_queue.cc:1035] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.132421  9937 raft_consensus.cc:1261] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.134310 10191 consensus_queue.cc:1035] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.144858 10042 raft_consensus.cc:1261] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.146250 10159 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.149745  9832 raft_consensus.cc:1261] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.151149 10218 consensus_queue.cc:1035] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.159981  9937 raft_consensus.cc:1261] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.161825 10159 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.165778 10042 raft_consensus.cc:1261] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 2df9f9c44dc3408c8d475c31e28684b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.167878 10203 consensus_queue.cc:1035] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.301004 10042 raft_consensus.cc:1261] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.302587 10159 consensus_queue.cc:1035] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:37.311342  9937 raft_consensus.cc:1261] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:37.312829 10144 consensus_queue.cc:1035] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:39.440559 22524 test_util.cc:268] Using random seed: -1968638833
I20260809 23:24:39.480536  9690 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35408:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260809 23:24:39.483076  9690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 23:24:39.541443  9816 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82248b0b7b426785f4e46d5a3b0467 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:39.542198  9915 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82248b0b7b426785f4e46d5a3b0467 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:39.543377  9816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82248b0b7b426785f4e46d5a3b0467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.543921  9915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82248b0b7b426785f4e46d5a3b0467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.544958  9919 tablet_service.cc:1467] Processing CreateTablet for tablet cef56cc4c7564305ae59cac349c349dc (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:39.546319  9919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef56cc4c7564305ae59cac349c349dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.547175  9913 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab82f8ca0bb46239fca3c46e42da97e (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:39.548738  9913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab82f8ca0bb46239fca3c46e42da97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.548854 10019 tablet_service.cc:1467] Processing CreateTablet for tablet 5b82248b0b7b426785f4e46d5a3b0467 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260809 23:24:39.550350 10022 tablet_service.cc:1467] Processing CreateTablet for tablet cef56cc4c7564305ae59cac349c349dc (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:39.552237  9813 tablet_service.cc:1467] Processing CreateTablet for tablet cef56cc4c7564305ae59cac349c349dc (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260809 23:24:39.553493  9813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef56cc4c7564305ae59cac349c349dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.557947 10026 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab82f8ca0bb46239fca3c46e42da97e (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:39.559912  9810 tablet_service.cc:1467] Processing CreateTablet for tablet 6ab82f8ca0bb46239fca3c46e42da97e (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260809 23:24:39.561710  9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab82f8ca0bb46239fca3c46e42da97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.550412 10019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b82248b0b7b426785f4e46d5a3b0467. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.563974 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ab82f8ca0bb46239fca3c46e42da97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.566435 10022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef56cc4c7564305ae59cac349c349dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.568572  9815 tablet_service.cc:1467] Processing CreateTablet for tablet 62d951cdf8bd429fabe71c0b759b92c5 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:39.570343  9920 tablet_service.cc:1467] Processing CreateTablet for tablet 62d951cdf8bd429fabe71c0b759b92c5 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:39.571780  9920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d951cdf8bd429fabe71c0b759b92c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.572377  9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d951cdf8bd429fabe71c0b759b92c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.575004 10021 tablet_service.cc:1467] Processing CreateTablet for tablet 62d951cdf8bd429fabe71c0b759b92c5 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260809 23:24:39.576269 10021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d951cdf8bd429fabe71c0b759b92c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.579532  9916 tablet_service.cc:1467] Processing CreateTablet for tablet 0d45b8d405f44dfa93a9df9bf307932b (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:39.579886  9812 tablet_service.cc:1467] Processing CreateTablet for tablet a2b74054639345d0b64bbb75cdcd23b7 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:39.580940  9916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d45b8d405f44dfa93a9df9bf307932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.581735  9812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b74054639345d0b64bbb75cdcd23b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.584085  9917 tablet_service.cc:1467] Processing CreateTablet for tablet a2b74054639345d0b64bbb75cdcd23b7 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:39.584923  9922 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d48e260ab40219722eaf4109a9af4 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:39.585728  9917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b74054639345d0b64bbb75cdcd23b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.586376  9922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d48e260ab40219722eaf4109a9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.587710 10025 tablet_service.cc:1467] Processing CreateTablet for tablet a2b74054639345d0b64bbb75cdcd23b7 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260809 23:24:39.589337 10025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2b74054639345d0b64bbb75cdcd23b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.591939  9809 tablet_service.cc:1467] Processing CreateTablet for tablet 0d45b8d405f44dfa93a9df9bf307932b (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:39.593364  9809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d45b8d405f44dfa93a9df9bf307932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.600664 10023 tablet_service.cc:1467] Processing CreateTablet for tablet 0d45b8d405f44dfa93a9df9bf307932b (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260809 23:24:39.602620 10023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d45b8d405f44dfa93a9df9bf307932b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.604287  9814 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d48e260ab40219722eaf4109a9af4 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:39.606437  9814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d48e260ab40219722eaf4109a9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.617367 10028 tablet_service.cc:1467] Processing CreateTablet for tablet eb4d48e260ab40219722eaf4109a9af4 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260809 23:24:39.618569  9818 tablet_service.cc:1467] Processing CreateTablet for tablet a90f4331ebd34e7a81d1a0b9d153e3f6 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:39.622822  9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f4331ebd34e7a81d1a0b9d153e3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.623376  9817 tablet_service.cc:1467] Processing CreateTablet for tablet 5840fbf0505c45dfbe1ad4545467bf47 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:39.620959 10027 tablet_service.cc:1467] Processing CreateTablet for tablet a90f4331ebd34e7a81d1a0b9d153e3f6 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:39.624764  9817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5840fbf0505c45dfbe1ad4545467bf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.626643  9920 tablet_service.cc:1467] Processing CreateTablet for tablet a90f4331ebd34e7a81d1a0b9d153e3f6 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260809 23:24:39.628202  9920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f4331ebd34e7a81d1a0b9d153e3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.618741 10028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb4d48e260ab40219722eaf4109a9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.633282 10027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90f4331ebd34e7a81d1a0b9d153e3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.640190  9924 tablet_service.cc:1467] Processing CreateTablet for tablet 5840fbf0505c45dfbe1ad4545467bf47 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:39.640234 10020 tablet_service.cc:1467] Processing CreateTablet for tablet 5840fbf0505c45dfbe1ad4545467bf47 (DEFAULT_TABLE table=soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260809 23:24:39.642115 10020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5840fbf0505c45dfbe1ad4545467bf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.642117  9924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5840fbf0505c45dfbe1ad4545467bf47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 23:24:39.702461 10238 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:39.737057 10238 tablet_bootstrap.cc:654] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.752914 10239 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:39.764940 10240 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:39.777108 10238 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:39.777729 10238 ts_tablet_manager.cc:1397] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.076s	user 0.017s	sys 0.010s
I20260809 23:24:39.778575 10240 tablet_bootstrap.cc:654] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.780421 10238 raft_consensus.cc:348] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.781212 10238 raft_consensus.cc:374] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.781490 10238 raft_consensus.cc:729] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.782366 10238 consensus_queue.cc:260] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.782994 10239 tablet_bootstrap.cc:654] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.829237 10240 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:39.829804 10240 ts_tablet_manager.cc:1397] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.065s	user 0.031s	sys 0.024s
I20260809 23:24:39.830107 10239 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:39.830525 10238 ts_tablet_manager.cc:1428] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.052s	user 0.017s	sys 0.030s
I20260809 23:24:39.830768 10239 ts_tablet_manager.cc:1397] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.078s	user 0.024s	sys 0.031s
I20260809 23:24:39.831678 10238 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:39.832345 10240 raft_consensus.cc:348] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.832999 10240 raft_consensus.cc:374] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.833248 10240 raft_consensus.cc:729] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.833847 10240 consensus_queue.cc:260] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.836030 10240 ts_tablet_manager.cc:1428] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260809 23:24:39.837018 10240 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:39.838884 10238 tablet_bootstrap.cc:654] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.842021 10239 raft_consensus.cc:348] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.843298 10240 tablet_bootstrap.cc:654] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.843264 10239 raft_consensus.cc:374] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.843923 10239 raft_consensus.cc:729] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.844975 10239 consensus_queue.cc:260] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.851441 10238 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:39.852216 10238 ts_tablet_manager.cc:1397] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
I20260809 23:24:39.855624 10238 raft_consensus.cc:348] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.856580 10238 raft_consensus.cc:374] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.856892 10238 raft_consensus.cc:729] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.857227 10239 ts_tablet_manager.cc:1428] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260809 23:24:39.857662 10238 consensus_queue.cc:260] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.859174 10240 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:39.859663 10240 ts_tablet_manager.cc:1397] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.012s
I20260809 23:24:39.860260 10238 ts_tablet_manager.cc:1428] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260809 23:24:39.861388 10238 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:39.862442 10240 raft_consensus.cc:348] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.863236 10240 raft_consensus.cc:374] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.863490 10240 raft_consensus.cc:729] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.864164 10240 consensus_queue.cc:260] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.866307 10239 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:39.868479 10238 tablet_bootstrap.cc:654] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.873113 10240 ts_tablet_manager.cc:1428] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260809 23:24:39.874131 10240 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:39.874570 10239 tablet_bootstrap.cc:654] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.878605 10191 raft_consensus.cc:482] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:39.879141 10191 raft_consensus.cc:504] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.881170 10238 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:39.881866 10238 ts_tablet_manager.cc:1397] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.005s
I20260809 23:24:39.881208 10191 leader_election.cc:288] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:39.882607  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:39.882606  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
W20260809 23:24:39.886138  9994 leader_election.cc:341] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:39.885816 10238 raft_consensus.cc:348] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.886870 10238 raft_consensus.cc:374] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.887388 10238 raft_consensus.cc:729] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
W20260809 23:24:39.888775  9995 leader_election.cc:341] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:39.889271  9995 leader_election.cc:302] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:39.890321 10238 consensus_queue.cc:260] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.894008 10191 raft_consensus.cc:2726] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:39.895501 10238 ts_tablet_manager.cc:1428] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.005s
I20260809 23:24:39.897182 10238 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:39.898869 10240 tablet_bootstrap.cc:654] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.905308 10239 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:39.905817 10239 ts_tablet_manager.cc:1397] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.040s	user 0.028s	sys 0.008s
I20260809 23:24:39.906524 10238 tablet_bootstrap.cc:654] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.908805 10239 raft_consensus.cc:348] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.909678 10239 raft_consensus.cc:374] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.910040 10239 raft_consensus.cc:729] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.910918 10239 consensus_queue.cc:260] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.911844 10240 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:39.912014 10238 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:39.912743 10240 ts_tablet_manager.cc:1397] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.039s	user 0.009s	sys 0.003s
I20260809 23:24:39.912746 10238 ts_tablet_manager.cc:1397] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.002s
I20260809 23:24:39.913900 10239 ts_tablet_manager.cc:1428] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260809 23:24:39.915336 10239 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:39.916671 10240 raft_consensus.cc:348] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.916937 10238 raft_consensus.cc:348] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.917748 10240 raft_consensus.cc:374] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.917986 10238 raft_consensus.cc:374] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.918188 10240 raft_consensus.cc:729] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.918844 10238 raft_consensus.cc:729] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.919438 10240 consensus_queue.cc:260] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.919773 10238 consensus_queue.cc:260] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.923182 10240 ts_tablet_manager.cc:1428] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.003s
I20260809 23:24:39.923790 10239 tablet_bootstrap.cc:654] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.924252 10240 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:39.924825 10238 ts_tablet_manager.cc:1428] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260809 23:24:39.925920 10238 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:39.932282 10238 tablet_bootstrap.cc:654] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.933770 10240 tablet_bootstrap.cc:654] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.937309 10239 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:39.937919 10239 ts_tablet_manager.cc:1397] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.007s
I20260809 23:24:39.941846 10239 raft_consensus.cc:348] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.942863 10239 raft_consensus.cc:374] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.943403 10239 raft_consensus.cc:729] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.944401 10239 consensus_queue.cc:260] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.947559 10239 ts_tablet_manager.cc:1428] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260809 23:24:39.948891 10239 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:39.957783 10240 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:39.958395 10240 ts_tablet_manager.cc:1397] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.034s	user 0.014s	sys 0.016s
I20260809 23:24:39.958499 10239 tablet_bootstrap.cc:654] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.962179 10240 raft_consensus.cc:348] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.963419 10240 raft_consensus.cc:374] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.964022 10240 raft_consensus.cc:729] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.965199 10240 consensus_queue.cc:260] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.968984 10240 ts_tablet_manager.cc:1428] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.005s
I20260809 23:24:39.970129 10240 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:39.971916 10239 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:39.972663 10191 raft_consensus.cc:482] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:39.973207 10191 raft_consensus.cc:504] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.975660 10191 leader_election.cc:288] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:39.977301 10240 tablet_bootstrap.cc:654] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:39.977607 10239 ts_tablet_manager.cc:1397] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.009s
I20260809 23:24:39.977794  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
W20260809 23:24:39.979790  9995 leader_election.cc:341] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:39.980937  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:39.981336 10239 raft_consensus.cc:348] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.982492 10239 raft_consensus.cc:374] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260809 23:24:39.982784  9994 leader_election.cc:341] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:39.982888 10239 raft_consensus.cc:729] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.983551  9994 leader_election.cc:302] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:39.984205 10239 consensus_queue.cc:260] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.985052 10191 raft_consensus.cc:2726] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:39.988032 10239 ts_tablet_manager.cc:1428] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260809 23:24:39.989235 10240 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:39.989790 10240 ts_tablet_manager.cc:1397] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.015s
I20260809 23:24:39.992902 10238 tablet_bootstrap.cc:492] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:39.992619 10240 raft_consensus.cc:348] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.993376 10240 raft_consensus.cc:374] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.993448 10238 ts_tablet_manager.cc:1397] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.068s	user 0.027s	sys 0.025s
I20260809 23:24:39.993685 10240 raft_consensus.cc:729] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.994903 10239 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:39.995473 10240 consensus_queue.cc:260] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.996403 10238 raft_consensus.cc:348] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.997143 10238 raft_consensus.cc:374] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:39.997427 10238 raft_consensus.cc:729] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:39.998345 10240 ts_tablet_manager.cc:1428] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.006s
I20260809 23:24:39.998162 10238 consensus_queue.cc:260] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:39.999603 10240 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:40.001008 10238 ts_tablet_manager.cc:1428] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.001s
I20260809 23:24:40.001992 10238 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:40.002693 10239 tablet_bootstrap.cc:654] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.006042 10240 tablet_bootstrap.cc:654] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.007951 10238 tablet_bootstrap.cc:654] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.015223 10239 tablet_bootstrap.cc:492] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:40.015738 10239 ts_tablet_manager.cc:1397] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.010s
I20260809 23:24:40.017403 10240 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:40.018261 10240 ts_tablet_manager.cc:1397] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.004s
I20260809 23:24:40.019387 10239 raft_consensus.cc:348] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.020275 10239 raft_consensus.cc:374] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.020627 10239 raft_consensus.cc:729] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.020736 10238 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:40.021380 10238 ts_tablet_manager.cc:1397] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.020s	user 0.017s	sys 0.001s
I20260809 23:24:40.021539 10239 consensus_queue.cc:260] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.022113 10240 raft_consensus.cc:348] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.022868 10240 raft_consensus.cc:374] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.023159 10240 raft_consensus.cc:729] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.023876 10240 consensus_queue.cc:260] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.024554 10239 ts_tablet_manager.cc:1428] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260809 23:24:40.024256 10238 raft_consensus.cc:348] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.025048 10238 raft_consensus.cc:374] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.025347 10238 raft_consensus.cc:729] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.025599 10239 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:40.026510 10240 ts_tablet_manager.cc:1428] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260809 23:24:40.026083 10238 consensus_queue.cc:260] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.027578 10240 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:40.028966 10238 ts_tablet_manager.cc:1428] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260809 23:24:40.030596 10238 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:40.033834 10191 raft_consensus.cc:482] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.034425 10191 raft_consensus.cc:504] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.036875 10240 tablet_bootstrap.cc:654] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.037262 10191 leader_election.cc:288] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.037782 10244 raft_consensus.cc:482] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.038355 10244 raft_consensus.cc:504] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.039146  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5840fbf0505c45dfbe1ad4545467bf47" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.039889  9833 raft_consensus.cc:2454] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:40.040421 10244 leader_election.cc:288] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.041771 10238 tablet_bootstrap.cc:654] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.041646 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.042462 10040 raft_consensus.cc:2454] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.043479 10244 raft_consensus.cc:482] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.043994 10244 raft_consensus.cc:504] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.046298 10244 leader_election.cc:288] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.048007  9994 leader_election.cc:302] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.048136  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.048893 10191 raft_consensus.cc:2781] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.049288 10191 raft_consensus.cc:482] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.049624 10191 raft_consensus.cc:3037] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.050634 10240 tablet_bootstrap.cc:492] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:40.051273 10240 ts_tablet_manager.cc:1397] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.007s
I20260809 23:24:40.052608 10238 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:40.053153 10238 ts_tablet_manager.cc:1397] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.004s
I20260809 23:24:40.054214  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.054450 10240 raft_consensus.cc:348] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.055413 10240 raft_consensus.cc:374] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.055755 10240 raft_consensus.cc:729] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.055795  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5840fbf0505c45dfbe1ad4545467bf47" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:40.056209 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.056092 10238 raft_consensus.cc:348] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.057045 10238 raft_consensus.cc:374] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.052973  9890 leader_election.cc:302] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.056773 10240 consensus_queue.cc:260] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
W20260809 23:24:40.057401  9995 leader_election.cc:341] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.057550 10238 raft_consensus.cc:729] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
W20260809 23:24:40.058130  9890 leader_election.cc:341] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.058486 10238 consensus_queue.cc:260] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.059445 10239 tablet_bootstrap.cc:654] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.059645 10218 raft_consensus.cc:2781] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.060135 10218 raft_consensus.cc:482] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.060469 10218 raft_consensus.cc:3037] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.061007 10238 ts_tablet_manager.cc:1428] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260809 23:24:40.062068 10238 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
W20260809 23:24:40.059216  9890 leader_election.cc:341] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.067435 10240 ts_tablet_manager.cc:1428] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.005s
I20260809 23:24:40.065619 10191 raft_consensus.cc:504] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.068523 10238 tablet_bootstrap.cc:654] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.070163  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5840fbf0505c45dfbe1ad4545467bf47" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.070317  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5840fbf0505c45dfbe1ad4545467bf47" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.070636 10244 raft_consensus.cc:482] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.070984  9833 raft_consensus.cc:3037] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:40.071861  9995 leader_election.cc:341] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.071455 10244 raft_consensus.cc:504] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.072476 10191 leader_election.cc:288] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.074364 10244 leader_election.cc:288] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.076530  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.077520  9832 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.079788  9833 raft_consensus.cc:2454] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:40.081228  9994 leader_election.cc:302] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:40.057000 10040 raft_consensus.cc:2454] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.083635 10191 raft_consensus.cc:2781] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.084438 10191 raft_consensus.cc:686] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:40.084489 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.085362 10191 consensus_queue.cc:237] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.085502 10040 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.092375 10240 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:40.069062 10218 raft_consensus.cc:504] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.101300 10240 tablet_bootstrap.cc:654] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.081468  9890 leader_election.cc:302] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.104182  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.106134  9890 leader_election.cc:302] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.090881 10239 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:40.107489 10218 leader_election.cc:288] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.107930 10239 ts_tablet_manager.cc:1397] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.083s	user 0.015s	sys 0.004s
I20260809 23:24:40.108110 10218 raft_consensus.cc:2781] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.108530 10218 raft_consensus.cc:482] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.108868 10218 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.108904 10244 raft_consensus.cc:2781] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.109354 10244 raft_consensus.cc:482] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.112599 10239 raft_consensus.cc:348] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.113754 10239 raft_consensus.cc:374] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.114063 10239 raft_consensus.cc:729] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.109690 10244 raft_consensus.cc:3037] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.128669 10238 tablet_bootstrap.cc:492] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:40.129417 10240 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:40.129263 10238 ts_tablet_manager.cc:1397] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.067s	user 0.025s	sys 0.024s
I20260809 23:24:40.130335 10240 ts_tablet_manager.cc:1397] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.024s
I20260809 23:24:40.132999 10238 raft_consensus.cc:348] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.133402 10240 raft_consensus.cc:348] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.134042 10238 raft_consensus.cc:374] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.134249 10240 raft_consensus.cc:374] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.134414 10238 raft_consensus.cc:729] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.134631 10240 raft_consensus.cc:729] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.135277 10238 consensus_queue.cc:260] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.135476 10240 consensus_queue.cc:260] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.137688 10238 ts_tablet_manager.cc:1428] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260809 23:24:40.125775 10218 raft_consensus.cc:504] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.137873 10240 ts_tablet_manager.cc:1428] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260809 23:24:40.139701 10240 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: Bootstrap starting.
I20260809 23:24:40.140237 10218 leader_election.cc:288] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.143091 10238 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: Bootstrap starting.
I20260809 23:24:40.145202 10240 tablet_bootstrap.cc:654] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.144841  9688 catalog_manager.cc:5495] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:40.150063 10238 tablet_bootstrap.cc:654] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.150806 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90f4331ebd34e7a81d1a0b9d153e3f6" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.151170  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.151726 10040 raft_consensus.cc:3037] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.152093  9833 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.152029 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.152927 10042 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.161232 10040 raft_consensus.cc:2454] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.114886 10239 consensus_queue.cc:260] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.165117 10239 ts_tablet_manager.cc:1428] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.057s	user 0.007s	sys 0.000s
I20260809 23:24:40.172648 10042 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.170128 10244 raft_consensus.cc:504] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
W20260809 23:24:40.173152  9890 leader_election.cc:341] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.173306 10239 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:40.173278 10218 raft_consensus.cc:482] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.178201  9890 leader_election.cc:302] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.178110 10218 raft_consensus.cc:504] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.179095 10281 raft_consensus.cc:2781] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.180554 10218 leader_election.cc:288] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.184815  9890 leader_election.cc:302] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.187973 10218 raft_consensus.cc:2781] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.188498 10218 raft_consensus.cc:686] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.189755 10239 tablet_bootstrap.cc:654] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.189414 10218 consensus_queue.cc:237] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.193177 10191 raft_consensus.cc:482] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.193883 10191 raft_consensus.cc:504] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.198042 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.199451  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.200381  9833 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.200341  9832 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.201290 10040 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.209478  9890 leader_election.cc:302] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.210100 10238 tablet_bootstrap.cc:492] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: No bootstrap required, opened a new log
I20260809 23:24:40.210604 10238 ts_tablet_manager.cc:1397] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent bootstrapping tablet: real 0.068s	user 0.028s	sys 0.028s
I20260809 23:24:40.212901 10240 tablet_bootstrap.cc:492] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: No bootstrap required, opened a new log
I20260809 23:24:40.212927 10292 raft_consensus.cc:2781] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.214078 10292 raft_consensus.cc:482] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.214646 10292 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.216564  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.213445 10238 raft_consensus.cc:348] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.217746 10238 raft_consensus.cc:374] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.217537  9833 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:40.218158 10238 raft_consensus.cc:729] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.219110 10238 consensus_queue.cc:260] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.219398  9994 leader_election.cc:302] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.220480 10276 raft_consensus.cc:2781] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.220851 10276 raft_consensus.cc:482] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.221166 10276 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.221539 10238 ts_tablet_manager.cc:1428] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260809 23:24:40.224225 10244 leader_election.cc:288] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.227883 10281 raft_consensus.cc:686] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.228906 10281 consensus_queue.cc:237] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.231621 10240 ts_tablet_manager.cc:1397] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: Time spent bootstrapping tablet: real 0.092s	user 0.008s	sys 0.011s
I20260809 23:24:40.233162 10191 leader_election.cc:288] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.234262  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:40.235857 10292 raft_consensus.cc:504] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.237179  9937 raft_consensus.cc:2379] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a2478dec1844a5481141d44f3136f26 in current term 1: Already voted for candidate 2df9f9c44dc3408c8d475c31e28684b4 in this term.
I20260809 23:24:40.239084  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.239426 10240 raft_consensus.cc:348] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.240479 10240 raft_consensus.cc:374] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.240748 10292 leader_election.cc:288] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.240798 10240 raft_consensus.cc:729] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.242623 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.242902  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.242511 10240 consensus_queue.cc:260] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.243481 10040 raft_consensus.cc:3037] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.245499 10240 ts_tablet_manager.cc:1428] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260809 23:24:40.245548 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.243494  9833 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:40.247677  9890 leader_election.cc:341] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Illegal state: must be running to vote when last-logged opid is not known
I20260809 23:24:40.265173 10191 raft_consensus.cc:482] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.265903 10191 raft_consensus.cc:504] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.270370  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.271085  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:40.271597  9832 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:40.272440  9937 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:40.269244 10191 leader_election.cc:288] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.274791  9994 leader_election.cc:302] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.275982 10276 raft_consensus.cc:504] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.277504 10042 raft_consensus.cc:2379] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2df9f9c44dc3408c8d475c31e28684b4 in current term 1: Already voted for candidate 8a2478dec1844a5481141d44f3136f26 in this term.
I20260809 23:24:40.279631  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.281414  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.282531  9937 raft_consensus.cc:2379] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a2478dec1844a5481141d44f3136f26 in current term 1: Already voted for candidate 2df9f9c44dc3408c8d475c31e28684b4 in this term.
I20260809 23:24:40.284531 10191 raft_consensus.cc:2781] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.285029 10191 raft_consensus.cc:482] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.285419 10191 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
W20260809 23:24:40.286721  9759 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260809 23:24:40.287894 10285 raft_consensus.cc:482] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.288391 10285 raft_consensus.cc:504] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.289345  9994 leader_election.cc:302] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:40.290467 10285 leader_election.cc:288] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.292908  9828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.293650  9828 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.294988  9890 leader_election.cc:302] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.296171 10292 raft_consensus.cc:2781] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.297071 10292 raft_consensus.cc:482] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.297645 10292 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.298324 10040 raft_consensus.cc:2454] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.292927  9833 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.299851 10276 leader_election.cc:288] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.303108  9890 leader_election.cc:302] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:40.303495 10276 raft_consensus.cc:2726] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260809 23:24:40.301934 10239 tablet_bootstrap.cc:492] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:40.304327 10218 raft_consensus.cc:2781] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.304653 10239 ts_tablet_manager.cc:1397] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.132s	user 0.015s	sys 0.000s
I20260809 23:24:40.304831 10218 raft_consensus.cc:686] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.305796 10218 consensus_queue.cc:237] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.309465 10285 raft_consensus.cc:482] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.309360  9688 catalog_manager.cc:5495] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.310048 10285 raft_consensus.cc:504] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.312096 10285 leader_election.cc:288] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.312463 10239 raft_consensus.cc:348] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.316234  9890 leader_election.cc:302] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.318027 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d48e260ab40219722eaf4109a9af4" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.315974  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d48e260ab40219722eaf4109a9af4" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.322283 10042 raft_consensus.cc:2454] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.323666 10281 raft_consensus.cc:2781] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.324252 10281 raft_consensus.cc:686] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.323961  9890 leader_election.cc:302] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.325059 10244 raft_consensus.cc:482] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.325381 10285 raft_consensus.cc:2781] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.325124 10281 consensus_queue.cc:237] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.326001 10285 raft_consensus.cc:482] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.325605 10244 raft_consensus.cc:504] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.326341 10296 raft_consensus.cc:482] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.326431 10285 raft_consensus.cc:3037] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.327950 10244 leader_election.cc:288] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.331084  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d45b8d405f44dfa93a9df9bf307932b" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.332441  9832 raft_consensus.cc:2454] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.337657 10296 raft_consensus.cc:504] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.341545  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd" is_pre_election: true
I20260809 23:24:40.342998  9832 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 0.
I20260809 23:24:40.344923 10292 raft_consensus.cc:504] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.347641 10292 leader_election.cc:288] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.348788 10296 leader_election.cc:288] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.350613  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:40.351876  9937 raft_consensus.cc:2379] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a2478dec1844a5481141d44f3136f26 in current term 1: Already voted for candidate 2df9f9c44dc3408c8d475c31e28684b4 in this term.
I20260809 23:24:40.355542  9994 leader_election.cc:302] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 2df9f9c44dc3408c8d475c31e28684b4
I20260809 23:24:40.357450  9890 leader_election.cc:302] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.357972 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d45b8d405f44dfa93a9df9bf307932b" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.359762 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.360769 10041 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.361208  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.362077  9832 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.359711 10042 raft_consensus.cc:2454] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.372109 10239 raft_consensus.cc:374] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.372591 10239 raft_consensus.cc:729] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.376456 10191 raft_consensus.cc:504] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.377772 10218 raft_consensus.cc:2781] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 23:24:40.319903 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.384209 10218 raft_consensus.cc:482] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 23:24:40.379448 10239 consensus_queue.cc:260] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.381735  9688 catalog_manager.cc:5495] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.384594 10218 raft_consensus.cc:3037] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.385075  9833 raft_consensus.cc:2454] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 0.
I20260809 23:24:40.385929 10239 ts_tablet_manager.cc:1428] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.081s	user 0.008s	sys 0.000s
I20260809 23:24:40.393322 10239 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:40.393712  9688 catalog_manager.cc:5495] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.395027  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.396009  9937 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.400431 10239 tablet_bootstrap.cc:654] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.388077  9832 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.405402 10218 raft_consensus.cc:504] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.406275 10191 leader_election.cc:288] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:40.407219  9937 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:40.407660 10218 leader_election.cc:288] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.409066  9995 leader_election.cc:302] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.410264 10191 raft_consensus.cc:2781] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.411191 10191 raft_consensus.cc:686] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:40.411388  9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.412060  9832 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.412276 10191 consensus_queue.cc:237] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.415464 10285 raft_consensus.cc:504] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.415745  9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d45b8d405f44dfa93a9df9bf307932b" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.415745 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d45b8d405f44dfa93a9df9bf307932b" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.418170 10040 raft_consensus.cc:3037] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.421504 10285 leader_election.cc:288] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.419838  9890 leader_election.cc:302] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.421936  9832 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 1.
I20260809 23:24:40.417347  9833 raft_consensus.cc:3037] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.423444 10285 raft_consensus.cc:2781] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.390532 10041 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.424376 10296 raft_consensus.cc:2741] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260809 23:24:40.424237 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d48e260ab40219722eaf4109a9af4" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.424899 10042 raft_consensus.cc:3037] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.425040  9688 catalog_manager.cc:5495] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.427381 10285 raft_consensus.cc:686] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.427538  9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb4d48e260ab40219722eaf4109a9af4" candidate_uuid: "2df9f9c44dc3408c8d475c31e28684b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:40.428467  9834 raft_consensus.cc:3037] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.428442 10285 consensus_queue.cc:237] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.434315 10042 raft_consensus.cc:2454] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.442472 10040 raft_consensus.cc:2454] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.442996  9833 raft_consensus.cc:2454] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.447005  9834 raft_consensus.cc:2454] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2df9f9c44dc3408c8d475c31e28684b4 in term 1.
I20260809 23:24:40.447875  9890 leader_election.cc:302] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.449086 10218 raft_consensus.cc:2781] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.449424 10239 tablet_bootstrap.cc:492] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:40.449340  9688 catalog_manager.cc:5495] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.449592 10218 raft_consensus.cc:686] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.450354 10239 ts_tablet_manager.cc:1397] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.057s	user 0.017s	sys 0.009s
I20260809 23:24:40.451459 10218 consensus_queue.cc:237] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.453260  9890 leader_election.cc:302] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.453729 10239 raft_consensus.cc:348] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.454499 10239 raft_consensus.cc:374] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.454756 10239 raft_consensus.cc:729] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.458366 10239 consensus_queue.cc:260] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.454180  9690 catalog_manager.cc:5495] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 1 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:40.454195 10285 raft_consensus.cc:2781] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 23:24:40.465590 10285 raft_consensus.cc:686] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.466723 10285 consensus_queue.cc:237] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.470547 10239 ts_tablet_manager.cc:1428] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260809 23:24:40.478091 10239 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Bootstrap starting.
I20260809 23:24:40.485126 10239 tablet_bootstrap.cc:654] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Neither blocks nor log segments found. Creating new log.
I20260809 23:24:40.488006  9688 catalog_manager.cc:5495] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.490002 10239 tablet_bootstrap.cc:492] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: No bootstrap required, opened a new log
I20260809 23:24:40.489641  9688 catalog_manager.cc:5495] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2). New cstate: current_term: 1 leader_uuid: "2df9f9c44dc3408c8d475c31e28684b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:40.490501 10239 ts_tablet_manager.cc:1397] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
I20260809 23:24:40.493216 10239 raft_consensus.cc:348] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.493901 10239 raft_consensus.cc:374] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 23:24:40.494145 10239 raft_consensus.cc:729] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Initialized, Role: FOLLOWER
I20260809 23:24:40.494733 10239 consensus_queue.cc:260] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.496589 10239 ts_tablet_manager.cc:1428] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260809 23:24:40.500478  9937 raft_consensus.cc:3037] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.507431 10191 consensus_queue.cc:1035] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.531762 10276 consensus_queue.cc:1035] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.554850 10226 raft_consensus.cc:482] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 23:24:40.555686 10226 raft_consensus.cc:504] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.559232 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2478dec1844a5481141d44f3136f26" is_pre_election: true
I20260809 23:24:40.559849 10226 leader_election.cc:288] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.560879  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2b74054639345d0b64bbb75cdcd23b7" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4" is_pre_election: true
I20260809 23:24:40.560067 10040 raft_consensus.cc:2379] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1da1957992c74e29b5f09aeb64fd88fd in current term 1: Already voted for candidate 2df9f9c44dc3408c8d475c31e28684b4 in this term.
I20260809 23:24:40.562415  9785 leader_election.cc:302] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd; no voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:40.563212 10226 raft_consensus.cc:3037] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.570585 10226 raft_consensus.cc:2726] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260809 23:24:40.571277  9937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:57882
I20260809 23:24:40.572029  9937 raft_consensus.cc:595] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.578114  9937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:57882
I20260809 23:24:40.578583  9937 raft_consensus.cc:595] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.583621 10040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:49402
I20260809 23:24:40.584231 10040 raft_consensus.cc:595] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.589350  9937 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
mode: GRACEFUL
new_leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:57882
I20260809 23:24:40.589869  9937 raft_consensus.cc:595] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Received request to transfer leadership to TS 1da1957992c74e29b5f09aeb64fd88fd
I20260809 23:24:40.614672  9834 raft_consensus.cc:3037] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 0 FOLLOWER]: Advancing to term 1
I20260809 23:24:40.621213 10218 consensus_queue.cc:1035] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.634133 10218 consensus_queue.cc:1035] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.766710 10218 consensus_queue.cc:1035] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.776237 10281 consensus_queue.cc:1035] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.787724 10281 consensus_queue.cc:1035] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.788178 10292 raft_consensus.cc:979] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.788965 10292 raft_consensus.cc:1067] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.791956  9834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47240
I20260809 23:24:40.792644  9834 raft_consensus.cc:482] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:40.793192  9834 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.793251 10292 consensus_queue.cc:1035] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.800122 10292 consensus_queue.cc:1035] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.803290  9834 raft_consensus.cc:504] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.805823  9834 leader_election.cc:288] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.806605  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.807276  9937 raft_consensus.cc:3032] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:40.807471 10042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef56cc4c7564305ae59cac349c349dc" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.808331 10042 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.807595  9937 raft_consensus.cc:729] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.812131 10244 consensus_queue.cc:1035] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.812336  9937 consensus_queue.cc:260] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.813786  9937 raft_consensus.cc:3037] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.824793  9937 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.830250 10042 raft_consensus.cc:2454] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.835361 10317 consensus_queue.cc:1035] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.832105 10285 raft_consensus.cc:979] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.836681 10218 raft_consensus.cc:1067] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.837705  9786 leader_election.cc:302] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.838639 10226 raft_consensus.cc:2781] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:40.843652 10226 raft_consensus.cc:686] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:40.845345  9828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47240
I20260809 23:24:40.848480 10226 consensus_queue.cc:237] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.853955 10285 consensus_queue.cc:1035] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.861184  9828 raft_consensus.cc:482] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:40.861980  9828 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.868778 10191 raft_consensus.cc:979] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.869349 10191 raft_consensus.cc:1067] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.868682 10285 consensus_queue.cc:1035] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260809 23:24:40.877285 10285 consensus_queue.cc:1035] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 23:24:40.878584 10296 consensus_queue.cc:1035] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.895431  9832 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47232
I20260809 23:24:40.896528  9832 raft_consensus.cc:482] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:40.897316  9832 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.916657 10285 consensus_queue.cc:1035] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.917464  9828 raft_consensus.cc:504] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.923820  9832 raft_consensus.cc:504] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.928236  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.929235  9937 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.934149  9832 leader_election.cc:288] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.936308 10040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ab82f8ca0bb46239fca3c46e42da97e" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.937058 10040 raft_consensus.cc:3032] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:40.937450 10040 raft_consensus.cc:729] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:40.941118 10040 consensus_queue.cc:260] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.941941 10281 consensus_queue.cc:1035] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.943617 10040 raft_consensus.cc:3037] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.946700  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:40.947748  9935 raft_consensus.cc:3032] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:40.948127  9935 raft_consensus.cc:729] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:40.949210  9935 consensus_queue.cc:260] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.950832  9935 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.958096  9937 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.959715  9786 leader_election.cc:302] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:40.960961 10338 raft_consensus.cc:2781] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:40.961573 10338 raft_consensus.cc:686] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:40.962566 10338 consensus_queue.cc:237] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:40.967142 10040 raft_consensus.cc:2454] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.973367  9828 leader_election.cc:288] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:40.974668 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d951cdf8bd429fabe71c0b759b92c5" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:40.975447 10041 raft_consensus.cc:3037] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.979188 10281 consensus_queue.cc:1035] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 23:24:40.982062 10281 raft_consensus.cc:979] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: : Instructing follower 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.982501 10281 raft_consensus.cc:1067] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Signalling peer 1da1957992c74e29b5f09aeb64fd88fd to start an election
I20260809 23:24:40.988495  9834 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
 from {username='slave'} at 127.0.0.1:47240
I20260809 23:24:40.989308  9834 raft_consensus.cc:482] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:40.989696  9834 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:40.993414 10041 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.995164  9785 leader_election.cc:302] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:40.996577 10338 raft_consensus.cc:2781] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:40.997263 10338 raft_consensus.cc:686] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:40.998217 10338 consensus_queue.cc:237] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:41.000756  9834 raft_consensus.cc:504] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:41.003736  9937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:41.004276  9937 raft_consensus.cc:3032] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Stepping down as leader of term 1
I20260809 23:24:41.004549  9937 raft_consensus.cc:729] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:41.005283  9937 consensus_queue.cc:260] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:41.006325  9937 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:41.008515  9935 raft_consensus.cc:2454] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:40.998471  9688 catalog_manager.cc:5495] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:41.012782 10041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b82248b0b7b426785f4e46d5a3b0467" candidate_uuid: "1da1957992c74e29b5f09aeb64fd88fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a2478dec1844a5481141d44f3136f26"
I20260809 23:24:41.013408 10041 raft_consensus.cc:3037] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Advancing to term 2
I20260809 23:24:41.018330  9937 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:41.019582  9786 leader_election.cc:302] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 2df9f9c44dc3408c8d475c31e28684b4; no voters: 
I20260809 23:24:41.019851  9834 leader_election.cc:288] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [CANDIDATE]: Term 2 election: Requested vote from peers 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409), 8a2478dec1844a5481141d44f3136f26 (127.21.255.3:40291)
I20260809 23:24:41.020694 10338 raft_consensus.cc:2781] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Leader election won for term 2
I20260809 23:24:41.021227 10338 raft_consensus.cc:686] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Leader. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:41.022082 10338 consensus_queue.cc:237] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:41.025421 10041 raft_consensus.cc:2454] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1da1957992c74e29b5f09aeb64fd88fd in term 2.
I20260809 23:24:41.040474  9691 catalog_manager.cc:5495] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 8a2478dec1844a5481141d44f3136f26 (127.21.255.3) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:41.042351  9691 catalog_manager.cc:5495] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:41.044234  9691 catalog_manager.cc:5495] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd reported cstate change: term changed from 1 to 2, leader changed from 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2) to 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1). New cstate: current_term: 2 leader_uuid: "1da1957992c74e29b5f09aeb64fd88fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260809 23:24:41.292490 10041 raft_consensus.cc:1261] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.293942 10338 consensus_queue.cc:1035] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.302660  9937 raft_consensus.cc:1261] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.304875 10336 consensus_queue.cc:1035] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260809 23:24:41.420460 10041 raft_consensus.cc:1261] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.422003 10338 consensus_queue.cc:1035] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.431053  9937 raft_consensus.cc:1261] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.432834 10336 consensus_queue.cc:1035] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.455912 10041 raft_consensus.cc:1261] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.457274 10336 consensus_queue.cc:1035] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.465605  9937 raft_consensus.cc:1261] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.467849 10338 consensus_queue.cc:1035] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.514222  9937 raft_consensus.cc:1261] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.515506 10338 consensus_queue.cc:1035] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:41.528406 10041 raft_consensus.cc:1261] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Refusing update from remote peer 1da1957992c74e29b5f09aeb64fd88fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260809 23:24:41.529968 10356 consensus_queue.cc:1035] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260809 23:24:43.608516  9690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35408:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260809 23:24:43.629917  9690 catalog_manager.cc:3333] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35408:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260809 23:24:43.740736 10407 tablet.cc:1722] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:43.794054 10409 tablet.cc:1722] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:43.849886 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260809 23:24:43.979408  9785 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.3:40291, user_credentials={real_user=slave}} blocked reactor thread for 81004.5us
I20260809 23:24:43.959688 10432 tablet.cc:1722] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260809 23:24:44.065232  9994 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.21.255.1:43405, user_credentials={real_user=slave}} blocked reactor thread for 84394.6us
I20260809 23:24:44.087656 10451 tablet.cc:1722] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.134287 10455 tablet.cc:1722] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260809 23:24:44.152760 10436 log.cc:924] Time spent T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260809 23:24:44.159192 10452 tablet.cc:1722] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260809 23:24:44.180353 10453 log.cc:924] Time spent T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260809 23:24:44.190011 10447 tablet.cc:1722] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.138103 10439 tablet.cc:1722] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.154322 10446 tablet.cc:1722] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.216156 10409 tablet.cc:1722] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.233608 10407 tablet.cc:1722] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.251721 10446 tablet.cc:1722] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.256182  9828 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3"
dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
mode: GRACEFUL
new_leader_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:57248
I20260809 23:24:44.257309  9828 raft_consensus.cc:595] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Received request to transfer leadership to TS 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:44.264137 10447 tablet.cc:1722] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.298084 10451 tablet.cc:1722] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.325552 10409 tablet.cc:1722] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.332293 10432 tablet.cc:1722] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.337530 10469 tablet.cc:1722] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.343194 10425 raft_consensus.cc:979] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: : Instructing follower 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:44.343792 10425 raft_consensus.cc:1067] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Signalling peer 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:44.345953 10446 tablet.cc:1722] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.346457 10038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:49384
I20260809 23:24:44.342854 10462 tablet.cc:1722] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.347342 10038 raft_consensus.cc:482] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:44.347926 10038 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.323670 10465 tablet.cc:1722] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.322232 10455 tablet.cc:1722] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.374912 10038 raft_consensus.cc:504] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.386099 10038 leader_election.cc:288] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 3 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:44.389330  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:44.390096  9935 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.395202 10469 tablet.cc:1722] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.398473  9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a48b130b22647e7b52f5f9f777896e3" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:44.399188  9834 raft_consensus.cc:3032] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:44.399580  9834 raft_consensus.cc:729] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1da1957992c74e29b5f09aeb64fd88fd, State: Running, Role: LEADER
I20260809 23:24:44.400539  9834 consensus_queue.cc:260] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.411226  9834 raft_consensus.cc:3037] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.416674  9933 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b609b47481a74aa58f03382178c4adad"
dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
mode: GRACEFUL
new_leader_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:35702
I20260809 23:24:44.417373  9933 raft_consensus.cc:595] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Received request to transfer leadership to TS 8a2478dec1844a5481141d44f3136f26
I20260809 23:24:44.419690 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260809 23:24:44.420392 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:44.423012 10432 tablet.cc:1722] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.423141 10439 tablet.cc:1722] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.393556 10451 tablet.cc:1722] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.429735 10462 tablet.cc:1722] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.431475 10447 tablet.cc:1722] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260809 23:24:44.446290  9834 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 3.
I20260809 23:24:44.437176  9935 raft_consensus.cc:2454] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 3.
I20260809 23:24:44.469560  9995 leader_election.cc:302] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2df9f9c44dc3408c8d475c31e28684b4, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:44.471199 10450 raft_consensus.cc:2781] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:44.471732 10450 raft_consensus.cc:686] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 3 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:44.473336 10450 consensus_queue.cc:237] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.497803  9691 catalog_manager.cc:6040] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: soft_deleted_table [id=63fe371fbcf346699ea46db6105b2980] alter complete (version 1)
I20260809 23:24:44.507895  9691 catalog_manager.cc:5495] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 2 to 3, leader changed from 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1) to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:44.927986 10478 raft_consensus.cc:979] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: : Instructing follower 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:44.928630 10411 raft_consensus.cc:1067] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Signalling peer 8a2478dec1844a5481141d44f3136f26 to start an election
I20260809 23:24:44.931218 10038 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b609b47481a74aa58f03382178c4adad"
dest_uuid: "8a2478dec1844a5481141d44f3136f26"
 from {username='slave'} at 127.0.0.1:49398
I20260809 23:24:44.931748 10038 raft_consensus.cc:482] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 23:24:44.932041 10038 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.939422 10038 raft_consensus.cc:504] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.941026 10038 leader_election.cc:288] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 3 election: Requested vote from peers 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405), 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409)
I20260809 23:24:44.942129  9935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2df9f9c44dc3408c8d475c31e28684b4"
I20260809 23:24:44.942304  9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b609b47481a74aa58f03382178c4adad" candidate_uuid: "8a2478dec1844a5481141d44f3136f26" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1da1957992c74e29b5f09aeb64fd88fd"
I20260809 23:24:44.942721  9935 raft_consensus.cc:3032] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Stepping down as leader of term 2
I20260809 23:24:44.943105  9834 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.943121  9935 raft_consensus.cc:729] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2df9f9c44dc3408c8d475c31e28684b4, State: Running, Role: LEADER
I20260809 23:24:44.944476  9935 consensus_queue.cc:260] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.946161  9935 raft_consensus.cc:3037] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Advancing to term 3
I20260809 23:24:44.952726  9933 raft_consensus.cc:1261] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:44.952610  9834 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 3.
I20260809 23:24:44.953634  9935 raft_consensus.cc:2454] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2478dec1844a5481141d44f3136f26 in term 3.
I20260809 23:24:44.954051  9994 leader_election.cc:302] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da1957992c74e29b5f09aeb64fd88fd, 8a2478dec1844a5481141d44f3136f26; no voters: 
I20260809 23:24:44.955084 10444 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:44.956014 10450 raft_consensus.cc:2781] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 23:24:44.960783 10450 raft_consensus.cc:686] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 3 LEADER]: Becoming Leader. State: Replica: 8a2478dec1844a5481141d44f3136f26, State: Running, Role: LEADER
I20260809 23:24:44.961582 10450 consensus_queue.cc:237] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } }
I20260809 23:24:44.976303  9834 raft_consensus.cc:1261] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:44.979627 10450 consensus_queue.cc:1035] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:44.983187  9690 catalog_manager.cc:5495] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 reported cstate change: term changed from 2 to 3, leader changed from 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2) to 8a2478dec1844a5481141d44f3136f26 (127.21.255.3). New cstate: current_term: 3 leader_uuid: "8a2478dec1844a5481141d44f3136f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2478dec1844a5481141d44f3136f26" member_type: VOTER last_known_addr { host: "127.21.255.3" port: 40291 } health_report { overall_health: HEALTHY } } }
I20260809 23:24:45.421465 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:45.427817 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:45.428294 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:45.456702  9935 raft_consensus.cc:1261] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:45.458195 10450 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df9f9c44dc3408c8d475c31e28684b4" member_type: VOTER last_known_addr { host: "127.21.255.2" port: 33409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:45.472752  9826 raft_consensus.cc:1261] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Refusing update from remote peer 8a2478dec1844a5481141d44f3136f26: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260809 23:24:45.478222 10492 consensus_queue.cc:1035] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da1957992c74e29b5f09aeb64fd88fd" member_type: VOTER last_known_addr { host: "127.21.255.1" port: 43405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260809 23:24:46.429345 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:46.434343 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:46.434691 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:47.435755 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:47.442023 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:47.442539 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:48.443746 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:48.449163 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:48.449774 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:49.450690 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:49.455765 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:49.456115 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:50.457037 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:50.464447 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:50.464951 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:51.466053 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:51.471544 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:51.471931 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:52.472956 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:52.479594 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:52.480051 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:53.480934 22524 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260809 23:24:53.485817 22524 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260809 23:24:53.486244 22524 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260809 23:24:54.560604 22524 tablet_server.cc:178] TabletServer@127.21.255.1:0 shutting down...
W20260809 23:24:54.569079  9994 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 18 similar messages]
W20260809 23:24:54.605155  9994 consensus_peers.cc:487] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:54.606458  9890 consensus_peers.cc:487] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:54.608603  9890 consensus_peers.cc:487] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:54.635010  9994 consensus_peers.cc:487] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.659147 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:54.660285 22524 tablet_replica.cc:330] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.661791 22524 raft_consensus.cc:2229] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.663360 22524 raft_consensus.cc:2258] T a2b74054639345d0b64bbb75cdcd23b7 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.666332 22524 tablet_replica.cc:330] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.667176 22524 raft_consensus.cc:2229] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.667774 22524 raft_consensus.cc:2258] T eb4d48e260ab40219722eaf4109a9af4 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.670723 22524 tablet_replica.cc:330] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.671897 22524 raft_consensus.cc:2229] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.672744 22524 raft_consensus.cc:2258] T 0d45b8d405f44dfa93a9df9bf307932b P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.677467 22524 tablet_replica.cc:330] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.678368 22524 raft_consensus.cc:2229] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Raft consensus shutting down.
W20260809 23:24:54.679427  9994 consensus_peers.cc:487] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.680505 22524 raft_consensus.cc:2258] T cef56cc4c7564305ae59cac349c349dc P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.685318 22524 tablet_replica.cc:330] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
W20260809 23:24:54.685551  9994 consensus_peers.cc:487] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.687239 22524 raft_consensus.cc:2229] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.689236 22524 raft_consensus.cc:2258] T 5b82248b0b7b426785f4e46d5a3b0467 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:54.691895  9890 consensus_peers.cc:487] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.692854 22524 tablet_replica.cc:330] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.693991 22524 raft_consensus.cc:2229] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.694821 22524 raft_consensus.cc:2258] T 5840fbf0505c45dfbe1ad4545467bf47 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:54.697230  9890 consensus_peers.cc:487] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.699045 22524 tablet_replica.cc:330] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.699746 22524 raft_consensus.cc:2229] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.701140 22524 raft_consensus.cc:2258] T 18ac43ba34c54f97877eee12223f43ce P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.704882 22524 tablet_replica.cc:330] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.705685 22524 raft_consensus.cc:2229] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.707043 22524 raft_consensus.cc:2258] T 6ab82f8ca0bb46239fca3c46e42da97e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.710210 22524 tablet_replica.cc:330] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.711016 22524 raft_consensus.cc:2229] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.711568 22524 raft_consensus.cc:2258] T 0a48b130b22647e7b52f5f9f777896e3 P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.715479 22524 tablet_replica.cc:330] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.716550 22524 raft_consensus.cc:2229] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.717281 22524 raft_consensus.cc:2258] T be3d5a988cf147e19133130c67fa914e P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.721220 22524 tablet_replica.cc:330] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.721935 22524 raft_consensus.cc:2229] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.722416 22524 raft_consensus.cc:2258] T 1a6a3113d7b540a18727995d759a4ada P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.724869 22524 tablet_replica.cc:330] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.725394 22524 raft_consensus.cc:2229] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.725981 22524 raft_consensus.cc:2258] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.728591 22524 tablet_replica.cc:330] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.729164 22524 raft_consensus.cc:2229] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.729712 22524 raft_consensus.cc:2258] T b609b47481a74aa58f03382178c4adad P 1da1957992c74e29b5f09aeb64fd88fd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.732200 22524 tablet_replica.cc:330] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.732863 22524 raft_consensus.cc:2229] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.733879 22524 raft_consensus.cc:2258] T 940f2aa4051d42d6aad243f3024117d6 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.737342 22524 tablet_replica.cc:330] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.737910 22524 raft_consensus.cc:2229] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.738461 22524 raft_consensus.cc:2258] T 348dba4300304e79a88b264279c3358e P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.741672 22524 tablet_replica.cc:330] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.742630 22524 raft_consensus.cc:2229] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.744138 22524 raft_consensus.cc:2258] T 62d951cdf8bd429fabe71c0b759b92c5 P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.747357 22524 tablet_replica.cc:330] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.747982 22524 raft_consensus.cc:2229] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.748761 22524 raft_consensus.cc:2258] T 7cd6557129314bd59fd84cdeed3211bf P 1da1957992c74e29b5f09aeb64fd88fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.751780 22524 tablet_replica.cc:330] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd: stopping tablet replica
I20260809 23:24:54.752382 22524 raft_consensus.cc:2229] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.753545 22524 raft_consensus.cc:2258] T 30c044125e014a6caf1be9b452d64b76 P 1da1957992c74e29b5f09aeb64fd88fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.808959 22524 tablet_server.cc:194] TabletServer@127.21.255.1:0 shutdown complete.
I20260809 23:24:54.860074 22524 tablet_server.cc:178] TabletServer@127.21.255.2:0 shutting down...
W20260809 23:24:54.888839  9995 consensus_peers.cc:487] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 -> Peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Couldn't send request to peer 2df9f9c44dc3408c8d475c31e28684b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:54.904410  9890 consensus_peers.cc:487] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 -> Peer 1da1957992c74e29b5f09aeb64fd88fd (127.21.255.1:43405): Couldn't send request to peer 1da1957992c74e29b5f09aeb64fd88fd. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.1:43405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:54.905429 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:54.906173 22524 tablet_replica.cc:330] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.906916 22524 raft_consensus.cc:2229] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.907972 22524 raft_consensus.cc:2258] T 0d45b8d405f44dfa93a9df9bf307932b P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.911667 22524 tablet_replica.cc:330] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.912642 22524 raft_consensus.cc:2229] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.913452 22524 raft_consensus.cc:2258] T cef56cc4c7564305ae59cac349c349dc P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.916625 22524 tablet_replica.cc:330] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.917335 22524 raft_consensus.cc:2229] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.918454 22524 raft_consensus.cc:2258] T a2b74054639345d0b64bbb75cdcd23b7 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.921360 22524 tablet_replica.cc:330] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.921983 22524 raft_consensus.cc:2229] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.923098 22524 raft_consensus.cc:2258] T eb4d48e260ab40219722eaf4109a9af4 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.926080 22524 tablet_replica.cc:330] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.927139 22524 raft_consensus.cc:2229] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.928714 22524 raft_consensus.cc:2258] T 7cd6557129314bd59fd84cdeed3211bf P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.933058 22524 tablet_replica.cc:330] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.933957 22524 raft_consensus.cc:2229] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.934778 22524 raft_consensus.cc:2258] T 62d951cdf8bd429fabe71c0b759b92c5 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.939139 22524 tablet_replica.cc:330] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.940747 22524 raft_consensus.cc:2229] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.942003 22524 raft_consensus.cc:2258] T 5840fbf0505c45dfbe1ad4545467bf47 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.945410 22524 tablet_replica.cc:330] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.946058 22524 raft_consensus.cc:2229] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.946605 22524 raft_consensus.cc:2258] T 18ac43ba34c54f97877eee12223f43ce P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.949043 22524 tablet_replica.cc:330] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.949652 22524 raft_consensus.cc:2229] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.950258 22524 raft_consensus.cc:2258] T 5b82248b0b7b426785f4e46d5a3b0467 P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.952839 22524 tablet_replica.cc:330] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.953476 22524 raft_consensus.cc:2229] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.954037 22524 raft_consensus.cc:2258] T 6ab82f8ca0bb46239fca3c46e42da97e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.956535 22524 tablet_replica.cc:330] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.957230 22524 raft_consensus.cc:2229] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.957772 22524 raft_consensus.cc:2258] T 0a48b130b22647e7b52f5f9f777896e3 P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.960228 22524 tablet_replica.cc:330] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.960831 22524 raft_consensus.cc:2229] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.961434 22524 raft_consensus.cc:2258] T 1a6a3113d7b540a18727995d759a4ada P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.963742 22524 tablet_replica.cc:330] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.964465 22524 raft_consensus.cc:2229] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.965965 22524 raft_consensus.cc:2258] T be3d5a988cf147e19133130c67fa914e P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.969424 22524 tablet_replica.cc:330] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.970279 22524 raft_consensus.cc:2229] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.970881 22524 raft_consensus.cc:2258] T 30c044125e014a6caf1be9b452d64b76 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.974952 22524 tablet_replica.cc:330] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.975677 22524 raft_consensus.cc:2229] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.977465 22524 raft_consensus.cc:2258] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.982921 22524 tablet_replica.cc:330] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.983528 22524 raft_consensus.cc:2229] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.984050 22524 raft_consensus.cc:2258] T b609b47481a74aa58f03382178c4adad P 2df9f9c44dc3408c8d475c31e28684b4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.986560 22524 tablet_replica.cc:330] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.987298 22524 raft_consensus.cc:2229] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:54.988412 22524 raft_consensus.cc:2258] T 348dba4300304e79a88b264279c3358e P 2df9f9c44dc3408c8d475c31e28684b4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:54.991384 22524 tablet_replica.cc:330] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4: stopping tablet replica
I20260809 23:24:54.991894 22524 raft_consensus.cc:2229] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:54.992375 22524 raft_consensus.cc:2258] T 940f2aa4051d42d6aad243f3024117d6 P 2df9f9c44dc3408c8d475c31e28684b4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 23:24:55.032516  9995 consensus_peers.cc:487] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 -> Peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Couldn't send request to peer 2df9f9c44dc3408c8d475c31e28684b4. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:33409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:55.045008 22524 tablet_server.cc:194] TabletServer@127.21.255.2:0 shutdown complete.
W20260809 23:24:55.045192  9995 consensus_peers.cc:487] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 -> Peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Couldn't send request to peer 2df9f9c44dc3408c8d475c31e28684b4. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:33409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260809 23:24:55.053557  9995 consensus_peers.cc:487] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 -> Peer 2df9f9c44dc3408c8d475c31e28684b4 (127.21.255.2:33409): Couldn't send request to peer 2df9f9c44dc3408c8d475c31e28684b4. Status: Network error: Client connection negotiation failed: client connection to 127.21.255.2:33409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 23:24:55.096189 22524 tablet_server.cc:178] TabletServer@127.21.255.3:0 shutting down...
I20260809 23:24:55.146181 22524 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260809 23:24:55.147218 22524 tablet_replica.cc:330] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.147955 22524 raft_consensus.cc:2229] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.148715 22524 raft_consensus.cc:2258] T 0d45b8d405f44dfa93a9df9bf307932b P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.151443 22524 tablet_replica.cc:330] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.152081 22524 raft_consensus.cc:2229] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.152655 22524 raft_consensus.cc:2258] T eb4d48e260ab40219722eaf4109a9af4 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.155474 22524 tablet_replica.cc:330] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.156055 22524 raft_consensus.cc:2229] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.156731 22524 raft_consensus.cc:2258] T cef56cc4c7564305ae59cac349c349dc P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.160270 22524 tablet_replica.cc:330] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.160808 22524 raft_consensus.cc:2229] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.161350 22524 raft_consensus.cc:2258] T a2b74054639345d0b64bbb75cdcd23b7 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.163901 22524 tablet_replica.cc:330] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.164453 22524 raft_consensus.cc:2229] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.164939 22524 raft_consensus.cc:2258] T be3d5a988cf147e19133130c67fa914e P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.167137 22524 tablet_replica.cc:330] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.167680 22524 raft_consensus.cc:2229] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 2 LEADER]: Raft consensus shutting down.
I20260809 23:24:55.168593 22524 raft_consensus.cc:2258] T 1a6a3113d7b540a18727995d759a4ada P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.171149 22524 tablet_replica.cc:330] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.171794 22524 raft_consensus.cc:2229] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.172353 22524 raft_consensus.cc:2258] T 62d951cdf8bd429fabe71c0b759b92c5 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.174710 22524 tablet_replica.cc:330] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.175308 22524 raft_consensus.cc:2229] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.175822 22524 raft_consensus.cc:2258] T 7cd6557129314bd59fd84cdeed3211bf P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.177992 22524 tablet_replica.cc:330] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.178601 22524 raft_consensus.cc:2229] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:55.179545 22524 raft_consensus.cc:2258] T 5840fbf0505c45dfbe1ad4545467bf47 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.181795 22524 tablet_replica.cc:330] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.182363 22524 raft_consensus.cc:2229] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.183115 22524 raft_consensus.cc:2258] T 5b82248b0b7b426785f4e46d5a3b0467 P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.185365 22524 tablet_replica.cc:330] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.185915 22524 raft_consensus.cc:2229] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.186446 22524 raft_consensus.cc:2258] T 18ac43ba34c54f97877eee12223f43ce P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.188551 22524 tablet_replica.cc:330] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.189086 22524 raft_consensus.cc:2229] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.189622 22524 raft_consensus.cc:2258] T 6ab82f8ca0bb46239fca3c46e42da97e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.192257 22524 tablet_replica.cc:330] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.193003 22524 raft_consensus.cc:2229] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:55.193912 22524 raft_consensus.cc:2258] T 0a48b130b22647e7b52f5f9f777896e3 P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.196341 22524 tablet_replica.cc:330] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.197202 22524 raft_consensus.cc:2229] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.197849 22524 raft_consensus.cc:2258] T 940f2aa4051d42d6aad243f3024117d6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.200429 22524 tablet_replica.cc:330] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.201028 22524 raft_consensus.cc:2229] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.201545 22524 raft_consensus.cc:2258] T a90f4331ebd34e7a81d1a0b9d153e3f6 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.203569 22524 tablet_replica.cc:330] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.204104 22524 raft_consensus.cc:2229] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 3 LEADER]: Raft consensus shutting down.
I20260809 23:24:55.204989 22524 raft_consensus.cc:2258] T b609b47481a74aa58f03382178c4adad P 8a2478dec1844a5481141d44f3136f26 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.207089 22524 tablet_replica.cc:330] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.207598 22524 raft_consensus.cc:2229] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.208134 22524 raft_consensus.cc:2258] T 348dba4300304e79a88b264279c3358e P 8a2478dec1844a5481141d44f3136f26 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.210255 22524 tablet_replica.cc:330] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26: stopping tablet replica
I20260809 23:24:55.210737 22524 raft_consensus.cc:2229] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 23:24:55.211277 22524 raft_consensus.cc:2258] T 30c044125e014a6caf1be9b452d64b76 P 8a2478dec1844a5481141d44f3136f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.260102 22524 tablet_server.cc:194] TabletServer@127.21.255.3:0 shutdown complete.
I20260809 23:24:55.304347 22524 master.cc:537] Master@127.21.255.62:44395 shutting down...
I20260809 23:24:55.336879 22524 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 1 LEADER]: Raft consensus shutting down.
I20260809 23:24:55.337653 22524 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 23:24:55.338014 22524 tablet_replica.cc:330] T 00000000000000000000000000000000 P f6d2ca9e8184414a8d9f866f81943aeb: stopping tablet replica
I20260809 23:24:55.360560 22524 master.cc:552] Master@127.21.255.62:44395 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (21354 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (44284 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 2 test suites ran. (510480 ms total)
[  PASSED  ] 5 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] LeaderRebalancerTest.TestMaintenanceMode
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260809 23:24:55.426559 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet_mm_ops.cc:221: suppressed but not reported on 2236 messages since previous log ~34 seconds ago
I20260809 23:24:55.426909 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet_mm_ops.cc:305: suppressed but not reported on 2236 messages since previous log ~34 seconds ago
I20260809 23:24:55.427426 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet_mm_ops.cc:140: suppressed but not reported on 2234 messages since previous log ~34 seconds ago
I20260809 23:24:55.427663 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 44 messages since previous log ~64 seconds ago
I20260809 23:24:55.427883 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 44 messages since previous log ~64 seconds ago
I20260809 23:24:55.428113 22524 logging.cc:424] LogThrottler Service Queue Overflow: suppressed but not reported on 1 messages since previous log ~266 seconds ago
I20260809 23:24:55.428337 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/service_pool.cc:137: suppressed but not reported on 1 messages since previous log ~266 seconds ago
I20260809 23:24:55.428599 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tserver/heartbeater.cc:644: suppressed but not reported on 2 messages since previous log ~151 seconds ago
I20260809 23:24:55.428861 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 14 messages since previous log ~0 seconds ago
I20260809 23:24:55.429191 22524 logging.cc:424] LogThrottler /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tablet/tablet.cc:2369: suppressed but not reported on 106 messages since previous log ~43 seconds ago